CN118228295A

CN118228295A - Risk assessment method and device for data leakage and storage medium

Info

Publication number: CN118228295A
Application number: CN202211575156.3A
Authority: CN
Inventors: 白离; 叶青青; 胡海波; 方成方; 徐珂; 时杰
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-12-08
Filing date: 2022-12-08
Publication date: 2024-06-21

Abstract

The present application relates to the field of machine learning, and in particular, to a risk assessment method and apparatus for data leakage, and a storage medium. The method comprises the following steps: acquiring a training data set and a model to be evaluated, and determining a reference model, wherein the reference model is a machine learning model constructed based on the training data set and is different from the model to be evaluated; acquiring respective corresponding relative loss differences of a plurality of training samples, wherein the relative loss differences indicate the relative differences between the risk degrees of data leakage of the training samples under a reference model and a model to be evaluated; and determining a risk result of the model to be evaluated according to the corresponding relative loss difference of the training samples. Aiming at a training data set of a machine learning model, the embodiment of the application provides a quantization method for risk assessment of data leakage, which can assess the risk degree of the machine learning model for leakage of data of the training data set under member reasoning attack, and improves the rationality and accuracy of risk assessment of data leakage.

Description

Risk assessment method and device for data leakage and storage medium

Technical Field

The present application relates to the field of machine learning (MACHINE LEARNING, ML), and in particular, to a risk assessment method, apparatus, and storage medium for data leakage.

Background

With the rapid development of machine learning technology, many companies have provided machine learning as a service to the public through their cloud platforms (MACHINE LEARNING AS A SERVICE, MLAAS). Users can train and develop machine learning models on the cloud platform using their own private data sets. To ensure the security of the machine learning model and the privacy of the data, the published machine learning model typically provides only one application program interface (Application Program Interface, API) interface to provide services to the user. In recent years, however, research has found that the API interface of a machine learning model still leaks sensitive information of training data or parameters of the model itself.

Machine learning models are applied in many scenarios, such as medical diagnosis, face recognition, etc. The training data of the machine learning model may contain some private information, such as a diagnosis book of a hospital patient, a personal certificate photograph, etc., which once compromised may seriously impair the privacy of the user. Therefore, people feel worry about the attack on the privacy of training data of the machine learning model. By analyzing the parameters and output parameters of the machine learning model, a tester makes inferences about whether a certain input sample is training data, and this attack is called a Membership inference attack (Membership INFERENCE ATTACK, MIA). Such attacks can present two problems, one being revealing the user's private data. If the training data includes privacy data for the user, then the membership inference attack may violate the relevant laws of privacy protection. Secondly, the source of the training data set is revealed, and when a tester performs member reasoning attack on the machine learning model, the tester can help the tester analyze the composition of the training data and presume the source of the data.

That is, the member reasoning attack can infer the data member information of the machine learning model, destroy the privacy security of the training data set, and how to measure the data leakage risk of the machine learning model under the member reasoning attack.

Disclosure of Invention

In view of this, a risk assessment method, apparatus and storage medium for data leakage are provided. Aiming at a training data set of a machine learning model, the embodiment of the application provides a quantization method for risk assessment of data leakage, which is used for assessing the risk degree of the machine learning model for leakage of data of the training data set under member reasoning attack.

In a first aspect, an embodiment of the present application provides a risk assessment method for data leakage, the method including:

acquiring a training data set and a model to be evaluated, wherein the training data set comprises a plurality of training samples;

Determining a reference model (REFERENCE MODEL, RM) which is a machine learning model constructed on the basis of the training dataset, the reference model being different from the model to be evaluated;

Acquiring respective corresponding relative loss differences of the plurality of training samples, wherein the relative loss differences indicate the relative differences between the risk degrees of data leakage of the training samples under the reference model and the model to be evaluated;

And determining a risk result of the model to be evaluated according to the corresponding relative loss difference of the training samples, wherein the risk result indicates the risk degree of the model to be evaluated for revealing the data of the training data set under member reasoning attack.

In this implementation, a training data set and a model to be evaluated are acquired, wherein the training data set comprises a plurality of training samples; determining a reference model, wherein the reference model is a machine learning model constructed based on a training data set and is different from a model to be evaluated; acquiring respective corresponding relative loss differences of a plurality of training samples, wherein the relative loss differences indicate the relative differences between the risk degrees of data leakage of the training samples under a reference model and a model to be evaluated; determining a risk result of the model to be evaluated according to the corresponding relative loss difference of the training samples, wherein the risk result indicates the risk degree of the model to be evaluated for revealing data of the training data set under member reasoning attack; in a risk assessment scene of a machine learning model, the absolute privacy risk of a model to be assessed is avoided, a reference model is introduced, the correlation of the data leakage risk of the model is quantified by taking the relative loss difference of a training set under the reference model and the model to be assessed as the risk assessment of the model to be assessed, in addition, the learning capacity of the model and the distribution characteristics of samples can be reflected by the loss output of the model, the data leakage risk of the model is measured by utilizing the loss output, and further, the risk measurement method is simpler and more efficient.

In one possible implementation, the determining the reference model includes:

obtaining a model list, wherein the model list comprises a plurality of candidate models, and the candidate models are machine learning models obtained through training based on the training data set;

Determining the reference model according to the model list, wherein the reference model is a candidate model with the highest risk index in the model list, or the reference model is a model with the highest risk index, subjected to parameter adjustment;

the risk index indicates the risk degree of the candidate model revealing the data of the training data set under the member reasoning attack.

In the implementation, a model list is obtained, wherein the model list comprises a plurality of candidate models, and the candidate models are machine learning models obtained based on training of a training data set; determining a reference model according to the model list, wherein the reference model is a candidate model with the highest risk index in the model list or a model with the highest risk index in the model list and subjected to parameter adjustment, and the risk index indicates the risk degree of the candidate model for revealing data of the training data set under member reasoning attack; the candidate model with the highest risk index in the model list is selected or fine-tuned, so that a model with high data leakage risk can be constructed as a reference model, so that obvious difference exists between the reference model and the model to be evaluated in training data set learning, the obtained relative loss difference can capture and reflect the difference between the reference model and the model to be evaluated with high probability, and the subsequent model sorting and selection are more accurate.

In another possible implementation, the reference model is a machine learning model that does not employ a defense mechanism, and/or the number of model parameters of the reference model is greater than a number threshold.

In this implementation, the reference model is a machine learning model that does not employ a defense mechanism, and/or the number of model parameters of the reference model is greater than a number threshold; empirically, if the machine learning model does not adopt any defense mechanism or has excessive model parameters, the model has a high risk of data leakage. Therefore, when the model to be evaluated adopts a certain defense mechanism, a machine learning model of any defense mechanism is not adopted, or a model with more parameters (such as increasing the depth of the model and selecting a complex network structure) has higher data leakage risk, the model can be used as a reference model, and a model with high data leakage risk can be constructed as the reference model.

In another possible implementation manner, the obtaining the difference in relative loss corresponding to each of the plurality of training samples includes:

For each training sample in the plurality of training samples, acquiring a first loss value and a second loss value of the training sample, wherein the first loss value indicates the risk degree of data leakage of the training sample under the reference model, and the second loss value indicates the risk degree of data leakage of the training sample under the model to be evaluated;

determining the relative loss difference for the training sample from a difference between the first loss value and the second loss value for the training sample.

In the implementation manner, for each training sample in a plurality of training samples, a first loss value and a second loss value of the training sample are obtained, wherein the first loss value indicates the risk degree of data leakage of the training sample under a reference model, and the second loss value indicates the risk degree of data leakage of the training sample under a model to be evaluated; determining a relative loss difference of the training sample according to the difference value of the first loss value and the second loss value of the training sample; because the training samples with smaller loss values are prone to have larger risks to be leaked, the risk degree of data leakage of the training samples is measured through the loss values of the models, namely the risk degree of the single training sample under the model to be evaluated is the difference of the loss values of the training samples on the reference model and the model to be evaluated compared with the reference model, and the risk quantification of the model to be evaluated is further guaranteed.

In another possible implementation manner, the determining the risk result of the model to be evaluated according to the respective relative loss differences of the plurality of training samples includes:

determining a loss stability result of the model to be evaluated according to the corresponding relative loss difference of the training samples;

And outputting a risk result of the model to be evaluated when the loss stability result meets a stability condition.

In the implementation mode, determining a loss stability result of a model to be evaluated according to the corresponding relative loss difference of a plurality of training samples; when the loss stability result meets the stability condition, outputting a risk result of the model to be evaluated, namely performing stability analysis on the model to be evaluated, and when the loss stability result meets the stability condition, directly outputting the risk result of the model to be evaluated, thereby ensuring the accuracy of the finally obtained risk result of the model to be evaluated.

In another possible implementation, the method further includes:

And when the loss stability result does not meet the stability condition, re-executing the step of determining the reference model.

In the implementation manner, when the loss stability result does not meet the stability condition, the step of determining the reference model is re-executed, namely stability analysis is performed on the model to be evaluated, and when the loss stability result does not meet the stability condition, iterative analysis is needed, so that the accuracy of the finally obtained risk result of the model to be evaluated is ensured.

In a second aspect, an embodiment of the present application provides a risk assessment apparatus for data leakage, the apparatus including:

The first acquisition unit is used for acquiring a training data set and a model to be evaluated, wherein the training data set comprises a plurality of training samples;

A first determining unit configured to determine a reference model, the reference model being a machine learning model constructed based on the training data set, the reference model being different from the model to be evaluated;

A second obtaining unit, configured to obtain a relative loss difference corresponding to each of the plurality of training samples, where the relative loss difference indicates a relative difference between risk degrees of data leakage of the training samples under the reference model and the model to be evaluated;

and the second determining unit is used for determining a risk result of the model to be evaluated according to the relative loss difference corresponding to each of the training samples, wherein the risk result indicates the risk degree of the model to be evaluated for revealing the data of the training data set under member reasoning attack.

In a possible implementation manner, the first determining unit is further configured to:

In another possible implementation manner, the second obtaining unit is further configured to:

In another possible implementation manner, the second determining unit is further configured to:

In a third aspect, an embodiment of the present application provides a risk assessment apparatus for data leakage, the apparatus including:

A processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method provided by the first aspect or any one of the possible implementations of the first aspect when executing the instructions.

In a fourth aspect, embodiments of the present application provide a non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method provided by the first aspect or any one of the possible implementations of the first aspect.

In a fifth aspect, embodiments of the present application provide a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying the computer readable code, which when run in an electronic device, a processor in the electronic device performs the method provided by the first aspect or any one of the possible implementations of the first aspect.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features and aspects of the application and together with the description, serve to explain the principles of the application.

FIG. 1 illustrates a schematic diagram of a computing device provided by an exemplary embodiment of the present application.

Fig. 2 is a flowchart illustrating a risk assessment method of data leakage according to an exemplary embodiment of the present application.

Fig. 3 is a flowchart illustrating a risk assessment method of data leakage according to another exemplary embodiment of the present application.

Fig. 4 shows a block diagram of a risk assessment apparatus for data leakage provided by an exemplary embodiment of the present application.

Detailed Description

Various exemplary embodiments, features and aspects of the application will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

In addition, numerous specific details are set forth in the following description in order to provide a better illustration of the application. It will be understood by those skilled in the art that the present application may be practiced without some of these specific details. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present application.

First, the terms related to the present application will be described.

1. Membership inference attack: by analyzing the internal parameters and the output parameters of the model, whether a certain test sample belongs to training data is inferred, and the attack is called membership inference attack.

In the related art, member inference attacks are mostly successfully implemented depending on the difference in output distribution of the machine learning model on the training data set and the test data set. Generally, the machine learning model output has a higher confidence level on the training dataset and a lower confidence level on the test dataset, the difference of which provides a multiplicative basis for membership inference attacks. There is evidence that a high generalization gap (English: generalization Gap) is likely to lead to machine learning model privacy disclosure. Generalization is an important indicator of predicting and understanding the performance of a model on unseen samples, while generalization gaps represent the difference between the performance of a model on training data sets and its performance on unseen samples. The generalization gap of the machine learning model becomes an important index for measuring the risk of membership reasoning of the model.

However, in the above method, the risk assessment method based on generalization capability only considers the learning capability of the model, and ignores the data distribution and sample characteristics. Studies have shown that models still present a risk of revealing data privacy even when the models have little generalized gaps. For some samples, such as outliers or samples located near the decision surface, these special samples still have a high risk of leakage even in the case of very good model generalization.

In addition, in the related art, the risk assessment method based on data distribution considers the data leakage risk existing in the model from the sample distribution, so that the reasons that the data leakage risk still exists in the well-generalized model can be explained. If the training data is representative, which has a significant impact on the Stability of the model (English: stability), then an attacker can easily infer whether these samples are present in the training dataset. The influence of the sample on the stability of the model can be utilized to evaluate the data leakage risk of the model. Optionally, a membership inference attack is performed on outliers in the training dataset, and it is found that membership information for these particular samples can be inferred even though the model is well-generalized across the test dataset.

However, in the above method, the risk assessment method based on data distribution has high computational complexity, and needs to perform operations such as preprocessing and clustering on the training data set. The evaluation method mainly focuses on special samples, such as outliers in a data set or samples near a decision surface, and often requires a data processing algorithm to pick the special samples. For large-scale data sets, this requires high computational resources and time support. Moreover, the risk assessment method based on data distribution only focuses on the duty ratio of a special sample in the whole training data set, and ignores the learning capacity of the model and leakage risks of other general samples.

In addition, in the related art, the risk assessment method based on the differential privacy (DIFFERENTIAL PRIVACY, DP) is to use a differential privacy mechanism to theoretically analyze that the model has the upper limit of member data leakage risk. Differential privacy is a common method of protecting data privacy. The method generally utilizes epsilon-differential privacy to analyze the relationship between member inference attacks and the existence of privacy budget epsilon in theory. Differential privacy severely limits the extent to which any point in the training data can affect the results of the calculation. Evidence exists to demonstrate the association of epsilon-differential privacy with member attack dominance (Membership Advantage, MA).

However, in the above method, on the one hand, the risk assessment method based on differential privacy requires that the model train the model using a differential privacy mechanism, and is not applicable to a model which has been trained or cannot be applied to a differential privacy mechanism. For models that have been trained, retraining incurs additional computational resources and time consumption, and therefore it is difficult to retrain in conjunction with the differential privacy mechanism. In addition, the introduction of differential privacy mechanisms is often accompanied by loss of model utility, which is unacceptable for models that are sensitive to loss of utility, such as medical disease prediction models, and so forth, and therefore this approach is not applicable to these models. On the other hand, the theoretical upper limit of the risk assessment method based on differential privacy is often loose, and the risk assessment method has more difference with the actual member reasoning attack effect.

Aiming at the problem that the machine learning model is easy to suffer from member reasoning attack, the embodiment of the application provides a data leakage risk quantification method aiming at training data, which can measure the data leakage risk of the machine learning model under the member reasoning attack.

Since memory and generalization are the main reasons for revealing data of the training dataset for machine learning models, and these are related to machine learning model structure, model parameters, data distribution, etc., it is difficult to directly calculate the exact degree of memory or generalization. In the embodiment of the application, the absolute privacy risk of the model to be evaluated is avoided being calculated, a reference model is introduced, and the correlation of the data leakage risk of the model is quantified by calculating the relative loss difference of the model to be evaluated relative to the reference model. In addition, the loss output of the model can reflect the learning ability of the model and the distribution characteristics of the samples, so that the embodiment of the application measures the data leakage risk of the model by using the loss output.

The embodiment of the application provides a risk assessment method for data leakage, wherein an execution subject is computing equipment. Referring to FIG. 1, a schematic diagram of a computing device according to an exemplary embodiment of the application is shown.

The computing device may be a terminal or a server. Terminals include mobile terminals or fixed terminals, such as terminals that may be cell phones, tablet computers, laptop portable computers, desktop computers, and the like. The server may be a server, a server cluster comprising a plurality of servers, or a cloud computing service center.

The computing device is used for analyzing privacy protection of the artificial intelligence model, and can be used for artificial intelligence evaluation, authentication service and the like.

As shown in fig. 1, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will appreciate that the architecture shown in fig. 1 is not limiting of the computing device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components. Wherein:

The processor 10 is a control center of the computing device, connects various portions of the overall computing device using various interfaces and lines, and performs various functions of the computing device and processes data by running or executing software programs and/or modules stored in the memory 20, and invoking data stored in the memory 20, thereby controlling the computing device overall. The processor 10 may be implemented by a CPU or by a graphics processor (Graphics Processing Unit, GPU).

The memory 20 may be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running software programs and modules stored in the memory 20. The memory 20 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system 21, a first acquisition unit 22, a first determination unit 23, a second acquisition unit 24, a second determination unit 25, and at least one application program 26 (such as machine learning model training, etc.) required for functions, and the like; the storage data area may store data created from the use of the computing device, and the like. The Memory 20 may be implemented by any type or combination of volatile or non-volatile Memory devices, such as static random access Memory (Static Random Access Memory, SRAM), electrically erasable programmable Read-Only Memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, EEPROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic or optical disk. Accordingly, the memory 20 may also include a memory controller to provide access to the memory 20 by the processor 10.

Wherein the processor 10 performs the following functions by operating the first acquisition unit 22: acquiring a training data set and a model to be evaluated, wherein the training data set comprises a plurality of training samples; the processor 10 performs the following functions by operating the first determination unit 23: determining a reference model, wherein the reference model is a machine learning model constructed based on the training data set and is different from the model to be evaluated; the processor 10 performs the following functions by operating the second acquisition unit 24: acquiring respective corresponding relative loss differences of the plurality of training samples, wherein the relative loss differences indicate the relative differences between the risk degrees of data leakage of the training samples under the reference model and the model to be evaluated; the processor 10 performs the following functions by operating the second determination unit 25: and determining a risk result of the model to be evaluated according to the corresponding relative loss difference of the training samples, wherein the risk result indicates the risk degree of the model to be evaluated for revealing the data of the training data set under member reasoning attack.

In the following, a risk assessment method for data leakage provided by the embodiment of the present application will be described using several exemplary embodiments.

Referring to fig. 2, a flowchart of a risk assessment method for data leakage according to an exemplary embodiment of the present application is shown, and the method is used in the computing device shown in fig. 1 for illustration. The method comprises the following steps.

Step 201, a training data set and a model to be evaluated are obtained, wherein the training data set comprises a plurality of training samples.

Optionally, the computing device obtains a training data set, a model to be evaluated, and a loss function. Wherein the training data set comprises a plurality of training samples.

The model to be evaluated is a machine learning model obtained based on training of a training data set. Optionally, the model to be evaluated includes one or more models input, and for convenience of description, the model to be evaluated is only described below as an example.

The loss function is set by default or is set by user. The loss function is used for calculating a loss value of the training sample under a model (to-be-evaluated model or reference model), wherein the loss value indicates the risk degree of data leakage of the training sample under the model. Optionally, the loss value of the training sample and the risk degree of data leakage of the training sample under the model are in a negative correlation relationship, that is, the smaller the loss value of the training sample, the higher the risk degree of data leakage of the training sample under the model.

Step 202, determining a reference model, wherein the reference model is a machine learning model constructed based on a training data set, and is different from a model to be evaluated.

Optionally, the computing device trains a machine learning model to obtain the reference model according to the model to be evaluated and the training data set.

In one possible implementation, the computing device selects or trims the candidate model with the highest risk indicator as the reference model. Optionally, the computing device obtains a model list, the model list including a plurality of candidate models, the candidate models being machine learning models trained based on the training data set; determining a reference model according to the model list, wherein the reference model is a candidate model with the highest risk index in the model list, or the reference model is a model with the highest risk index after parameter adjustment; the risk index indicates the risk degree of the candidate model revealing the data of the training data set under the member reasoning attack.

There may be different sensitivities due to the variability between different reference models and models to be evaluated. If the reference model and the model to be evaluated have obvious difference in training data set learning, the relative loss difference can capture and reflect the difference between the reference model and the model to be evaluated with high probability, so that the model ordering and selection are more accurate. If the reference model is not properly selected, comparable to the learning behavior of the model to be evaluated on the data set, subtle differences between the models to be evaluated are easily ignored. Therefore, in order to avoid errors caused by improper selection of the reference model as much as possible, a model with high risk of data leakage is constructed as the reference model.

If the machine learning model does not adopt any defense mechanism or has excessive model parameters, the model has higher data leakage risk. Therefore, when the model to be evaluated adopts a certain defense mechanism (such as regularization, dropout, differential privacy and the like), the machine learning model which does not adopt any defense mechanism has higher data leakage risk and can be used as a reference model; when the model to be evaluated does not take any defensive mechanism, then a machine learning model with more model parameters (such as increasing the depth of the model and selecting a complex network structure) is selected as a reference model.

Optionally, the reference model is a machine learning model with a data leakage risk higher than a risk threshold. The risk threshold is set by default or is set by user. The embodiment of the present application is not limited thereto.

Optionally, the reference model is a machine learning model that does not employ a defense mechanism, and/or the number of model parameters of the reference model is greater than a number threshold. The quantity threshold is set by default or is set by user. The embodiment of the present application is not limited thereto.

The model to be evaluated and the reference model are two different machine learning models, and the model to be evaluated and the reference model are both models obtained based on training of a training data set.

Optionally, the model to be evaluated and the reference model are both used for the image classification task. Or the model to be evaluated and the reference model can be used for non-image classification tasks. The embodiment of the present application is not limited thereto.

In step 203, a relative loss difference corresponding to each of the plurality of training samples is obtained, where the relative loss difference indicates a relative difference between risk levels of data leakage of the training samples under the reference model and the model to be evaluated.

Optionally, for each training sample in the training dataset, the computing device calculates a relative loss variance for the training sample, the relative loss variance being indicative of a relative variance between the risk level of data leakage for the training sample under the reference model and the model to be evaluated.

Optionally, for each training sample in the plurality of training samples, the computing device obtains a first loss value and a second loss value of the training sample, the first loss value indicating a risk level of data leakage of the training sample under the reference model, the second loss value indicating a risk level of data leakage of the training sample under the model to be evaluated; and determining the relative loss difference of the training samples according to the difference value of the first loss value and the second loss value of the training samples.

Because machine learning models often exhibit higher confidence levels for already learned samples, less loss values, and higher losses on unseen test data sets, embodiments of the present application utilize loss values of the model to measure the risk of data leakage in training samples. Training data set samples with smaller loss values tend to be compromised at greater risk because they are easily speculated by an attacker. In addition, the distribution of the samples themselves may have an influence on the loss value, in addition to the learning ability of the model. For those samples at the center of the distribution, the model easily learns the distribution of those samples, while for those samples at the edges of the distribution, such as those near the decision hyperplane, most models have difficulty learning the information of that portion of the sample. In order to avoid the influence of sample distribution on risk assessment, a reference model is introduced, and a risk index for measuring data leakage of a certain training sample under a model to be evaluated and the reference model is established aiming at a single training sample of a training data set. The risk index of data leakage of a single training sample under a to-be-evaluated model compared with that of a reference model is as follows: the relative loss difference of the training sample on the reference model and the model to be evaluated.

Optionally, the relative loss difference of each training sample is a probability value, that is, the computing device determines the relative loss difference of the training sample according to the difference between the first loss value and the second loss value of the training sample, including: the computing device determines a difference between the first loss value and the second loss value of the training sample, and converts the difference to a probability value using an activation function.

Illustratively, the computing device calculates the difference in loss values of a training sample x under the model to be evaluated T and the reference model T by the following formula, and converts the difference in loss values into probability values using an activation function (e.g., sigmoid function)

Wherein T is a model to be evaluated, R is a reference model, x is a training sample in a training data set, and l is a loss value of the training sample x in the reference model R and the model to be evaluated T.

And 204, determining a risk result of the model to be evaluated according to the corresponding relative loss difference of the training samples, wherein the risk result indicates the risk degree of the model to be evaluated for revealing the data of the training data set under the member reasoning attack.

Optionally, the computing device analyzes the risk gain of the model to be evaluated above the reference model data leakage risk by calculating the loss difference of each model to be evaluated, and performs stability analysis to determine whether iterative analysis is needed. Optionally, the risk gain is an average of the relative loss differences corresponding to each of the plurality of training samples.

Optionally, the computing device determines a risk result of the model to be evaluated according to an average value of the relative loss differences corresponding to each of the plurality of training samples.

Optionally, the computing device performs stability analysis, including: determining a loss stability result of the model to be evaluated according to the corresponding relative loss difference of the training samples; and when the loss stability result meets the stability condition, outputting a risk result of the model to be evaluated.

Optionally, the step of determining the reference model is re-performed when the loss stability results in not meeting the stability condition.

Optionally, the computing device acquires a candidate model set, where the candidate model set includes a plurality of candidate models to be evaluated; and determining a candidate model to be evaluated as a reference model from the candidate model set, wherein the stability condition indicates that the determined reference model is the model with the greatest risk degree in the current model to be evaluated.

Optionally, the computing device performing the iterative analysis includes: the computing equipment acquires a plurality of candidate reference models, sorts the plurality of candidate reference models according to the risk results, determines the candidate reference model with the largest risk degree as the reference model according to the sorting results, re-measures the candidate reference model, and sorts and selects the candidate reference model.

Optionally, for the model T to be evaluated trained on the training data set D _tr, the computing device determines, according to the average value of the relative loss differences corresponding to each of the plurality of training samples, a risk result of the model T to be evaluated by the following formula

In summary, the embodiment of the application provides a risk assessment method based on loss differences, which comprises the steps of firstly constructing a reference model with high data leakage risk, then sequentially calculating the relative loss differences of training samples in a training dataset under the reference model and a model to be assessed based on the data leakage risk of the reference model, and finally determining a risk result of the model to be assessed according to the relative loss differences corresponding to a plurality of training samples, for example, taking the average value of the relative loss differences as the risk of the data of the training dataset of the model to be assessed, thereby improving the rationality and accuracy of risk assessment of data leakage.

Referring to fig. 3, a flowchart of a risk assessment method for data leakage according to another exemplary embodiment of the present application is shown, and this embodiment is illustrated by using the method in the computing device shown in fig. 1. The method comprises the following steps.

Step 301, a training data set and a model to be evaluated are obtained.

The computing device obtains a training data set and a model to be evaluated, the training data set including a plurality of training samples.

Step 302, it is determined whether to train the reference model.

Optionally, the computing device determines whether the number of models to be evaluated is one, and if the number of models to be evaluated is one, the reference model needs to be trained. If the reference model needs to be trained, step 303 is executed; if the reference model does not need to be trained, step 304 is performed.

Step 303, training to obtain a reference model according to the training data set without adopting any defense mechanism.

The computing device trains to obtain a reference model according to the training data set without adopting any defense mechanism.

And 304, selecting or fine-tuning the candidate model with the highest risk index as a reference model.

The computing equipment determines the candidate model with the highest risk index as a reference model, or adjusts parameters of the candidate model with the highest risk index, and determines the candidate model with the highest risk index after adjustment as the reference model.

In step 305, a relative loss difference corresponding to each of the plurality of training samples is calculated.

The computing device calculates a relative loss difference for each of the plurality of training samples, the relative loss difference indicating a relative difference between the risk level of data leakage of the training samples under the reference model and the model to be evaluated.

And 306, determining a loss stability result of the model to be evaluated.

And the computing equipment determines a loss stability result of the model to be evaluated according to the corresponding relative loss difference of the training samples.

Step 307, it is determined whether the stability loss result satisfies the stability condition.

Optionally, the computing device determines whether the loss stability result satisfies a stability condition to determine whether iterative analysis is required. If the loss stability result satisfies the stability condition, step 308 is performed. When the loss stability results do not meet the stability condition, step 302 is re-executed.

And step 308, outputting a risk result of the model to be evaluated.

Optionally, the computing device outputs a risk result of the model to be evaluated.

It should be noted that, the details of steps 301 to 308 may be referred to the description of the above embodiments, and are not repeated here.

In summary, the embodiment of the application creates a risk assessment example (Loss Difference Measure, LDM) based on a loss difference for the problem of data leakage of the training data set in the machine learning model, and assesses the risk of the model to be assessed leaking member information of the training data set under member reasoning attack. Compared with a general measurement method based on generalization degree in the related art, the embodiment has higher correlation with actual member attack risk.

The embodiment of the application provides a risk assessment method based on loss difference, which takes the relative loss difference of a training data set on a reference model and a model to be evaluated as the risk assessment of the model in a risk assessment scene of a machine learning model, and simultaneously considers the learning capacity of the model and the distribution characteristics of samples, so that the risk measurement method is simple and efficient.

The embodiment of the application provides a risk assessment method based on loss difference, on one hand, in member reasoning attack, the risk assessment of LDM is more consistent with the actual attack effect in distribution, and in particular, for a Multi-layer Perceptron (MLP) model, LDM and MIA accuracy and MIA advantage distribution are almost linearly distributed. Meanwhile, for a complex convolutional neural network (Convolutional Neural Networks, CNN) model, the actual leakage risk of the model is difficult to evaluate by the overfitting evaluation mode, and the LDM can still reflect the data leakage risk of the model to be evaluated to a certain extent.

On the other hand, the LDM risk assessment method has higher correlation with the actual attack effect. For complex CNN model and STL10 data, the pearson correlation coefficient (Pearson Correlation Coefficient, PCC) of LDM metric method and actual attack effect is 0.87 and 0.80, respectively, much higher than the baseline method. In addition, for non-image data and an MLP model, the correlation between the LDM method and MIA accuracy reaches more than 0.90, which shows that the LDM can effectively indicate the data leakage risk of the model to be detected. The PCC results for LDM and actual MIA on the 4 data sets are shown in table one.

List one

On the other hand, the framework of the risk assessment method for data leakage provided by the embodiment of the application is of a general design, is not limited by data types and training methods, and is suitable for image classification tasks and non-image classification tasks.

The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.

Referring to fig. 4, a block diagram of a risk assessment apparatus for data leakage according to an exemplary embodiment of the present application is shown. The apparatus may be implemented as all or part of a computing device by software, hardware, or a combination of both. The apparatus may include: the first acquisition unit 410, the first determination unit 420, the second acquisition unit 430, the second determination unit 440.

A first obtaining unit 410, configured to obtain a training data set and a model to be evaluated, where the training data set includes a plurality of training samples;

A first determining unit 420, configured to determine a reference model, where the reference model is a machine learning model constructed based on a training data set, and the reference model is different from a model to be evaluated;

A second obtaining unit 430, configured to obtain relative loss differences corresponding to the plurality of training samples, where the relative loss differences indicate relative differences between risk degrees of data leakage of the training samples under the reference model and the model to be evaluated;

The second determining unit 440 is configured to determine a risk result of the model to be evaluated according to the respective relative loss differences of the plurality of training samples, where the risk result indicates a risk degree of the model to be evaluated leaking data of the training data set under a membership inference attack.

In a possible implementation manner, the first determining unit 420 is further configured to:

obtaining a model list, wherein the model list comprises a plurality of candidate models, and the candidate models are machine learning models obtained based on training of a training data set;

determining a reference model according to the model list, wherein the reference model is a candidate model with the highest risk index in the model list, or the reference model is a model with the highest risk index after parameter adjustment;

In another possible implementation, the reference model is a machine learning model that does not employ a defense mechanism and/or the number of model parameters of the reference model is greater than a number threshold.

In another possible implementation manner, the second obtaining unit 430 is further configured to:

for each training sample in the plurality of training samples, acquiring a first loss value and a second loss value of the training sample, wherein the first loss value indicates the risk degree of data leakage of the training sample under a reference model, and the second loss value indicates the risk degree of data leakage of the training sample under a model to be evaluated;

And determining the relative loss difference of the training samples according to the difference value of the first loss value and the second loss value of the training samples.

In another possible implementation, the second determining unit 440 is further configured to:

and when the loss stability result meets the stability condition, outputting a risk result of the model to be evaluated.

when the loss stability results do not meet the stability condition, the step of determining the reference model is re-performed.

It should be noted that, in the apparatus provided in the foregoing embodiment, when implementing the functions thereof, only the division of the foregoing functional modules is used as an example, in practical application, the foregoing functional allocation may be implemented by different functional modules, that is, the internal structure of the device is divided into different functional modules, so as to implement all or part of the functions described above. In addition, the apparatus and the method embodiments provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the apparatus and the method embodiments are detailed in the method embodiments and are not repeated herein.

The embodiment of the application provides a risk assessment device for data leakage, which comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the methods performed by the computing device in the various embodiments described above when executing the instructions.

Embodiments of the present application provide a non-transitory computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the methods performed by a computing device in the various embodiments described above.

Embodiments of the present application provide a computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying the computer readable code, which when run in a computing device, a processor in the computing device performs the methods performed by the computing device in the various embodiments described above.

The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disk, hard disk, random Access Memory (Random Access Memory, RAM), read Only Memory (ROM), erasable programmable Read Only Memory (ELECTRICALLY PROGRAMMABLE READ-Only-Memory, EPROM or flash Memory), static Random Access Memory (SRAM), portable compact disk Read Only Memory (Compact Disc Read-Only Memory, CD-ROM), digital versatile disk (Digital Video Disc, DVD), memory stick, floppy disk, mechanical coding devices, punch cards or in-groove bump structures such as instructions stored thereon, and any suitable combination of the foregoing.

The computer readable program instructions or code described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present application may be assembler instructions, instruction set architecture (Instruction Set Architecture, ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C ++ or the like and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (Local Area Network, LAN) or a wide area network (Wide Area Network, WAN), or may be connected to an external computer (e.g., through the internet using an internet service provider). In some embodiments, aspects of the application are implemented by personalizing electronic circuitry, such as Programmable logic circuitry, field-Programmable gate arrays (GATE ARRAY, FPGA), or Programmable logic arrays (Programmable Logic Array, PLA), with state information for computer-readable program instructions.

Various aspects of the present application are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by hardware, such as circuits or ASIC (Application SPECIFIC INTEGRATED circuits) which perform the corresponding functions or acts, or combinations of hardware and software, such as firmware and the like.

Although the application is described herein in connection with various embodiments, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

The foregoing description of embodiments of the application has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the improvement of technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A risk assessment method for data leakage, the method comprising:

determining a reference model, wherein the reference model is a machine learning model constructed based on the training data set and is different from the model to be evaluated;

2. The method of claim 1, wherein the determining a reference model comprises:

3. The method according to claim 1 or2, wherein the reference model is a machine learning model that does not employ a defense mechanism and/or the number of model parameters of the reference model is greater than a number threshold.

4. A method according to any one of claims 1 to 3, wherein said obtaining respective relative loss differences for said plurality of training samples comprises:

5. The method according to any one of claims 1 to 4, wherein determining the risk result of the model to be evaluated according to the respective relative loss differences of the plurality of training samples comprises:

6. The method of claim 5, wherein the method further comprises:

7. A risk assessment apparatus for data leakage, the apparatus comprising:

8. A risk assessment apparatus for data leakage, the apparatus comprising:

A processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method of any of claims 1-6 when executing the instructions.

9. A non-transitory computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1-6.

10. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying the computer readable code, characterized in that a processor in an electronic device performs the method of any one of claims 1-6 when the computer readable code is run in the electronic device.