WO2021121128A1 - Artificial intelligence-based sample evaluation method, apparatus, device, and storage medium - Google Patents

Abstract

An artificial intelligence-based sample evaluation method, an apparatus, a computer device, and a storage medium. The method comprises: obtaining a training data set; using a training model to be evaluated to train N test training samples so as to obtain the sample loss function corresponding to the training model to be evaluated; selecting K target training samples; inputting the N-K test training samples other than the K target training samples in the training data set into the training model to be evaluated for training, so as to obtain a first influence function; inputting K feature updated samples and the N-K test training samples other than the K target training samples in the training data set into the training model to be evaluated for training, so as obtain a second influence function; obtaining the sample influence result on the basis of the first influence function and the second influence function. The present invention allows for interpretability of the influence of a target training sample on the output result of a training model to be evaluated, thereby facilitating ensuing optimization and improvement of the training model to be evaluated.

Description

Artificial intelligence-based sample evaluation method, device, equipment and storage medium

This application is based on the Chinese invention application with the application number 202010514014.0 filed on June 8, 2020 and titled "Artificial Intelligence-based Sample Evaluation Method, Apparatus, Computer Equipment and Storage Medium", and claims its priority.

Technical field

This application relates to the field of artificial intelligence, and in particular to a sample evaluation method, device, computer equipment, and storage medium based on artificial intelligence.

Background technique

In the field of artificial intelligence, the predictive performance of the training model to be evaluated is an important performance indicator. However, the sample output of the training model to be evaluated affects the interpretability of the result is also an important performance indicator. Because by understanding the reasons why the output samples of the training model to be evaluated affect the results, the factors that affect the output samples of the training model to be evaluated affect the results can be directly changed to improve the performance of the training model to be evaluated, and at the same time, it can also provide users with an explanation of the sample’s impact on the results. This is particularly important when the business involves sensitive user information.

However, existing training models to be evaluated in many fields, such as deep neural network models for image and speech recognition, are a complex black box model, and it is difficult to explain the impact of the output samples on the results. The prior art mainly focuses on understanding how the fixed training model to be evaluated corresponds to a specific sample impact result, for example, by locally fitting a simpler training model to be evaluated around the test data points or observing the output by adding interference to the test data The sample affects the results. The inventor realizes that the prior art only explains the impact of the sample output of the training model to be evaluated from the perspective of the training model to be evaluated, but does not have the impact of the output result of the training model to be evaluated from the perspective of the training sample, which is not conducive to the evaluation of the training sample. Follow-up optimization and improvement.

Summary of the invention

The embodiments of the present application provide an artificial intelligence-based sample evaluation method, device, computer equipment, and storage medium to solve the problem of the inability to explain the impact of the training sample on the output result of the training model to be evaluated.

A sample evaluation method based on artificial intelligence, including:

Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.

An artificial intelligence-based sample evaluation device, including:

A data acquisition module for acquiring a training data set, the training data set includes N test training samples, where N is a positive integer;

The sample training module is configured to train the N test training samples by using the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

The sample detection module is used to detect the N test training samples and select K target training samples, where K is a positive integer;

The first influence function module is configured to input NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first loss function corresponding to the sample loss function. An influence function;

The second influence function module is used to change the sample characteristics of the K target training samples, obtain K updated characteristic samples, and divide the K updated characteristic samples and the training data set by the K target training samples NK outside test training samples are input into the training model to be evaluated for training, and a second influence function corresponding to the sample loss function is obtained;

The result acquisition module is configured to acquire the sample influence results of the K target training samples on the training model to be evaluated based on the first influence function and the second influence function.

A computer device includes a memory, a processor, and computer-readable instructions that are stored in the memory and can run on the processor, and the processor implements the following steps when the processor executes the computer-readable instructions:

Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.

One or more readable storage media storing computer readable instructions, the computer readable storage medium storing computer readable instructions, and when the computer readable instructions are executed by one or more processors, the one Or multiple processors perform the following steps:

Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.

In the above-mentioned artificial intelligence-based sample evaluation method, device, computer equipment and storage medium, the server compares the output predicted value of the test training sample after training on the training model to be evaluated with the actual value corresponding to the N test training samples to obtain the corresponding The sample loss function for the next step is to analyze the reasons that affect the output prediction value of the training model to be evaluated; through the selected K target training samples to further test the impact of the output prediction value of the training model to be evaluated, target training The analysis or evaluation of the influence of the output result of the training model to be evaluated from the angle of the sample is helpful for subsequent optimization and improvement of the training model to be evaluated. Further, the server calculates the first influencing function and the second influencing function, while maintaining a relatively high fitting accuracy, so that the training model to be evaluated that cannot be derivable for calculus can be calculated at a lower computational cost. The first influence function and the second influence function obtain K target training samples' sample influence results of the training model to be evaluated. By analyzing the sample influence results, the influence of the target training samples to the training model to be evaluated is obtained, and the target training samples are realized The interpretability of the impact of the output results of the training model is helpful for subsequent optimization and improvement of the training model to be evaluated.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings that need to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

FIG. 1 is a schematic diagram of an application environment of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

2 is a flowchart of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

FIG. 3 is another flowchart of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

FIG. 4 is another flowchart of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

FIG. 5 is another flowchart of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

FIG. 6 is another flowchart of a sample evaluation method based on artificial intelligence in an embodiment of the present application;

FIG. 7 is a schematic diagram of a sample evaluation device based on artificial intelligence in an embodiment of the present application;

Fig. 8 is a schematic diagram of a computer device in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

According to the artificial intelligence-based sample evaluation method provided by the embodiments of the present application, the artificial intelligence-based sample evaluation method can be applied to the application environment shown in FIG. Specifically, the artificial intelligence-based sample evaluation method is applied to a sample evaluation system, which includes a client and a server as shown in FIG. Intelligent sample evaluation method. Among them, the client is also called the client, which refers to the program that corresponds to the server and provides local services to the client. The client can be installed on, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster composed of multiple servers. The server compares the output prediction value of the test training sample after training on the training model to be evaluated with the actual value corresponding to the N test training samples, and obtains the corresponding sample loss function, so that the next step can affect the output prediction of the training model to be evaluated Analyze the cause of the value; by selecting K target training samples to further test the impact of the output prediction value of the training model to be evaluated, the angle of the target training sample can be analyzed or evaluated for the impact of the output result of the training model to be evaluated. Helps subsequent optimization and improvement of the training model to be evaluated. Further, the server calculates the first influencing function and the second influencing function, while maintaining a relatively high fitting accuracy, so that the training model to be evaluated that cannot be derivable for calculus can be calculated at a lower computational cost. The first influence function and the second influence function obtain K target training samples' sample influence results of the training model to be evaluated. By analyzing the sample influence results, the influence of the target training samples to the training model to be evaluated is obtained, and the target training samples are realized The interpretability of the impact of the output results of the training model is helpful for subsequent optimization and improvement of the training model to be evaluated.

In an embodiment, as shown in FIG. 2, a sample evaluation method based on artificial intelligence is provided. The application of the method to the server in FIG. 1 is taken as an example for description, including the following steps:

S10: Obtain a training data set. The training data set includes N test training samples, where N is a positive integer.

Among them, the training data set is a user-defined set for storing test training samples. In this example, the training data set stores N test training samples for analysis based on the N test training samples. The test training sample includes the training data and the label corresponding to the training data. As an example, the test training sample may be a car damage training sample, where a car damage training sample specifically includes a car damage image and a label corresponding to the car damage image. In this case, the car damage image is training data.

S20: Use the training model to be evaluated to train N test training samples, and obtain a sample loss function corresponding to the training model to be evaluated.

Among them, the training model to be evaluated is a model that needs to be evaluated and analyzed, and specifically may be a deep learning model for training a test training sample. Optionally, the training model to be evaluated includes but is not limited to the Faster RCNN model or the SS D model. The sample loss function is a function that calculates the difference between the output predicted value of the training model to be evaluated and the actual value of the test training sample. The output prediction value is the value obtained after the training model to be evaluated trains the test training sample. The actual value is the actual value corresponding to the test training sample, and the actual value here can be understood as the label of the training data. For example, the actual value corresponding to the test training sample is A, the output prediction value obtained by the training model to be evaluated on the test training sample is B, and the sample loss function is a function that measures the difference between A and B

Specifically, the server inputs N test training samples to the training model to be evaluated for training. After the training model to be evaluated trains the N test training samples, the output prediction values corresponding to the N test training samples are obtained. The server compares the N output predicted values with the actual values corresponding to the N test training samples, obtains the sample loss values corresponding to the N test training samples, and then builds the sample loss function based on the N sample loss values for the next step The reasons that affect the output prediction value of the training model to be evaluated are analyzed.

S30: Detect N test training samples, and select K target training samples, where K is a positive integer.

Wherein, the target training sample is a sample used to test the influence of the output prediction value of the training model to be evaluated.

Specifically, the server detects N test training samples, and selects K target training samples from the training data set. Optionally, selecting the target training samples may be randomly selecting K target training samples from the training data set, or selecting K target training samples determined by screening N test training samples according to a preset selection method. Among them, the preset selection method is a user-defined selection method, which is used to select the target training sample.

Understandably, the server further tests the influence of the output prediction value of the training model to be evaluated by selecting K target training samples, which can improve the efficiency of analyzing the influence of the target training sample on the output prediction value of the training model to be evaluated.

S40: Input the N-K test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first influence function corresponding to the sample loss function.

Among them, the first influence function is a function for calculating the influence of N-K test training samples on the output prediction value of the training model to be evaluated.

Specifically, in order to analyze the influence of the target training sample on the output prediction value of the training model to be evaluated, the server passes the selected K target training samples, and then inputs the remaining NK test training samples in the training data set into the training model to be evaluated for testing. According to the output prediction value of the training model to be evaluated, further, the server calculates the output prediction value of the training model to be evaluated and the sample loss function to obtain the first influence function, so as to follow the output of the training model to be evaluated according to the first influence function The predicted value is analyzed, and the sample impact result is obtained to realize the analysis of the training samples that affect the output predicted value of the training model to be evaluated, which is helpful for optimizing and improving the training model to be evaluated from the perspective of training samples.

S50: Perform sample feature changes on K target training samples, obtain K updated feature samples, and input K updated feature samples and NK test training samples in the training data set except for K target training samples into the training model to be evaluated for training , Obtain the second influence function corresponding to the sample loss function.

Wherein, the second influence function is a function for calculating the influence of K update feature samples and N-K test training samples on the output prediction value of the training model to be evaluated.

Specifically, in order to analyze the impact of the target training sample on the output prediction value of the training model to be evaluated, the server obtains K updated feature samples by changing the sample characteristics of the target training sample, and trains the remaining NK test samples in the training data set. The samples and K updated feature samples are input to the training model to be evaluated for testing, and the second model parameters are obtained according to the output predicted value of the training model to be evaluated. Further, the server obtains the second influence function by calculating the second model parameters and the sample loss function, so as to subsequently analyze the output prediction value of the training model to be evaluated according to the second influence function, and obtain the influence of the sample influence result. Realize the analysis of the reasons that affect the output prediction value of the training model to be evaluated.

As an example, sample feature changes are performed on the sample features β of K target training samples to obtain K updated feature samples whose sample features are δ.

S60: Based on the first influence function and the second influence function, obtain K target training samples sample influence results of the training model to be evaluated.

Wherein, the sample influence result is the result of evaluating or analyzing the influence of the output prediction value of the training model to be evaluated.

Specifically, the server performs comprehensive analysis and processing on the first influence function and the second influence function based on the preset processing logic, and obtains the sample influence results of the K target training samples of the training model to be evaluated. Wherein, the preset processing logic is to perform weighting or difference processing on the first influencing function and the second influencing function. That is, after acquiring the first influence function and the second influence function, the server can analyze the sample influence result of the output prediction value of the training model to be evaluated by the target training sample through the first influence function and the second influence function. The server obtains the impact of the target training sample on the training model to be evaluated by analyzing the sample impact results, and realizes the evaluation or analysis of the impact of the output predicted value of the training model to be evaluated.

In this embodiment, the server compares the output predicted value of the test training sample after the training model to be evaluated is trained with the actual value corresponding to the N test training samples, and obtains the corresponding sample loss function, so that the next step can affect the expected value. Analyze the reason for the output prediction value of the evaluation training model; further test the impact of the output prediction value of the evaluation training model by selecting K target training samples, and the impact of the target training sample angle on the output prediction value of the evaluation training model Analysis or evaluation is helpful for subsequent optimization and improvement of the training model to be evaluated. Further, the server calculates the first influencing function and the second influencing function, while maintaining a relatively high fitting accuracy, so that the training model to be evaluated that cannot be derivable for calculus can be calculated at a lower computational cost. The first influence function and the second influence function obtain K target training samples' sample influence results of the training model to be evaluated. By analyzing the sample influence results, the influence of the target training samples to the training model to be evaluated is obtained, and the target training samples are realized The interpretability of the impact of the output results of the training model is helpful for subsequent optimization and improvement of the training model to be evaluated.

In one embodiment, as shown in FIG. 3, step S20, that is, using the training model to be evaluated to train N test training samples to obtain the sample loss function corresponding to the training model to be evaluated includes:

S21: Use the training model to be evaluated to train N test training samples, and obtain output prediction values corresponding to the N test training samples.

Specifically, the server trains N test training samples through the training model to be evaluated, and obtains the output prediction value of the training model to be evaluated. Understandably, the server can further calculate the sample loss function of the difference between the output prediction value of the training model to be evaluated and the actual value of the test training sample through the N test training samples and the predicted output value of the training model to be evaluated.

S22: Obtain a sample loss function based on the test training sample and the output prediction value.

Specifically, the server obtains N sample loss values by calculating the actual values of N test training samples and the output prediction values of the training model to be evaluated, and obtains the corresponding sample loss function based on the N sample loss values for subsequent The sample loss function is used to analyze the reasons that affect the output prediction value of the training model to be evaluated.

As an example, X is the input space car damage image, Y is the output space such as the label corresponding to the car damage image, and each test training sample is defined as Z ₁ ,...,Z _n , where Z _i =(X _i ,Y _i )∈X×Y. For a test training sample Z and wait

Suppose the empirical risk can be quadratic calculus and the initial model parameter θ is a convex function.

In this embodiment, the server obtains the test training sample after the training model to be evaluated is trained, and obtains the sample loss function used to calculate the difference between the output prediction value of the training model to be evaluated and the actual value corresponding to the test training sample, through The sample loss function further analyzes the reasons that affect the output prediction value of the training model to be evaluated to ensure the accuracy and validity of the analysis result.

In one embodiment, as shown in FIG. 4, step S30, that is, detecting N test training samples and selecting K target training samples includes:

S31: Obtain current sample parameters corresponding to N test training samples, and determine whether the current sample parameters meet the screening parameter threshold.

Among them, the current sample parameter is the data parameter in the test training sample. The filter parameter threshold is a value set by the user and is used to filter the current sample parameters.

Specifically, after the server obtains the current sample parameters corresponding to the N test training samples, it judges the current sample parameters and judges whether the current sample parameters are sufficient to meet the screening parameter threshold, so as to filter the test training samples by the screening parameter threshold, so that K target training samples that meet the screening parameter threshold are selected from N test training samples, and the K target training samples are used to analyze the reasons that affect the output prediction value of the training model to be evaluated, so that the subsequent K target training samples are to be evaluated The samples of the training model affect the results, and the corresponding training samples are updated to improve the accuracy of the training model to be evaluated.

As an example, the test training samples are car damage training samples, and the server obtains the current sample parameters corresponding to the car damage images in each car damage training sample in the training data set. The current sample parameters can be the image resolution size and the image level resolution. At least one of the evaluation features of the image rate, image vertical resolution, image brightness, and contrast. The screening parameter threshold corresponding to each evaluation feature is set to X. If the current sample parameter obtained is Y, and X<Y, the current sample parameter Determined as the target training sample.

S32: If the current sample parameter meets the screening parameter threshold, the test training sample is determined as the target training sample.

In this example, the conditions for filtering parameter thresholds can be user-defined thresholds. As an example, determining that the current sample parameter satisfies the screening parameter threshold is a screening condition for screening K target training samples from N test training samples. Specifically, the current sample parameter may be greater than or equal to the screening parameter threshold.

Specifically, when among the current sample parameters corresponding to the N test training samples, if there is a current sample parameter that meets the screening parameter threshold, the test training sample corresponding to the current sample parameter that meets the screening parameter threshold is determined as the target training sample.

In this embodiment, the server screens the test training samples through the screening parameter threshold, so as to screen out K target training samples that meet the screening parameter threshold from the N test training samples, and use K target training samples to influence The reason for the output prediction value of the training model to be evaluated is analyzed, so that the subsequent K target training samples affect the results of the sample of the training model to be evaluated, and the corresponding training samples are updated to improve the accuracy of the training model to be evaluated.

In one embodiment, as shown in FIG. 5, in step S40, NK test training samples in the training data set except the K target training samples are input into the training model to be evaluated for training, and the first training model corresponding to the sample loss function is obtained. An influence function, including:

S41: Input the N-K test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first change weight of the training model to be evaluated.

Wherein, the first change weight is the weight of each model parameter in the training model to be evaluated after training the training model to be evaluated using N-K test training samples in addition to the K target training samples.

Specifically, the server inputs N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, obtains the empirical risk based on the sample loss function, and obtains the first change weight based on the empirical risk. Among them, the empirical risk is the average value obtained by accumulating the target training sample through the sample loss function. It is understandable that after the K target training samples are eliminated, NK test training samples other than the K target training samples are trained in the training model to be evaluated, and the weights of the model parameters are changed accordingly, which is obtained through empirical risk The first change weight can further obtain the change of the sample weight of the target training sample to analyze the reasons that affect the output prediction value of the training model to be evaluated.

S42: Acquire the first model parameter of the training model to be evaluated according to the initial model parameters and the first change weight corresponding to the training model to be evaluated.

Among them, the initial model parameter is the initial parameter that minimizes the difference in the calculation of the sample loss function, and is a parameter obtained based on empirical risk. The first model parameter is a test parameter that minimizes the calculation difference between the sample loss function and the N-K test training samples.

As an example, the empirical risk is specifically

The initial model parameters are obtained by calculating the empirical risk

among them,

Is the initial model parameter, Θ is the collection of all models in the database, θ is the test model to be evaluated, and L(Z _i , θ) is the sample loss function.

S43: Obtain a first influence function corresponding to the sample loss function based on the first model parameter and the sample loss function corresponding to the training model to be evaluated.

Specifically, in order to analyze the impact of the target training sample on the output result of the training model to be evaluated, the server deletes the selected K target training samples, and then inputs the remaining NK test training samples in the training data set into the training model to be evaluated for testing. The initial model parameters and the first change weight corresponding to the training model are updated to obtain the first model parameters. Further, the server obtains the first influence function through calculations based on the amount of change between the initial model parameters and the first model parameters and the sample loss function, and analyzes the output predicted value of the training model to be evaluated through the first influence function, and obtains the sample influence The influence of the result to realize the analysis of the reason that influences the output prediction value of the training model to be evaluated.

Seek guidance.

Is the loss function

The second derivative of the Hesse matrix.

Is the first derivative of the Hessian matrix. _{Understandably, the server obtains the change amount J up,params} (Z) between the initial model parameter and the first model parameter, which can realize that without testing the training samples, it can evaluate the first model parameter after removing certain K target training samples. Impact.

In this embodiment, the server calculates the first model parameters and the sample loss function to maintain a relatively good fitting accuracy, so that the training model to be evaluated that cannot be derived and calculus can be evaluated at a lower computational cost. Calculate, obtain the first influence function, analyze the output prediction value of the training model to be evaluated through the first influence function, and obtain the influence of the sample on the result, so as to realize the analysis of the reasons that affect the output prediction value of the training model to be evaluated, and improve the The efficiency of the artificial intelligence sample evaluation method.

As an example, step S43 is to obtain the first influence function corresponding to the sample loss function based on the first model parameter and the sample loss function corresponding to the training model to be evaluated, including: based on the Hesse vector product and the preset number of iterations, The first model parameter and the sample loss function corresponding to the training model to be evaluated are processed, and the first influence function corresponding to the sample loss function is obtained.

Among them, the Hesse vector product is a method used to calculate the first influential function and the second influential function. The preset number of iterations is the number of iterative calculations on the first influencing function and the second influencing function set by the user.

Specifically, to obtain the first influence function J _up,loss (Z,Z _test ) of changing the sample weight on the target training sample Z _test , it is necessary to calculate the inverse of the Hessian matrix, which will consume huge computing resources. For this reason, this embodiment uses the Hessel vector product (H

To avoid directly calculating the inverse of the Hessian matrix, by efficiently estimating the Hessian vector product (H

To calculate the first influencing function J _up,loss (Z,Z _test ). and

The calculation of, can be achieved by stochastic estimation. The random parameter estimation method only needs to sample one sample point in each iteration, so it can greatly increase the calculation speed and reduce the computing resources; at the same time

middle

And use

Means

Taylor's estimate of the first j items:

From the nature of Taylor expansion, we know that when j→∞,

therefore

Unbiased estimate of

Still have

Here, the embodiment is based on

To calculate the first influencing function J _up,loss (Z,Z _test ).

Specifically, select K target training samples from the training data set, Z ₁ ... Z _k , and define

The initial value of the Hesse vector product is

The first influence function corresponding to the selected K target training samples is iteratively calculated according to the preset number of iterations

Determine the result of the last iteration calculation as the first influence function

In this embodiment, the server calculates the first influence function through the Hessian vector product to improve the calculation efficiency, so as to improve the efficiency of the server to evaluate or analyze the influence of the output prediction value of the training model to be evaluated through the sample influence result. And through the high-efficiency calculation of the Hessian vector product, the first influence function is obtained to realize the evaluation or analysis of the influence of the output prediction value of the training model to be evaluated.

In one embodiment, as shown in FIG. 6, in step S50, the K update feature samples and NK test training samples in the training data set except the K target training samples are input into the training model to be evaluated for training, and the samples are obtained The second influence function corresponding to the loss function includes:

S51: Input the K update feature samples and N-K test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the second change weight corresponding to the training model to be evaluated.

Wherein, the second change weight is that after the sample feature of the target training sample is changed, the weight of the feature sample is updated. Since the sample features of the K target training samples are changed, the weights of the K updated feature samples in the training model to be evaluated are changed accordingly, and the second change weight can further obtain the change of the sample weight and affect the training model to be evaluated The reason for the output predicted value is analyzed.

Understandably, after the sample feature of the target training sample is changed, the response to the updated feature sample Z _δ weight ∈ is also changed. The server can further obtain the sample weight change through the second change weight and affect the output of the training model to be evaluated. The reason for the predicted value is analyzed.

S52: Obtain a second model parameter corresponding to the training model to be evaluated according to the initial model parameter and the second change weight corresponding to the training model to be evaluated.

Wherein, the second model parameter is a test parameter that minimizes the calculation difference between the K update feature samples and the N-K test training samples of the sample loss function.

Specifically, the server calculates the initial, initial model parameters and the second change weight, and obtains the training model to be evaluated.

S53: Obtain a second influence function corresponding to the sample loss function according to the second model parameter and the sample loss function corresponding to the training model to be evaluated.

Specifically, after the server changes the sample characteristics of the target training sample, based on the second change weight, obtains the second model parameters corresponding to the training model to be evaluated, and calculates the second model parameters and the sample loss function to obtain the second impact function.

As an example, the server first calculates the second model parameters based on the second change weight to obtain the initial model

The influence of type parameters. Further, the second influence function is used to update the feature by the sample loss function of the test training sample

In this embodiment, the server can further obtain the change in sample weight through the second change weight to analyze the reasons that affect the output prediction value of the training model to be evaluated; then, use the second change weight to obtain the corresponding training model to be evaluated. The second model parameter is to calculate the second model parameter and the sample loss function, while maintaining a better fitting accuracy, so that the training model to be evaluated that cannot be derived can be calculated and obtained at a lower computational cost. The influence function, so that the server can evaluate or analyze the influence of the output prediction value of the training model to be evaluated through the second influence function.

In one embodiment, step S53 is to obtain the second influence function corresponding to the sample loss function based on the second model parameter and the sample loss function corresponding to the training model to be evaluated, including: based on the Hessian vector product and the preset iteration The number of times, the second model parameter and the sample loss function corresponding to the training model to be evaluated, and the second influence function corresponding to the sample loss function is obtained.

Specifically, to obtain the second influence function J _pert,loss (Z,Z _test ) ^T of the target training sample Z _test by changing the sample characteristics, it needs

The corresponding second influence function is iteratively calculated according to the preset number of iterations

Determine the result of the last iteration calculation as the second influence function

In this embodiment, the server calculates the second influence function through the Hessian vector product to improve the calculation efficiency, so as to improve the efficiency of the server to evaluate or analyze the influence of the output prediction value of the training model to be evaluated through the sample influence result. And through the high-efficiency calculation of the Hessian vector product, the second influence function is obtained to realize the evaluation or analysis of the influence of the output prediction value of the training model to be evaluated.

It should be understood that the size of the sequence number of each step in the foregoing embodiment does not mean the order of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

In one embodiment, an artificial intelligence-based sample evaluation device is provided, and the artificial intelligence-based sample evaluation device corresponds to the artificial intelligence-based sample evaluation method in the above-mentioned embodiment in a one-to-one correspondence. As shown in FIG. 7, the artificial intelligence-based sample evaluation device includes a data acquisition module 10, a sample training module 20, a sample detection module 30, a first influence function module 40, a second influence function module 50 and a result acquisition module 60. The detailed description of each functional module is as follows:

The data acquisition module 10 is used to acquire a training data set, the training data set includes N test training samples, where N is a positive integer;

The sample training module 20 is used to train N test training samples using the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

The sample detection module 30 is used to detect N test training samples and select K target training samples, where K is a positive integer;

The first influence function module 40 is configured to input N-K test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first influence function corresponding to the sample loss function;

The second influence function module 50 is used to change the sample characteristics of the K target training samples, obtain K updated feature samples, and combine the K updated feature samples and the training data set with NK test training samples other than the K target training samples Input the training model to be evaluated for training, and obtain the second influence function corresponding to the sample loss function;

The result acquisition module 60 is configured to acquire K target training samples based on the first influence function and the second influence function, and sample influence results of the training model to be evaluated.

Further, the sample training module 20 includes:

The prediction value acquisition sub-module is used to train N test training samples using the training model to be evaluated, and obtain the output prediction values corresponding to the N test training samples;

The loss function sub-module is used to obtain the sample loss function based on the test training sample and the output prediction value.

Further, the sample detection module 30 includes:

The threshold judgment sub-module is used to obtain the current sample parameters corresponding to the N test training samples, and judge whether the current sample parameters meet the screening parameter threshold;

The sample determination sub-module is used to determine the test training sample as the target training sample when the current sample parameter meets the screening parameter threshold.

Further, the first influencing function module 40 includes:

The first weight sub-module is used to input N-K test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first change weight of the training model to be evaluated;

The first parameter sub-module is used to obtain the first model parameter of the training model to be evaluated according to the initial model parameters and the first change weight corresponding to the training model to be evaluated;

The first function sub-module is used to obtain the first influence function corresponding to the sample loss function based on the first model parameter and the sample loss function corresponding to the training model to be evaluated.

Further, the first influencing function module 40 also includes:

The first parameter processing sub-module is used to process the first model parameter and sample loss function corresponding to the training model to be evaluated based on the Hessian vector product and the preset number of iterations to obtain the first influence function corresponding to the sample loss function.

Further, the second influence function module 50 includes:

The second weight sub-module is used to input the K update feature samples and NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the second change weight corresponding to the training model to be evaluated ；

The second parameter sub-module is used to obtain the second model parameter corresponding to the training model to be evaluated according to the initial model parameter and the second change weight corresponding to the training model to be evaluated;

The second function sub-module is used to obtain the second influence function corresponding to the sample loss function based on the second model parameter and the sample loss function corresponding to the training model to be evaluated.

Further, the second influencing function module 50 further includes:

The second parameter processing sub-module is used to obtain the second influence function corresponding to the sample loss function based on the Hessian vector product and the preset number of iterations, the second model parameter and the sample loss function corresponding to the training model to be evaluated.

Regarding the specific limitations of the artificial intelligence-based sample evaluation device, please refer to the above limitations on the artificial intelligence-based sample evaluation method, which will not be repeated here. Each module in the above-mentioned artificial intelligence-based sample evaluation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules may be embedded in the form of hardware or independent of the processor in the computer equipment, or may be stored in the memory of the computer equipment in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. 8. The computer equipment includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The computer equipment database is used for sample evaluation. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer-readable instructions are executed by the processor to realize an artificial intelligence-based sample evaluation method.

In one embodiment, one or more readable storage media storing computer readable instructions are provided, the computer readable storage medium storing computer readable instructions, and the computer readable instructions are processed by one or more When the processor executes, the one or more processors execute the computer-readable instructions to implement the artificial intelligence-based sample evaluation method in the foregoing embodiment, such as steps S10 to S60. To avoid repetition, details are not described herein again. Or, when the processor executes the computer-readable instructions, the functions of the modules/units in the embodiment of the artificial intelligence-based sample evaluation device, such as modules 10 to 60, are implemented. To avoid repetition, details are not described herein again. The readable storage medium in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

In one embodiment, a computer-readable storage medium is provided, and computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are executed by a processor, the artificial intelligence-based sample evaluation method in the above-mentioned embodiment is implemented. For example, step S10 to step S60, in order to avoid repetition, the details are not repeated here. Alternatively, when the computer-readable instruction is executed by the processor, the function of each module/unit in the embodiment of the artificial intelligence-based sample evaluation device, such as module 10 to module 60, is realized. In order to avoid repetition, details are not described herein again.

A person of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be implemented by instructing relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile computer. In a readable storage medium, the computer readable instruction may be stored in a non-volatile readable storage medium or may be stored in a volatile readable storage medium. When the computer readable instruction is executed, it may include The flow of the embodiment of each method. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, only the division of the above functional units and modules is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed. Module completion, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, a person of ordinary skill in the art should understand that it can still implement the foregoing The technical solutions recorded in the examples are modified, or some of the technical features are equivalently replaced; these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of protection of this application.

Claims (20)

1. A sample evaluation method based on artificial intelligence, which includes:

   Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

   Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

   Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

   Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

   Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

   Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.
2. The artificial intelligence-based sample evaluation method according to claim 1, wherein the training the N test training samples with the training model to be evaluated, and obtaining the sample loss function corresponding to the training model to be evaluated, comprises:

   Training the N test training samples with the training model to be evaluated, and obtain the output prediction values corresponding to the N test training samples;

   Obtain the sample loss function based on the test training sample and the output prediction value.
3. The method for sample evaluation based on artificial intelligence according to claim 1, wherein said detecting the N test training samples and selecting K target training samples comprises:

   Acquiring current sample parameters corresponding to the N test training samples, and judging whether the current sample parameters meet the screening parameter threshold;

   If the current sample parameter meets the screening parameter threshold, the test training sample is determined as the target training sample.
4. The artificial intelligence-based sample evaluation method according to claim 1, wherein said inputting NK test training samples in said training data set except K said target training samples into said training model to be evaluated for training, Obtaining the first influence function corresponding to the sample loss function includes:

   Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining the first change weight of the training model to be evaluated;

   Acquiring the first model parameter of the training model to be evaluated according to the initial model parameters corresponding to the training model to be evaluated and the first change weight;

   Based on the first model parameter corresponding to the training model to be evaluated and the sample loss function, a first influence function corresponding to the sample loss function is obtained.
5. The artificial intelligence-based sample evaluation method according to claim 4, wherein the first model parameter corresponding to the training model to be evaluated and the sample loss function are obtained based on the first model parameter corresponding to the sample loss function. An influence function, including:

   Based on the Hessian vector product and the preset number of iterations, the first model parameter corresponding to the training model to be evaluated and the sample loss function are processed to obtain the first influence function corresponding to the sample loss function.
6. The artificial intelligence-based sample evaluation method according to claim 3, wherein the K update feature samples and NK test training samples other than the K target training samples in the training data set are input into the training data set. The training model to be evaluated is trained to obtain the second influence function corresponding to the sample loss function, including:

   Input the K update feature samples and the NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the second training model corresponding to the training model to be evaluated. Change weight

   Acquiring the second model parameter corresponding to the training model to be evaluated according to the initial model parameter corresponding to the training model to be evaluated and the second change weight;

   Based on the second model parameter corresponding to the training model to be evaluated and the sample loss function, a second influence function corresponding to the sample loss function is obtained.
7. The artificial intelligence-based sample evaluation method of claim 6, wherein the second model parameter corresponding to the training model to be evaluated and the sample loss function are used to obtain the first sample loss function corresponding to the sample loss function. Two influence functions, including:

   Based on the Hessian vector product and the preset number of iterations, for the second model parameter corresponding to the training model to be evaluated and the sample loss function, a second influence function corresponding to the sample loss function is obtained.
8. A sample evaluation device based on artificial intelligence, which includes:

   A data acquisition module for acquiring a training data set, the training data set includes N test training samples, where N is a positive integer;

   The sample training module is configured to train the N test training samples by using the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

   The sample detection module is used to detect the N test training samples and select K target training samples, where K is a positive integer;

   The first influence function module is configured to input NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the first loss function corresponding to the sample loss function. An influence function;

   The second influence function module is used to change the sample characteristics of the K target training samples, obtain K updated characteristic samples, and divide the K updated characteristic samples and the training data set by the K target training samples NK outside test training samples are input into the training model to be evaluated for training, and a second influence function corresponding to the sample loss function is obtained;

   The result acquisition module is configured to acquire the sample influence results of the K target training samples on the training model to be evaluated based on the first influence function and the second influence function.
9. A computer device includes a memory, a processor, and computer-readable instructions that are stored in the memory and can run on the processor, wherein the processor implements the following steps when the processor executes the computer-readable instructions:

   Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

   Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

   Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

   Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

   Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

   Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.
10. 9. The computer device according to claim 9, wherein the training the N test training samples with the training model to be evaluated, and obtaining the sample loss function corresponding to the training model to be evaluated, comprises:

    Training the N test training samples with the training model to be evaluated, and obtain the output prediction values corresponding to the N test training samples;

    Obtain the sample loss function based on the test training sample and the output prediction value.
11. 9. The computer device according to claim 9, wherein said detecting the N test training samples and selecting K target training samples comprises:

    Acquiring current sample parameters corresponding to the N test training samples, and judging whether the current sample parameters meet the screening parameter threshold;

    If the current sample parameter meets the screening parameter threshold, the test training sample is determined as the target training sample.
12. The computer device according to claim 9, wherein the NK test training samples except the K target training samples in the training data set are input into the training model to be evaluated for training, and the samples are obtained The first influence function corresponding to the loss function includes:

    Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining the first change weight of the training model to be evaluated;

    Acquiring the first model parameter of the training model to be evaluated according to the initial model parameters corresponding to the training model to be evaluated and the first change weight;

    Based on the first model parameter corresponding to the training model to be evaluated and the sample loss function, a first influence function corresponding to the sample loss function is obtained.
13. The computer device according to claim 12, wherein the obtaining the first influence function corresponding to the sample loss function based on the first model parameter corresponding to the training model to be evaluated and the sample loss function comprises :

    Based on the Hessian vector product and the preset number of iterations, the first model parameter corresponding to the training model to be evaluated and the sample loss function are processed to obtain the first influence function corresponding to the sample loss function.
14. The computer device according to claim 11, wherein the K update feature samples and NK test training samples in the training data set except the K target training samples are input into the training model to be evaluated Training to obtain the second influence function corresponding to the sample loss function includes:

    Input the K update feature samples and the NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the second training model corresponding to the training model to be evaluated. Change weight

    Acquiring the second model parameter corresponding to the training model to be evaluated according to the initial model parameter corresponding to the training model to be evaluated and the second change weight;

    Based on the second model parameter corresponding to the training model to be evaluated and the sample loss function, a second influence function corresponding to the sample loss function is obtained.
15. One or more readable storage media storing computer readable instructions, the computer readable storage medium storing computer readable instructions, where the computer readable instructions when executed by one or more processors cause all The one or more processors execute the following steps:

    Acquiring a training data set, the training data set including N test training samples, where N is a positive integer;

    Training the N test training samples with the training model to be evaluated, and obtain the sample loss function corresponding to the training model to be evaluated;

    Perform detection on the N test training samples, and select K target training samples, where K is a positive integer;

    Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining a first influence function corresponding to the sample loss function;

    Perform sample feature changes on the K target training samples, obtain K updated feature samples, and input the K updated feature samples and NK test training samples in the training data set except the K target training samples Training the training model to be evaluated to obtain a second influence function corresponding to the sample loss function;

    Based on the first influence function and the second influence function, obtain sample influence results of the K target training samples on the training model to be evaluated.
16. 15. The readable storage medium according to claim 15, wherein the training the N test training samples using the training model to be evaluated and obtaining the sample loss function corresponding to the training model to be evaluated comprises:

    Training the N test training samples with the training model to be evaluated, and obtain the output prediction values corresponding to the N test training samples;

    Obtain the sample loss function based on the test training sample and the output prediction value.
17. 15. The readable storage medium according to claim 15, wherein said detecting the N test training samples and selecting K target training samples comprises:

    Acquiring current sample parameters corresponding to the N test training samples, and judging whether the current sample parameters meet the screening parameter threshold;

    If the current sample parameter meets the screening parameter threshold, the test training sample is determined as the target training sample.
18. The readable storage medium according to claim 15, wherein the NK test training samples except the K target training samples in the training data set are input into the training model to be evaluated for training, and the training data is obtained The first influence function corresponding to the sample loss function includes:

    Inputting N-K test training samples in the training data set excluding the K target training samples into the training model to be evaluated for training, and obtaining the first change weight of the training model to be evaluated;

    Acquiring the first model parameter of the training model to be evaluated according to the initial model parameters corresponding to the training model to be evaluated and the first change weight;

    Based on the first model parameter corresponding to the training model to be evaluated and the sample loss function, a first influence function corresponding to the sample loss function is obtained.
19. The readable storage medium of claim 18, wherein the first model parameter corresponding to the training model to be evaluated and the sample loss function are used to obtain a first influence function corresponding to the sample loss function ,include:

    Based on the Hessian vector product and the preset number of iterations, the first model parameter corresponding to the training model to be evaluated and the sample loss function are processed to obtain the first influence function corresponding to the sample loss function.
20. The readable storage medium according to claim 17, wherein the K pieces of the updated feature samples and the NK pieces of test training samples in the training data set excluding the K pieces of the target training samples are input into the to-be-evaluated The training model is trained to obtain the second influence function corresponding to the sample loss function, including:

    Input the K update feature samples and the NK test training samples in the training data set except the K target training samples into the training model to be evaluated for training, and obtain the second training model corresponding to the training model to be evaluated. Change weight

    Acquiring the second model parameter corresponding to the training model to be evaluated according to the initial model parameter corresponding to the training model to be evaluated and the second change weight;

    Based on the second model parameter corresponding to the training model to be evaluated and the sample loss function, a second influence function corresponding to the sample loss function is obtained.

Images