CN114880206A

CN114880206A - Interpretability method of mobile application program code submission fault prediction model

Info

Publication number: CN114880206A
Application number: CN202210038845.4A
Authority: CN
Inventors: 陈翔; 胡新宇; 翟瀚丰; 高朝阳; 夏鸿崚; 顾亚锋; 杨少宇
Original assignee: Nantong University
Current assignee: Nantong University
Priority date: 2022-01-13
Filing date: 2022-01-13
Publication date: 2022-08-09

Abstract

The invention provides an interpretable method for submitting a fault prediction model by mobile application program codes, which comprises the following steps: (1) collecting code submissions from the mobile application program project through a mining version control system, and then measuring and labeling the code submissions to form a data set; (2) constructing a mobile application program code submitting fault prediction model by means of random forests according to the data set; (3) when a new code is submitted, measuring the code submission, and inputting the measured code submission into a code submission fault prediction model to obtain a prediction result; (4) and carrying out super-parameter optimization on the LIME of the local interpretability technology, and generating an explanation of the prediction result by using an ExplaiApp (finite element analysis) method of the super-parameter optimization. The invention has the beneficial effects that: whether a new code submission introduces a fault is predicted, and meanwhile, a corresponding high-quality explanation can be given to assist developers in completing understanding, positioning and repairing of subsequent faults.

Description

Interpretability method of mobile application program code submission fault prediction model

Technical Field

The invention relates to the technical field of computers, in particular to an interpretable method of a mobile application program code submission failure prediction model.

Background

In recent years, with the explosion of mobile internet and the rapid popularization of smart mobile devices, more and more smart phone users communicate with each other, obtain information and entertain through their devices. The user can download various application software in the application market at any time, and developers can continuously develop new application software or update and optimize released software to meet the requirements of the user and optimize the experience of the user. However, frequent updates of the application, which would involve a large amount of code submission, may introduce failures to new versions of the mobile application, thereby affecting the quality of the application. During the development and maintenance process of the mobile application program, faults are found in time and are solved by developers, and unnecessary loss can be avoided. The task of detecting a fault is called fault prediction.

Researchers have proposed commit-level software failure prediction to identify whether newly committed code changes will introduce failures, thereby providing developers with timely feedback to discover and repair failures as early as possible. At present, the code submission failure prediction is applied to the mobile application program in the prior art, and because the code submission failure prediction has the advantages of fine granularity, instantaneity, good traceability and the like, the code submission failure prediction is suitable for the mobile application program which is frequently updated.

However, in the field of mobile application code submission failure prediction, most research focuses only on the performance of the prediction model, and ignores the model's interpretability. The fault prediction model can well detect whether the target code segment contains faults by applying a machine learning technology, but some machine learning models are based on black box models (such as random forests, neural networks and the like) and therefore are not interpretable. That is, we give the model an input and the model gives us a prediction result, but we do not know what the basis behind the model gives us this result. This may cause the developer to lack confidence in the prediction result of the failure prediction model and prevent the failure prediction model from being applied in practice.

How to solve the above technical problems is the subject of the present invention.

Disclosure of Invention

The invention aims to provide an interpretable method of a mobile application program code submission failure prediction model, which can better help developers to understand the prediction result of the code submission failure prediction model on a test case.

The idea of the invention is as follows: the invention applies the interpretable method after the super-parametric optimization to the mobile application program code submission fault prediction, provides visual interpretation to the prediction result, constructs a mobile application program code submission fault prediction model through a data set, carries out the super-parametric optimization to the local interpretable model LIME, then uses the LIME after the super-parametric optimization to locally interpret the single sample result obtained by the model, and as for the fitting goodness (R2) of the local linear regression model, the LIME model after the super-parametric optimization is superior to the original LIME model.

The invention is realized by the following measures: an interpretable method of a mobile application code submission failure prediction model, comprising the steps of:

(1) code submissions from within the mobile application project are gathered by the mining version control system. And designing characteristics of the five dimensions of the first dispersion degree of the code submission, the modification amount of the code submission, the modification purpose of the code submission, the history of the code submission and the experience of developers related to the code submission, and measuring the collected code submission by using the characteristics. The code submissions are then marked, i.e., marked as both code submissions with and without introducing faults. After completing the measurement and marking of the code submission, a data set D is formed.

(2) And constructing a code submission fault prediction model M by means of random forests based on the data set D.

(3) When a new code submission x is faced, the code submission is measured by using the same characteristics in step 1, and then the measured value is input into a code submission failure prediction model M to obtain a prediction result (i.e., whether the code submission introduces a failure is detected).

(4) An interpretation of the predicted result is generated using the hyperparametric optimized LIME method (i.e., the explaiapps method). The local model independent interpretability method LIME method is an explanation given to a prediction result of a test case, and n adjacent virtual cases surrounding a code submission x are randomly generated through the code submission x needing prediction; then using the n virtual instances to obtain a prediction result through a code submission prediction model M, and forming a data set D by the n virtual instances and the prediction result _lime (ii) a From a data set D _lime And constructing a local regression model ML, and identifying the contribution of the corresponding characteristics to the prediction result based on the coefficients of the local regression model ML. Because the LIME method has the hyperparameter (namely the virtual instance number k needing to be generated), the value of the optimal hyperparameter of the LIME method is found by means of a differential evolution algorithm, and the optimization goal is to construct the goodness of fit (goodness of fit) of a local regression model ML.

Hyperparametric optimization of LIME by differential evolution algorithm finds the value of the number of instances generated when the linear model goodness of fit is best, where the objective function is to maximize the goodness of fit of the local regression model (R2). And obtaining the goodness-of-fit of the model by using a score () function in the LIME toolkit, and setting a random seed so that the goodness-of-fit values obtained each time for the models with the same virtual instance number are the same. The method for carrying out differential evolution on the LIME by giving a code submission instance x and a code submission failure prediction model M specifically comprises the following steps:

4-1) initializing the population, and randomly generating the initial population with the size. Since the number of instances is generated when the goodness of fit of the linear model is to be found to be optimal, each individual in the population is the number of randomly generated instances k (k random generation) around the submitted instance x, i.e., each individual is 1-dimensional, the initial population may be represented as S ═ { k ═ k ₁ ,k ₂ ,k ₃ ,…,k _size }. Let k be the upper and lower boundaries of an individual _max And k _min Then the individual is initialized as follows:

k _i,t ＝k _min +rand(0,1)*(k _max -k _min )

wherein, i is 1,2, size, t is iteration number, and the initial value is 1. k is a radical of _i,t Denotes the ith individual in the tth generation, and rand (0,1) denotes a random number that follows a uniform distribution over the interval (0, 1).

(4-2) for each candidate individual, constructing a local regression model ML using LIME based on the randomly generated virtual instances and the prediction results of the M model.

(4-3) selecting the optimal individual from the population, which has the optimal goodness of fit of the regression model.

(4-4) generating a new population based on the population through mutation, crossover and selection operations using a scaling factor pr and a crossover probability pc.

Mutation operation, the mutation strategy used is as follows:

v _i,t ＝k _r1,t +pr*(k _r2,t -k _r3,t )

wherein k is _r1,t ，k _r2,t And k _r3,t Randomly selecting three different individuals in the population in the t iteration, v _i,t Is a target variant individual; r1, r2, r3 is different from {1,2, ·, size } and is different from the current target index i, so size is equal to or larger than 4; the scaling factor pr is between (0,1), the value of pr is selected to be moderate, too small pr can reduce the convergence speed of the algorithm, and too large pr can cause the non-convergence of the population.

Crossover operation, after mutation operation, on the t-th generation of the population { k _i,t And variant intermediates thereof { v } _i,t Performing intercross operation among individuals:

wherein the crossover probability pc is a constant in the interval (0, 1).

And (6) selecting operation. Test subject u _i,t And target individual k _i,t And comparing the objective function values of (a) to select a better individual. That is, the goodness-of-fit values of the two are compared, and an individual having a larger goodness-of-fit value is selected. Selection operationThe formula of (1) is as follows:

(4-5) re-executing the program until the program reaches the specified number of iterations iter.

(4-6) finally, the number of instances k required to produce the highest goodness of fit in all iterations and the model at the highest goodness of fit is derived _best . The number k of the best examples _best Applied to a regression model and generates a corresponding interpretation based on the coefficients of the regression model.

The parameter value setting in the method is as follows:

k ranges from 100 to 10000

size of 10

The cross probability pc is 0.5

The scaling factor pr is 0.8

The maximum number of iterations iter is 100.

Compared with the prior art, the invention has the beneficial effects that: the invention provides an interpretable method of a mobile application program code submission fault prediction model, wherein a LIME model after super-parameter optimization is applied to the mobile application program code submission fault prediction model, so that whether a new code submission introduces a fault can be predicted, a predicted result can be reasonably explained, developers can be helped to locate and repair the fault in time, other researchers can be helped to understand what the basis behind the model prediction result is, and the trust of a user on the prediction model is enhanced.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.

FIG. 1 is a system framework diagram of an interpretable method of a mobile application code submission failure prediction model according to the present invention.

Fig. 2 is a visualization explanatory diagram based on an application example of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative of the invention and are not intended to be limiting.

Example 1

Referring to fig. 1, the present invention provides an interpretable method of a mobile application code submission failure prediction model, which specifically includes the following contents:

(1) code submissions from within the mobile application project are gathered by the mining version control system. The invention selects 14 Android mobile applications from different application fields and scales from a COMMIT GURU platform by using an open source data set provided by previous research. Table 1 summarizes the basic statistics of these applications, including the application name, the number of code lines (# LOC), the total number of committed instances (# TC), the number of faulty instances (# DC), the number of non-faulty instances (# CC), and the proportion of faulty instances (% DR).

14 metrics from five dimensions were used for the collected code submissions. These 5 dimensions are: the method comprises the following steps of distributing degree of code submission, modification amount of code submission, modification purpose of code submission, history of code submission and experience of developers related to code submission. Table 2 gives a brief description of 14 characteristics that measure code changes. That is, features are first designed from five dimensions and used to measure the collected code submissions. The code submissions are then marked, i.e., marked as both code submissions with introduced faults and code submissions without introduced faults. After completing the measurement and marking of the code submission, a data set D is formed.

Basic information of table 114 Android applications

Table 214 profiles

(2) And dividing the data set D into a training set and a testing set, and constructing a code submission fault prediction model M through a random forest by using the training set.

(3) And selecting a code submission instance in the test set to input into the code submission failure prediction model M to obtain a prediction result (namely detecting whether the code submission introduces a failure or not).

(4) An interpretation of the predicted result is generated using the hyperparametric optimized LIME method (i.e., the explaiapps method). Because the LIME method has the hyperparameter (namely the virtual instance number k needing to be generated), the value of the optimal hyperparameter of the LIME method is found by means of a differential evolution algorithm, and the optimization goal is to construct the goodness of fit (goodness of fit) of a local regression model ML. And (3) obtaining the goodness of fit of the model by using a score () function in the LIME toolkit, and setting a random seed so that the goodness of fit values obtained each time for the models with the same virtual instance number are the same. Given a code submission instance x and a code submission failure prediction model M, the differential evolution of the LIME specifically comprises the following steps:

4-1) initializing the population, and randomly generating the initial population with the size. Since the number of instances is generated when the goodness of fit of the linear model is to be found to be optimal, each individual in the population is the number of randomly generated instances k (k random generated) around the submitted instance x, i.e., each individual is 1-dimensional, the initial population can be represented as S ═ { k ═ k { ₁ ,k ₂ ,k ₃ ,…,k _size }. Let k be the upper and lower boundaries of an individual _max And k _min Then the individual is initialized as follows:

k _i,t ＝k _min +rand(0,1)*(k _max -k _min )

wherein i is 12, size, t is the number of iterations with an initial value of 1. k is a radical of _i,t Denotes the ith individual in the tth generation, and rand (0,1) denotes a random number that follows a uniform distribution over the interval (0, 1).

And (5) performing mutation operation. The mutation strategy used is as follows:

v _i,t ＝k _r1,t +pr*(k _r2,t -k _r3,t )

And (4) performing a crossover operation. After mutation operation, for the t-th generation of population { k } _i,t And variant intermediates thereof { v } _i,t Performing intercross operation among individuals:

wherein the crossover probability pc is a constant in the interval (0, 1).

And (6) selecting operation. Test subject u _i,t And target individual k _i,t Comparing the objective function values of the two groups of the same group, and selecting a better individual. That is, the goodness-of-fit values of the two are compared, and an individual having a larger goodness-of-fit value is selected. The formula for the selection operation is as follows:

(5) Selecting an example of a test set in a Firewall application program, and interpreting a prediction result of the selected example in the test set by using a LIME technology (namely ExplaiApp) after hyperparameter optimization to obtain a visual interpretation result. Fig. 2 gives a visual explanation of the example generated by the LIME model after hyperparametric optimization (i.e. the explaiapps method), the example being predicted to be faulty with a probability of 73%. The right bar (+) of figure 2 represents the score of the metric that is predicted to be fault-supporting for the instance, while the left bar (-) represents the score of the metric that is predicted to be fault-objecting for the instance.

(6) For each instance of the test data set for each application, a hyperparametric optimization was performed, and we analyzed the goodness of fit (R2) for LIME and explainps for the same instance. Table 3 gives the mean of the goodness of fit of the local linear regression models constructed by 14 applications using LIME and explaiapps methods.

TABLE 3 mean of goodness of fit for 14 applications using LIME and ExplaiApp methods

Experiments show that the hyperparametrically optimized LIME (i.e. the explainpap method) is always better than the original LIME in terms of goodness of fit of the local regression model, as can be seen from the table. For these 14 applications, an average improvement of 94.50% was achieved for all test cases. Alfresco increased the most, by 1.43 times. The application lozic is improved by 0.18% with a minimum, because the goodness of fit obtained by the application lozic using the original LIME is high, and the average value is 0.976, which is difficult to improve.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. An interpretable method for submitting a failure prediction model based on mobile application code, comprising the steps of:

the method comprises the following steps: collecting code submissions from a mobile application program project through a mining version control system, firstly measuring the collected code submissions by using five dimensional design characteristics of the code submissions, such as the dispersion degree of the code submissions, the modification quantity of the code submissions, the modification purpose of the code submissions, the history of the code submissions and the experience of related developers of the code submissions, marking the code submissions as the code submissions with introduced faults and the code submissions without introduced faults, and forming a data set D after finishing the measurement and marking of the code submissions;

step two: constructing a code submission fault prediction model M by means of a random forest based on the data set D;

step three: when a new code submission x is faced, measuring the code submission by adopting the same characteristics in the step one, then inputting the measured value into a code submission fault prediction model M to obtain a prediction result, and detecting whether a fault is introduced into the code submission;

step four: using a hyperparametric optimization LIME method to generate an explanation of the prediction result, wherein a local model independent interpretability method LIME method is an explanation given to the prediction result of the test case, and randomly generating n adjacent virtual cases surrounding the code submission x through the code submission x needing to be predicted; then using the n virtual instances to obtain a prediction result through a code submission prediction model M, and forming the n virtual instances and the prediction resultData set D _lime (ii) a From a data set D _lime And constructing a local regression model ML, identifying the contribution of corresponding characteristics to a prediction result based on the coefficient of the local regression model ML, and searching the value of the optimal hyperparameter of the LIME method by means of a differential evolution algorithm due to the fact that the LIME method has the hyperparameter, wherein the optimization goal is to construct the goodness of fit of the local regression model ML.

2. The interpretability method of the mobile application code-submission failure prediction model of claim 1, wherein the step four is to optimize parameter settings of the LIME algorithm by using a hyperparametric optimization algorithm, find the value of the number of instances generated when the goodness of fit of the linear model is best by a differential evolution algorithm, wherein the objective function is to maximize the goodness of fit of the local regression model (R2), obtain the goodness of fit of the model by score () function in the LIME toolkit, set a random seed so that the model with the same number of virtual instances has the same goodness of fit each time, and differentially evolve the LIME by giving one code-submission instance x and the code-submission failure prediction model M, which specifically comprises the steps of:

(4-1) initializing a population, randomly generating an initial population of size, generating a number of instances when the goodness of fit of the linear model is to be found to be optimal, wherein each individual in the population is the number k of randomly generated instances surrounding a submitted instance x, and each individual is 1-dimensional, and the initial population can be represented as S ═ { k ═ k { ₁ ,k ₂ ,k ₃ ,…,k _size Let k be the upper and lower boundaries of the individual _max And k _min Then the individual is initialized as follows:

k _i,t ＝k _min +rand(0,1)*(k _max -k _min )

wherein i is 1,2, t is an iteration number, and the initial value is 1, k _i,t Representing the ith individual in the tth generation, rand (0,1) representing a random number subject to uniform distribution over the interval (0, 1);

(4-2) for each candidate individual, constructing a local regression model ML based on the randomly generated virtual instances and the prediction result of the M model by using LIME;

(4-3) selecting an optimal individual having an optimal goodness-of-fit of the regression model from the population;

(4-4) generating a new population through mutation, intersection and selection operations using a scaling factor pr and an intersection probability pc based on the population;

mutation operation, the mutation strategy used is as follows:

v _i,t ＝k _r1,t +pr*(k _r2,t -k _r3,t )

wherein k is _r1,t ，k _r2,t And k _r3,t Randomly selecting three different individuals in the population in the t iteration, v _i,t Is a target variant individual; r1, r2, r3 belongs to {1,2, ·, size } and is different from the current target index i, so size is larger than or equal to 4; the scaling factor pr is between (0,1), the value of pr is selected to be moderate, the convergence speed of the algorithm is reduced when pr is too small, and the population is not converged when pr is too large;

crossover operation, after mutation operation, on the t-th generation of the population { k _i,t } and variant intermediates { v ] thereof _i,t Performing intercross operation among individuals:

wherein, the cross probability pc is a constant within the interval (0, 1);

selecting and operating to test the individual u _i,t And target individual k _i,t Comparing the objective function values of the two groups of the same group, and selecting a better individual; comparing the goodness-of-fit values of the two, selecting an individual with a larger goodness-of-fit value, and selecting an operation formula as follows:

(4-5) re-executing the program until the program reaches a specified number of iterations iter;

(4-6) finally obtainingThe number of instances k required to produce the highest goodness of fit and model at the highest goodness of fit in all iterations _best Number of best examples k _best Applied to a regression model and generates a corresponding interpretation based on the coefficients of the regression model.