CN110780878A

CN110780878A - Method for carrying out JavaScript type inference based on deep learning

Info

Publication number: CN110780878A
Application number: CN201911020317.0A
Authority: CN
Inventors: 孙建华; 刘利娜; 陈浩
Original assignee: Hunan University
Current assignee: Hunan University
Priority date: 2019-10-25
Filing date: 2019-10-25
Publication date: 2020-02-11

Abstract

The invention relates to deep learning in the field of artificial intelligence, in particular to learning of source codes. The content of the invention comprises: the method comprises the following steps of data collection and processing, model construction, model training, model evaluation and type inference. The collection and processing of data includes: firstly, downloading a certain amount of source code on Github, then screening the source code with rich types as a final data set, then converting the data into a format with aligned matching of words (token) and types (type), simultaneously generating a token vocabulary library and a type vocabulary library, and finally expressing the source code into a data format suitable for learning, such as a vector, by utilizing token-type mapping. The construction of the model comprises the following steps: firstly, determining the type of the neural network, then determining the number of layers of the neural network, and finally determining the number of neurons in each layer. The training of the model comprises: and tracking the loss function value and the classification error, and updating the model parameters until a model with higher accuracy is obtained. The model evaluation comprises the following steps: the accuracy and consistency of the model is statistical. The type inference includes: and loading the trained model with higher accuracy, marking the type inference result behind the corresponding identifier of the model, and finally outputting the type inference result in a file form. The flow is shown in figure 1.

Description

Method for carrying out JavaScript type inference based on deep learning

Technical Field

The invention relates to deep learning in the field of artificial intelligence, in particular to learning of source codes. Specifically, after the source code is processed, the source code is used as the input of a neural network, and then training is performed to obtain a model with high accuracy, so that a JavaScript type inference method is performed.

Background

In recent years, deep learning has been widely used in the fields of speech recognition, machine translation, automatic driving, and the like, and learning of source codes is attracting attention at present. The advantage of deep learning is that highly complex features can be extracted through deep neural networks, whereas machine learning requires self-defining features. Deep neural networks have been successful in a series of natural language tasks. Unlike Natural Language, a program contains rich, clear and complex structural information, and a traditional Natural Language Processing (NLP) model is not suitable for a source code task, so that a neural network and a Natural Language Processing model can be combined to process the source code task.

Javascript, as a language for web development, remains one of the most popular languages. JavaScript is a weak type language and has certain disadvantages in static type checking, code restructuring and language service. Type inference can make static verification of code very different by type-labeling identifiers (referring to the names of variables, functions, attributes, or parameters of functions) and the like, and type inference can improve code quality. Typescript is a strong type language, and the types are restrained through type annotation, so that some type problems can be avoided, and the code robustness is improved. Flow is a static type checker used to quickly discover errors in JavaScript applications and validate the basic data types in JavaScript. There are other code checking tools, but these tools have some disadvantages, such as the inability to detect type errors, such as reference types, custom types, etc., so it is necessary to make type inference on JavaScript.

The traditional JavaScript type inference needs to consider a lot of factors, and generally, types are continuously refined through various analysis technologies such as syntax analysis, semantic analysis and the like, so as to perform type inference. And the JavaScript type inference based on deep learning only needs to construct a corpus with aligned and matched words and types, and then the model is constructed and trained based on the corpus and a data set, so that the model with higher accuracy is finally obtained.

Disclosure of Invention

The invention relates to a method for carrying out JavaScript type inference by adopting a deep learning technology. Firstly, collecting and processing source code data, dividing the data into training data, verification data and test data, and expressing the data in a data format suitable for learning, such as a vector form; secondly, constructing a model, selecting a proper neural network, such as a gate control cycle Unit (GRU) with bidirectional propagation, a cyclic neural network (RNN, Recurrent neural network), a Long-Term Memory neural network (LSTM, Long Short-Term Memory) and the like, capturing more source code information, and enabling input data to be reasonably output after passing through the neural network; thirdly, training the model, continuously updating network parameters according to the loss function value and the verification set accuracy after each training, and continuously improving the precision of the model until obtaining a model with higher accuracy; fourthly, evaluating the model, and counting the accuracy and consistency of the model; fifthly, type inference is carried out, data to be inferred are input into the trained model with high accuracy, and an inference result is output. The method is shown in figure 1 and comprises the following steps:

1. data collection and processing

Aiming at the deep learning task, the source code data which is not processed can not be directly used, and the source code data is used as the input of the neural network after being subjected to the processing of screening, converting, segmenting, representing and the like. The method comprises the following specific steps:

screening: aiming at a source code learning task, the diversity of a source code design mode and style needs to be ensured; for the task of type inference, it is necessary to ensure that the types in the code are rich and the number of the various types remains balanced. Based on the above two requirements, a certain amount of open source project source code is downloaded on the GitHub.

Conversion: the type of each identifier in each source file is analyzed by using a type checker, the identifiers are labeled behind the identifiers, finally, the identifiers and the corresponding types are converted into a data format with words and types (token-type) aligned and matched, and meanwhile, a word vocabulary and a type vocabulary with a certain size are generated, so that subsequent learning is facilitated.

Cutting: according to the configuration of an experimental machine, data are segmented into training data, verification data and test data with proper size and proportion, so that the training speed is improved.

Represents: the data is normalized by word and type vocabulary mapping as input to the neural network.

2. Model construction

Aiming at a source code task, the traditional NLP model is not suitable, and the deep neural network is adopted in the invention to combine the neural network with the natural language processing model. The construction of the model comprises the following parts:

a. a neural network: the invention adopts neural networks of GRU, LSTM, RNN and the like which are propagated in two directions to construct a deep neural network. Firstly, observing the influence of any neural network on the accuracy of the whole model through experiments to obtain the influence degree of various neural networks on the accuracy of the model; and then combining the influence of each neural network on the accuracy of the model, combining two or more neural networks, observing the influence on the accuracy of the whole model, finally balancing various factors, and selecting the most appropriate neural network combination.

b. The network structure is as follows: the whole model is divided into an input layer, a hidden layer and an output layer. The input layer inputs the data into the neural network after normalization processing. The hidden layer performs various operations on input data and the like. And the output layer outputs the data result of the hidden layer. And setting the proper number of neural network layers and the number of neurons in each layer according to the difficulty and the complexity of the network to be solved.

c. Network parameters: training a data set in batches, setting the size of each batch of data, and performing regularization processing on the data to improve the training speed; setting activation functions of an input layer, a hidden layer and an output layer and Dropout values of the hidden layer, wherein the activation functions are used for adding nonlinear factors and solving the problem which cannot be solved by a linear model, and when the Dropout is used for model training, certain nodes of a neural network do not work at random; selecting a proper loss function according to the problem type, wherein the loss function is a cross entropy function generally for multi-classification tasks; the training process adopts a gradient descent algorithm, sets a proper Learning Rate (Learning Rate) and Momentum (Momentum) value for accelerating gradient descent, or directly adopts a learner to automatically adjust the Learning Rate, and sets a gradient cut-off value at the same time to ensure that the gradient value is in a certain range.

In a word, based on common neural networks such as GRU, LSTM, RNN and the like, a reasonable neural network structure is constructed by combining a data set, and initial neural network parameters are set.

3. Model training

The training of the model is divided into the following steps: batch processing, model parameter initialization, tracking loss function value updating model parameters, testing model generalization capability and improving the model generalization capability.

a. Batch treatment: when the data size of the training set is too large and the whole data is input into the neural network at one time, the training set is not beneficial to learning, so that the data are divided into a plurality of batch data (minibrates) with certain sizes, and the training speed is improved to a certain extent.

b. Initializing model parameters: and selecting an initialization method according to the size of the activation function value and the size of the gradient value, and initializing the Weight (Weight) and the Bias (Bias) of each layer of nodes. For RNN, for example, the orthogonal initialization (SVD) method may work better.

c. Tracking loss function values, updating model parameters: for multi-classification tasks, a cross entropy function is adopted by a general loss function, and the gradient value is cut down after the loss function is calculated, so that the gradient value is kept in a certain range. And calculating the classification error rate of the system. Based on the loss function value and the classification error rate, the learning rate is automatically adjusted by a learner, and the like, so that model training is performed.

d. Generalization ability of test model: and taking the verification set data as the input of the model, counting the accuracy of the model inference result and the consistency of the statistic same identifier type inference result at different positions as the standard for measuring the generalization ability of the model.

e. Improving the generalization capability of the model: the model is adjusted mainly aiming at the phenomena of model over-fitting and under-fitting.

Generally, the difference between the training error and the verification error is not large, but both errors are large, and under-fitting phenomenon occurs in the case. Usually, the number of hidden layers of the neural network can be increased, the number of neurons can be increased, the training time can be prolonged, other more complex neural network models can be selected, the training error and the verification error can be reduced, and the generalization capability of the models can be improved.

The training error and the verification error are generally greatly different, and in this case, an overfitting phenomenon occurs. Generally, Dropout operation can be performed on neurons in a hidden layer part, that is, neurons in each layer in the hidden layer are temporarily discarded from the network according to a certain probability, so that the function of simplifying a complex network model is achieved. Meanwhile, the scale of batch processing can be changed, data can be regularized, and the generalization capability of the model can be improved.

4. Model evaluation

The evaluation of the model, namely the measurement of the generalization ability of the model, has strong generalization ability and shows that the accuracy of the model inference result is high. The generalization ability is weak, which indicates that the accuracy of the model inference result is low. And (3) inputting the test set data into the model, and counting the accuracy and consistency of the model inference result so as to determine the generalization capability of the model.

5. Type inference

For type inference, firstly, data to be inferred is used as model input, then type inference is carried out by using an obtained model with higher accuracy, and finally an inference result is output. The whole process is as follows:

(1) inputting: the source code is first converted into word sequence data using a tokenizer as input to the model.

(2) And (3) deducing: and mapping the hidden state in the hidden layer to a vector with the same dimensionality as the type vocabulary by adopting a trained model with higher accuracy, and then normalizing the hidden state into the probability distribution of the type by utilizing an output layer activation function.

(3) And (3) outputting: the type inference result for each identifier is labeled behind it.

Drawings

FIG. 1: flow chart of the system

Detailed Description

The hardware environment of the invention is mainly a server with a GPU model of GeForce GTX 1080 Ti. The software implementation takes ubuntu 16.04 as a platform, is based on a deep learning framework CNTK (computerized Network toolkit), and is developed by adopting python language. The CNTK supports various neural network models and algorithms, and great convenience is brought to the realization of the whole system. Experimental data was derived from a number of open source projects on Github. The operation is mainly divided into four parts: data preprocessing, model construction and training, model evaluation and type inference. The method comprises the following specific steps:

1. data pre-processing

The data preprocessing mainly comprises the steps of screening, converting, segmenting, representing and the like of data. The specific implementation is as follows:

screening: selecting a certain amount of open source items with rich types and a large number of stars; the files in each open source project are checked, and the files with fewer types and excessive identifiers are deleted.

Conversion: the type information of the identifier is obtained by using a type checker, and the type information is marked behind the identifier.

Cutting: the data are processed according to the following steps of 8: 1: the scale of 1 is divided into training data, validation data and test data.

Represents: a vocabulary of words (token vocabularies) and a vocabulary of types (typevocabularies) of a certain size are generated, because the size of the vocabulary is strongly correlated with the complexity of the model, and in order to ensure that the training time and the required storage are reasonable, a vocabulary of the proper size is selected as much as possible. And converting the data file in the TXT format into a CTF format suitable for learning by utilizing the mapping relation between the two vocabularies.

2. Model construction and training

The model construction comprises a neural network structure, a trainer and training. And constructing the whole model according to functions in modules such as io, sequences, layers, ops, learners, train, metrics and the like in the deep learning library CNTK. The specific implementation is as follows:

creating a model: the whole model is created into an input layer, a hidden layer and an output layer. Create an input container by sequence input _ variable () first before creating a model; the input layer converts input data into embedded vectors with certain dimensionality through layer. The hidden layer calls the existing GRU, RNN and LSTM neural network models through a layer.Recurrence () function to construct an encoder and a decoder, the encoder calculates input data to obtain a hidden state, the final hidden state is used as an initial state of the decoder, and the decoder correspondingly calculates the data of the hidden state to be used as the input of an output layer; the output layer creates a fully connected layer through the layers 'detect () function and processes the final result with ops' softmax (). A certain drop rate is also set by layers. The network structure of the hidden layer is simplified by the discarding operation.

Creating a trainer: the whole training process is encapsulated by a Trainer () function in a train module, and parameters such as a loss function and a learner are added to the training process. The loss function is directly realized by using a calculation function in the CNTK, the learner constructs a learner example through adadelta () and the like in a learners module in a deep learning library, and various network parameters such as initial learning rate, momentum value, gradient threshold value and the like are added into the learner example;

training a model: firstly, reading input data through functions such as MinibatchSource () in an io module, calling a trainer to learn the input data, tracking the value of a loss function and the accuracy of a model in the training process, and continuously updating model parameters until the model with higher accuracy is obtained.

3. Model evaluation

Model evaluation includes accuracy and consistency of the model. The specific implementation is as follows:

firstly, loading a trained model with higher accuracy, then iteratively inputting test data, recording a real type corresponding to each identifier in the test data and a type obtained by a type checker, deducing the obtained type, and finally carrying out statistics to obtain the accuracy of the model. And simultaneously recording the number of the tokens appearing in a single time and the number of the tokens appearing in multiple times, deducing the number of inconsistency by type, and finally counting the result of the inconsistency.

Accuracy:

where tp represents the number of inferred types that are consistent with the true type and fp represents the number of inferred types that are inconsistent with the true type.

Inconsistency:

where inconsistence denotes the number of type inference inconsistencies in tokens that occur multiple times and multi _ occurrents denotes the number of tokens that occur multiple times.

4. Type inference

Type inference is based on a trained, more accurate model to infer the type of identifier in JavaScript.

The specific implementation is as follows:

first, input data is processed and represented as a word sequence, such as "var str ═ hello"; "sentence decomposition into [ var, str, ═ hello"; these words are then analyzed to remove keywords var, etc., leaving only the identifier data. Finally, loading the model, performing type inference to obtain the type of the corresponding identifier, and marking the type information corresponding to each identifier behind the model, such as "var str: String ═ hello"; ".

Claims

1. A method for JavaScript type inference by adopting a deep learning technology comprises the following steps:

(1) and (4) data. Screening, converting, segmenting and representing data, and finally using the data as the input of a model;

(2) and (4) modeling. The method comprises the steps of neural network composition, initialization of hyper-parameters of the neural network composition and the like;

(3) and (5) training. Tracking loss functions of the model and the like, and determining the influence of factors such as the type of a neural network, the number of network layers, the hyper-parameters and the like on the accuracy of the model, thereby continuously adjusting the parameters of the model and further continuously improving the accuracy of the model;

(4) and (4) type inference. And taking the source code file as the input of the model, loading the model with higher accuracy in the trained model, performing type inference on each identifier in the input, and marking the inferred type information behind the identifier.

2. The data required by the system according to claim 1, wherein the following transformation is required to conform to the input of the model, wherein:

(1) screening source codes which are rich in types and keep the number of the various types balanced;

(2) obtaining type information of the identifier in the source code by using a type checker, converting the source code and the type information thereof into TXT text data by using a word segmentation device, and constructing a vocabulary with proper size;

(3) setting training data, verification data and test data with proper size and proportion according to the configuration of an experimental machine;

(4) the TXT text data is represented as deep learning library CNTK-compatible CTF format data.

3. The model for source code according to claim 1, having the following features:

(1) the traditional natural language processing model is not suitable for a source code task, and the deep neural network is adopted in the invention, and the neural network is combined with the traditional natural language processing model to capture more code characteristics;

(2) selecting a proper number of neural network layers, the number of neurons contained in each hidden layer and the like by combining the size of the data set;

(3) and setting reasonable initial neural network parameters by combining the task types. Such as settings for activation functions, loss functions, learners, etc. For the classification task, the output layer generally selects softmax () function to process the output result, and the learning rate generally attenuates with the training, so that an adaptive gradient method, such as using adadelta learner, can automatically adjust the learning rate, and the like.

4. Training for models according to claim 1 or 3, the main features comprising:

(1) tracking loss function values, training set accuracy and verification set accuracy, further continuously adjusting model parameters and optimizing the performance of the model;

(2) training a model by adopting a single neural network;

(3) training a model by combining a plurality of neural networks;

(4) and aiming at the two models, the hyper-parameters are changed, the influence of each hyper-parameter on the final result is observed, various influences are integrated, and the model with higher accuracy is obtained.

5. The type inference of claim 1, characterized by the following:

(1) extracting each identifier from a source file serving as an input parameter by using a word segmentation tool;

(2) loading a trained model with higher accuracy, mapping a hidden state in a hidden layer to a vector with the same dimensionality as a type vocabulary library aiming at each identifier, then processing an output result by using a softmax () function, and normalizing the output result into effective type classification probability distribution;

(3) the type inference result for each identifier is tagged behind its corresponding identifier.