CN117951038A - Rust language document test automatic generation method and device based on code large model - Google Patents
Rust language document test automatic generation method and device based on code large model Download PDFInfo
- Publication number
- CN117951038A CN117951038A CN202410357258.0A CN202410357258A CN117951038A CN 117951038 A CN117951038 A CN 117951038A CN 202410357258 A CN202410357258 A CN 202410357258A CN 117951038 A CN117951038 A CN 117951038A
- Authority
- CN
- China
- Prior art keywords
- function
- code
- model
- document test
- cases
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 127
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 title claims abstract description 80
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000006870 function Effects 0.000 claims abstract description 225
- 230000000694 effects Effects 0.000 claims abstract description 10
- 238000011156 evaluation Methods 0.000 claims abstract description 8
- 230000006978 adaptation Effects 0.000 claims abstract description 7
- 238000005516 engineering process Methods 0.000 claims abstract description 5
- 239000011159 matrix material Substances 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 14
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 12
- 230000014509 gene expression Effects 0.000 claims description 12
- 238000004458 analytical method Methods 0.000 claims description 7
- 230000003068 static effect Effects 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000001502 supplementing effect Effects 0.000 claims description 6
- 238000013077 scoring method Methods 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 230000008859 change Effects 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 14
- 239000000523 sample Substances 0.000 description 7
- 235000002198 Annona diversifolia Nutrition 0.000 description 6
- 241000282842 Lama glama Species 0.000 description 6
- 230000018109 developmental process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 239000012634 fragment Substances 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000005406 washing Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013074 reference sample Substances 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Landscapes
- Devices For Executing Special Programs (AREA)
Abstract
The invention provides a method and a device for automatically generating a Rust language document test based on a large code model, wherein the method comprises the following steps: collecting a code-annotation-document test dataset of the Rust language; searching the function use cases of the functions in the items in which the functions are located in the data set; clustering the function cases, merging the repeated function cases, scoring and sorting according to the quality of the function cases, and selecting the typical function case with the highest score; and fine tuning the code large model by using a low-rank adaptation technology, and evaluating the generation effect of the code large model on a test set by taking the similarity and the operability of a document test and a reference document test generated by the code large model as evaluation indexes. The document test generated by the invention can help a user of the code library to better understand the using method of the code, and fills the blank of the intelligent code generation technology in the field of Rust document test.
Description
Technical Field
The invention relates to the field of code generation based on deep learning, in particular to an automatic generation method and device for Rust language document test based on a large code model.
Background
In the software field, the system software is computer software for directly executing or controlling hardware, and is used as bottom software for supporting the development and operation of upper-layer application software; the stability and the safety of system software are ensured, and the method has important significance. In the past, C/C++ was the dominant language for developing system software; however, the use of memory and pointers is not limited by the C/c++ language, so that software developed by the C/c++ language is easy to have hidden memory leakage, pointer use security and other problems, and great maintenance pressure is brought to software development and maintenance personnel.
Rust is an emerging general-purpose system level programming language, and through unique ownership and life cycle mechanisms, memory problems introduced during programming can be effectively avoided, and high performance similar to that of the C/C++ language is maintained. Rust has been listed as the most favored programming language for six consecutive years in the global developer survey report of the programming language question-and-answer community Stack over flow organization. The Rust language is also considered to be one of the next generation programming languages most suitable for developing system software in recent years due to the excellent memory security mechanism, performance advantages not inferior to the C/C++ language, and the like.
In the Rust programming language, there is a feature called document testing that allows a developer to insert code fragments for demonstrating class or function functions, often containing several assertion statements, in the annotation fields at the beginning of the class and function, run the code fragments through rustdoc tools and verify that the results are correct. When running, rustdoc tool will extract the code segment from the comment area, prepare the required running environment, then compile and run the code segment, and reach the test effect according to whether the runtime assertion statement is satisfied. In addition to testing, document testing may also be read by a user of the code segment as an auxiliary document for reference by the user. The high-quality document test can clearly and concisely show the expected behavior of the code, and effectively assist developers to understand the use scene. Through high-quality document testing, developers can more deeply understand the design ideas and expected behaviors of codes, so that development work can be performed more efficiently. In addition, the document test can also be used as the instant verification of the code, so as to ensure the consistency and stability of the behavior of the written code under different environments.
However, writing a document test requires more time and effort for the developer than writing a code annotation, and thus, there is often a problem that after the developer completes the development of the code entity, the developer forgets or ignores the maintenance of the code document test, resulting in missing or outdated code document test. The absence of document testing can cause other developers to lack guidance when invoking the code entity, requiring more time and effort to understand the code; the outdated document test results in misleading of other developers, mismatching of understanding of the code entity with the intended function of the code entity, and thus incorrect invocation of the code entity, resulting in software bugs and even more serious problems. Therefore, it is very important to maintain the integrity of the document testing and consistency with the code entity content.
Disclosure of Invention
The invention aims at overcoming the defects of the prior art and provides a Rust language document test automatic generation method and device based on a large code model.
In order to achieve the above purpose, the invention provides an automatic generation method for Rust language document test based on a large code model, which comprises the following steps:
(1) Collecting code-annotation-document test data of Rust language, and preprocessing the collected data to obtain a data set; dividing the data set into a training set, a verification set and a test set;
(2) Searching a function use case of a function in a project in which the function is located in the data set, and supplementing the function use case into the data set;
(3) Clustering the function cases obtained in the step (2), and merging repeated function cases; scoring and sorting the combined function use cases according to the quality of the combined function use cases, and selecting k function use cases with highest scores;
(4) Constructing a code large model, and performing fine adjustment on the code large model by utilizing the data set and a low-rank adaptation technology; substituting the function codes, the function comments and the k function use cases of the training set into a prompt word template to be used as the input of a large code model;
(5) Inputting the function codes, function comments and k function use cases of the test set into the trimmed code large model to generate a corresponding document test; the similarity and the operability of a document test and a reference document test generated by the trimmed code large model are used as evaluation indexes, and the generation effect of the trimmed code large model is evaluated on a test set;
(6) And (3) extracting function codes and function comments which need to generate a document test, repeating the steps (2) to (4), and generating a Rust language document test through the trimmed code large model.
Further, in the step (1), a grammar parser of Rust language is utilized to parse the source code into an abstract grammar tree, and the positions of the function code and the function document are determined; and extracting plain text notes and document tests from the function document by using the regular expression, and cleaning data according to the requirements of the document test generation task.
Further, the step (2) specifically comprises: for each piece of data in the data set, acquiring a function call graph of the item in which the static analysis tool is positioned by utilizing a static analysis tool; searching a function calling the target function in the function call graph as an alternative function use case; and supplementing the alternative function use cases into the data set.
Further, the step (3) specifically comprises: extracting an external API sequence called by the inside of the function use case, and taking the external API sequence as the clustering characteristic of the function use case; calculating editing distance between function cases, clustering the function cases by utilizing HDBSCAN algorithm, merging repeated function cases, and only reserving the function cases of the middle point of each class cluster; and calculating the quality scores of the remaining function cases by using a scoring method, sorting the function cases according to the quality scores from high to low, and reserving k function cases with the highest scores.
Further, the scoring method has the expression:
；
Wherein, Representing function use case/>Mass fraction of/>Representing function use case/>The actual number of code lines involved,/>Representing function use case/>Number of external APIs called in,/>Representing function use case/>The number of operators contained in the list.
Further, in the step (4), fine tuning the large code model by using a low rank adaptation technique includes:
for a parameter matrix in a large code model Two low rank matrices/>, are createdAndWherein/>Representing the matrix order after decomposition,/>Representing the number of rows of the original parameter matrix,/>Representing the number of columns of the original parameter matrix; two low rank matrices are injected into the code big model and the parameters of the code big model are replaced by:
；
Wherein, Representing a new parameter matrix,/>Representing the amount of change of the parameter matrix,/>Representing a matrix transposition operation.
Further, in the step (5), backfilling the document test generated by the code large model to a corresponding position in the original project, and executing the generated document test by utilizing rustdoc tools in the Rust language tool set; and judging whether the generated document test can be correctly operated or not according to the output of the rustdoc tool, and taking the document test as an operability index for evaluating the generation effect of the code large model.
In order to achieve the above purpose, the invention also provides an automatic Rust language document test generation device based on the code large model, which comprises one or more processors, and is used for realizing the automatic Rust language document test generation method based on the code large model.
To achieve the above object, the present invention also provides an electronic device including a memory and a processor, the memory being coupled to the processor; the memory is used for storing program data, and the processor is used for executing the program data to realize the automatic generation method of the Rust language document test based on the code large model.
To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described automatic generation method of a Rust language document test based on a large code model.
The beneficial effects of the invention are as follows: although various intelligent generation methods of code completion and code annotation exist at present, the generation effect is limited, no attempt is made to use a large code model for intelligent generation of document test at present, and the invention makes an initial attempt in this respect and proves that the correct and high-quality code document test can be generated by the method. The invention can generate high-quality document test for a large number of existing Rust code libraries and Rust codes written in the future, thereby saving the time of developers and improving the development efficiency; meanwhile, the document test is generated, so that a user of the code library can be helped to better understand the using method of the codes; the blank of the intelligent code generation technology in the field of Rust document testing is filled.
Drawings
FIG. 1 is an overall frame diagram of the method of the present invention;
FIG. 2 is a diagram of an exemplary algorithm LoRA for fine-tuning a large code model in accordance with the present invention;
FIG. 3 is a schematic view of the structure of the device of the present invention;
Fig. 4 is a schematic diagram of an electronic device of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
The method comprises the steps of firstly crawling an open source Rust code from a Github open source code hosting platform to serve as an experimental data set. Specifically, the invention firstly calls the item query API of the Github platform to query the Rust items meeting the conditions, then clones the items to the local through the git tool, analyzes all source codes of the Rust items through the tree-sitter tool, extracts all code functions in the Rust items, and records item names, item versions (commit IDs), file paths, function codes and function comments. And then extracting the document test from the function annotation.
Compared with the traditional code and annotation generation method, the method for automatically generating the Rust language document test based on the code large model additionally retrieves the use cases (function use cases) of the target codes in the existing codes and inputs the use cases into the code large model together so as to enhance the generation effect of the large model.
Referring to fig. 1, the automatic generation method for the test of the Rust language document based on the large code model provided by the invention comprises the following steps:
Step one: collecting code-annotation-document test data of Rust language from an open source code hosting platform, and preprocessing the collected data to obtain a data set; the data set is divided into a training set, a validation set and a test set.
The method collects an open source Rust code from an open source code hosting platform Github, analyzes the source code into an Abstract Syntax Tree (AST) by utilizing a syntax analyzer of Rust language, and determines the positions of function codes and function documents; extracting plain text notes and document tests from the function document by using the regular expression; and performing targeted data cleaning aiming at the requirements of the document test generation task.
The method comprises the steps of firstly calling a query API of the Github platform, and screening satisfied items according to the following query conditions:
(A) The project creation time is not earlier than 1 month and 1 day in 2020;
(B) The number of target stars (Star) is more than or equal to 10;
(C) The item is not a fork from other items;
(D) The language mainly used for the project is Rust.
Up to now, 20895 Rust items were collected in total, and the item names and the item addresses of the Rust items were recorded; these items are then cloned locally by a git tool; for each Rust item, traversing all Rust source code files (ending with. Rs), and parsing file contents through a tree-sitter tool; all of the functions therein are then extracted, the item name, the item version (commit ID), the file path, the function code, and the function document (document), and these data are saved in the database.
During the process, the following is skipped:
(a) The file content cannot be properly parsed into Rust code;
(b) Too long or too short data of which the function length is more than 100000 characters or less than 3 characters (token);
(c) The function document contains special symbols, such as < img … > or http:// and the like;
(d) The functional document language contains non-english characters.
After extracting the function information, function notes (Comment) and document tests (Documentation test) are further parsed from the function document. A function document is a piece of multi-line text that precedes the function, beginning with the annotation tag "////". During processing, firstly removing annotation marks "////" atthe beginning of each line, and then matching the first plain text paragraph by using a regular expression as function annotation; the paragraphs wrapped with "" are matched as code block paragraphs. There may be multiple code block segments in a function document, the present invention retaining the first of them.
Preprocessing the collected data. Specifically, the data is cleaned by the extracted data, and the cleaning process comprises the following steps: washing off samples with too long or too short length; washing off samples containing nonsensical repeated special symbols; washing off samples containing non-english text; samples that do not contain any assertion (assert) statements are flushed.
The Rust language expresses the expected outcome of the computation using an assertion statement of asset | (), asset_eq | (), asset_ne | (), etc., where asset | (expr) indicates that assertion expression expr is true, asset_eq | (expr 1, expr 2) indicates that the outcome of assertion expressions expr1 and expr2 is equal, and similarly, asset_ne | (expr ) indicates that the outcome of assertion of both expressions is unequal. For each document test collected, if it does not contain any predicate statements therein, it is removed.
Up to now, a total of 7264139 different Rust function fragments were collected, of which 1918316 contained annotated function fragments and 65581 contained document test function fragments; as a dataset. The data set is divided into a training set, a validation set and a test set according to the ratio of 8:1:1.
Step two: the function use case searcher is used for searching the real use cases (function use cases) of the functions in the items where the functions are located in the data set and supplementing the real use cases (function use cases) to the original data set.
For each piece of data in the dataset, traversing all function call nodes by using a static analysis tool (tree-sitter-rust grammar parser), extracting function names of a calling party and a called party, and obtaining a function call graph of a project where the function call graph is located; searching a function calling the target function in the function call graph as an alternative function use case; then supplementing the collected alternative function use cases into the original data set; and collecting codes of alternative function use cases.
In order to obtain the function use case, it is first necessary to comprehensively understand the function call condition in the whole item. The invention performs a static analysis on the entire item to obtain a function call graph (CALL GRAPH) for the entire item. Static analysis refers to: the control flow, data flow, pointer pointing, etc. information of the program when running is analyzed without running the program to determine whether the program meets a certain property of interest, such as whether there is a defect or security hole in the program, etc. The function call graph is a directed graph, wherein vertices represent functions in source code, edges represent call relationships between functions, the start point of the edges is a caller, and the end point is a target method of call.
The present embodiment uses a simplified function call graph construction algorithm that considers only explicit function calls; specifically, the Rust source code of the whole item is parsed by using a tree-sitter-Rust grammar parser, and an Abstract Syntax Tree (AST) of each Rust source code file is generated. In AST, a syntax tree node of which type is call_expression is identified, and a value of a function field of the node is extracted to determine callee information. According to the type of the function field, different processing methods are adopted:
If the type of the function field is field_expression (member expression), continuing to record the value of the field in the expression;
if the type of the function field is scoped _identifier, the value of the name field is recorded, and the value of the path field is additionally recorded as auxiliary information;
if the type of the function field is identifier, the text content corresponding to the identifier is recorded.
Information of the current caller function is then determined. Starting from the calling expression node, searching the root node along AST until a node with the type of function_item (function definition) is found, and recording the value of the name field of the node as the function name of the current calling party; then, searching the root node continuously, searching for the latest node with the type impl _item (type implementation), and if the node is found, describing that the current function is a member method of a Rust structure; the values of the track field and the type field of the node are recorded as additional information of the current function.
According to the method, all calling party-called party relation edges are collected, and then a directed graph, namely a function call graph of the whole project, is constructed.
Searching the called points of the function (namely the target function) which needs to generate the document test from the project after the function call graph is ready; all the functions which call the objective function are collected and used as alternative function cases of the objective function. Specifically, in the function call graph, a corresponding vertex is found according to the objective function information, and then all the functions calling the objective function are determined along the entering edge of the vertex, namely the alternative function use cases.
For each function in the data set, collecting alternative function use cases according to the method; the collected alternative function use cases are then supplemented to the original dataset.
Step three: and clustering the collected function use cases for the functions in each data set, and merging repeated use cases to reduce the number of the use cases. And then, scoring and sorting the function cases according to quality, and selecting a plurality of compact and most representative function cases.
In large projects, a function may be called in a similar manner in multiple places, and thus the collected function use cases may have similar or duplicate situations. The method comprises the steps of firstly clustering the collected function cases to combine similar function cases, and reducing redundant input. Specifically, for each function use case, the API call sequence inside the function is extracted as the clustering feature of the function use case. For function use casesClustering features/>Can be expressed as:
；
Wherein, Any one of/>For the function use case/>External function name called in (a). Then, the distance between the function cases is calculated by using the editing distance of the word level, and the function case/> isgeneratedDistance matrix between. The present embodiment adopts the following distance formula to measure the distance:
；
Wherein EDITDISTANCE is the word-level Levenshtein distance (i.e., edit distance) between two API call sequences. After the calculation is completed, the HDBSCAN algorithm is utilized to cluster the function cases, and the HDBSCAN variable density design is added on the basis of the original DBSCAN algorithm, so that the clustering effect is more robust for different parameter selections. For each class cluster, only reserving the function use cases at the center point of the class cluster; to merge duplicate or similar use cases, the number of function use cases is reduced. In the embodiment, hdbscan packages of Python language are adopted to cluster the function use cases.
After clustering similar function use cases, the number of function use cases may still be large, far beyond the input length limit of the code large model. Therefore, the invention further ranks the function cases, selects the most representative and concise function cases, and inputs the function cases into the code large model. The invention adopts the following formulas to score the function use cases:
；
Wherein, Representing function use case/>Mass fraction of/>Representing function use case/>The actual number of lines of code contained in the file, excluding blank lines and comment lines,/>Representing function use case/>The number of external APIs called in (c) is,Representing function use case/>The number of operators contained in the list. The scoring method considers three factors, namely the number of code lines of the function use case, the number of external API calls contained in the function use case and the number of operators contained in the function use case. The method aims at selecting a plurality of compact and representative function cases, and then inputting the function cases into a code large model.
Scoring the clustered function cases, and sorting the function cases according to the score from high to low to select k cases with highest scores. Where k is a parameter that can be adjusted according to the length constraint of the large model input, k=10 is used in this embodiment.
Step four: a Code lalama Code large model is built based on transformers and fine-tuned using the collected dataset and LoRA (low-rank adaptation) technique.
And substituting the function codes, the function comments and the typical function use cases obtained through clustering and sequencing of the training set into a prompt term (prompt) template to obtain the input of the large code model. The document test is then processed as a reference output of the code large model. Constructing a Code Llama 7B Code large model, loading pre-trained weights, and performing fine adjustment on the Code large model by using a LoRA method.
It should be noted that Code Llama is a large model of the Code foundation of the open source of Meta company. Based on a more general Llama 2 natural language large model, the method additionally utilizes 500B Token-scale public codes to pretrain, and is suitable for various programming related tasks. Code Llama shows excellent results in both Code evaluation tasks HumanEval and MBPP. The model has a number of versions of different parameter scales, including 7B, 13B and 34B. The larger the model parameters, the better the model performs, but the more memory is required for training and reasoning. The invention selects the 7B version with smaller parameters for fine tuning.
And substituting the function codes, the function notes and the typical function use cases obtained through clustering and sequencing of the training set into a prompt term (prompt) template to obtain the input of the large model. Through testing, the invention adopts the following prompt word templates:
"""You are an experienced Rust programmer. Write a documentation test for the following Rust function. The corresponding comment and some usage examples of the function are given.
### Function:
{code}
### Comment:
{comment}
### Usage Example 1:
{usage[0]}
### Usage Example 2:
{usage[1]}
(more usage sample is omitted)
### Doctest:
"""
Wherein ,"You are an experienced Rust programmer. Write a documentation test for the following Rust function. The corresponding comment and some usage examples of the function are given." has the chinese meaning of "you are an experienced Rust programmer. Please write a document test for the following Rust function. Corresponding notes and some examples of use of the function are given. "; function, comment, usage sample, doctest represent functions, notes, usage examples (i.e., function use cases), document testing, respectively; { code }, { comment }, and { user [ n ] } are placeholders, representing the function code, the function annotation, and the nth typical function use case, respectively. The input-corresponding reference output is modified by the real document test. First, an annotation tag "////", and then a document test code block tag "///////'") are added at the beginning and end in response to input data, before each line of the real document test.
Then, a word segmentation device (tokenizer) matched with the Code Llama 7B is used for segmenting the training corpus, and the word elements are converted into digital IDs according to a word list.
Due to the huge number of parameters, fine tuning all parameters of a large code model consumes very computing resources. The invention adopts a lightweight LoRA (Low-Rank Adaption, low rank adaptation) method for fine tuning. The LoRA method achieves fine tuning by adding a small number of learnable parameters to the model, the principle of which is shown in figure 2,Input representing neural network,/>Representing values of hidden layers of neural network,/>Represents an average value of 0 and a standard deviation of/>Is a normal distribution of (c). A parameter matrix/>, for the large model of the original codeThe LoRA method creates two new low rank matrices/>And/>Wherein/>Representing the matrix order after decomposition,/>，/>Representing the number of rows of the original parameter matrix,/>Representing the number of columns of the original parameter matrix, injecting the columns into the original model, and then replacing the code large model parameters with:
；
Wherein, Representing a new parameter matrix,/>Representing the amount of change of the parameter matrix,/>Representing a matrix transpose operation, relative to a fine-tuning pre-trained complete parameter/>LoRA fine-tune only the injected low-rank decomposition matrices a and B. Because the scale of matrices A and B is much smaller than the model parameters/>Therefore, the number of parameters to be trained can be greatly reduced, and the calculation cost required for fine tuning of a large model is reduced.
Firstly, creating a Code Llama 7B model by utilizing a transformers library provided by huggingface communities, and loading pre-trained model weights; then, the peft library is utilized to configure the model, and a fine tuning mode is set to LoRA; finally, training the model by utilizing Trainer tools in transformers libraries, and setting a training set and a verification set. In training, the word segmentation device parameter adopted by the invention is max_length=4096; the LoRA parameters used were lora_rank=8, lora_alpha=256, lora_dropout=0.1; the learning rate adopted is learning_rate=5e-4; training 1 epoch.
Step five: and inputting the function codes, the function comments and the typical use cases of the test set into a code large model to generate corresponding document tests. And evaluating the generation effect of the code large model on the test set, and using the similarity and the runnability of the document test and the reference document test sample generated by the code large model as two evaluation indexes.
The invention adopts two evaluation indexes to measure the generation quality of the large model, namely the similarity of the generation sample and the reference document test sample and the operability of the document test.
Metrics that measure the similarity of the generated samples to the reference samples include BLEU (bilingual evaluation study) and ROUGE (recall-oriented basic evaluation study), among others. The invention adopts BLEU index, which is one of the indexes commonly used for evaluating code generation tasks. The BLEU index calculation generates, as the similarity, the ratio of the occurrence of phrases (n-grams) of consecutive n words in the sample in the reference sample. The calculation formula is as follows:
；
Wherein, And/>All generate one of a set of generated samples,/>Representation of the generated samples/>Phrase consisting of consecutive n words,/>Representing all the generated sample sets, n is the length of the n-gram,Representing the number of occurrences of an n-gram in the generated samples,/>Indicating the number of occurrences of a certain n-gram in the reference sample. There are various variants of BLEU depending on the value of n in the n-gram, with the commonly used values of n ranging from 1 to 4.BLEU-1 is typically used to measure word level accuracy, and higher order BLEUs can measure sentence fluency. The invention adopts BLEU-4.
The runnability index indicates whether the generated document test can function correctly on the original project. To evaluate the runnability of the document test, the present invention utilizes rustdoc tools in the tool set of the Rust programming language. rustdoc is an official document management tool in the Rust language that, in addition to generating code documents, provides a convenient way to run document testing; during execution rustdoc captures the output of the test code and compares it with the expected result; if the test result is consistent with the expectation, rustdoc marks the test as pass; if there is an inconsistency or a cast exception (panic), rustdoc reports the failed test cases for further review and repair by the developer. Specifically, the document test executor firstly backfills the generated document test to the document area of the corresponding function in the original project according to the Rust specification, then runs rustdoc tools, monitors the output of rustdoc tools, and judges the operability of the generated document test.
Step six: and extracting function codes and comments which need to generate the document test from the real project, repeating the second to fourth steps, and generating the document test by using the trimmed code large model.
Corresponding to the embodiment of the automatic generation method of the Rust language document test based on the code large model, the invention also provides an embodiment of the automatic generation device of the Rust language document test based on the code large model.
Referring to fig. 3, the automatic generation device for the Rust language document test based on the large code model provided by the embodiment of the invention comprises one or more processors, which are used for realizing the automatic generation method for the Rust language document test based on the large code model in the embodiment.
The embodiment of the Rust language document test automatic generation device based on the code large model can be applied to any device with data processing capability, and the device with data processing capability can be a device or a device such as a computer. The apparatus embodiments may be implemented by software, or may be implemented by hardware or a combination of hardware and software. Taking software implementation as an example, the device in a logic sense is formed by reading corresponding computer program instructions in a nonvolatile memory into a memory by a processor of any device with data processing capability. In terms of hardware, as shown in fig. 3, a hardware structure diagram of an apparatus with data processing capability according to the present invention, where the automatic run language document test generating device based on a large code model is located, is shown in fig. 3, and in addition to a processor, a memory, a network interface, and a nonvolatile memory shown in fig. 3, any apparatus with data processing capability in the embodiment generally includes other hardware according to an actual function of the any apparatus with data processing capability, which is not described herein again.
The implementation process of the functions and roles of each unit in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present invention. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Corresponding to the embodiment of the automatic generation method of the Rust language document test based on the large code model, the embodiment of the application also provides electronic equipment, which comprises the following steps: one or more processors; a memory for storing one or more programs; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the Rust language document test auto-generation method based on the code Large model as described above. As shown in fig. 4, a hardware structure diagram of any device with data processing capability, where the automatic generation method for testing a Rust language document based on a large code model according to the embodiment of the present application is located, is shown in fig. 4, and in addition to a processor, a memory, a DMA controller, a magnetic disk, and a nonvolatile memory shown in fig. 4, any device with data processing capability in the embodiment is generally according to an actual function of the any device with data processing capability, and may further include other hardware, which is not described herein.
Corresponding to the foregoing embodiment of the automatic generation method of the Rust language document test based on the large code model, the embodiment of the invention further provides a computer-readable storage medium having a program stored thereon, which when executed by a processor, implements the automatic generation method of the Rust language document test based on the large code model in the foregoing embodiment.
The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any of the data processing enabled devices described in any of the previous embodiments. The computer readable storage medium may also be any device having data processing capabilities, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), an SD card, a flash memory card (FLASH CARD), or the like, provided on the device. Further, the computer readable storage medium may include both internal storage units and external storage devices of any data processing device. The computer readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing apparatus, and may also be used for temporarily storing data that has been output or is to be output.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather to enable any modification, equivalent replacement, improvement or the like to be made within the spirit and principles of the invention.
The above embodiments are merely for illustrating the design concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement the same, the scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications according to the principles and design ideas of the present invention are within the scope of the present invention.
Claims (10)
1. The automatic Rust language document test generation method based on the large code model is characterized by comprising the following steps of:
(1) Collecting code-annotation-document test data of Rust language, and preprocessing the collected data to obtain a data set; dividing the data set into a training set, a verification set and a test set;
(2) Searching a function use case of a function in a project in which the function is located in the data set, and supplementing the function use case into the data set;
(3) Clustering the function cases obtained in the step (2), and merging repeated function cases; scoring and sorting the combined function use cases according to the quality of the combined function use cases, and selecting k function use cases with highest scores;
(4) Constructing a code large model, and performing fine adjustment on the code large model by utilizing the data set and a low-rank adaptation technology; substituting the function codes, the function comments and the k function use cases of the training set into a prompt word template to be used as the input of a large code model;
(5) Inputting the function codes, function comments and k function use cases of the test set into the trimmed code large model to generate a corresponding document test; the similarity and the operability of a document test and a reference document test generated by the trimmed code large model are used as evaluation indexes, and the generation effect of the trimmed code large model is evaluated on a test set;
(6) And (3) extracting function codes and function comments which need to generate a document test, repeating the steps (2) to (4), and generating a Rust language document test through the trimmed code large model.
2. The automatic generation method of a Rust language document test based on a large code model according to claim 1, wherein in the step (1), a grammar parser of Rust language is utilized to parse a source code into an abstract grammar tree, and the positions of a function code and the function document are determined; and extracting plain text notes and document tests from the function document by using the regular expression, and cleaning data according to the requirements of the document test generation task.
3. The automatic generation method of the Rust language document test based on the large code model according to claim 1, wherein the step (2) is specifically: for each piece of data in the data set, acquiring a function call graph of the item in which the static analysis tool is positioned by utilizing a static analysis tool; searching a function calling the target function in the function call graph as an alternative function use case; and supplementing the alternative function use cases into the data set.
4. The automatic generation method of a Rust language document test based on a large code model according to claim 1, wherein the step (3) is specifically: extracting an external API sequence called by the inside of the function use case, and taking the external API sequence as the clustering characteristic of the function use case; calculating editing distance between function cases, clustering the function cases by utilizing HDBSCAN algorithm, merging repeated function cases, and only reserving the function cases of the middle point of each class cluster; and calculating the quality scores of the remaining function cases by using a scoring method, sorting the function cases according to the quality scores from high to low, and reserving k function cases with the highest scores.
5. The automatic generation method of a Rust language document test based on a large code model according to claim 4, wherein the expression of the scoring method is:
；
Wherein, Representing function use case/>Mass fraction of/>Representing function use case/>The actual number of code lines involved,/>Representing function use case/>Number of external APIs called in,/>Representing function use case/>The number of operators contained in the list.
6. The automatic generation method of a Rust language document test based on a large code model according to claim 1, wherein in the step (4), the fine tuning of the large code model by using a low-rank adaptation technique comprises:
for a parameter matrix in a large code model Two low rank matrices/>, are createdAndWherein/>Representing the matrix order after decomposition,/>Representing the number of rows of the original parameter matrix,/>Representing the number of columns of the original parameter matrix; two low rank matrices are injected into the code big model and the parameters of the code big model are replaced by:
；
Wherein, Representing a new parameter matrix,/>Representing the amount of change of the parameter matrix,/>Representing a matrix transposition operation.
7. The automatic generation method of the Rust language document test based on the large code model according to claim 1, wherein in the step (5), the document test generated by the large code model is backfilled to the corresponding position in the original project, and the generated document test is executed by utilizing rustdoc tools in the Rust language tool set; and judging whether the generated document test can be correctly operated or not according to the output of the rustdoc tool, and taking the document test as an operability index for evaluating the generation effect of the code large model.
8. An automatic generation device for a Rust language document test based on a large code model is characterized by comprising one or more processors for realizing the automatic generation method for a Rust language document test based on a large code model according to any one of claims 1 to 7.
9. An electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used for storing program data, and the processor is used for executing the program data to realize the Rust language document test automatic generation method based on the code big model as set forth in any one of claims 1-7.
10. A computer-readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the code large model-based Rust language document test automatic generation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410357258.0A CN117951038B (en) | 2024-03-27 | 2024-03-27 | Rust language document test automatic generation method and device based on code large model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410357258.0A CN117951038B (en) | 2024-03-27 | 2024-03-27 | Rust language document test automatic generation method and device based on code large model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117951038A true CN117951038A (en) | 2024-04-30 |
| CN117951038B CN117951038B (en) | 2024-06-25 |
Family
ID=90796400
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410357258.0A Active CN117951038B (en) | 2024-03-27 | 2024-03-27 | Rust language document test automatic generation method and device based on code large model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117951038B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515444A (en) * | 2021-04-21 | 2021-10-19 | 平安银行股份有限公司 | Test case generation method and device, computer equipment and storage medium |
| US20230040412A1 (en) * | 2021-08-05 | 2023-02-09 | Sap Se | Multi-language source code search engine |
| WO2023022727A1 (en) * | 2021-08-20 | 2023-02-23 | Google Llc | Prompt tuning using one or more machine-learned models |
| CN116974916A (en) * | 2023-07-07 | 2023-10-31 | 支付宝(杭州)信息技术有限公司 | Method and device for generating test cases |
| CN117009232A (en) * | 2023-07-31 | 2023-11-07 | 平安银行股份有限公司 | Code coverage rate testing method and device |
| CN117010345A (en) * | 2023-08-15 | 2023-11-07 | 网易（杭州）网络有限公司 | Method, device, equipment and storage medium for generating function document |
| CN117032722A (en) * | 2023-08-18 | 2023-11-10 | 上海澜码科技有限公司 | Code generation method based on API (application program interface) document |
| CN117194258A (en) * | 2023-09-15 | 2023-12-08 | 支付宝(杭州)信息技术有限公司 | Method and device for evaluating large code model |
| CN117453915A (en) * | 2023-10-17 | 2024-01-26 | 上海触脉数字医疗科技有限公司 | Complex task processing method of large language model based on programmable workflow |
| CN117648093A (en) * | 2023-12-28 | 2024-03-05 | 中电鸿信信息科技有限公司 | RPA flow automatic generation method based on large model and self-customized demand template |
-
2024
- 2024-03-27 CN CN202410357258.0A patent/CN117951038B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515444A (en) * | 2021-04-21 | 2021-10-19 | 平安银行股份有限公司 | Test case generation method and device, computer equipment and storage medium |
| US20230040412A1 (en) * | 2021-08-05 | 2023-02-09 | Sap Se | Multi-language source code search engine |
| WO2023022727A1 (en) * | 2021-08-20 | 2023-02-23 | Google Llc | Prompt tuning using one or more machine-learned models |
| CN116974916A (en) * | 2023-07-07 | 2023-10-31 | 支付宝(杭州)信息技术有限公司 | Method and device for generating test cases |
| CN117009232A (en) * | 2023-07-31 | 2023-11-07 | 平安银行股份有限公司 | Code coverage rate testing method and device |
| CN117010345A (en) * | 2023-08-15 | 2023-11-07 | 网易（杭州）网络有限公司 | Method, device, equipment and storage medium for generating function document |
| CN117032722A (en) * | 2023-08-18 | 2023-11-10 | 上海澜码科技有限公司 | Code generation method based on API (application program interface) document |
| CN117194258A (en) * | 2023-09-15 | 2023-12-08 | 支付宝(杭州)信息技术有限公司 | Method and device for evaluating large code model |
| CN117453915A (en) * | 2023-10-17 | 2024-01-26 | 上海触脉数字医疗科技有限公司 | Complex task processing method of large language model based on programmable workflow |
| CN117648093A (en) * | 2023-12-28 | 2024-03-05 | 中电鸿信信息科技有限公司 | RPA flow automatic generation method based on large model and self-customized demand template |
Non-Patent Citations (2)
| Title |
|---|
| XING HU等: "Deep Code Comment Generation", 2018 ACM/IEEE 26TH INTERNATIONAL CONFERENCE ON PROGRAM COMPREHENSION, 28 May 2018 (2018-05-28) * |
| 胡成海: "基于标签与代码结构的动态需求跟踪研究", 中国优秀硕士学位论文全文数据库, 15 June 2020 (2020-06-15) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117951038B (en) | 2024-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wang et al. | Reinforcement-learning-guided source code summarization using hierarchical attention | |
| Agashe et al. | JuICe: A large scale distantly supervised dataset for open domain context-based code generation | |
| US8417709B2 (en) | Automatic refinement of information extraction rules | |
| US20080221870A1 (en) | System and method for revising natural language parse trees | |
| Lawrie et al. | Quantifying identifier quality: an analysis of trends | |
| Liu et al. | Deep learning based program generation from requirements text: Are we there yet? | |
| CN115576984A (en) | Method for generating SQL (structured query language) statement and cross-database query by Chinese natural language | |
| CN113076133B (en) | Deep learning-based Java program internal annotation generation method and system | |
| Thomas et al. | Mining unstructured software repositories | |
| CN113609838B (en) | Document information extraction and mapping method and system | |
| Jones et al. | Case studies in systematic software development | |
| JP2021060800A (en) | Data extraction method and data extraction device | |
| US20220374212A1 (en) | Indexing and accessing source code snippets contained in documents | |
| Santos et al. | Finding and correcting syntax errors using recurrent neural networks | |
| Wang et al. | Delving into commit-issue correlation to enhance commit message generation models | |
| Liu et al. | Adaptivepaste: Code adaptation through learning semantics-aware variable usage representations | |
| CN117951038B (en) | Rust language document test automatic generation method and device based on code large model | |
| CN116360794A (en) | Database language analysis method, device, computer equipment and storage medium | |
| Jacobs et al. | GE-CMU: Description of the SHOGUN System used for MUC-5 | |
| CN114661616A (en) | Target code generation method and device | |
| Mohanan et al. | Natural Language Processing Approach for UML Class Model Generation from Software Requirement Specifications via SBVR. | |
| Bull | Software maintenance by program transformation in a wide spectrum language | |
| Kasegn et al. | Spatial locality based identifier name recommendation | |
| Loureiro et al. | Learning to resolve geographical and temporal references in text | |
| Chittimalli et al. | BuRRiTo: a framework to extract, specify, verify and analyze business rules |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |