CN116302919A - A multi-language scalable code dependency parsing model and parsing method - Google Patents

Abstract

The present invention discloses a multi-language scalable code dependency parsing model and parsing method. Firstly, an abstract syntax tree is generated by using a parser, based on traversing the abstract syntax tree, information is obtained at an appropriate node, and entity extraction is completed. and save, secondly, in this process, some dependencies that can be directly judged and incomplete dependencies that can judge half of the information are retained. After other" to complete dependency backfilling, the incomplete dependencies that complete dependency backfilling will be supplemented as complete dependency storage, and the incomplete dependencies that fail to depend on backfilling will remain as unresolved dependency storage.

Description

A multi-language scalable code dependency parsing model and parsing method

technical field

The invention relates to the field of trusted software and static code analysis, in particular to a language-independent, test item application-scenario-independent, scalable static code dependency analysis field.

Background technique

Static code analysis is an important part of the software engineering field. Software products need static code analysis in the process of development and maintenance. The dependency analysis results obtained by static code analysis provide a solid foundation for upper-level software and architecture analysis, and can be widely used in Vulnerability detection, architecture measurement, software bad taste and other fields can guarantee the reliability and security of software products. Compared with dynamic analysis, static code analysis does not need to run software products, and it has better performance in terms of time complexity and space complexity.

Research in academic circles tends to focus on certain specific languages, and tends to focus on the construction of call (call) dependency graphs and analysis in specific scenarios (such as Linux operating system). The entity granularity is relatively coarse, and it is often analyzed at the function level. , there are relatively few dependent types, and the dependent types focus on the calling relationship and inheritance relationship. More fine-grained entities and more types of dependencies can provide richer information and provide solid support for upper-layer applications such as architectural metrics. At present, the research in the industry is often designed based on its own needs. The tools that are highly developed and open to the outside world are mostly closed-source and fee-based projects, which have high costs and poor openness. Diversity and the inability to determine the type of accepted objects during compilation, etc. In some scenarios, the accuracy rate is low, and supplementary and iterative development cannot be carried out. The rest of the projects are mostly in the early stage of development, with fewer supported functions and relatively weak capabilities . At this stage, there are still some protocols used to describe the intermediate results of static analysis, but their popularity and use are not enough, and it is difficult to verify their feasibility.

Contents of the invention

The purpose of the present invention is to provide a multi-language, scalable code dependency parsing method, which can achieve a certain balance between sorting performance and time overhead, and can achieve both performance and time overhead in large-scale software The advantages of static code analysis are used to overcome the shortcomings of dynamic code analysis that the program must have no running errors and consume computing power and time, so as to solve the above technical problems.

In order to achieve the above object, the present invention adopts the following technical solutions to achieve:

S1 traverses the file system

S2 generates an abstract syntax tree

S3 Entity Extraction Based on Abstract Syntax Tree Traversal

S4 Intermediate processing based on entity extraction results

S5 symbol table-based dependency backfilling

S6 generates results and outputs

A multi-language scalable code dependency parsing model, including:

The code depends on the graph model, which is composed of nodes and edges. Nodes represent entity objects and store all information related to entities, including but not limited to entity types, unique identification indexes, full names, attribute information, and location information. Edges It is a bridge linking two entities that are dependent. Here, the edge is a directed edge, and the direction of the edge refers to the direction in which the dependency occurs, that is, a certain dependency relationship occurs between the source entity and the target entity. In addition , the edge also retains other dependency-related information, including but not limited to the type of dependency and where it occurs.

A multi-language scalable code dependency parsing method, traversing the file system module, traversing the input project path, and obtaining the file information and processing order to be processed;

The abstract syntax tree generation module, according to the file list and processing order, sets appropriate compilation options to generate an abstract syntax tree;

The abstract syntax tree-based entity extraction module extracts entity information from the corresponding nodes on the abstract syntax tree according to the generated abstract syntax tree and the visitor design pattern;

The intermediate processing module is used to identify and solve the unfinished processing information in the entity list;

The dependency backfill module based on the symbol table completes the entity information according to the entity information in the entity warehouse and the incomplete dependency obtained through processing, and obtains the final entity dependency graph;

Result output module.

Each node in the dependency graph can identify a unique entity. Complex situations such as one dependency occurring multiple times or multiple dependencies occurring at the same time are allowed between nodes, that is, edges can overlap. In addition, there is an association relationship between dependency information , deduce the circular dependency/indirect dependency according to the dependency information. The location of the dependency can be in the entity definition file or not. The location of the dependency and the definition location of the source entity and the destination entity can be in different files.

The method optimizes the internal storage structure and processing logic. For large-scale software products with complex dependencies, the method can complete the analysis in ideal time and space.

In the case that other forms need to be parsed and other related types of dependencies are required, the abstract syntax tree generated during the method process can be extended. The entire framework has clear functional isolation, allowing and friendly support for module extensions.

There is no restriction on the language of the input software product, and the method has good robustness for tolerating grammatical errors and semantic errors of the software product.

The entity granularity extracted by it is the entity at the symbol level, and the granularity is finer. The retained entity information takes the variable as the minimum entity level, and the retained dependency information takes the dependency information that occurs between entities as the minimum dependency level.

A finer-grained information format, including entities as object structures with attributes including but not limited to names, code locations, and types. In addition to types, entities can also have subdivided types, which reflect finer-grained entity characteristics:

Extensive dependency types, including dynamic dependencies and static dependencies;

The sequence of dependency backfilling, including the comprehensive symbol table, is completed in the order of "Export first, then Import, and finally others". The incomplete dependencies that complete dependency backfilling will be supplemented as complete dependency storage.

Compared with the prior art, the present invention has the following beneficial effects:

The multi-language scalable code dependency parsing method of the present invention, firstly, uses the parser to generate an abstract syntax tree, and based on the traversal of the abstract syntax tree, obtains information at an appropriate node, and completes the extraction and storage of entities; secondly, this process Some dependencies that can be directly judged and incomplete dependencies that can judge half of the information are retained. Finally, the incomplete dependencies obtained in the analysis of the pre-order process are processed, and the comprehensive symbol table is completed in the order of "Export first, then Import, and finally others". Backfill, the incomplete dependency that completes dependency backfill will be supplemented as a complete dependency store, and the incomplete dependency that fails to depend on backfill will remain as an unresolved dependency store.

Description of drawings

FIG. 1 is an entity dependency classification diagram of a multilingual extensible code dependency analysis method according to an embodiment of the present invention;

Fig. 2 is a flow chart of a multi-language scalable code dependency parsing method according to an embodiment of the present invention, which includes three sub-graphs, sub-graph (a) is a general flow chart, sub-graph (b) is a dependency backfill flow chart, and sub-graph (c) is an extension module, implicit dependency extraction flow chart;

Fig. 3 is a multilingual extensible entity dependency model diagram of an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The present invention is described in further detail below in conjunction with accompanying drawing:

Referring to Figure 1, the embodiment of the present invention divides all possible entities and dependencies into 3 types as shown in (a)-(c) and 3 types as shown in (d)-(f) according to their properties and extraction methods. 3 types shown. For entities, Figure 1(a) is part of the structural entity, which is generally a software project’s constituent files and some folders with specific patterns; Figure 1(b) is a part of the code entity, which is generally a software project code file Elements conforming to a specific grammatical format and undertaking specific software function logic; Figure 1(c) is a partial scope entity, which is generally an element in a software project code file that conforms to a specific grammatical format but only functions as a logical control flow conversion. For dependencies, Figure 1(d) is a partial dependency without binding, which is generally a dependency that can be fully resolved simultaneously with the extraction of entities during the traversal of the abstract syntax tree of the code; Figure 1(e) is a partial The need to bind dependencies, which is generally a dependency that cannot be fully resolved when traversing the code abstract syntax tree, usually requires subsequent secondary analysis to find the entity pointed to by the dependency according to the code string; Figure 1(f) Implicit dependency is a part of the implicit dependency, which is generally a dependency constructed by inferring the possible type of a variable based on the subsequent use of attributes of a variable in a programming language without a type annotation system. Compared with the aforementioned dependency, the implicit Formal dependency resolution also requires additional steps to implement solution and constraint refinement.

Referring to Fig. 2 (a) the overall flowchart of the embodiment of the present invention, the input of this method includes project source code, project configuration and third-party library list, etc., and the output is entity warehouse, dependent warehouse and custom analysis results (if any), etc. , including the following procedures:

Preparations Select a suitable parser. This method needs to investigate existing parsers. The selected parser needs to be able to generate an abstract syntax tree with high efficiency, high accuracy, and low energy consumption under the condition of fault tolerance. The parser needs to be a consensus within the industry, universally verified, and according to the language version. Change the tools for timely maintenance. For example, in the present invention, Eclipse's JDT and CDT are used for Java and C++ languages, the language official parser is used for Python, and babel is used for Javascript and TypeScript.

S1 traverses the software project directory to obtain all structural entities. Specifically, the following steps are included:

S101: Given multiple directory paths of a software project, first use the tree traversal algorithm to traverse all the files in the directory. For programming language source files, configuration files of specific programming languages or frameworks, and other files that will affect dependency analysis, respectively: It creates a File entity;

S102: For directory structures with specific functional meanings in some specific programming languages, for example, the directory containing the __init__.py file in Python is specialized as Python Module and the directory containing package.json file in JavaScript is specialized as Node.js Package, Additionally create corresponding structural entities for it. If a structural entity has a corresponding configuration file, the information in the configuration file that will affect the dependency analysis is saved in the entity in the form of attributes.

S2 traverses the code abstract syntax tree nodes to obtain all code entities and scope entities. Specifically, the following steps are included:

S201: A preprocessing link defined by a specific programming language. If there is an action involving rewriting code files in a specific programming language, the embodiment of the present invention will perform the same rewriting according to relevant definitions here. For example, in C/C++, all header files (.h files) need to be processed before processing (macro expansion) other code files, and specific program environment variables need to be accessed and analyzed.

S202: Use the Parser to parse the content of the code file corresponding to the File entity obtained in S101 into a code abstract syntax tree. Depending on the Parser selected, this step can either parse a single file one by one, or directly use the API provided by a specific Parser to build a complete abstract syntax tree for a complete project at one time for one-off parsing;

S203: Use the visitor mode-based node traversal interface provided by Parser to sequentially traverse all code abstract syntax tree nodes, and register corresponding extraction methods for the abstract syntax tree nodes corresponding to each code entity and scope entity. A specific extraction method can create an entity of its type when it is called, and extract the required information from the corresponding abstract syntax tree node and save it in the entity. During this process, all entities are stored hierarchically according to the code scope inclusion relationship;

S204: Initialize and continuously maintain the cross-node context information stack along with the node traversal process in S203. Some programming language features will make the attribute calculation of the current node depend on the related attributes of sibling nodes of the same layer or parent-child nodes spanning multiple layers. The node traversal interface provided by Parser is usually stateless, so it is necessary to use an additional one that can be used in A stack structure that maintains and transmits specific information when a node jumps;

S205: Partial immediate resolution without binding dependencies. The grammatical mode of a specific programming language makes the entity to be bound actually appear after the grammatical identifier of the specific dependency. When traversing the abstract syntax tree nodes in order, the dependency can be completed while creating the entity to be bound according to S203 The creation and binding of , making it a complete dependency;

S206: A record that needs to be bound dependent (incompletely dependent). Contrary to S204, most grammatical patterns make the entity to be bound appear before the grammatical identifier of a specific dependency, and when traversing to the grammatical identifier of the dependency, only a reference to the entity in the form of a string can be obtained , the string needs to be parsed into the entity saved in S203 in a subsequent step, but in this step, only the incomplete dependency (incomplete dependency) and other relevant information that is helpful for binding can be temporarily saved in a separate structure .

S3 performs intermediate processing according to the entity extraction result. Complete the entity attribute information saved in the process of not processing due to not all entities being extracted in 2. For example: (1) bind the type information of the object defined by the data aggregation entity; (2) realize the identification and recording of the function overload information, etc.

S4 is based on hierarchical symbol table dependency binding. Traverse the incomplete dependencies recorded in 2 in the order of "Export->Import->Others", and complete the dependencies according to the module path resolution rules and scope-based symbol table hierarchical search. Specifically, the following steps are included:

S401: Process the binding information that can be resolved by the abstract syntax tree, and backfill the dependencies in the same file according to the binding information provided by the abstract syntax tree, and store them in the dependency warehouse;

S402: Handle cross-file and unbound dependencies between files. This process searches and backfills from the symbol table. Classification discussion is carried out here: the first step is based on the language needs, first check whether there is a corresponding Export dependency for the Import dependency; the second step is classified according to the life cycle, for the case where the dependent target binding entity is a full life cycle such as a function/class , first use the method of searching upwards according to the scope, and then search in files with reference relationships (Import, Include, etc.) after the search fails, and finally search for the built-in Built-in entity of the programming language; for those with life cycle restrictions For entities such as local variables, since the memory will be released at the end of the local scope and will not be accessed by external entities, only short names are used for setting, and only the current scope or upper layer functions during the search process Search within the domain and then bind. Dependency information that successfully binds using the above is stored in the entity warehouse as a complete dependency. Dependency information that fails to bind is classified as unknown dependencies. The reasons for unknown dependencies include but are not limited to standard API calls and If the version of the third-party library is inconsistent, etc., it is also stored in the dependency warehouse, and the unbound entity can be marked as unresolved, and the dependency is also marked as unresolved.

Referring to FIG. 2( b ), it is a flowchart of dependency binding for incomplete dependencies in this method embodiment. Generally speaking, regarding dependency binding, the steps that may be implemented include local scope search, module path resolution, Exported scope search, Imported scope search, tripartite library list search, and language Built-in list search. Different types of dependencies may be separately Single or multiple of all the above steps are used. For all the incomplete dependencies recorded in S206, they are sorted by type and in a specific processing sequence before processing, and generally follow the order of "Export first, then Import, and then others". The steps required to handle each type of dependency are identified in the figure with a "√" (single step) or a number (steps are performed in numerical order).

The following is an illustration and explanation of the scalability of the method:

Referring to Fig. 2(c), it is an example of an extensible part in this method embodiment, and this embodiment presents an implicit dependency extraction module. Implicit dependencies are dependencies that are difficult to index through conventional static code analysis, and this type of dependency can only be obtained through dynamic analysis under normal circumstances. In this method, the semantics can be abstracted with the help of the generated AST, for example, x=y can be expressed as Move(x,y), and x.f=y can be expressed as Store(x,f,y); according to the semantic abstraction, the above The explicit dependencies and heap model mentioned above can obtain the parsed code dependencies according to the pointing relationship, and then reduce the implicit dependencies constrained by member calls.

Referring to Fig. 3, it is an entity dependency model diagram in the method embodiment, and the entity is an object structure including but not limited to properties such as name (Name), code position (Location) and type (Type), except the type entity There can also be a sub-type (Sub-type), which reflects finer-grained entity characteristics; dependency D=<E _src , Type, E _dest > is a triple structure including the starting entity, dependent type and terminating entity , which also has attributes including but not limited to code location (Location). This model divides the traditional simple enumeration model {E={E _a ,E _b ,…}, D={D _a ,D _b ,…}} into finer-grained divisions, and utilizes the dependent triple structure The start entity type and end entity type in divide the dependencies with the same name that may occur between multiple entity pairs into different two-dimensional coordinates (E _src , E _dest ) according to different entity types. The entity dependency model can not only show the complete entity and dependency type classification, but also clearly show the fine-grained relationship between which entities can generate which dependencies, and can prompt attention to avoid omissions when guiding coding implementation.

The division of modules in the embodiments of the present invention is schematic, and is only a logical function division. In actual implementation, there may be other division methods. In addition, each functional module in each embodiment of the present invention can be integrated into a processing In the controller, it can also be physically present separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall fall within the protection scope of the claims of the present invention.

Claims (10)

1. A multi-language scalable code dependency parsing model, characterized in that it includes: The code depends on the graph model, which is composed of nodes and edges. Nodes represent entity objects and store all information related to entities, including but not limited to entity types, unique identification indexes, full names, attribute information, and location information. Edges It is a bridge linking two entities that are dependent. Here, the edge is a directed edge, and the direction of the edge refers to the direction in which the dependency occurs, that is, a certain dependency relationship occurs between the source entity and the target entity. In addition , the edge also retains other dependency-related information, including but not limited to the type of dependency and where it occurs. 2. Adopt a kind of multi-language scalable code dependency parsing method of model as claimed in claim 1, it is characterized in that, traverse the file system module, traverse the input project path, obtain the file information and processing sequence that need to be processed; The abstract syntax tree generation module, according to the file list and processing order, sets appropriate compilation options to generate an abstract syntax tree; The abstract syntax tree-based entity extraction module extracts entity information from the corresponding nodes on the abstract syntax tree according to the generated abstract syntax tree and the visitor design pattern; The intermediate processing module is used to identify and solve the unfinished processing information in the entity list; The dependency backfill module based on the symbol table completes the entity information according to the entity information in the entity warehouse and the incomplete dependency obtained through processing, and obtains the final entity dependency graph; Result output module. 3. A multi-language scalable code dependency analysis method according to claim 2, characterized in that each node in the dependency graph can identify a unique entity, and a dependency between nodes is allowed to occur multiple times or It is a complex situation where multiple dependencies occur at the same time, that is, edges can overlap. In addition, there is a relationship between dependency information, and circular dependency/indirect dependency is derived based on dependency information. The location of dependency can be in the entity definition file or No, the location where the dependency occurs and the definition location of the source entity and the target entity can be in different files. 4. A multi-language scalable code dependency parsing method according to claim 2, characterized in that the method optimizes the internal storage structure and processing logic, and for large-scale software products with complex dependencies, the method can Complete analysis in ideal time and space. 5. A multi-language scalable code dependency parsing method according to claim 2, characterized in that, when other forms need to be parsed and other related types of dependencies are needed, the abstract syntax tree generated in the method process can be For extension processing, the functional isolation of the entire framework is clear, allowing and friendly support for module extensions. 6. A multilingual scalable code dependency parsing method according to claim 2, characterized in that, there is no restriction on the language of the input software product, and the method has good performance in tolerating grammatical errors and semantic errors of the software product. robustness. 7. A multi-language scalable code dependency parsing method according to claim 2, characterized in that the entity granularity extracted by it is a symbol-level entity, and the granularity is finer, and the retained entity information takes the variable as the minimum entity Hierarchy, the retained dependency information takes the dependency information that occurs between entities as the minimum dependency level. 8. A multi-language scalable code dependency parsing method according to claim 2, characterized in that the finer-grained information format includes entities as objects with attributes including but not limited to names, code locations, and types Structure, in addition to types, entities can also have subdivision types, which reflect finer-grained entity characteristics. 9. A multi-language scalable code dependency parsing method according to claim 2, characterized in that the wide variety of dependencies includes dynamic dependencies and static dependencies. 10. A multi-language scalable code dependency parsing method according to claim 2, characterized in that the sequence of dependency backfilling includes a comprehensive symbol table, and the dependency backfilling is completed in the order of "Export first, then Import, and finally others" , the incomplete dependencies that complete dependency backfilling will be supplemented as complete dependency storage.

Images