CN112988217B - Code base design method and detection method for rapid full-network code traceability detection - Google Patents

Abstract

The invention discloses a code library design method for rapid full-network code traceability detection, which is used for obtaining a code library by carrying out efficient storage on Git objects in a full-network using Git open source project through project discovery, data extraction, data storage, code information mapping construction and data updating processes, and realizing efficient updating of the code library; comprising the following steps: a storage mode of dividing and storing according to the types of the Git objects is adopted; constructing a relation mapping between code files and code file information, and rapidly searching the whole network information of the code files; and a high-efficiency updating mode is adopted for the constructed ultra-large-scale code library, a customized Git fetch protocol is proposed based on the Libgit2 function library, and the constructed ultra-large-scale code library is taken as the rear end to efficiently obtain newly added Git object data of a remote warehouse. The code library generated by the method can be updated regularly and efficiently, and supports quick full-network traceability detection of codes on file granularity, and the detection efficiency is high.

Description

Code library design method and detection method for rapid network-wide code traceability detection

Technical field

The invention provides a code base design method for fast whole-network code traceability detection and a fast whole-network code traceability detection method based on the code library, and belongs to the technical field of software engineering.

Background technique

With the vigorous development of open source software, a large number of excellent open source software resources have accumulated on the Internet, and open source code is increasingly used in software development. While the use of open source code improves the efficiency of software development, it also introduces risks. For example, if you do not understand the source of the open source code, you will not be able to update the subsequent vulnerability fixes of the open source code simultaneously, and you will also be exposed to license compliance. Risks and legal risks such as intellectual property rights bring varying degrees of security threats and economic or reputational losses. A well-known open source risk case is the Heartbleed vulnerability. It is a security vulnerability in the encryption library OpenSSL, which is widely used to implement the Internet's Transport Layer Security protocol. It was introduced into OpenSSL in 2012 and first disclosed to the public in April 2014. As long as a defective OpenSSL instance is used, both the server and the client may be vulnerable to attack. Therefore, traceability detection of the code in software products is crucial for software products.

In order to implement code traceability detection of software products, it is necessary to build a code library for code matching search. The number of codes and construction methods contained in the code library directly affect the accuracy and efficiency of code traceability detection. Due to the difficulty of constructing large code bases, existing code traceability detection technologies mostly propose efficient code detection algorithms based on the assumption that there are already massive code libraries. For example, research on selecting codes from massive open source software libraries is most likely to be The open source software used participates in traceability comparison. And there is a lack of effective techniques on how to structure a code base. In the existing technology, several open source software are usually downloaded locally to form a code base. However, these code libraries have problems such as small project coverage and insufficient support for network-wide traceability detection of codes, and poor code library architecture design leading to inefficient code traceability detection.

Contents of the invention

In view of the technical problems existing in the prior art, the present invention provides a code library design method for rapid network-wide code traceability detection, and realizes rapid network-wide code traceability detection based on the code library. The code library generated by the method of the present invention can Supports rapid network-wide traceability detection of code at file granularity, with high detection efficiency. At the same time, the code base can be updated regularly and efficiently.

In the present invention, "whole network code" refers to the collected code data of most open source code hosting platforms. The warehouse hosted on the code hosting platform is called a remote warehouse; the remote warehouse is cloned locally to become a local warehouse; the code library designed by the present invention for rapid network-wide code traceability detection is to clone the remote warehouse locally to form a local warehouse, and then A database formed by extracting data from a local warehouse.

The code library design provided by the present invention for rapid network-wide code traceability detection utilizes the internal principles and hash values of Git, and specifically includes:

1) The remote warehouse can be downloaded to the local through the git clone command, and the updates of the remote warehouse can be transferred back to the local through the git fetch command. git fetch calculates what objects are missing in the local warehouse compared to the remote warehouse by comparing the heads of the local warehouse and the remote warehouse, and then the remote warehouse transfers these missing objects back to the local.

2) Git uses four types of data objects for version control, and the references to these objects are SHA1 values calculated based on the contents of the object. The commit object represents a change to the project and contains a SHA1 including the commit parent object (if any), the folder (tree object), author ID and timestamp, submitter ID and timestamp, and submission information. The tree object represents a folder within the project and is a list that contains the SHA1 of the files (blobs) and subfolders (other tree objects) in the folder, as well as their related modes, types and names. A blob object is a compressed version of a certain version of a file's contents (source code). The tag object is a string used to associate a human-readable name with a specific version of the repository. A commit represents a code change, usually including modifications to several files (blobs).

3) The hash value is a hash value of several bits calculated based on the file content. Different file contents will generate different hash values, so the hash value can be used to uniquely index the file.

The present invention adopts a code library design method for code traceability detection on the entire network, and efficiently stores Git objects in open source projects that use Git on the entire network to obtain a code library, which can be used for code traceability detection and analysis, and at the same time provides code library Efficient update solution. Specifically, the design and construction of the code library for code network-wide traceability detection includes: project discovery, data extraction, data storage, code information mapping construction and data update. As for the data storage part, the present invention designs a storage mode that is different from the traditional Git storage mode and blocks storage according to Git object classification. This storage mode can greatly reduce the storage space of the code base and improve the efficiency of the entire network retrieval. The original method of the invention; the code information mapping construction part is also the original method of the present invention. The present invention proposes to construct the code file to the information of the code file (including its project and commit, the author and time when it was created, and its file name). Relational mapping can quickly retrieve the entire network information of code files; the present invention proposes an efficient update method for the constructed ultra-large-scale code library, and proposes a customized git fetch protocol based on the Libgit2 function library, using the constructed code library as On the back end, this customized protocol can correctly obtain the new Git object data of the remote warehouse with minimal time and space costs. Finally, the present invention also provides a fast network-wide traceability detection solution for codes at file granularity.

The technical solution of the present invention is:

A code base design method for rapid network-wide code traceability detection, which uses Git objects in Git open source projects to efficiently store the entire network to obtain a code base, and achieve efficient updates of the code base; it is proposed to classify Git objects by type Block storage storage mode to reduce the storage space of the code library and improve the efficiency of the entire network retrieval; build a relationship mapping between code files to code file information, which can quickly retrieve the entire network information of code files; and build The ultra-large code base adopts an efficient update method. Based on the Libgit2 function library, a customized git fetch protocol is proposed. With the built code base as the backend, new Git object data of the remote warehouse can be obtained efficiently; used for fast whole-network code The code base design for traceability detection includes: project discovery, data extraction, data storage, code information mapping construction and data update process; specifically includes the following steps:

A. Obtain a list of open source software projects across the entire network through a variety of project discovery methods;

Open source software projects are mostly hosted on some popular development collaboration platforms such as GitHub, Bitbucket, GitLab and SourceForge. The present invention uses a variety of methods, including using the API provided by the development collaboration platform, parsing the web pages of the platform, etc. to discover projects, and then uses the union of the discovered project sets as the final open source project list, thereby obtaining the open source project list.

In specific implementation, it can be completed on an ordinary server (such as Intel E5-2670 CPU server), and the requirements for hardware are low. This invention packages the script of the project discovery process into the docker image.

B. Data extraction: Download the projects in the open source project list obtained in step A to the local computer and extract the Git objects in them;

In specific implementation, a copy of the remote warehouse is created locally through the git clone command. After batch copying open source projects, batch extract all Git objects in the cloned open source projects through Git.

Data extraction can be done in parallel on (cloud) servers. This invention uses Git's C language interface Libgit2 to first list all Git objects in the project, then classify them according to object type, and finally extract the contents of each object. The present invention specifically uses a cluster with 36 nodes, each node has a 16-core Intel E5-2670 CPU, and a memory of 256GB. Each node starts 16 threads to complete the above Git object extraction work. One node can process approximately 50,000 items in 2 hours. After extracting the Git data in the cloned project, delete the cloned project, and then start a new cloning-extraction process.

C.Git object data storage: Store Git object data in blocks according to Git object type classification, reducing data storage space and improving parallel processing efficiency; specifically including:

a. Do not save binary files (such as PDFs and images) included in open source projects;

b. Store Git object data according to the Git object type, that is, the database types include commit database, tree database, blob database (excluding binary blobs) and tag database. This storage method reduces the data storage space to hundreds of terabytes, and at the same time can quickly retrieve whether the data is saved in the code base.

c. The database of each type of Git object includes cache data and content data, which are stored in the cache database and content database respectively to speed up retrieval; the cache included in each type of database (i.e. commit database, tree database, blob database, tag database) The database and content database can be divided into multiple copies (such as 128 copies) for parallelization; the cache database is used to quickly determine whether a Git object is already stored in the database and is necessary for data extraction (if it exists, this Git object will not be extracted objects, thereby saving time). In addition, the cache database can also help determine whether a repository needs to be cloned. If a repository's head (the commit object pointed to by each branch stored in .git/refs/heads) is already in the cache database, there is no need to clone.

d. The cache database is a key-value database; the content database is saved in a splicing manner to facilitate updates.

The cache database is a key-value database, where the key is the SHA1 value (20 bytes) of the Git object, and the value is the offset position and size of the Git object in the content database after being compressed using Perl's compress library. The content database contains the compressed contents of Git objects spliced together contiguously. The content database is saved in a splicing manner, which can be updated quickly by splicing the new content to the end of the corresponding file. For commit and tree objects, a separate random search key-value database is created, where the key is the SHA1 of the Git object and the value is the compressed content of the corresponding Git object. The random query performance of the key-value database is relatively fast, and each thread can query more than 170K git objects per second.

e. Utilize SHA1 values for parallelization.

This invention uses the last 7 bits of the first byte of the SHA1 value of the Git object to divide each type of database into 128 parts. In this way, there are 128 cache databases and 128 content databases for the four types of Git objects. In addition, the commit object and the tree object each have 128 random search key value databases, with a total of 128*(4+4+2) databases. These databases can be placed on one server to accelerate parallelism. During specific implementation, the size of a single content database ranges from 20MB for tag objects to 0.8TB for blob objects. The maximum size of a single cache database is a tree object with a size of 2Gb.

f. The present invention uses the database TokyoCabinet written in C language (similar to berkeley db).

TokyoCabinet uses hashes as indexes, which can provide read query performance about ten times faster than various common key-value databases such as MongoDB or Cassandra. Faster reading query speed and strong portability just meet the construction requirements of a code library for network-wide code traceability detection. Therefore, the present invention uses the database TokyoCabinet instead of the NoSQL database with more complete functions.

D. Code information mapping construction:

The goal of the code library designed by the present invention is to quickly conduct network-wide traceability detection of codes and support the analysis of the security and compliance of software projects. The present invention constructs a code file (blob) to the project containing it, and the code file to Contains its commit, the relationship mapping of the code file to its author, the code file to its file name, and the code file to its creation time. These relationship mappings are saved in the form of a database, and the entire network information of a code file can be quickly obtained. For example, the project and commit that contains it, the author who created it and the time when it was created are implemented to construct a mapping of these code information. Obtaining this information for a code file is useful for a comprehensive assessment of the security and compliance of a software project.

This invention builds relationship mapping centered on commit, which specifically includes:

Build the mutual mapping between commits and projects, the relationship mapping between commits and authors, and time, the relationship mapping between authors and commits, the mutual mapping between commits to code files (blobs), and the mutual mapping between commits and file names. mapping.

The list of projects containing a code file (blob) can be determined by the combination of the code file (blob) to commit and commit to project; the creation time of the code file (blob) can be determined by the code file (blob) to commit and commit to time. The relationship combination is determined. The author of the code file (blob) can be determined through the relationship combination of the code file (blob) to commit and commit to the author.

A mapping of the relationship between code files and file names is also constructed to support traceability of specific code fragments.

Utilize the TokyoCabinet database to save these relational maps for quick retrieval. The present invention still uses block storage to improve retrieval efficiency. Specifically, the present invention divides each type of relationship mapping into 32 sub-databases. For commits and (code file) blobs, use the last 5 digits of the first character of their SHA1 to divide. For author, project and file names, the present invention uses the last 5 bits of the first byte of their FNV-1Hash for partitioning.

E.Data update

Git objects are immutable (that is, existing Git objects will remain unchanged, only new Git objects will exist), so you only need to obtain these new Git objects. This invention specifically uses two methods to update the code base:

a. Identify the new Git project, clone and extract the Git objects in it.

b. Identify updated projects by obtaining the latest commit of the branch of the remote repository that has been collected, and then modify the git fetch protocol so that the protocol can be cloned without a local Git repository (after extracting the data in step B) When the Git repository is deleted (to save space), the built code base is used as the backend to obtain the new Git objects of the remote repository and extract the new Git objects into the code base. This invention restores the git fetch process through the source code that implements the gitfetch function in Libgit2, which specifically includes the following steps:

b1) Add the remote warehouse to the local warehouse. In Libgit2, the git_remote structure is used to represent the remote warehouse. When this structure is created, all branch references in the .git/refs/heads folder in the local warehouse will be filled into a member variable (ref )middle;

b2) The local warehouse establishes a connection to the remote warehouse;

b3) After the connection is established, the remote warehouse will respond (respond) and send all branch references (contents in the .git/refs/heads folder) of the remote warehouse to the local;

b4) After the local warehouse receives the references sent back by the remote warehouse, it will check one by one whether the objects pointed to by these references are in the local warehouse. If it is in the local warehouse, it will mark it to indicate that this branch has not been updated, and there is no need to request the remote warehouse to send it. renew. Then insert these references into the member variables mentioned in step b1

b5) After the local warehouse has checked all these references, it will send this member variable back to the remote warehouse (including the marked references) to "negotiate" with the remote warehouse. Here the local will wait for the ACK signal sent back by the remote warehouse. The way Libgit2 waits here is to sort the commit objects in the local warehouse in chronological order, and then traverse forward from the most recent commit. For each commit object, send it to the remote warehouse to tell it that the object exists locally, and then Then send the checked reference to the remote warehouse. This is repeated up to 256 times until an ACK signal is received from the remote warehouse.

b6) After negotiating with the remote warehouse (that is, telling the remote warehouse: what are the latest commits of the local warehouse branch and which ones do you want), the remote warehouse can calculate which Git objects to send back to the local. The remote warehouse packages these objects into files in packfile format and sends them back to the local computer.

b7) After receiving the data sent back, the local warehouse will parse it according to the format of the packfile and build the corresponding index file to facilitate retrieval. When building the index file, it needs to be restored based on the Git object in the local Git repository.

It can be seen from the steps of git fetch that except for steps b5) and b7), other processes do not involve other Git objects except the branch references pointed to. git fetch is performed by comparing the branch references of the remote warehouse with the local ones. Determine whether the remote warehouse has updates. We propose to make the following modifications to git fetch:

1) Modify the original git fetch step b3: Save the branch references sent back by the remote warehouse to the local, and determine whether these branch references are saved in the local code library. If they exist, it means that the remote warehouse has not been updated. If they do not exist, it means that the remote warehouse has. Update and go to the next step.

2) Modify the original git fetch step b5: The original git fetch protocol sorts the commits and sends them to the remote warehouse, just to wait for the ACK signal of the remote warehouse, and has no special effect, so the present invention changes the waiting method: every Send the latest commit object of the main branch each time, repeating up to 256 times until the ACK signal from the remote warehouse is received.

3) Modify the original git fetch step b6: Save the packfile format file sent back by the remote warehouse to the local, parse the packfile file according to the Git object in the code library, and do not proceed to step b7.

After making the above modifications to git fetch, git fetch can be updated with the built code base as the backend, without the need to clone the complete warehouse for each update, while reducing network bandwidth overhead and time overhead.

During specific implementation, the present invention also provides a fast network-wide code traceability detection method based on the code library at the file granularity, including the following steps:

1) For a code file, calculate its SHA1 value

2) Based on the code information mapping constructed in step D, use the SHA1 of the code file as the key to query the entire network information of the code file, including the project list, commit list, corresponding file name, author and other information containing the code file, and feedback to user.

Compared with the prior art, the beneficial effects of the present invention are:

Through the code library design provided by the present invention, efficient network-wide traceability detection of codes can be supported. Through the technical solutions and provided embodiments of the present invention, it is possible to complete the construction of local code libraries for open source Git warehouses on multiple code hosting platforms across the entire network, including GitHub, without the need for a large number of servers; without the need for particularly large amounts of bandwidth, Complete incremental updates to the code base.

The technical solution and provided embodiments of the present invention provide detailed guidance for constructing a code library for network-wide code traceability detection, filling the gap in massive code library construction technology in the field of code traceability detection.

Description of the drawings

Figure 1 is a flow chart of a code base design method for rapid network-wide code traceability detection in an embodiment of the present invention.

Figure 2 is a flow chart of a code base update strategy in an embodiment of the present invention.

Figure 3 is a flow chart of the customized git fetch process in the embodiment of the present invention.

Figure 4 is a flow chart for obtaining remote warehouse updates based on the customized git fetch protocol in an embodiment of the present invention.

Figure 5 is a flow chart of a fast network-wide code traceability detection method based on the constructed code library in an embodiment of the present invention.

Detailed ways

The present invention is further described below through examples in conjunction with the accompanying drawings, but does not limit the scope of the present invention in any way.

The present invention provides a code library design method for rapid network-wide code traceability detection, which specifically includes the following steps:

A. Obtain a list of open source software projects across the entire network through a variety of project discovery methods. The implementation method is:

At present, most open source software projects are hosted on some popular development collaboration platforms such as GitHub, Bitbucket, GitLab and SourceForge. There are also some open source projects that are hosted on personal or project-specific websites. Therefore, in order to support network-wide traceability detection of code, it is necessary to obtain as complete a list of open source projects as possible. In response to this challenge, the present invention combines multiple methods, such as using APIs provided by the platform, parsing the web pages of the platform, to discover projects. Finally, the union of the project sets discovered by these methods is used as the final open source project list.

B. Data extraction: Download the projects in the open source project list in step A locally and extract the Git objects in them;

This step is responsible for downloading the project discovered in step A to the local and extracting the Git objects in it. Use the gitclone command to create a copy of the remote repository locally. After batch copying the project, use Git to batch extract all the Git objects in the cloned project. This step can be done in parallel on the (cloud) server.

C. Data storage: Store in blocks according to Git object types, reducing data storage space and improving parallel processing efficiency;

There may be many duplicate Git objects between open source projects due to code reuse, pull-request development model and other reasons. At the same time, open source projects will also include many binary files, such as PDFs and images. If this redundancy and binary files are not removed, it is estimated that the required data storage space will exceed 1.5PB. Such a huge amount of data will make the code traceability task almost impossible to achieve. In order to avoid the redundancy of Git objects between warehouses, and the design of the code library is oriented to code traceability detection on the entire network, the present invention does not save binary files. The present invention stores them according to the types of Git objects, that is, commit database, tree database, blob database (excluding binary blobs) and tag database. This storage method reduces the data storage space to hundreds of terabytes and can also quickly retrieve whether the data is saved in the code base.

D. Code information mapping construction:

The goal of this code library is to quickly conduct network-wide traceability detection of codes and support analysis of the security and compliance of software projects. To this end, the present invention constructs a code file (blob) to the project containing it, and the code file to the project containing it. Its commit, the relationship mapping of the code file to its author, the code file to its file name, and the code file to its creation time. These relationship mappings are saved in the form of a database, and the entire network information of a code file can be quickly obtained, such as Contains its project and commit, the author who created it and the time when it was created, to achieve the construction of a mapping of these code information. Obtaining this information for a code file is useful for a comprehensive assessment of the security and compliance of a software project.

E.Data update

Keeping the code base up to date is critical to the task of code traceability detection. As existing repositories grow in size and new repositories appear, the process of cloning all repositories takes longer and longer. Currently, to clone all git repositories (more than 130 million including forks), it is estimated that the total time will require 600 single-threaded servers to run for a week, and the result will be more than 1.5PB of disk space. Fortunately, git objects are immutable (that is, existing Git objects will remain unchanged, only new Git objects will exist), so just get these new Git objects. Specifically, the present invention proposes to use two strategies to update the code base:

1. Identify the new Git project, clone and extract the Git objects in it.

2. Identify updated projects by obtaining the latest commits of all branches of the remote repository that has been collected, and then modify the git fetch protocol so that the protocol can be cloned without a local Git repository (after extracting the data in step B) (The Git repository has been deleted to save space), use the built code base as the backend to obtain updates from the remote repository, and extract the new Git objects into the code base. This invention restores the git fetch process through the source code that implements the git fetch function in Libgit2, as shown in Figure 2, which specifically includes the following 7 steps:

1) Add the remote warehouse to the local warehouse. In Libgit2, the git_remote structure is used to represent the remote warehouse. When this structure is created, all branch references in the .git/refs/heads folder in the local warehouse will be filled into a member variable (ref )middle;

2) The local warehouse establishes a connection to the remote warehouse;

3) After the connection is established, the remote warehouse will respond (respond) and send all branch references (contents in the .git/refs/heads folder) of the remote warehouse to the local;

4) After the local warehouse receives the references sent back by the remote warehouse, it will check one by one whether the objects pointed to by these references are in the local warehouse. If it is in the local warehouse, it will mark it to indicate that this branch has not been updated, and there is no need to request the remote warehouse to send it. renew. Then insert these references into the member variables mentioned in step 1

5) After the local warehouse has checked all these references, it will send this member variable back to the remote warehouse (including the marked references) to "negotiate" with the remote warehouse. Here the local will wait for the ACK signal sent back by the remote warehouse. The way Libgit2 waits here is to sort the commit objects in the local warehouse in chronological order, and then traverse forward from the most recent commit. For each commit object, send it to the remote warehouse to tell it that the object exists locally, and then Then send the checked reference to the remote warehouse. This is repeated up to 256 times until an ACK signal is received from the remote warehouse.

6) After negotiating with the remote warehouse (that is, telling the remote warehouse: what are the latest commits of the local warehouse branch and which ones do you want), the remote warehouse manager can calculate which Git objects to send back to the local. The remote warehouse packages these objects into files in packfile format and sends them back to the local computer.

7) After receiving the data sent back, the local warehouse will parse it according to the format of the packfile and build the corresponding index file to facilitate retrieval. When building the index file, it needs to be restored based on the Git object in the local Git repository.

It can be seen from the steps of git fetch that except for steps 5) and 7), other processes do not involve other Git objects except the branch reference pointed to. git fetch is by comparing the branch reference of the remote warehouse with the branch of the local warehouse. The difference in references is used to determine whether the remote repository is updated. The present invention proposes to modify git fetch as follows:

1) Modify the original git fetch Step 3: Save the branch references sent back by the remote warehouse to the local, and determine whether these branch references are saved in the local code library. If they exist, it means that the remote warehouse has no new Git object data. If not, Existence indicates that the remote warehouse has new Git object data. Go to the next step.

2) Modify the original git fetch step 5: The original git fetch protocol sorts the commits and sends them to the remote warehouse, just to wait for the ACK signal of the remote warehouse, which has no special effect, so the present invention changes the waiting method: every Send the latest commit object of the main branch each time, repeating up to 256 times until the ACK signal from the remote warehouse is received.

3) Modify the original git fetch step 6: Save the packfile format file sent back by the remote warehouse to the local, parse the packfile file according to the Git object in the code library, do not proceed to step 7.

After making the above modifications to git fetch, git fetch can be updated with the built code base as the backend, without the need to clone the complete warehouse for each update, while reducing network bandwidth overhead and time overhead.

Finally, the present invention provides a fast network-wide traceability detection solution for code at file granularity, which specifically includes two steps:

1. Calculate the SHA1 value of a code file

2. Based on the code information mapping tool database built in step D, use the SHA1 of the code file as the key to query the entire network information of the code file, including the project list containing the code file, the commit list and the corresponding file name and author. , feedback to the user.

As a preferred solution, step B uses Git's C language interface Libgit2 (because C language is more efficient and faster) to complete the extraction task.

As a preferred solution, step C and step D use the TokyoCabinet database.

As a preferred solution, step E uses Git's C language interface Libgit2 to implement the customized Gitfetch protocol.

Figure 2 shows the process of a code base design method for rapid network-wide code traceability detection in an embodiment of the present invention, which includes the following specific implementation steps:

A. Project Discovery:

In order to obtain as complete a list of open source projects as possible, the present invention combines a variety of heuristic methods, including using the API of the development collaboration platform, parsing the web pages of the platform, etc. to discover projects. Finally, the union of the project sets discovered by these methods is used as the final open source project list. This invention packages the script of the project discovery process into the docker image. Specifically, the project discovery method adopted in this invention is as follows:

1. Use the API of the development collaboration platform. Some code hosting platforms such as GitHub provide APIs that can be used to discover the complete collection of open source projects on the platform. These APIs are platform-specific and can be used in different ways, so different API queries need to be designed for different platform APIs. However, these APIs generally have access rate restrictions for users or IP addresses. This restriction can be overcome by building a user ID pool. For the GitHub platform, we use GitHub's GraphQL API to obtain a list of updated GitHub warehouses. The specific operation is to divide the time period of the warehouses that need to be obtained equally according to the number of user IDs in the user ID pool, and then each user ID is responsible for one time The number of updated warehouses in the segment. The query condition is: {is:public archived:false pushed:start_time..end_time}. Here, start_time and end_time are replaced with 10 minutes as an interval in each time period to obtain the number of updates in each 10-minute interval. The number of updated repositories; for the Bitbucket platform, the api query used is https://api.bitbucket.org/2.0/repositories/? pagelen=100&after=date, replace date with a specific time such as 2017-11-18, and you can get the Bitbucket warehouse created after 2017-11-18; for the SourceForge platform, the platform provides a project list in XML format , the address of the XML file is https://sourceforge.net/sitemap.xml, download the XML parsing to get a list of all projects on SourceForge; for the GitLab platform, the API query used is https://gitlab.com/api/ v4/projects? archived=false&membership=false&order_by=created_a t&owned=false&page={}&per_page=99&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_request s_enabled=false, here set the page parameter to 1, and then obtain all git incrementally on the lab project.

2. Parse the web pages of the website. For the Bioconductor platform, by parsing the http://git.bioconductor.org web page, you can get all the projects on the website; for the repo.or.cz platform, by parsing https://repo.or.cz/? a=project_list web page, you can get all the projects on the website; for the Android platform, you can get all the projects on the website by parsing the https://android.googlesource.com/ web page;

For the ZX2C4 platform, parse the https://git.zx2c4.com web page to get all projects on the platform; for the eclipse platform, parse the http://git.eclipse.org/ web page to get all the projects on the platform All projects; for the PostgreSQL platform, parse the http://git.postgresql.org web page to get all projects on the platform;

For the Kernel.org platform, parse the http://git.kernel.org web page to get all the projects on the platform; for the Savannah platform, parse the http://git.savannah.gnu.org/cgit web page to get all the projects on the platform. Get all the projects on the platform.

This step can be completed on an ordinary server (such as Intel E5-2670 CPU server), and the hardware requirements are very low. As of September 2020, we have retrieved more than 130 million different repositories (excluding GitHub repositories marked as fork and repositories with no content).

B. Data extraction:

This step can be done in parallel on many servers, but requires a lot of network bandwidth and storage space. Use the git clone command to batch clone the remote warehouse to the local. After calculation, on an Intel E5-2670 CPU server, a single-threaded shell process can clone 20,000 to 50,000 randomly selected objects in 24 hours without network bandwidth restrictions. of projects (times vary greatly depending on warehouse size and platform). In order to clone all projects (more than 130 million) in one week, about 400-800 servers are needed, which is very expensive. Therefore, the present invention optimizes retrieval by running multiple threads on each server and retrieves only a small portion of the repository that has changed since the last retrieval. The present invention currently uses 5 data transmission nodes on a computing cluster platform with 300 nodes and a bandwidth of up to 56Gb/s to complete the cloning task. In addition, this step can be completed using cloud servers instead of computing clusters. You can purchase customized cloud service resources that meet your needs during cloning, and then release these resources after the batch cloning is completed. Cloud servers can achieve higher bandwidth and clone faster.

After cloning the project locally, you need to extract all Git objects in the project. The Git client can only display the contents of one Git object one by one, which is not conducive to automated batch processing. This invention uses Git's C language interface Libgit2 to first list all Git objects in the project, then classify them according to object type, and finally extract the contents of each object. The present invention is currently on a cluster with 36 nodes, each node has a 16-core Intel E5-2670 CPU and a memory of 256GB. Each node starts 16 threads to complete the above Git object extraction work. One node can process approximately 50,000 items in 2 hours. After extracting the Git data in the cloned project, delete the cloned project, and then start a new cloning-extraction process.

C. Data storage: Store in blocks according to Git object types, and do not save binary files, reducing data storage space and increasing parallel processing speed.

The invention stores git objects by type to avoid redundancy and reduce storage overhead; it is oriented to code traceability detection and does not save binary files during storage; each Git object database contains cache data and content data, which are stored in the cache database and content database respectively. in order to speed up retrieval; in order to enable parallelization, the cache database and content database of each Git object can be divided into multiple copies (such as 128 copies) for parallelization; the content database is saved in a spliced manner to facilitate updates.

Specifically, the present invention stores Git objects separately according to their types to avoid redundancy, so there are four types of databases: commit database, blob database, tree database and tag database. Each database contains cache data and content data, which are stored in the cache database and content database respectively. The cache database is used to quickly determine if a specific object is already stored in our database and is required for the above data extraction (if it exists, the Git object is not extracted, thus saving time). In addition, the cache database can also help determine whether a repository needs to be cloned. If the head of a warehouse (the commit object pointed to by each branch stored in .git/refs/heads) is already in our cache database, it means that the warehouse has not been updated, and there is no need to clone the warehouse.

The cache database is a key-value database, where the key is the SHA1 value (20 bytes) of the Git object, and the value is the offset position and size of the Git object in the content database after being compressed using Perl's compress library. The content database contains the compressed contents of Git objects spliced together contiguously. The content database is saved in a splicing manner, which ensures that the update is completed quickly. You only need to splice the new content to the end of the corresponding file. Although this storage method can quickly scan the entire database, it is not the best choice for the random search required. For example, when calculating the modifications made by a commit, we need to traverse the commit database twice to obtain the commit object pointing to The tree object and the tree object pointed to by its parent commit object, and then traverse the tree database multiple times to obtain the content contained in the two tree objects, find the files with differences, and finally traverse the blob database once to calculate the modifications. Each traversal will cause repeated additional time overhead. Therefore, for commit and tree, the present invention also creates a random search key value database respectively, where the key is the SHA1 of the git object and the value corresponds to the compressed content of the Git object. The random query performance of this key-value database is relatively fast. After testing: a single thread on a server with a CPU of Intel E5-2623 can randomly query 1 million git objects in 6 seconds, that is, each thread queries more than 170K per second. git object.

Currently, the present invention has retrieved more than 20 billion Git objects (including more than 2.3 billion commit objects, more than 9.1 billion blob objects, more than 9.4 billion tree objects, and more than 18 million tag objects), and the data storage space is about 150TB. Without parallel processing, processing such a large amount of data would become extremely inefficient. The present invention utilizes SHA1 values to achieve parallelization. This invention uses the last 7 bits of the first byte of the SHA1 value of the Git object to divide each type of database into 128 parts. In this way, there are 128 cache databases and 128 content databases for the four types of Git objects. In addition, the commit object and the tree object each have 128 random search key value databases, with a total of 128*(4+4+2) databases. These databases can be placed on one server to accelerate parallelism. Currently, the size of a single content database ranges from 20MB for tag objects to 0.8TB for blob objects. The largest single cache database is a tree object with a size of 2Gb.

Nonetheless, the size of the database limits the choice of database. For example, a graph database like neo4j is very useful for storing and querying relationships, including transitive relationships, but it cannot (at least on a normal server) handle hundreds of billions of relationships. In addition to neo4j, the present invention also tried many traditional databases. This invention evaluates the common relational databases MySQL and PostgreSQL as well as the key-value database (NoSQL) databases MongoDB, Redis and Cassandra. SQL, like all centralized databases, has limitations in handling petabytes of data. The present invention therefore focuses on NoSQL databases, which are designed for large-scale data storage and large-scale parallel data processing on a large number of commercial servers.

After testing, this invention uses a database written in C language called TokyoCabinet (similar to Berkeley db). TokyoCabinet uses hashes as indexes, which can provide read query performance about ten times faster than various common key-value databases such as MongoDB or Cassandra. The faster reading query speed and strong portability just meet the requirements for building a code base for network-wide code traceability detection, so we use it to replace the NoSQL database with more complete functions.

D. Code information mapping construction, including:

Design and generate a relationship mapping that can quickly map a code file (blob) to its information. The information of the code file includes the project and commit that contains it, the author and time when it was created, and its file name. These relationships are mapped to the database's Save in form, you can quickly retrieve the entire network information of code files

The goal of this code base is to quickly conduct network-wide traceability testing of codes and support analysis of the security and compliance of software projects. Therefore, the present invention generates a relationship mapping between the code file (blob) and its information (including the project and commit containing it, the author and time when it was created, and the file name), and saves it in the form of a database, so that the code file can be Search the entire network information. . Network-wide information of code files is useful for comprehensive assessment of the security and compliance of software projects, and is an important part of network-wide code traceability detection.

Information about a code file includes the project and commit that contains it, its file name, the author who created it and the time it was created. Among them, the author and time when it was created are included in the commit that created it. At the same time, the relationship mapping from commit to project and the relationship mapping from project to commit can be completed in step B data extraction. Therefore, the present invention takes commit as the center to build relationship mapping, specifically: building the mutual mapping between commit and project, building the relationship mapping between commit and author and time, building the relationship mapping from author to commit, and building the relationship mapping from commit to code file. The mutual mapping between (blob) and the mutual mapping between commit and file name. Then, the list of projects containing a code file (blob) can be determined by the combination of the code file (blob) to commit and commit to projects; similarly, the creation time of a code file (blob) can be determined by the code file (blob) to The combination of the relationship between commit and commit to time is determined. The author of the code file (blob) can be determined by the combination of the relationship between the code file (blob) and commit and commit to the author.

The mapping from commit to author, time and project is not difficult to achieve, because the author and time are part of the commit object, and the mapping between commit and project can be obtained during step B data extraction. However, the code files (blobs) introduced or deleted by a commit are not directly related to the commit and need to be calculated by recursively traversing the tree objects of the commit and its parent commit. A commit contains a snapshot of the warehouse, including all trees (folders) and blobs (code files). In order to calculate the difference between a commit and its parent commit, that is, the new code file (blob), we start from the tree object pointed to by the commit object, traverse each sub-tree and extract all code files (blob). By comparing all code files (blobs) of each commit, you can get the new code files (blobs) introduced by a commit. On average, it takes about 1 minute to get the changed file names and code files (blobs) of 10,000 commits in a single thread. It is estimated that for more than 2.3 billion commits, the overall time for a single thread takes 104 days. By running 16 threads on a server with a 16-core Intel E5-2623 CPU, it can be completed in one week. In addition, these relationships are incremental and only need to be generated once, then the above operations are performed on each updated commit, and then inserted into the existing database. Based on the combination of the relationship between code file (blob) and commit and commit to file name, the corresponding relationship between code file (blob) and file name cannot be determined because one commit will modify multiple files. The present invention also constructs a mutual relationship mapping between code files and file names to support traceability of specific code fragments. For example, if you want to perform a traceability check on a piece of Python code, then all Python files must be checked. Then the mapping of file names to code files can obtain all Python files ending with .py, and then perform code traceability checks on these files.

Similar to the data storage part of step C, the present invention uses the TokyoCabinet database to save these relationship mappings for fast retrieval. The present invention uses block storage to improve retrieval efficiency. Specifically, the present invention divides each type of relationship mapping into 32 sub-databases. For commits and (code file) blobs, the present invention uses the last 5 bits of the first character of their SHA1 for partitioning. For author, project and file names, the present invention uses the last 5 bits of the first byte of their FNV-1Hash for partitioning.

E.Data update

Keeping the code base up to date is critical to the task of code traceability detection. In order to obtain an acceptable update time, the present invention uses the following methods to complete the data update:

1. Identify the new Git project, clone and extract the Git objects in it. The new open source project list discovered through step A is then compared with the last open source project list to determine the newly added projects, then clone it locally and extract the Git objects in it.

2. Identify the updated project, then clone only the updated project, and extract the new Git object. The present invention makes the following modifications to the git fetch protocol based on Libgit2:

1) Modify the original git fetch step 3: Save the branch reference sent back by the remote warehouse to the local. After the filter_wants function in the src/fetch.c file of Libgit2 calls the git_remote_ls function, the SHA1 value of the heads sent back by the remote warehouse received by git_remote_ls is saved to the file.

2) Modify the original git fetch Step 5: Modify the src/transports/smart_protocol.c file of Libgit2, modify the git_smart__negotiate_fetch function: comment out the call about git_revwalk_next, add the git_reference_name_to_id call, so that the latest commit object of the main branch is sent every time, repeat At most 256 times until the ACK signal from the remote warehouse is received.

3) Modify the original git fetch Step 6: Modify the git_smart__negotiate_fetch function in the /src/transports/smart_protocol.c file of Libgit2 to send data back to the remote warehouse (git_pkt_progress

*p) Save to a local file and return directly without proceeding to step 7.

After making the above modifications, recompile the Libgit2 library, and then use the modified git fetch protocol to obtain the new Git object data of the remote warehouse. The specific steps are as follows:

1. Initialize an empty Git repository

2. Extract the SHA1 values and contents referenced by all branches of a warehouse from the built code base, and fill them into the empty git warehouse. The filling method is as follows: construct the header information of the branch reference, the format is: object type + space

+Object content length+a null byte, such as "blob 12\u0000". Then the header information and the original data are spliced together, and the spliced content is compressed using zlib's compress function. Finally, create a subdirectory named with the first two digits of SHA1 in the .git/objects folder in the empty git warehouse, create a file named with the last 38 digits of SHA1 in this subdirectory, and write the compressed content into this in file

3. Create a file named branch name (such as master) in the .git/refs/heads folder in this empty Git repository, and then write the SHA1 value of the commit referenced by the branch into this file

This invention directly splices the newly added Git object data into the corresponding content database according to the type and its SHA1 value, records its SHA1 value, offset and size in the content file in the cache database, and updates the corresponding Relational mapping database.

After the code base is built, the code can be quickly traced back to the entire network at the file granularity. Proceed as follows:

1. Calculate the SHA1 value of the code file. Here we use the sha1 function of the hashlib library of python2 for calculation. For example, the file https://github.com/fchollet/deep-learning-models/blob/master/resnet50.py contains the implementation of the deep learning model ResNet50, and its SHA1 value is calculated to be e8cf3d7c248fbf6608c4947dc53cf368449c8c5f

2. Based on the code information mapping tool database built in step D, use the SHA1 of the code file as the key to query the entire network information of the code file, including the project list containing the code file, the commit list and the corresponding file name and author. , feedback to users. Through the mapping from blob to commit, we get 192 commits containing the blob, and through the mapping from commit to project, we get 377 projects containing the blob. The above process only takes 0.831s.

It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined by the claims.

Claims (10)

1. A code base design method for quick whole network code traceability detection is used for obtaining a code base by carrying out efficient storage on Git objects in a whole network using Git open source project through project discovery, data extraction, data storage, code information mapping construction and data updating processes, and realizing efficient updating of the code base;

Comprising the following steps: a storage mode of dividing and storing according to the types of the Git objects is adopted; constructing a relation mapping between code files and code file information, and rapidly searching the whole network information of the code files; a high-efficiency updating mode is adopted for the constructed ultra-large-scale code library, a customized gate fetch protocol is proposed based on a Libgit2 function library, and the constructed ultra-large-scale code library is taken as a rear end to efficiently obtain newly-added gate object data of a remote warehouse;

the method specifically comprises the following steps:

A. a server is utilized to obtain a full-network open source software project list through a plurality of project discovery methods, and scripts of the project discovery process are packed into a dock mirror image;

B. and (3) data extraction: b, downloading the items in the open source item list acquired in the step A to the local and extracting the Git objects in the items; the extraction is completed in parallel in a multithreading manner on the server cluster;

git object data store: according to the type division and block storage of the Git object types, the data storage space is reduced, and the parallel processing efficiency is improved; the method specifically comprises the following steps:

a. the binary files included in the open source items are not saved;

b. classifying and storing the Git object data according to the Git object type, namely, the types of the database comprise a commit database, a tree database, a blob database and a tag database, so that the data storage space is reduced to hundred TB levels, and meanwhile, whether the data are stored in a code base or not can be quickly searched;

c. The database of each Git object comprises cache data and content data which are respectively stored in the cache database and the content database, so that the retrieval speed is increased; dividing a cache database and a content database contained in each type of database into a plurality of parts for parallelization; the cache database is used for quickly determining whether a certain Git object is already stored in the database and is necessary for data extraction; if a certain Git object exists in the database, the Git object is not extracted; the cache database is also used for determining whether a warehouse needs to be cloned; if the commit object pointed to by the head of a repository is already in the cache database, i.e., no cloning is required;

d. the cache database is a key value database; the content database is saved in a splicing mode so as to be convenient to update;

the key in the cache database is the SHA1 value of the Git object, and the value in the cache database is the offset position and the size of the Git object in the content database after compression by using the Perl's express library;

the content database contains compressed content of the Git objects which are continuously spliced together; the content database is saved in a splicing mode, and only new content is spliced to the tail of a corresponding file;

Creating a random lookup key value database for the commit and tree objects respectively, wherein the key is SHA1 of the Git object and the value is the compression content of the corresponding Git object;

e. dividing each type of database into a plurality of parts by using SHA1 values, and realizing parallelization acceleration;

f. using a database TokyoCabinet indexed by hash;

D. constructing a mapping of the code information relationship by taking a commit as a center; comprising the following steps: a mapping of code files to items containing it, code files to completions containing it, code files to its author, code files to its filename, code files to its creation time; storing the relation maps by using a TokyoCapbinet database in a block storage mode so as to perform quick retrieval;

E. acquiring a new Git object, and updating data of a code base; the method comprises two methods:

a. identifying a new Git item, cloning, and extracting a Git object in the new Git item;

b. identifying an updated item by acquiring the latest commit of a branch of a remote warehouse of the collected warehouse, and then modifying the gate fetch so that when the local gate warehouse is not available, a built code library is taken as the rear end, a newly added gate object of the remote warehouse is acquired, and the newly added gate object is extracted into the code library; the method specifically comprises the following steps:

b1 Adding the remote repository to the local repository; the remote warehouse is represented in Libgit2 by the git _ remote structure,

all branch references within the local warehouse. Git/refs/heads folder are filled into one member variable ref within the structure when the structure is created;

b2 A connection from the local warehouse to the remote warehouse is established;

b3 After the connection is established, the remote warehouse replies a response, and all branch references of the remote warehouse, namely, contents in the file folders of the git/refs/heads, are sent to the local;

storing branch references sent back by the remote warehouse to the local, judging whether the branch references are stored in a local code base, if so, indicating that the remote warehouse is not updated, and if not, entering the next step;

b4 After the local warehouse receives the references sent back by the remote warehouse, checking whether the objects pointed by the references are in the local warehouse one by one; if in the local repository, a flag is made indicating that the remote repository need not be requested to send updates, then these references are inserted into the member variables;

b5 The local warehouse orders the commit, sends the member variable, including the marked references, back to the remote warehouse, and negotiates with the remote warehouse; locally waiting for an ACK signal sent back by the remote warehouse; specifically, the most up-to-date commit object of the main branch is sent each time, and the execution is repeated for a plurality of times until an ACK signal of the remote warehouse is received;

b6 After negotiating with the remote repository, the remote repository may calculate the Git object to be sent back to the local; the remote warehouse packages the objects into files in a pack format and sends the files back to the local;

storing the file in the pack file format sent back by the remote warehouse to the local, and analyzing the pack file according to the Git object in the code library;

through the steps after the modification of the gate latch, the gate latch is updated by taking the constructed code base as the back end, a complete warehouse does not need to be cloned for each update, and network bandwidth overhead and time overhead are reduced.

2. The code library design method for rapid full-network code traceability detection according to claim 1, wherein the method for rapid full-network code traceability detection of codes on file granularity based on the code library comprises the following steps:

1) Calculating the SHA1 value of one code file;

2) And D, inquiring the whole network information of the code file by taking SHA1 of the code file as a key according to the code information mapping constructed in the step D, wherein the whole network information comprises a project list, a commit list, a corresponding file name and author information of the code file, and feeding back the project list, the commit list and the corresponding file name and author information to a user.

3. The code library design method for rapid full-network code traceability detection of claim 1, wherein in step a, said server comprises Intel E5-2670 CPU server; the development collaboration platform for hosting the open source software project comprises a Github, a Bitbucket, a GitLab and a SourceForge; the obtaining the full-network open source software project list through the multiple project discovery methods comprises the following steps: and using the API provided by the development collaboration platform and the webpage method of the analysis platform to take the union of the discovered project sets as a final open source project list, thereby obtaining the open source project list.

4. The code library design method for rapid full-network code traceability detection according to claim 1, wherein the step B is to perform data extraction, specifically to create a copy of a remote warehouse locally through a gitclone command, and then to extract all the Git objects in the cloned open source item in batches through Libgit 2.

5. The code library design method for rapid full-network code traceability detection according to claim 4, wherein the data extraction specifically adopts an Intel E5-2670 CPU with 36 nodes, the CPU of each node is 16 cores, the memory is a cluster with 256GB, and each node starts 16 threads; and using a Git C language interface Libgit2 to list all Git objects in the project, classifying according to object types, and extracting the content of each object.

6. The code library design method for rapid full-network code traceability detection according to claim 1, wherein in step C, the cache database and the content database contained in each class of databases can be divided into 128 parts for parallelism;

the parallelization is realized by using the SHA1 value, specifically, the last 7 bits of the first byte of the SHA1 value of the Git object are used for dividing each type of database into 128 parts, so that the four types of Git objects have 128 cache databases and 128 content databases, the commit object and the tree object also have 128 random lookup key value databases respectively, and 128 (4+4+2) databases are shared; these databases may be placed on a server to accelerate parallelism.

7. The method for rapid full-network code traceability detection of claim 6, wherein the size of a single content database is from 20MB of tag object to 0.8TB of blob object, and the size of a single cache database is up to 2Gb.

8. The code library design method for rapid full-network code traceability detection of claim 1, wherein step D builds a relationship map centered on commit, comprising:

constructing a mutual mapping between a commit and an item, constructing a relationship mapping between a commit and an author, time, constructing a relationship mapping between an author to a commit, constructing a mutual mapping between a commit to a code file blob, and constructing a mutual mapping between a commit to a file name;

determining an item list containing one code file blob by combining the relations from the code file blob to the commit and from the commit to the item; determining the creation time of the code file blob by the combination of the relationship from the code file blob to the commit and from the commit to the time; determining an author of the code file blob by a combination of the code file blob to commit and commit to author relationships;

reconstructing a correlation mapping between the code file and the file name to support the tracing of the specific code segment;

Storing the relation maps by using a TokyoCapbinet database and using a block storage, and particularly dividing each type of relation map into 32 sub-databases; for commit and code file blob, partitioning is performed using the last 5 bits of the first character of SHA 1; for author, item, and file name, the last 5 bits of the first byte of FNV-1Hash are used for partitioning.

9. The code library design method for rapid full-network code traceability detection according to claim 1, wherein step E recompiles Libgit2 library after modifying the gate fetch by method b; acquiring newly added Git object data of a remote warehouse; the method comprises the following specific steps:

E1. initializing an empty Git warehouse;

E2. extracting SHA1 values and contents of all branch references of a warehouse from the constructed code library, and filling the SHA1 values and contents into an empty git warehouse;

E3. creating a file named a branch name in a Git/refs/heads folder in an empty Git repository, and then writing the SHA1 value of the commit referenced by the branch into the file;

and directly splicing the data of the newly added Git object into a corresponding content database according to the type and the SHA1 value thereof, and recording the SHA1 value thereof, the offset and the size in the content file in a cache database, namely updating the corresponding relation mapping database.

10. The code library design method for rapid full-network code traceability detection according to claim 9, wherein in step E2, the filling mode is specifically as follows:

constructing header information of branch references in the format of: object type + space + object content length + one empty byte;

then splicing the head information and the original data, and compressing the spliced content by using a compression function of zlib;

finally, the first two subdirectories named SHA1 are created in the. Git/objects folder in the empty git repository, a file named SHA1 last 38 is created in the subdirectory, and the compressed content is written into the file.