CN117519724A

CN117519724A - Concurrent compiling method and device

Info

Publication number: CN117519724A
Application number: CN202311707350.7A
Authority: CN
Inventors: 刘伸伸; 程黎辉; 关亚东
Original assignee: Shanghai Longcheer Technology Co Ltd
Current assignee: Shanghai Longcheer Technology Co Ltd
Priority date: 2023-12-12
Filing date: 2023-12-12
Publication date: 2024-02-06

Abstract

The application aims to provide a method and equipment for concurrent compiling. Compared with the prior art, the method and the device have the advantages that the dependency relationship among the application program modules is determined, then the dependency relationship among the application program modules is split into independent modules, so that the dependency among the application program modules is independent, tasks which can be executed in parallel are determined based on the independent modules and the dependency relationship, the tasks which can be executed in parallel are defined by using parallel blocks in Jenkins Pipeline, and then the concurrent compiling task is triggered through Jenkins, so that the concurrent compiling of the tasks which are executed in parallel is realized. By the method, the technical problem of long compiling time in a software development project is solved, the compiling efficiency is remarkably improved by utilizing concurrent execution and dependency management, the total time required by the construction process is reduced, the utilization of computing resources can be maximally improved, and the idle time is reduced.

Description

Concurrent compiling method and device

Technical Field

The present application relates to the field of software development and continuous integration, and in particular, to a technique for concurrent compilation.

Background

In the conventional compiling process, a serial execution mode is generally adopted, and tasks are generally executed sequentially, so that the technical problem of low compiling efficiency exists.

Disclosure of Invention

The application aims to provide a method and equipment for concurrent compiling.

According to one aspect of the present application, there is provided a method for concurrent compilation, wherein the method comprises:

determining a dependency relationship between application program modules;

splitting the dependency relationship between the application program modules into independent modules so as to enable the dependency relationship between the application program modules to be independent;

based on the independent modules and the dependency relationships, determining tasks which can be executed in parallel, and defining the tasks which can be executed in parallel in a Jenkins Pipeline by using parallel blocks;

and triggering concurrent compiling tasks through Jenkins to realize concurrent compiling of the tasks which are executed in parallel.

Optionally, the splitting the dependency relationship between the application program modules into independent modules, so that the dependency between the application program modules independently includes:

splitting the dependency relationship between the application program modules into independent modules;

and adjusting the code position and the structure of the modules according to the split independent modules so as to enable the dependence among the application program modules to be independent.

Further wherein the method further comprises:

the dynamic compiling judges whether the application program module is completely and independently split, wherein the determining the tasks which can be executed in parallel based on the independent module and the dependency relationship, and the defining the tasks which can be executed in parallel in the Jenkins Pipeline by using parallel blocks comprises the following steps:

and when the tasks are split completely and independently, determining tasks which can be executed in parallel based on the independent modules and the dependency relationships, and defining the tasks which can be executed in parallel by using parallel blocks in Jenkins Pipeline.

Further, wherein the determining the dependency relationship between the application modules includes:

the dependency between the application modules is determined based on the source code and the configuration file using static analysis.

Further wherein the static analysis comprises at least any one of:

import statement analysis, class inheritance relationship analysis, method call analysis, build profile analysis, resource file reference analysis, and data flow analysis.

Optionally, the analysis of the imported sentence includes:

extracting names of modules or classes in the import statement based on the import statement in the source code;

determining a dependency relationship between source codes based on the names of the modules or classes and preset indexes, wherein the preset indexes comprise at least any one of the following: the name of the belonging package, the complete path of the class, the signature of the method.

Optionally, the method further includes:

dependency on keywords are used in Jenkins Pipeline to specify dependencies between tasks to ensure that tasks are performed in the correct order.

Optionally, the method further includes:

the compiling progress and results were monitored by Jenkins.

According to another aspect of the present application, there is also provided a computer readable medium having stored thereon computer readable instructions executable by a processor to implement operations as the aforementioned method.

According to yet another aspect of the present application, there is also provided an apparatus for concurrent compiling, wherein the apparatus includes:

one or more processors; and

a memory storing computer readable instructions that, when executed, cause the processor to perform operations of the method as described above.

Compared with the prior art, the method and the device have the advantages that the dependency relationship among the application program modules is determined, then the dependency relationship among the application program modules is split into independent modules, so that the dependency among the application program modules is independent, tasks which can be executed in parallel are determined based on the independent modules and the dependency relationship, the tasks which can be executed in parallel are defined by using parallel blocks in Jenkins Pipeline, and then the concurrent compiling task is triggered through Jenkins, so that the concurrent compiling of the tasks which are executed in parallel is realized. By the method, the technical problem of long compiling time in a software development project is solved, the compiling efficiency is remarkably improved by utilizing concurrent execution and dependency management, the total time required by the construction process is reduced, the utilization of computing resources can be maximally improved, and the idle time is reduced.

Drawings

Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the accompanying drawings in which:

FIG. 1 illustrates a method flow diagram for concurrency compilation, in accordance with an aspect of the subject application.

The same or similar reference numbers in the drawings refer to the same or similar parts.

Detailed Description

The invention is described in further detail below with reference to the accompanying drawings.

In one typical configuration of the present application, the terminal, the device of the service network, and the trusted party each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer readable media, as defined herein, does not include non-transitory computer readable media (transmission media), such as modulated data signals and carrier waves.

In order to further describe the technical means and effects adopted by the present application, the following description will be made in detail and complete with reference to the accompanying drawings and preferred embodiments.

FIG. 1 shows a flow chart of a method for concurrent compilation provided by an aspect of the present application, the method comprising the steps of:

s11, determining the dependency relationship between application program modules;

s12, splitting the dependency relationship between the application program modules into independent modules so as to enable the dependency relationship between the application program modules to be independent;

s13, based on the independent modules and the dependency relationships, determining tasks which can be executed in parallel, and defining the tasks which can be executed in parallel in Jenkins Pipeline by using parallel blocks;

s14, triggering concurrent compiling tasks through Jenkins to achieve concurrent compiling of the tasks executed in parallel.

In this embodiment, the present solution may be initially configured, for example, to set up the necessary compilation environment, including installation and configuration of build tools, and preparation of relevant build scripts and configuration files.

Further, in the step S11, a dependency relationship between application modules is determined, where the dependency relationship is used to define an execution order between application modules, for example, if the module a depends on the module B and the module C, the task of the module a should be started after the task execution of the module B and the module C is completed. If the module A does not depend on other modules, the execution of the tasks of the module is started after the task execution of the other modules is completed.

Preferably, the step S11 includes: the dependency between the application modules is determined based on the source code and the configuration file using static analysis.

Preferably, wherein the static analysis comprises at least any one of:

Specifically, for the analysis of the import sentence, the method comprises the following steps: 1) Traversing an import statement in a source code file; 2) Extracting the names of modules or classes in the import statement; 3) And determining the direct dependency relationship between the source code files according to the extracted names of the modules or classes.

For example, in a Java source code file there are the following import statements:

import com.example.moduleA；

import com.example.moduleB；

by traversing the import statements in the source code files, the names of the modules or classes that are imported, such as com.sample.module A and com.sample.module B, can be extracted, and then the direct dependencies between the source code files are determined from the extracted names of the modules or classes, i.e., module A depends on module B.

In one embodiment, if the dependency determination is likely to be inaccurate based solely on the name of the module or class, then in a preferred embodiment, the import statement analysis includes:

For class inheritance relationship analysis, comprising the steps of: 1) Traversing class definitions in the source code file; 2) Checking whether the class definition contains an extensions key; 3) If there is an extensions key, identifying the inherited class; 4) And determining the dependency relationship between the classes according to the identified inherited classes.

For example, the following class definitions exist in the source code file:

public class ClassA extends ClassB{

definition of/(ClassA)

}

By traversing the class definitions in the source code file, it is checked whether the class definitions contain extensions keywords. If there is an extensions key, the inherited class, such as ClassB, is identified. From the identified inherited classes, dependencies between the classes are determined, i.e. class a depends on class b.

Analysis for method calls, comprising the steps of: 1) Traversing a method definition in a source code file; 2) Analyzing a method call statement defined by a method; 3) Other methods invoked in the method invocation statement are identified. 4) And determining the dependency relationship between the methods according to the identified method call relationship.

For example, the following method definitions exist in the source code file:

public void methodA(){

methodB()；

}

the method call statement is analyzed by traversing the method definitions in the source code file. Other methods called in the method call statement, such as method db, are identified. And determining the dependency relationship between the methods according to the identified method call relationship, namely, the methodA depends on the methodB.

For build profile analysis, comprising the steps of: 1) Parsing the build configuration file (e.g., gradle or Maven configuration file); 2) Searching a statement of a dependent library, wherein the statement comprises the name and version information of the library; 3) Analyzing the reference relation between the modules, such as the dependency declaration between the modules or the introduction statement of the modules; 4) And determining the dependency relationship stated in the construction configuration file and the dependency relationship between the modules.

For example, in the Gradle profile there are the following dependent library declarations:

the Gradle profile is parsed to find the declarations of the dependent libraries, including the library name and version information, such as com.example.library A:1.0.0 and com.example.library B:2.0.0. The reference relationships between the modules, such as dependency declarations between the modules or introduction statements of the modules, are analyzed. The dependencies declared in the build profile and the dependencies between the modules, i.e. the application modules depend on the relevant libraries, are determined.

Analysis for resource file references, comprising the steps of: 1) Traversing a reference statement to a resource file in a source code file; 2) Extracting the name or resource ID of the resource file in the reference statement; 3) And determining the dependency relationship of the source code file on the resource file according to the extracted name or resource ID of the resource file.

For example, there are the following references to resource files in the source code file:

int layoutId＝R.layout.activity_main；

String stringResource＝getResources().getString(R.string.app_name)；

traversing the reference statement of the source code file to the resource file, and extracting the name or the resource ID of the resource file in the reference statement, such as activity_main and app_name. And determining the dependency relationship of the source code file on the resource file according to the extracted name or resource ID of the resource file, namely that the source code file depends on the related resource file.

For data flow analysis, comprising the steps of: 1) Tracking the definition and use of variables by static analysis techniques; 2) The flow path of the data in the program is analyzed, including assignment and use of variables. 3) And determining the dependency relationship of the data according to the analyzed data flow path.

For example, the following method definitions exist in the source code file:

by static analysis techniques, the definition and use of variables is tracked, and the flow path of data in the program is analyzed. The assignment and use of the analysis variables, such as the value variable, is assigned as the return value of the call method db. From the analyzed data flow paths, the dependency of the data is determined, i.e. the return value of the methodA method depends on the return value of the methodB method.

Through the logic judgment process, the dependency relationship between the application program modules can be accurately identified and determined, and a foundation is provided for subsequent concurrent compiling optimization.

In this embodiment, in the step S12, the dependency relationship between the application modules is split into independent modules, so that the dependencies between the application modules are independent.

Preferably, the step S12 includes:

Specifically, according to the dependency relationship, the modules of the application program are split, and the highly coupled modules are split into independent and low-coupled modules. The goal of the splitting here is to reduce dependencies between modules as much as possible so that each module can be compiled and deployed independently.

In this embodiment, in step S13, tasks that can be executed in parallel are determined based on the independent modules and the dependency relationships, and the tasks that can be executed in parallel are defined using parallel blocks in Jenkins Pipeline. Specifically, parallel block (parallel block) configuration is used in Jenkins Pipeline to configure the stages and steps to be performed concurrently. Tasks that can be executed in parallel are determined according to the split modules and the dependency relationships, and the tasks are defined in parallel blocks.

Concurrent execution configured in Jenkins Pipeline may use parallel blocks (parallel blocks) to define stages and steps of parallel execution. Tasks that can be executed in parallel are determined according to the split modules and the dependency relationships, and the tasks are defined in parallel blocks.

The specific configuration steps are as follows:

1) Creating a Pipeline item in Jenkins, and defining a construction flow in a Pipeline script.

2) And determining tasks which can be executed in parallel according to the split modules and the dependency relationships. For example, assume that we split the application into module A, module B, and module C, with the dependency between them being that A depends on B and C, and B depends on C.

3) The parallel blocks are used in the Pipeline script to configure the stages and steps of concurrent execution. For example, parallel blocks may be defined using parallel keys and individual tasks defined therein. According to the above example, the build tasks of modules A, B and C, respectively, can be defined as individual tasks in parallel blocks.

4) In each task, corresponding construction steps are defined according to specific construction requirements of the modules. For example, shell steps may be used to execute build commands, test scripts, and the like.

Through the configuration, the Jenkins Pipeline can execute defined tasks in parallel, and efficiency of a construction process is improved. In the parallel execution process, jenkins can automatically manage the execution sequence of tasks according to the dependency relationship of the tasks, so that the dependency relationship is ensured to be correct.

It should be noted that parallel execution may increase the load of system resources, so when parallel execution is configured, the availability of system resources and the number of tasks allocated reasonably need to be evaluated according to the actual situation, so as to avoid resource contention and performance problems.

Preferably, the dependencies between tasks are specified using the dependendson key in Jenkins Pipeline. The tasks are guaranteed to be executed in the correct order to satisfy the dependency relationship.

Dependency on keywords are used in Jenkins Pipeline to specify dependencies between tasks, ensuring that tasks are executed in the correct order to satisfy the dependencies. The rules specifying the dependency relationship are as follows:

1) And determining the dependency relationship between tasks according to the result of the static analysis and the dependency relationship of the modules. For example, if module A relies on module B and module C, then the tasks of module A should not begin until the execution of the tasks of module B and module C is completed.

2) In Jenkins Pipeline, dependencies are specified by using the dependendson key in the definition of the task. Behind the dependendson key, the names or references of other tasks that the task depends on are listed.

The following is an example of how a rule-based specification of dependencies is described in Chinese:

in the above example we assume three tasks, module a task, module B task and module C task, respectively. By using the dependendson key in the definition of task a, task a is specified to be dependent on task B and task C. Thus, jenkins will ensure that task B and task C will begin to execute task a after their execution is completed when Pipeline is executed.

By specifying the dependency relationship through the rule, the task can be ensured to be executed according to the correct sequence, and the dependency relationship among the tasks is satisfied. Therefore, in the concurrent compiling process, each task can be executed after the dependent task is completed, and the compiling accuracy and reliability are ensured.

Preferably, the method further comprises:

the dynamic compiling determines whether the application program module is completely independently split, wherein the step S13 includes:

Preferably, the method further comprises: the compiling progress and result are monitored by Jenkins monitoring. Through the real-time monitoring and reporting functions provided by Jenkins, a developer can view the compiling progress and results in real time. This can help to quickly discover problems and make timely adjustments and repairs.

In a preferred embodiment, the technical solution for the concurrent compilation optimization framework based on jenkins+pipeline is as follows:

1) Initial setting: the necessary compilation environment is set up, including the installation and configuration of build tools, and the preparation of the associated build scripts and configuration files.

2) Static analysis: by statically analyzing the source code and constructing the configuration file, the dependency relationship between the application program modules is identified. This includes import statement analysis, class inheritance relationship analysis, method call analysis, build profile analysis, resource file reference analysis, and data flow analysis.

3) Dependent on splitting and adjustment: and splitting the dependency relationship of the application program into independent modules according to the result of the static analysis. And adjusting the code position and the structure of the modules according to the split dependency relationship, so that the dependencies among the modules are independent as much as possible.

4) Concurrent execution configuration: the parallel block (parallel block) configuration is used in Jenkins Pipeline to configure the stages and steps to be performed concurrently. Tasks that can be executed in parallel are determined according to the split modules and the dependency relationships, and the tasks are defined in parallel blocks.

5) Dependency management: dependency relationships between tasks are specified using the dependendson key in Jenkins Pipeline. The tasks are guaranteed to be executed in the correct order to satisfy the dependency relationship.

6) Concurrent compilation execution: triggering and compiling tasks through Jenkins, and monitoring execution progress and results of the tasks. And parallel execution and dependency management are used to realize efficient concurrent compiling.

7) Real-time monitoring and reporting: through the real-time monitoring and reporting functions provided by Jenkins, a developer can view the compiling progress and results in real time. This can help to quickly discover problems and make timely adjustments and repairs.

Through the technical means, the concurrent compiling optimization framework based on Jenkins+pipeline realizes efficient concurrent compiling. Each step and technical means in the technical solution play a key role, and from static analysis to dependency management and concurrent execution, the compiling speed and efficiency are improved, and the correct construction sequence and dependency relationship are ensured. These descriptions may assist those skilled in the art in implementing the framework without impediment.

Furthermore, embodiments of the present application provide a computer readable medium having stored thereon computer readable instructions executable by a processor to implement the foregoing method.

The embodiment of the application also provides a management device for concurrent compiling, wherein the device comprises:

one or more processors; and

a memory storing computer readable instructions that, when executed, cause the processor to perform the operations of the aforementioned method.

For example, computer-readable instructions, when executed, cause the one or more processors to: determining a dependency relationship between application program modules; splitting the dependency relationship between the application program modules into independent modules so as to enable the dependency relationship between the application program modules to be independent; based on the independent modules and the dependency relationships, determining tasks which can be executed in parallel, and defining the tasks which can be executed in parallel in a Jenkins Pipeline by using parallel blocks; and triggering concurrent compiling tasks through Jenkins to realize concurrent compiling of the tasks which are executed in parallel.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.

Claims

1. A method for concurrent compilation, wherein the method comprises:

determining a dependency relationship between application program modules;

2. The method of claim 1, wherein the splitting the dependencies between the application modules into independent modules such that the dependencies between application modules are independent comprises:

3. The method of claim 1, wherein the method further comprises:

4. The method of claim 1, wherein the determining the dependencies between application modules comprises:

5. The method of claim 4, wherein the static analysis comprises at least any one of:

6. The method of claim 5, wherein the import statement analysis comprises:

7. The method of claim 1, wherein the method further comprises:

8. The method of claim 1, wherein the method further comprises:

the compiling progress and results were monitored by Jenkins.

9. A computer readable medium having stored thereon computer readable instructions executable by a processor to implement the method of any of claims 1 to 8.

10. An apparatus for concurrent compilation, wherein the apparatus comprises:

one or more processors; and

a memory storing computer readable instructions that, when executed, cause the processor to perform the operations of the method of any one of claims 1 to 8.