CN110018830A - A kind of large software compilation device based on distributed type assemblies - Google Patents
A kind of large software compilation device based on distributed type assemblies Download PDFInfo
- Publication number
- CN110018830A CN110018830A CN201910270190.1A CN201910270190A CN110018830A CN 110018830 A CN110018830 A CN 110018830A CN 201910270190 A CN201910270190 A CN 201910270190A CN 110018830 A CN110018830 A CN 110018830A
- Authority
- CN
- China
- Prior art keywords
- project
- sub
- host
- node
- mapping table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
Abstract
The invention discloses a kind of large software compilation device based on distributed type assemblies, including NFS share storage file system, calculating main frame cluster, configuration file, sub-project-node mapping table, controller.NFS shares storage file system and stores sub-project source code；Calculating main frame cluster is the computing cluster being made of several hosts；Configuration file stores the configuration information of every sub-project；Host map relationship in sub-project-node mapping table storage sub-project and calculating main frame cluster；Controller is found by sub-project-node mapping table executes sub-project institute host to be used, it is logged in by ssh client remote, sub-project is mounted on corresponding host node by way of Network File System, enter the catalogue where sub-project on host node, carries out running environment configuration according to configuration information required for sub-project；Execute the compiling work of every sub-project.The present invention has the characteristics that fast compilation speed, rational management resource, multi-platform support, result are predictable.
Description
Technical field
The present invention relates to large software technique of compiling fields, more particularly to a kind of large software based on distributed type assemblies
Compilation device.
Background technique
With the development of computer technology, the scale of a large software usually reaches million line code ranks, sub-project
Quantity reaches up to a hundred or even thousands of.There may be various dependences between each sub-project, also be had between code each
The nesting of kind level.Usually we can distribute the preferable server of configuration dedicated for compiling, this kind of extensive compiling
When item code, when especially from initial stage, exploitation iterates to certain scale, we can encounter two stubborn problems:
1. compiling building will once wait for a long time；
2. frequently encountering the mistake (because being mounted with too many software, having done many configurations) that can not track source, it is difficult to lean on
Manpower goes to clear relationship and influence of the Chu between them.The problem of compiling efficiency and iteration efficiency, has become software developer
Very important problem.
When compiling large software, software company would generally configure a high performance server dedicated for compiling.
When project is expanded to certain scale, some problems will be exposed, the parallel processing capability of single machine is insufficient, environment and
Configuration is not easy to comb clear, and maintenance is got up meticulous, cannot be altered the test of environment easily.In addition, after the completion of compiling
Resource is idle, causes the waste of computing resource.
In this context, there are some distributed compilation tools in industry, it is representational have IncrediBuild and
distcc.Their realization principle is that source code is distributed on several client machines from server-side to be compiled, and is completed
Output result is passed back to server-side again afterwards.IncrediBuild is specially to be designed to compiling Visual C++ program, no
Support the compiling of linux environment.The load-balancing algorithm of Distcc is too simple coarse, only according to the host name in configuration information
Sequence successively distributes compiler task, it is easy to cause overall performance to decline because certain machine burden is overweight.
For the mode of traditional separate unit host compiling:
Speed is slow.There are bottlenecks for single machine parallel ability, and once there is still a need for wait long time for complete compiling
Translation and compiling environment is difficult to safeguard.Environment is once destroyed or loses, and building again will take a long time
Mistake is difficult to track, iteration low efficiency.Since project is too big, all configurations, file are all placed on a machine
On, when encountering a more intractable mistake, usually it is difficult to position because range is too big, meanwhile, the efficiency of iterative development
Also it can reduce
Compiling host is idle after being finished, and is resulted in waste of resources
Distributed compilation mode is carried out using third party's tool for existing:
IncrediBuild only supports Visual C++, does not support Linux environment to compile
The load-balancing algorithm of Distcc is too simple, can not accurately obtain the resource service condition of every machine,
It is easy to influence overall performance because certain machine burden is overweight, compiler task distribution policy is not quite reasonable.
Compiling requires to carry out transmission back file between server-side and the client of execution compiler task every time, generates one
Fixed disk and network consumption.
Summary of the invention
The technical problem to be solved in the present invention is to provide it is a kind of quickly, can rational management resource, multi-platform support based on
The large software compilation device of distributed type assemblies.
The application provides a kind of large software compilation device based on distributed type assemblies, including NFS shares storage file system
System, calculating main frame cluster, configuration file, sub-project-node mapping table, controller；
NFS shares storage file system and stores sub-project source code, and the sub-project source code is torn open for Large-Scale Projects
Several mutually independent sub-projects being divided into；
Calculating main frame cluster is the computing cluster being made of several hosts；
Configuration file stores the configuration information of every sub-project, the essential information of the information configuration packet enclosed tool project
And demand information；The essential information includes sub-project ID, sub-project title and storage catalogue position, and the demand information includes
The configuration information of system model, resource information and running environment required for compilation run that sub-project needs；
Host map relationship in sub-project-node mapping table storage sub-project and calculating main frame cluster, builds with sub-project
The host of vertical mapping relations meets:
Condition 1, system model, configuration information and the sub-project of host are consistent；
Condition 2, host resource occupancy are less than the threshold value of setting；
Condition 3, residual capacity is maximum in the All hosts of the condition that meets 1 and condition 2；
Controller traverses the demand information of every sub-project in configuration file, logs on collection by ssh client remote
Each host in group selects the host for meeting the mapping relations, and sub-project-node mapping table is written in mapping relations；
Controller is found by sub-project-node mapping table executes sub-project institute host to be used, is stepped on by ssh client remote
Sub-project is mounted on corresponding host node by land by way of Network File System, and subitem is entered on host node
Catalogue where mesh carries out running environment configuration according to configuration information required for sub-project；Execute the compiling of every sub-project
Work；
Further, resource information required for sub-project compilation run includes the memory space that sub-project occupies, compiling
When maximum consumption memory headroom, the cpu resource of maximum consumption when compiling；The configuration information of running environment includes operation sub-project
Required environment setting, third party software packet to be mounted needed for running sub-project.
Further, the mapping table of host ip and corresponding sub-project ID is stored in sub-project-node mapping table.
The invention has the following beneficial effects:
1. speed is fast: promoting compilation speed using the cluster resource of magnanimity, reduce the waiting time
2. rational management resource, elastic telescopic: accurate analysis cluster loads and rationally assigns compiler task, dynamic application money
Source discharges after being finished
3. multi-platform support: while executing compiler task parallel in different platform
4. result is predictable: the corresponding independent node of each sub-project, this node only includes least necessity
File.All compiling, test are all run in node, which ensure that the predictable result unrelated with environment
Detailed description of the invention
Fig. 1 is the functional block diagram of compilation device of the present invention.
Specific embodiment
Present invention is further described in detail with reference to the accompanying drawing.
Although the present invention has been described by means of preferred embodiments, the present invention is not limited to as described herein
Embodiment further includes made various changes and variation without departing from the present invention.
Fig. 1 is the functional block diagram of compilation device of the present invention, as shown in Figure 1, compilation device includes that NFS shares storage file system
System, calculating main frame cluster, configuration file, sub-project-node mapping table, controller.
Several mutually independent sub-projects that Large-Scale Projects are split into, sub-project A, sub-project B until sub-project N,
The source code of sub-project is stored in NFS and shares in storage file system.
Calculating main frame cluster is made of several nodes, and such as figure interior joint A, node B, until node N, each node is only pacified
Least necessary software package has been filled, specific compiling work is responsible for.
Configuration file stores the configuration information of every sub-project, essential information and demand information including sub-project；Substantially
Packet enclosed tool item id, sub-project title and storage catalogue position, demand information include system model, the volume that sub-project needs
Translate the configuration information of resource information required for running and running environment；Resource information packet required for sub-project compilation run
It includes, the memory space that sub-project occupies, the memory headroom of maximum consumption when compiling, the cpu resource of maximum consumption when compiling；Operation
The configuration information of environment includes that environment required for operation sub-project is arranged, third party software to be mounted needed for running sub-project
Packet.
Mapping relations of the sub-project-between node mapping table storage sub-project and calculating main frame clustered node, for establishing
Connection.
Controller is the hinge of this compilation device, using sub-project-node mapping table as bridge, using configuration file as foundation,
Node resource in rational management distributed type assemblies, operating procedure are as follows:
1. controller traverses the demand information of every sub-project in the configuration file in configuration file, demand information includes:
Required system model, occupied memory space, the memory headroom of maximum consumption when compiling, maximum consumption when compiling
Cpu resource runs environment required for sub-project and is arranged, third party software packet to be mounted needed for running sub-project.
2. after controller gets the detailed requirements information of sub-project, being logged in cluster by ssh client remote
Each host, screened host exit at once if being unsatisfactory for, and match next, step are as follows:
(1) first ensure that system model is consistent with configuration information
(2) service condition of resource type (CPU, memory, disk) needed for obtaining, it is ensured that current hosts node resource occupies
Amount is less than threshold value (it is assumed that 80%)
(3) if occupancy < current hosts residual capacity of resource needed for sub-project (CPU, memory, disk), screening
It is used as a candidate host node out
(4) the maximum fictitious host computer of resource (CPU, memory, disk) residual capacity is picked out in all candidate hosts
(5) it will be singled out the host ip come and corresponding sub-project ID charge to sub-project-node mapping table
(6) mapping relations and note with a certain host node in cluster are established for each sub-project according to the above process
Enter sub-project-node mapping table
Sub-project institute host to be used is executed 3. finding by sub-project-node mapping table, ssh telnet is right up
It executes afterwards:
(1) sub-project is mounted on corresponding host node by way of NFS Network File System
(2) enter the catalogue where sub-project on host node, execute environment configurations required for sub-project, institute is installed
The third party software needed
(3) compiling, the construction work of sub-project are executed
(4) according to the above process, compiling, the building task of every sub-project are executed.
Claims (3)
1. a kind of large software compilation device based on distributed type assemblies, which is characterized in that share storage file system including NFS
System, calculating main frame cluster, configuration file, sub-project-node mapping table, controller；
NFS shares storage file system and stores sub-project source code, and the sub-project source code splits into for Large-Scale Projects
Several mutually independent sub-projects；
Calculating main frame cluster is the computing cluster being made of several hosts；
Configuration file stores the configuration information of every sub-project, the essential information and need of the information configuration packet enclosed tool project
Seek information；The essential information includes sub-project ID, sub-project title and storage catalogue position, and the demand information includes subitem
The configuration information of system model, resource information and running environment required for compilation run that mesh needs；
Host map relationship in sub-project-node mapping table storage sub-project and calculating main frame cluster, reflects with sub-project foundation
The host for penetrating relationship meets:
Condition 1, system model, configuration information and the sub-project of host are consistent；
Condition 2, host resource occupancy are less than the threshold value of setting；
Condition 3, residual capacity is maximum in the All hosts of the condition that meets 1 and condition 2；
Controller traverses the demand information of every sub-project in configuration file, is logged in cluster by ssh client remote
Each host selects the host for meeting the mapping relations, and sub-project-node mapping table is written in mapping relations；Control
Device is found by sub-project-node mapping table executes sub-project institute host to be used, is logged in by ssh client remote, will
Sub-project is mounted on corresponding host node by way of Network File System, is entered where sub-project on host node
Catalogue, according to configuration information required for sub-project carry out running environment configuration；Execute the compiling work of every sub-project；
2. a kind of large software compilation device based on distributed type assemblies as described in claim 1, which is characterized in that sub-project
Resource information required for compilation run includes, the memory space that sub-project occupies, and the memory headroom of maximum consumption, is compiled when compiling
The cpu resource of maximum consumption when translating；The configuration information of running environment includes that environment required for operation sub-project is arranged, operation
Third party software packet to be mounted needed for project.
3. a kind of large software compilation device based on distributed type assemblies as described in claim 1, which is characterized in that subitem
The mapping table of host ip and corresponding sub-project ID is stored in mesh-node mapping table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910270190.1A CN110018830A (en) | 2019-04-04 | 2019-04-04 | A kind of large software compilation device based on distributed type assemblies |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910270190.1A CN110018830A (en) | 2019-04-04 | 2019-04-04 | A kind of large software compilation device based on distributed type assemblies |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110018830A true CN110018830A (en) | 2019-07-16 |
Family
ID=67190645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910270190.1A Pending CN110018830A (en) | 2019-04-04 | 2019-04-04 | A kind of large software compilation device based on distributed type assemblies |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110018830A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110716898A (en) * | 2019-10-15 | 2020-01-21 | 中国地质调查局发展研究中心 | Method and system for gathering field geological survey data in real time |
| CN115640968A (en) * | 2022-10-18 | 2023-01-24 | 中电金信软件有限公司 | Job scheduling method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100037035A1 (en) * | 2008-08-11 | 2010-02-11 | International Business Machines Corporation | Generating An Executable Version Of An Application Using A Distributed Compiler Operating On A Plurality Of Compute Nodes |
| CN103440136A (en) * | 2013-09-05 | 2013-12-11 | 迈普通信技术股份有限公司 | Distributed authoring and versioning method and system |
| CN106055377A (en) * | 2016-06-03 | 2016-10-26 | 北京奇虎科技有限公司 | Method for achieving distributed compiling and distributed compiling system |
| CN107391221A (en) * | 2017-07-28 | 2017-11-24 | 迈普通信技术股份有限公司 | Dispatch server, compiler server and distributed compilation method |
| CN108279976A (en) * | 2017-12-26 | 2018-07-13 | 努比亚技术有限公司 | A kind of compiling resource regulating method, computer and computer readable storage medium |
| CN109254842A (en) * | 2017-07-12 | 2019-01-22 | 腾讯科技（深圳）有限公司 | Method for managing resource, device and the readable storage medium storing program for executing of distributive type system |
-
2019
- 2019-04-04 CN CN201910270190.1A patent/CN110018830A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100037035A1 (en) * | 2008-08-11 | 2010-02-11 | International Business Machines Corporation | Generating An Executable Version Of An Application Using A Distributed Compiler Operating On A Plurality Of Compute Nodes |
| CN103440136A (en) * | 2013-09-05 | 2013-12-11 | 迈普通信技术股份有限公司 | Distributed authoring and versioning method and system |
| CN106055377A (en) * | 2016-06-03 | 2016-10-26 | 北京奇虎科技有限公司 | Method for achieving distributed compiling and distributed compiling system |
| CN109254842A (en) * | 2017-07-12 | 2019-01-22 | 腾讯科技（深圳）有限公司 | Method for managing resource, device and the readable storage medium storing program for executing of distributive type system |
| CN107391221A (en) * | 2017-07-28 | 2017-11-24 | 迈普通信技术股份有限公司 | Dispatch server, compiler server and distributed compilation method |
| CN108279976A (en) * | 2017-12-26 | 2018-07-13 | 努比亚技术有限公司 | A kind of compiling resource regulating method, computer and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 童亚拉: "分布式编译的方法和系统研究", 《计算机技术与发展》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110716898A (en) * | 2019-10-15 | 2020-01-21 | 中国地质调查局发展研究中心 | Method and system for gathering field geological survey data in real time |
| CN115640968A (en) * | 2022-10-18 | 2023-01-24 | 中电金信软件有限公司 | Job scheduling method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Kazanavičius et al. | Migrating legacy software to microservices architecture | |
| JP7063289B2 (en) | Optimal software placement method and program for offload servers | |
| US10437885B2 (en) | Graph modeling of applications for detection of duplicate modules | |
| CN113010393A (en) | Fault drilling method and device based on chaotic engineering | |
| De Gooijer et al. | An industrial case study of performance and cost design space exploration | |
| Cheng et al. | Efficient event correlation over distributed systems | |
| US20190228009A1 (en) | Information processing system and information processing method | |
| CN110018830A (en) | A kind of large software compilation device based on distributed type assemblies | |
| Callaghan et al. | Reducing time-to-solution using distributed high-throughput mega-workflows-experiences from SCEC CyberShake | |
| Moreno et al. | Load balancing in homogeneous pipeline based applications | |
| CN115994085A (en) | Code coverage rate test processing method, device, equipment and storage medium | |
| Su et al. | Changing engines in midstream: A Java stream computational model for big data processing | |
| CN107168795B (en) | Codon deviation factor model method based on CPU-GPU isomery combined type parallel computation frame | |
| Wang et al. | Federated MapReduce to transparently run applications on multicluster environment | |
| CN108243238B (en) | Method and device for acquiring performance data | |
| CN104834734A (en) | Efficient data analysis and processing method | |
| Valvåg et al. | Oivos: Simple and efficient distributed data processing | |
| CN106843822B (en) | Execution code generation method and equipment | |
| CN110750582B (en) | Data processing method, device and system | |
| Bhatele et al. | Applying graph partitioning methods in measurement-based dynamic load balancing | |
| Benoit et al. | Scheduling skeleton-based grid applications using PEPA and NWS | |
| CN104834733A (en) | Big data mining and analyzing method | |
| Hazekamp et al. | Scaling up bioinformatics workflows with dynamic job expansion: A case study using galaxy and makeflow | |
| Zhao et al. | Code refactoring from OpenMP to MapReduce model for big data processing | |
| Raval et al. | Dynamic load balancing strategies for graph applications on gpus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |