US20170097821A1 - Distributed parallel build system - Google Patents

Distributed parallel build system Download PDF

Info

Publication number
US20170097821A1
US20170097821A1 US15/378,613 US201615378613A US2017097821A1 US 20170097821 A1 US20170097821 A1 US 20170097821A1 US 201615378613 A US201615378613 A US 201615378613A US 2017097821 A1 US2017097821 A1 US 2017097821A1
Authority
US
United States
Prior art keywords
build
software component
builds
servers
server
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US15/378,613
Other versions
US10048961B2 (en
Inventor
Kevin Gu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
eBay Inc
Original Assignee
eBay Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US74403906P priority Critical
Priority to US11/526,310 priority patent/US8458661B2/en
Priority to US13/903,614 priority patent/US8972937B2/en
Priority to US14/635,818 priority patent/US9529589B2/en
Application filed by eBay Inc filed Critical eBay Inc
Priority to US15/378,613 priority patent/US10048961B2/en
Assigned to EBAY INC. reassignment EBAY INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GU, KEVIN
Publication of US20170097821A1 publication Critical patent/US20170097821A1/en
Application granted granted Critical
Publication of US10048961B2 publication Critical patent/US10048961B2/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/70Software maintenance or management
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/40Transformation of program code
    • G06F8/41Compilation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/70Software maintenance or management
    • G06F8/71Version control; Configuration management

Abstract

This document describes, among other things, systems and methods for managing distributed parallel builds. A computer-implemented method to manage parallel builds, comprises identifying one or more software components in a software project, wherein each software component includes an executable binary file; determining a build configuration for each software component, wherein the build configuration includes a mapping from each software component to one or more build servers; and building each software component using the mapped one or more build servers in the corresponding build configuration, wherein the building includes compiling one or more source files associated with each software component to one or more object files, by distributing the one or more source files to one or more compilation machines.

Description

    RELATED PATENT DOCUMENTS
  • This application is a continuation of U.S. patent application Ser. No. 14/635,818 filed Mar. 2, 2015, which is a continuation of U.S. patent application Ser. No. 13/903,614 filed May 28, 2013, now U.S. Pat. No. 8,972,937 which is a continuation of U.S. patent application Ser. No. 11/526,310 filed Sep. 25, 2006, now U.S. Pat. No. 8,458,661, which claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 60/744,039, filed on Mar. 31, 2006, the contents of which are hereby incorporated by reference in their entirety.
  • TECHNICAL FIELD
  • Embodiments relate generally to the field of software development, and more specifically to methods and systems that build software projects in parallel.
  • BACKGROUND
  • The software development process usually involves several steps including analyzing requirements, drafting specifications, designing the software architecture, coding, testing and debugging, and maintenance. During the coding, testing, and debugging stages some or all of a software project is built using tools such as a compiler and a linker. In a complex software project, builds may take long periods of time, causing an inefficient use of software development resources.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 illustrates a schematic block diagram of a network-based system, in accordance with an example embodiment;
  • FIG. 2 illustrates a schematic block diagram of a build manager, in accordance with an example embodiment;
  • FIG. 3 illustrates a schematic block diagram of a component-based arrangement, in accordance with an example embodiment;
  • FIG. 4 is a chart illustrating build configurations for software components, in accordance with an example embodiment;
  • FIG. 5 illustrates a method for component-based distributed parallel builds, in accordance with an example embodiment;
  • FIG. 6 illustrates a schematic block diagram of build servers and compilation machines, in accordance with an example embodiment; and
  • FIG. 7 illustrates a diagrammatic representation of a machine in the exemplary form of a computer system, within which a set or sequence of instructions for causing the machine to perform any one of the methodologies discussed herein may be executed.
  • DETAILED DESCRIPTION
  • Methods and systems to manage software builds in a network-based system are described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the inventive subject matter. It will be evident, however, to one skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details.
  • For the purposes of this document, “software component” includes any independent binary or independent executable software module, such as library files (e.g., dynamically linked libraries or DLL), executable files (e.g., applications or .exe files), or services (e.g., daemons). Other types of independent binaries are included as understood by one of ordinary skill in the art.
  • FIG. 1 illustrates a schematic block diagram of a network-based system 100, in accordance with an example embodiment. The network-based system 100 includes a web server 102, which can communicate over a network 104 with one or more terminals 106A, 106B, 106C, . . . , 106N. In various embodiments, portions of the network 104 may include wired or wireless networking. The terminals 106 can connect to the network 104 using wired or wireless communication. The web server 102 is communicatively coupled to a database 108 and other backend servers, such as an email server 110, a database engine server 112, and a file server 114. Additionally, the system 100 includes one or more build servers 116A, 116B, 116C, . . . , 116N. In embodiments, the build servers 116 may include any type of computer including a laptop, desktop, blade server, network server, or the like. In addition, build servers 116 may include one or more software compilers 118 or linkers 120.
  • In an embodiment, a user (e.g., a software developer) can use a terminal machine 106 to control software builds using a web-based user-interface provided by the web server 102. In an embodiment, a user (e.g., a software developer) can use a terminal machine 106 to control software builds using a web-based user-interface provided by the web server 102. During a typical software development phase, the user may write and edit files, which are part of a software project. At some time, the user may desire to build the project. Building the project may involve compiling one or more files and then linking them into one or more files (e.g., executable or library files). In an embodiment, the user can initiate such a build process using the user-interface provided by the web server 102. Upon receiving such a request, the web server 102 communicates with the database engine server 112. Communication may include information such as user identification, project identification, build destination, and other pertinent information. The database engine server 112 can then communicate with the database 108 to determine the files needed and the correct commands to issue to the build servers 116. Files may be stored in the database 108 or on a file server 114. Files for each particular software component are associated with one or more build servers 116 in a distributed manner. In an embodiment, each build server 116 contains a compiler and a linker. The files are transmitted to the associated build servers 116, where they are compiled and linked. After each software component is compiled and linked on the distributed build servers 116, the software project (e.g., the collection of the particular software components) is transmitted to a development or a release area.
  • FIG. 2 illustrates a schematic block diagram of a build manager 200, in accordance with an example embodiment. The build manager 200 may operate, in some embodiments, as a build server 220. In such an embodiment, the build manager 200 can compile one or more files while also coordinating the distributed parallel build process. Alternatively, the build manager 200 may be solely operated as a controller machine: taking commands from a client machine 202, managing the build process, and placing final builds on either a development system 204 or a release system 206.
  • In an embodiment, the build manager 200 includes a user interface module an error handling module 210, a scheduling module 212, a queuing module 214, a file transfer module 216, a compiler 215, and a linker 217. Users (e.g., software developers) may connect with the build machine 200 from their client machines 202 to issue one or more commands that control software builds. For example, a user may issue a command via the user interface module 208 to schedule a build or a series of builds, cancel a build or check the status of a build.
  • The scheduling module 212 may be used to schedule one or more builds in the future. In an embodiment, users can schedule one or more builds to commence at a specific time or date. In another embodiment, users can schedule periodic or recurring builds. Schedules of periodic or recurring builds may have a terminating date or time, such that software projects will regularly re-build until the terminating date or time.
  • The error handling module 210 detects errors that may occur before or during a build. In an embodiment, detected errors are logged to a database 218 or a file. Users may, in some embodiments, view the errors stored in the database 218 using the user interface module 208. In some embodiments, the error handling module 210 may communicate with one or more modules to control current or later builds. For example, if a build fails and the error handling module 210 detects a certain type of error or a certain degree of error, the error handling module 210 may communicate with the scheduling module 212 to discontinue future builds or defer the next scheduled build until, for example, the user is notified and issues a command to continue the scheduled builds. In another embodiment, after detecting an error, the error handling module 210 may communicate with the queuing module 214 to remove any queued portions of the build not yet in progress, thereby terminating the build.
  • The queuing module 214 manages user build requests using one or more queues, in embodiments. In an embodiment, queuing is sorted by a first-come, first-served priority basis. In other embodiments, queuing is prioritized using factors such as a project type, a requestor, a project size, a project priority, or the like. In an embodiment, the queuing module 214 may communicate with the scheduling module 212 to receive scheduled builds. The queuing module 214 may then insert the scheduled builds into a queue based on one or more of the factors noted above. In an embodiment, queuing is performed at the project level, such that a project may not begin building until a previous project is completed. In another embodiment, queuing is performed at a more granular level, such as at a component or file level. It may be advantageous to manage queues at a finer granularity to reduce build server 220 idle time. In an embodiment, the queuing module 214 communicates with the database 218 to update the build status, For example, the queuing module 214 may communicate with the user interface module 208 to provide an indication of which software project or software component is currently being built, which project or component is next in the queue, or indications of success or error conditions existing in the current software build.
  • In an embodiment, the file transfer module 216 can transfer source files to one or more build servers 220. Source files may be stored on a file server, removable storage media, or a structured storage system, such as a database, version control system, or the like. In one embodiment, source files are stored in Rational ClearCase provided by IBM, Inc. In an embodiment, the build servers 220 compile the source files to object files. In another embodiment, the build servers 220 manage the distributed compilation of the source files across one or more compilation machines. The file transfer module 216 may then transfer the resultant object file from each build server 220 to another server, which may in some embodiments be the build manager 200, where linking is performed. In an alternative embodiment, linking is performed by one or more build servers 220 and the file transfer module 216 accesses the linked executable file. In either embodiment, the linked executable file is eventually transferred to the development system 204 or the release system 206. In embodiments, file transfers are performed using secured (e.g., Secure Shell (SSH)) or unsecured (e.g., TCP/IP) protocols. In an embodiment, the file transfer module 216 communicates with the database 218 to update the status of file transfers.
  • FIG. 3 illustrates a schematic block diagram of a component-based arrangement 300, in accordance with an example embodiment. In an embodiment, a project 302 may be divided into several software components. In one example embodiment, the project may include an online commerce system, where the software components include a user account manager 304A, a payment system 304B, a shopping cart 304C, a catalog interface 304D, and a feedback module 304E. Each component 304 may be logically divided further into separate files or sets of files 306A, 306B, . . . , 306N representing a subdivision of the component 304 related in some manner, such as by related functionality, complexity, or the like. In an embodiment, the subdivided files or sets of files 306 are organized in a manner that increases the speed or efficiency of the build process. For example, one or more complex files may be associated with a more powerful build server 116 (FIG. 1), whereas less complex files may be associated with a less powerful build server 116, in order to seek overall efficiency gains.
  • FIG. 4 is a chart 400 illustrating build configurations for software components, in accordance with an example embodiment. As depicted in FIG. 4, a build configuration for a software component includes a mapping from a particular software component to one or more servers. For example, each software component in the component column 402 is associated with one or more computing machines in the server column 404. A command can be issued to compile a component in a distributed manner using the machines identified in the server column 404. In an embodiment, the machines, as described in column 404, include a build server. In another embodiment, the machines include a computing device dedicated to compiling or assembling source code to object code, such as a compilation machine. In an embodiment, a software tool is used to manage and facilitate distributed compilation, such as for example, distcc by Martin Pool. The number of servers and which servers are used for each component may be determined by one or more factors, including the size of the component, the complexity of the component, the computational capability of each build server, the typical frequency of changes to a particular component, the network capabilities to one or more servers, and other factors that may be considered when attempting to optimize build times. In an embodiment, where two or more components are of sufficiently simple complexity to maintain overall build efficiency, portions or all of the components may be compiled or built on a single build server.
  • In an embodiment, mapping is performed by the build manager 200 on a first-come first-served basis. For example, a list of one or more software components may be submitted to a user interface provided by the user interface module 208. The build manager 200 may process the list of software components from first to last and map each software component to one or more build servers depending on one or more factors, as previously described. The build manager 200 may be aware of how many build servers are available and what each build server's configuration is, for example memory, processing capability, or the like. Using such information, the build manager 200 may adaptively map (assign) software components to build servers in an effort to balance processing duties or an attempt to achieve an overall maximized operating efficiency. In another embodiment, mapping is performed based on one or more characteristics of the software components, such as the size, complexity, age, name, priority, language, or the like. Using characteristics of the software components may be advantageous by providing the best resources to the most complex components before mapping other less complex components, which may not fully maximize a build server's capabilities. In another embodiment, mapping is performed by dividing the number of available build servers by the number of software components and then assigning the allotted number of build servers to each software component.
  • FIG. 5 illustrates a method 500 for component-based distributed parallel builds, in accordance with an example embodiment. At 502, each component is identified. In one embodiment, a command is issued by a user of the system 100 using a user-interface provided on a terminal machine 106. The command can include an indication of one or more components to build. The method 500 can parse the command to determine the components.
  • For each component 504, the method 500 determines the build configuration for the component 506. For example, using a table similar to the one illustrated in FIG. 4, the method 500 can determine which build servers 116 (FIG. 1) will be targeted for the distributed compilation of the particular component.
  • At 508, one or more commands are issued to distributively compile the component. In an embodiment, the command identifies one or more build servers 116 to be used to compile the particular component. In an embodiment, one or more components are built using one or more assigned build servers 116, where building a component includes compiling and linking the component's source files. In another embodiment, component source files are only compiled on build servers 116 in a distributed manner, and linking the resulting object code is performed on a different computer. In another embodiment, build servers 116 control the distributed compilation of one or more components using one or more compilation machines.
  • At 510, the current build status is updated for later use, for example, by a report or a user-interface screen to be provided to a user showing the current status of each component build.
  • In certain embodiments, some or all of the builds can be scheduled. In other embodiments, some or all of the builds can be performed in serial, parallel, or a combination. This may be necessary, for example, because of inherent dependencies between different units of software.
  • In an embodiment, a build is distributed over two or more CPUs in a single machine, such that, for example, each component is assigned and associated with one or more CPUs in the machine. In another embodiment, a machine has a multiple-core CPU (e.g., AMD Athlon X2 series and Intel Pentium D processors) and a build can be distributed over two or more CPU cores in a single machine. In a further embodiment, a machine may have multiple CPUs, each with multiple CPU cores, and systems and methods described herein can adaptively associate and assign components in a project to particular CPUs or CPU cores or any combination of the two. In a further embodiment, builds can be distributed over several multi-processor machines, where each machine's processors may or may not have multiple cores. Components can then be assigned to a particular processor on a particular machine or even with more granularities, such as by assigning a certain component build to one or more processor cores on a particular machine or across several machines.
  • FIG. 6 illustrates a schematic block diagram of build servers 600 and compilation machines 602, in accordance with an example embodiment. In one configuration, multiple build servers 600A, 600B, 600C, . . . , 600N are arranged in a hierarchal tree, such that a root build server 600A has the task of distributing one or more software components of a software project to one or more component-level build servers 600B, 600C, . . . , 600N in some embodiments, the root build server 600A includes a linker, a packager (e.g., Red Hat Package Manager) and other software to distribute and manage distributed compilations such that the root build server 600A may act as a component-level build server during a separate project build. In an embodiment, the packager includes software to create a software package using a particular file format, such that a management tool can install, update, uninstall, verify and query software packaged in the format. Component-level build servers 600B, 600C, . . . , 600N include a linker, software to distribute and manage distributed compilations, and a packager. Compilation machines 602A, 602B, 602C, . . . , 602N includes one or more compilers and are configured as dedicated machines with a primary task of compiling source code to object code, in an embodiment.
  • In an embodiment, after a build command is submitted to the root build server 600A, components of the software project are distributed to the component-level build servers 600B, 600C, . . . , 600N. In another embodiment, each component-level build server 600B, 600C, . . . , 600N is provided with a component identifier, such as a component name, and may retrieve the source files associated with the component from a central repository, such as a version control system. At each component-level build server 60013, 600C, . . . , 600N, the source files that are associated with the component are distributed across the compilation machines 602 using software, such as distcc, where the source files are compiled to object files. In one embodiment, a configuration file maps software components to one or more compilation machines 602. The configuration file may be stored in a central repository. When a component-level build server 600B, 600C, . . . , 600N receives a build command identifying the component to build, the associated configuration file may be retrieved and used to determine the target compilation machines 602 for the distributed compilation. After the source files are compiled, the object files are linked at the component-level build server 600B, 600C, . . . , 600N and the resulting software component (e.g., executable file or library file) is packaged using the packager. The package can then be transferred to a development platform 604 or a release platform 606, where it can be installed and used or tested. In an embodiment, development platform 604 may include one or more development servers 604A, 604B to provide parallel development and testing. In an embodiment, packages from component-level build server 600B, 600C, . . . , 600N are mustered at a staging area 608, before being distributed to the development platform 604 or the release 606. In various embodiments, the staging area may include one or more file servers, database servers, or the like, to temporarily store the packages before distribution. The staging area 608 may validate that all packages in a software project are present before transferring the project in the form of a complete set of packages) to the appropriate target platform.
  • FIG. 7 illustrates a diagrammatic representation of a machine in the exemplary form of a computer system 700 within which a set or sequence of instructions, for causing the machine to perform any one of the methodologies discussed herein, may be executed. In alternative embodiments, the machine may comprise a computer, a network router, a network switch, a network bridge, Personal Digital Assistant (PDA), a cellular telephone, a web appliance, set-top box (STB) or any machine capable of executing a sequence of instructions that specify actions to be taken by that machine.
  • The computer system 700 includes a processor 702, a main memory 706 and a static memory 708, which communicate with each other via a bus 724. The computer system 700 may further include a video display unit 712 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 700 also includes an alphanumeric input device 714 (e.g., a keyboard), a cursor control device 716 (e. g., a mouse), a disk drive unit 717, a signal generation device 722. (e.g., a speaker) and a network interface device 710 to interface the computer system to a network 726.
  • The disk drive unit 718 includes a machine-readable medium 720 on which is stored a set of instructions or software 704 embodying any one, or all, of the methodologies described herein. The software 704 is also shown to reside, completely or at least partially, within the main memory 706 and/or within the processor 702. The software 704 may further be transmitted or received via the network interface device 710. For the purposes of this specification, the term “machine-readable medium” shall be taken to include any medium which is capable of storing or encoding a sequence of instructions for execution by the machine and that cause the machine to perform any one of the methodologies of the inventive subject matter. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic disks, and carrier wave signals. Further, while the software is shown in FIG. 7 to reside within a single device, it will be appreciated that the software could be distributed across multiple machines or storage media, which may include the machine-readable medium.
  • The foregoing description of specific embodiments reveals the general nature of the inventive subject matter sufficiently that others can, by applying current knowledge, readily modify and/or adapt it for various applications without departing from the generic concept. Therefore, such adaptations and modifications are within the meaning and range of equivalents of the disclosed embodiments. The phraseology or terminology employed herein is for the purpose of description and not of limitation. Accordingly, the inventive subject matter embraces all such alternatives, modifications, equivalents and variations as fall within the spirit and broad scope of the appended claims.
  • Method embodiments described herein may be computer-implemented. Some embodiments may include computer-readable media encoded with a computer program (e.g., software), which includes instructions operable to cause an electronic device to perform methods of various embodiments. A software implementation (or computer-implemented method) may include microcode, assembly language code, or a higher-level language code, which further may include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, the code may be tangibly stored on one or more volatile or non-volatile computer-readable media during execution or at other times. These computer-readable media may include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMS), read only memories (ROMs), and the like.
  • In the foregoing description of various embodiments, reference is made to the accompanying drawings, which form a part hereof and show, by way of illustration, specific embodiments in which the inventive subject matter may be practiced. Various embodiments are described in sufficient detail to enable those skilled in the art to practice the inventive subject matter, and it is to be understood that other embodiments may be utilized, and that process or mechanical changes may be made, without departing from the scope of the inventive subject matter.
  • Embodiments of the inventive subject matter may be referred to, individually and/or collectively, herein by the term “inventive subject matter” merely for convenience and without intending to voluntarily limit the scope of this application to any single inventive subject matter or inventive concept if more than one is, in fact, disclosed. It will be recognized that the methods of various embodiments can be combined in practice, either concurrently or in succession. Various permutations and combinations may be readily apparent to those skilled in the art.

Claims (20)

1. A system for managing software component builds, comprising:
a hardware processor;
a memory comprising instructions which, when executed by the processor, cause the system to:
access multiple files associated with a software component;
map the software component to a subset of a set of build servers based on a typical frequency of changes to the software component; and
build the software component by using each build server, of the subset of build servers, to compile one or more of the multiple files.
2. The system of claim 1, further comprising instructions which, when executed by the processor, cause the system to:
determine a level of complexity of each of the multiple files;
determine a level of power of each of the build servers;
map each of the multiple files to a build server, of the subset of build servers, by mapping one or more of the multiple files with a higher level of complexity to a build server with a higher power level; and
compile one or more of the multiple files by using each build server, of the subset of build servers, to compile the one or more files that are mapped to the build server.
3. The system of claim 1, further comprising instructions which, when executed by the processor, cause the system to schedule one or more of the builds.
4. The system of claim 3, wherein the scheduling is based on a build requestor or a build request time of each of the scheduled builds.
5. The system of claim 3, wherein the scheduling is based on build type or a build size of each of the scheduled builds.
6. The system of claim 1, further comprising instructions which, when executed by the processor, cause the system to detect one or more error conditions and perform one or more actions in response to the detected error conditions.
7. The system of claim 6, wherein the actions include one or more of reporting the error condition, logging the error condition, or ignoring the error condition.
8. A method for managing software component builds, comprising:
accessing multiple files associated with a software component;
mapping the software component to a subset of a set of build servers based on a typical frequency of changes to the software component; and
building the software component by using each build server, of the subset of build servers, to compile one or more of the multiple files.
9. The method of claim 8, further comprising:
determining a level of complexity of each of the multiple files;
determining a level of power of each of the build servers;
mapping each of the multiple files to a build server, of the subset of build servers, by mapping one or more of the multiple files with a higher level of complexity to a build server with a higher power level; and
compiling one or more of the multiple tiles by using each build server, of the subset of build servers, to compile the one or more files that are mapped to the build server.
10. The method of claim 8, further comprising scheduling one or more of the builds.
11. The method of claim 10, wherein the scheduling is based on a build requestor or a build request time of each of the scheduled builds.
12. The method of claim 10, wherein the scheduling is based on build type or a build size of each of the scheduled builds.
13. The method of claim 8, further comprising detecting one or more error conditions and perform one or more actions in response to the detected error conditions.
14. The method of claim 13, wherein the actions include one or more of reporting the error condition, logging the error condition, or ignoring the error condition.
15. A non-transitory machine-readable medium comprising instructions which, when executed by a processor of the machine, cause the machine to manage software component builds by:
accessing multiple files associated with a software component;
mapping the software component to a subset of a set of build servers based on a typical frequency of changes to the software component; and
building the software component by using each build server, of the subset of build servers, to compile one or more of the multiple files.
16. The machine-readable medium of claim 15, further comprising instructions which, when executed by the processor, cause the machine to manage software component builds by:
determining a level of complexity of each of the multiple files;
determining a level of power of each of the build servers;
mapping each of the multiple files to a build server, of the subset of build servers, by mapping one or more of the multiple files with a higher level of complexity to a build server with a higher power level; and
compiling one or more of the multiple files by using each build server, of the subset of build servers, to compile the one or more files that are mapped to the build server.
17. The machine-readable medium of claim 15, further comprising instructions which, when executed by the processor, cause the machine to manage software component builds by scheduling one or more of the builds.
18. The machine-readable medium of claim 17, wherein the scheduling is based on a build requestor or a build request time of each of the scheduled builds.
19. The machine-readable medium of claim 17, wherein the scheduling is based on build type or a build size of each of the scheduled builds.
20. The machine-readable medium of claim 15, further comprising instructions which, when executed by the processor, cause the machine to manage software component builds by detecting one or more error conditions and performing one or more actions in response to the detected error conditions.
US15/378,613 2006-03-31 2016-12-14 Distributed parallel build system Active US10048961B2 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US74403906P true 2006-03-31 2006-03-31
US11/526,310 US8458661B2 (en) 2006-03-31 2006-09-25 Distributed parallel build system
US13/903,614 US8972937B2 (en) 2006-03-31 2013-05-28 Distributed parallel build system
US14/635,818 US9529589B2 (en) 2006-03-31 2015-03-02 Distributed parallel build system
US15/378,613 US10048961B2 (en) 2006-03-31 2016-12-14 Distributed parallel build system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/378,613 US10048961B2 (en) 2006-03-31 2016-12-14 Distributed parallel build system
US16/043,380 US20180365005A1 (en) 2006-03-31 2018-07-24 Distributed parallel build system

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US14/635,818 Continuation US9529589B2 (en) 2006-03-31 2015-03-02 Distributed parallel build system

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/043,380 Continuation US20180365005A1 (en) 2006-03-31 2018-07-24 Distributed parallel build system

Publications (2)

Publication Number Publication Date
US20170097821A1 true US20170097821A1 (en) 2017-04-06
US10048961B2 US10048961B2 (en) 2018-08-14

Family

ID=38561051

Family Applications (5)

Application Number Title Priority Date Filing Date
US11/526,310 Active 2031-06-09 US8458661B2 (en) 2006-03-31 2006-09-25 Distributed parallel build system
US13/903,614 Active US8972937B2 (en) 2006-03-31 2013-05-28 Distributed parallel build system
US14/635,818 Active US9529589B2 (en) 2006-03-31 2015-03-02 Distributed parallel build system
US15/378,613 Active US10048961B2 (en) 2006-03-31 2016-12-14 Distributed parallel build system
US16/043,380 Pending US20180365005A1 (en) 2006-03-31 2018-07-24 Distributed parallel build system

Family Applications Before (3)

Application Number Title Priority Date Filing Date
US11/526,310 Active 2031-06-09 US8458661B2 (en) 2006-03-31 2006-09-25 Distributed parallel build system
US13/903,614 Active US8972937B2 (en) 2006-03-31 2013-05-28 Distributed parallel build system
US14/635,818 Active US9529589B2 (en) 2006-03-31 2015-03-02 Distributed parallel build system

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/043,380 Pending US20180365005A1 (en) 2006-03-31 2018-07-24 Distributed parallel build system

Country Status (1)

Country Link
US (5) US8458661B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180081661A1 (en) * 2016-09-16 2018-03-22 Microsoft Technology Licensing, Llc Optimization for Multi-Project Package Manager

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8458661B2 (en) 2006-03-31 2013-06-04 Ebay Inc. Distributed parallel build system
KR100871563B1 (en) * 2007-02-14 2008-12-02 삼성전자주식회사 Apparatus and method for developing software based on component
GB2459690A (en) * 2008-04-30 2009-11-04 Sony Comp Entertainment Europe Data conversion apparatus and method
US8819266B2 (en) * 2008-05-22 2014-08-26 Hartford Fire Insurance Company Dynamic file transfer scheduling and server messaging
CA2698066A1 (en) * 2009-07-31 2011-01-31 Nitobi Software Inc. System and method for remotely compiling multi-platform native applications for mobile devices
US9235458B2 (en) 2011-01-06 2016-01-12 International Business Machines Corporation Methods and systems for delegating work objects across a mixed computer environment
US9052968B2 (en) * 2011-01-17 2015-06-09 International Business Machines Corporation Methods and systems for linking objects across a mixed computer environment
US8856724B2 (en) 2011-06-20 2014-10-07 Ebay Inc. Systems and methods for incremental software development
US9250866B2 (en) 2011-06-20 2016-02-02 Ebay Inc. Systems and methods for incremental software deployment
US8635607B2 (en) 2011-08-30 2014-01-21 Microsoft Corporation Cloud-based build service
US8949802B1 (en) * 2011-10-18 2015-02-03 Google Inc. Sharding program compilation for large-scale static analysis
US9448780B1 (en) * 2011-12-13 2016-09-20 Zynga Inc. Package manager verifier
CN102662655B (en) * 2012-03-12 2018-09-25 南京中兴软件有限责任公司 The structure system and method for version
US8959484B2 (en) * 2012-06-21 2015-02-17 Microsoft Corporation System for hosted, shared, source control build
US9645807B2 (en) * 2012-08-28 2017-05-09 International Business Machines Corporation Automated deployment of a configured system into a computing environment
CN104123184B (en) 2013-04-28 2017-12-22 国际商业机器公司 A kind of method and system for being used to distribute resource for the task in building process
US9329858B2 (en) * 2014-09-30 2016-05-03 Linkedin Corporation Managing access to resource versions in shared computing environments
CN106325966B (en) * 2015-06-29 2019-08-13 南京中兴软件有限责任公司 Software compilation method and device
CN105373430A (en) * 2015-10-13 2016-03-02 惠州Tcl移动通信有限公司 System for dynamic allocation of compiler and method thereof
US9910649B2 (en) * 2016-01-19 2018-03-06 International Business Machines Corporation Integrating and sharing software build component targets
CN105893105A (en) * 2016-03-29 2016-08-24 乐视控股(北京)有限公司 Compiling system and method for multiple compiling platforms
US10048954B2 (en) 2016-09-21 2018-08-14 International Business Machines Corporation Accelerating software builds
US10282275B2 (en) 2016-09-22 2019-05-07 Microsoft Technology Licensing, Llc Method and system for managing code
US10372441B2 (en) 2016-11-28 2019-08-06 Microsoft Technology Licensing, Llc Build isolation system in a multi-system environment
US10223099B2 (en) * 2016-12-21 2019-03-05 Palantir Technologies Inc. Systems and methods for peer-to-peer build sharing

Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6223342B1 (en) * 1996-11-06 2001-04-24 A. Chacko George Object-oriented sequencing using hierarachical configuration streams
US6405364B1 (en) * 1999-08-31 2002-06-11 Accenture Llp Building techniques in a development architecture framework
US7076784B1 (en) * 1997-10-28 2006-07-11 Microsoft Corporation Software component execution management using context objects for tracking externally-defined intrinsic properties of executing software components within an execution environment
US7093232B1 (en) * 2001-10-17 2006-08-15 Oracle International Corporation (Oic) Component stager
US7266805B2 (en) * 2004-12-22 2007-09-04 Timesys Corporation Systems and methods for generating software and hardware builds
US7412685B2 (en) * 2000-12-06 2008-08-12 Microsoft Corportation Data structures and related methods for facilitating media content processing in user-defined development projects
US7448022B1 (en) * 2004-02-10 2008-11-04 Prasad Ram Dynamic software composition in a component-based software system
US7516439B2 (en) * 2001-01-12 2009-04-07 Robinson Marck R Method and system for creating reusable software components through a uniform interface
US7614040B2 (en) * 2004-05-22 2009-11-03 Bea Systems, Inc. System and method for efficiently analyzing and building interdependent resources in a software project
US7930676B1 (en) * 2007-04-27 2011-04-19 Intuit Inc. System and method for adapting software elements based on mood state profiling
US8042089B2 (en) * 2006-10-03 2011-10-18 Electric Cloud, Inc. Process automation system and method employing multi-stage report generation
US8065658B1 (en) * 2007-08-07 2011-11-22 Network Appliance, Inc. Tool for visualizing software architecture
US8219974B2 (en) * 2007-12-07 2012-07-10 Sap Ag Enforcing legal holds of heterogeneous objects for litigation
US8589868B2 (en) * 2005-12-22 2013-11-19 Ncr Corporation Creating a terminal application
US8694968B2 (en) * 2009-12-30 2014-04-08 Foneclay, Inc. System for creating personalized and customized mobile devices
US8726231B2 (en) * 2011-02-02 2014-05-13 Microsoft Corporation Support for heterogeneous database artifacts in a single project
US9134962B1 (en) * 2013-03-15 2015-09-15 Sanctum Solutions, Inc. Interactive content development
US9189227B2 (en) * 2012-12-14 2015-11-17 Telefonaktiebolaget L M Ericsson (Publ) Systems, methods, and computer program products for a software build and load process using a compilation and deployment service
US9442717B2 (en) * 2014-07-15 2016-09-13 Vmware, Inc. Techniques for automatically identifying input files used to generate output files in a software build process
US9524192B2 (en) * 2010-05-07 2016-12-20 Microsoft Technology Licensing, Llc Distributed workflow execution

Family Cites Families (50)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5649200A (en) * 1993-01-08 1997-07-15 Atria Software, Inc. Dynamic rule-based version control system
US6321158B1 (en) * 1994-06-24 2001-11-20 Delorme Publishing Company Integrated routing/mapping information
US6298476B1 (en) * 1995-12-04 2001-10-02 International Business Machines Corporation Object oriented software build framework mechanism
US7506020B2 (en) * 1996-11-29 2009-03-17 Frampton E Ellis Global network computers
EP0998708B1 (en) * 1997-07-25 2002-01-23 BRITISH TELECOMMUNICATIONS public limited company Visualisation in a modular software system
US6093215A (en) * 1997-08-12 2000-07-25 International Business Machines Corporation Method and apparatus for building templates in a component system
US6775829B1 (en) * 1998-06-04 2004-08-10 Gateway, Inc. Method for configuring software for a build to order system
US6438744B2 (en) * 1998-07-15 2002-08-20 Microsoft Corporation Dynamic mapping of component interfaces
US8042098B2 (en) * 2000-12-06 2011-10-18 Axiomatic Design Software, Inc. Method and apparatus for producing software
US7428726B1 (en) * 2000-04-12 2008-09-23 Compuware Corporation Techniques for software configuration tracking
AU8859701A (en) * 2000-09-01 2002-03-13 Togethersoft Corp Methods and systems for animating a workflow and a project plan
US7131112B1 (en) * 2000-11-21 2006-10-31 Microsoft Corporation Managing code changes for software development
EP1331554B1 (en) * 2001-06-13 2017-03-08 Accenture Global Services Limited Method of producing a software product
US7003759B2 (en) * 2001-06-21 2006-02-21 Codefast, Inc. Collection makefile generator
US7055144B2 (en) * 2001-07-12 2006-05-30 International Business Machines Corporation Method and system for optimizing the use of processors when compiling a program
US6928637B2 (en) * 2001-08-10 2005-08-09 Wind River Systems, Inc. Version control adapter interface to support integration of multiple vendors integrated development environments (IDEs)
US7516438B1 (en) * 2001-09-12 2009-04-07 Sun Microsystems, Inc. Methods and apparatus for tracking problems using a problem tracking system
EP2256622A1 (en) * 2002-02-07 2010-12-01 Sap Ag Method and apparatus for parallel distributed compilation
US7069541B2 (en) * 2002-03-01 2006-06-27 Bellsouth Intellectual Property Corporation System and method for a web-based application development and deployment tracking tool
US6721652B1 (en) * 2002-11-22 2004-04-13 Electronic Data Systems Corporation (EDS) Implementing geo-fencing on mobile devices
US7168064B2 (en) * 2003-03-25 2007-01-23 Electric Cloud, Inc. System and method for supplementing program builds with file usage information
US7539976B1 (en) * 2003-03-25 2009-05-26 Electric Cloud, Inc. System and method for intelligently distributing source files within a distributed program build architecture
US7225437B2 (en) * 2003-03-26 2007-05-29 Sun Microsystems, Inc. Dynamic distributed make
US7546575B1 (en) * 2003-06-09 2009-06-09 Dillman Frederick J System and method for using blueprints to provide a software solution for an enterprise
US7596782B2 (en) * 2003-10-24 2009-09-29 Microsoft Corporation Software build extensibility
US9009658B2 (en) * 2004-03-15 2015-04-14 Ramco Systems Limited Component based software system
US7428588B2 (en) * 2004-04-08 2008-09-23 International Business Machines Corporation Method for distributing and geographically load balancing location aware communication device client-proxy applications
US7673295B1 (en) * 2004-04-27 2010-03-02 Sun Microsystems, Inc. System and method for compile-time non-concurrency analysis
US7802228B2 (en) * 2004-08-19 2010-09-21 Microsoft Corporation Systems and methods for varying software build properties using primary and supplemental build files
US8151245B2 (en) * 2004-12-17 2012-04-03 Computer Associates Think, Inc. Application-based specialization for computing nodes within a distributed processing system
US7904877B2 (en) * 2005-03-09 2011-03-08 Microsoft Corporation Systems and methods for an extensive content build pipeline
US7640533B1 (en) * 2005-03-30 2009-12-29 Emc Corporation System and methods for defining a software build
US7761851B2 (en) * 2005-05-31 2010-07-20 International Business Machines Corporation Computer method and system for integrating software development and deployment
US7805706B1 (en) * 2005-06-21 2010-09-28 Unisys Corporation Process for optimizing software components for an enterprise resource planning (ERP) application SAP on multiprocessor servers
US7840944B2 (en) * 2005-06-30 2010-11-23 Sap Ag Analytical regression testing on a software build
US7730448B2 (en) * 2005-08-11 2010-06-01 Microsoft Corporation Layered type systems
US7937692B2 (en) * 2005-11-30 2011-05-03 Red Hat, Inc. Methods and systems for complete static analysis of software for building a system
US8458661B2 (en) 2006-03-31 2013-06-04 Ebay Inc. Distributed parallel build system
US7539972B2 (en) * 2006-08-18 2009-05-26 Cisco, Technology Inc. Method of improving user interaction with an object management tool
US8037453B1 (en) * 2006-09-13 2011-10-11 Urbancode, Inc. System and method for continuous software configuration, test and build management
US8201148B2 (en) * 2007-03-05 2012-06-12 Sap Ag Guided development and testing for computer code
KR101470319B1 (en) * 2008-02-15 2014-12-08 삼성전자주식회사 Method and apparatus for generating virtual software platform based on component model and validating software platform architecture using thereof
US8291378B2 (en) * 2008-07-29 2012-10-16 International Business Machines Corporation Simplified deployment modeling
US8613936B2 (en) * 2009-03-13 2013-12-24 Bavarian Nordic A/S Replication deficient recombinant viruses expressing antigens regulated by transcriptional control elements comprising multiple elements
US8327330B1 (en) * 2010-01-11 2012-12-04 Google Inc. System and method of generating build instructions
US8762944B2 (en) * 2011-03-23 2014-06-24 International Business Machines Corporation Build process management system
US8719767B2 (en) * 2011-03-31 2014-05-06 Commvault Systems, Inc. Utilizing snapshots to provide builds to developer computing devices
US8438532B2 (en) * 2011-04-19 2013-05-07 Sonatype, Inc. Method and system for scoring a software artifact for a user
US8683433B2 (en) * 2011-06-22 2014-03-25 Business Objects Software Limited Adaptive change management in computer system landscapes
US8850391B1 (en) * 2012-01-27 2014-09-30 Google Inc. System and method for building components of a software product in a distributed system

Patent Citations (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6223342B1 (en) * 1996-11-06 2001-04-24 A. Chacko George Object-oriented sequencing using hierarachical configuration streams
US7076784B1 (en) * 1997-10-28 2006-07-11 Microsoft Corporation Software component execution management using context objects for tracking externally-defined intrinsic properties of executing software components within an execution environment
US6405364B1 (en) * 1999-08-31 2002-06-11 Accenture Llp Building techniques in a development architecture framework
US7412685B2 (en) * 2000-12-06 2008-08-12 Microsoft Corportation Data structures and related methods for facilitating media content processing in user-defined development projects
US7516439B2 (en) * 2001-01-12 2009-04-07 Robinson Marck R Method and system for creating reusable software components through a uniform interface
US7093232B1 (en) * 2001-10-17 2006-08-15 Oracle International Corporation (Oic) Component stager
US7448022B1 (en) * 2004-02-10 2008-11-04 Prasad Ram Dynamic software composition in a component-based software system
US7614040B2 (en) * 2004-05-22 2009-11-03 Bea Systems, Inc. System and method for efficiently analyzing and building interdependent resources in a software project
US7266805B2 (en) * 2004-12-22 2007-09-04 Timesys Corporation Systems and methods for generating software and hardware builds
US8589868B2 (en) * 2005-12-22 2013-11-19 Ncr Corporation Creating a terminal application
US8042089B2 (en) * 2006-10-03 2011-10-18 Electric Cloud, Inc. Process automation system and method employing multi-stage report generation
US7930676B1 (en) * 2007-04-27 2011-04-19 Intuit Inc. System and method for adapting software elements based on mood state profiling
US8065658B1 (en) * 2007-08-07 2011-11-22 Network Appliance, Inc. Tool for visualizing software architecture
US8219974B2 (en) * 2007-12-07 2012-07-10 Sap Ag Enforcing legal holds of heterogeneous objects for litigation
US8694968B2 (en) * 2009-12-30 2014-04-08 Foneclay, Inc. System for creating personalized and customized mobile devices
US9524192B2 (en) * 2010-05-07 2016-12-20 Microsoft Technology Licensing, Llc Distributed workflow execution
US8726231B2 (en) * 2011-02-02 2014-05-13 Microsoft Corporation Support for heterogeneous database artifacts in a single project
US9189227B2 (en) * 2012-12-14 2015-11-17 Telefonaktiebolaget L M Ericsson (Publ) Systems, methods, and computer program products for a software build and load process using a compilation and deployment service
US9134962B1 (en) * 2013-03-15 2015-09-15 Sanctum Solutions, Inc. Interactive content development
US9442717B2 (en) * 2014-07-15 2016-09-13 Vmware, Inc. Techniques for automatically identifying input files used to generate output files in a software build process

Non-Patent Citations (8)

* Cited by examiner, † Cited by third party
Title
Bachiller et al, "Building Smart Environments with LooCI", ACM, pp 1-2, 2012 *
Batory et al, "The Design and Implementation of Hierarchical Software Systems with Reusable Components ", ACM TransactIons on Software Engmeermg and Methodology, vol. 1, no 4, pp, 354-398, 1992. *
Burg et al, " Tracing Software Build Processes to Uncover License Compliance Inconsistencies", ACM, pp 731-741, 2014 *
Coulson et al, " A Generic Component Model for Building Systems Software", ACM Transactions on Computer Systems, Vol. 26, No. 1, Article 1, pp 1-42, 2008. *
Ommering, " Building Product Populations with Software Components", ACM, pp 255-265, 2002 *
Ornburn et al, " Building, Modifying and Using Component Generators", IEEE, pp 391-402, 1993 *
Woodside et al, "Performance by Unified Model Analysis (PUMA)", ACM, pp 1-12, 2005 *
Yoon et al, "Towards Incremental Component Compatibility Testing", ACM, pp 119-128, 2011 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180081661A1 (en) * 2016-09-16 2018-03-22 Microsoft Technology Licensing, Llc Optimization for Multi-Project Package Manager
US10095499B2 (en) * 2016-09-16 2018-10-09 Microsoft Technology Licensing, Llc Optimization for multi-project package manager

Also Published As

Publication number Publication date
US8972937B2 (en) 2015-03-03
US9529589B2 (en) 2016-12-27
US20130268915A1 (en) 2013-10-10
US10048961B2 (en) 2018-08-14
US8458661B2 (en) 2013-06-04
US20070234320A1 (en) 2007-10-04
US20150169322A1 (en) 2015-06-18
US20090070750A9 (en) 2009-03-12
US20180365005A1 (en) 2018-12-20

Similar Documents

Publication Publication Date Title
Warneke et al. Nephele: efficient parallel data processing in the cloud
Kerrisk The Linux programming interface: a Linux and UNIX system programming handbook
US10019256B2 (en) Systems and methods for incremental software development
CN1647043B (en) Method and equipment for continuous editting files accroding to source file priority list
US8739150B2 (en) Systems and methods for dynamically replacing code objects via conditional pattern templates
US8799477B2 (en) Hypervisor selection for hosting a virtual machine image
US8230426B2 (en) Multicore distributed processing system using selection of available workunits based on the comparison of concurrency attributes with the parallel processing characteristics
US7774747B2 (en) System and associated methods for software assembly
US20060075407A1 (en) Distributed system interface
US9491117B2 (en) Extensible framework to support different deployment architectures
CN100449524C (en) Servicing a component-based software product
US8812627B2 (en) System and method for installation and management of cloud-independent multi-tenant applications
US8903943B2 (en) Integrating cloud applications and remote jobs
Venner Pro hadoop
US20020147855A1 (en) Method and system for cross platform, parallel processing
US8037471B2 (en) Systems and methods for constructing relationship specifications from component interactions
US8219987B1 (en) Optimized virtual machine specification for provisioning application specific runtime environment
EP2650783A1 (en) On-demand database service system, method, and computer program product for verifying that a developed application will operate properly with at least one other application
US20120131567A1 (en) Systematic migration of workload based on classification
US8572566B2 (en) Systems and methods for analyzing changes in application code from a previous instance of the application code
US20020144249A1 (en) Method and apparatus for installing and upgrading an application in a computer system
US8185903B2 (en) Managing system resources
DE112010005705T5 (en) Reschedule workload in a hybrid computing environment
Warneke et al. Exploiting dynamic resource allocation for efficient parallel data processing in the cloud
CN103069385B (en) System and method for dynamic load calculating based on figure

Legal Events

Date Code Title Description
AS Assignment

Owner name: EBAY INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GU, KEVIN;REEL/FRAME:040734/0310

Effective date: 20060922

STCF Information on status: patent grant

Free format text: PATENTED CASE