HK1237463B - Compiling graph-based program specifications - Google Patents

Description

Compiling graph-based program specifications

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Application Serial No. 62/044,645, filed September 2, 2014, and U.S. Application Serial No. 62/164,175, filed May 20, 2015.

Technical Field

This specification relates to a method for compiling a graph-based program specification.

Background Art

A method for dataflow computation utilizes a graph-based representation in which computational components corresponding to nodes (vertices) of the graph are coupled via dataflows corresponding to links (directed edges) of the graph (referred to as a "dataflow graph"). Downstream components connected to upstream components via dataflow links receive an ordered stream of input data elements and processes the input data elements in the order received, optionally generating one or more corresponding streams of output data elements. A system for performing such graph-based computations is described in prior U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference. In an implementation related to the method described in that prior patent, each component is implemented as a process residing on one of typically a plurality of computer servers. Each computer server can have multiple such component processes active at any one time, and an operating system (e.g., Unix) scheduler shares resources (e.g., processor time and/or processor cores) between components hosted on that server. In this implementation, the data flow between components can be implemented using the data communication services of the operating system and the data network (e.g., named pipes, TCP/IP sessions, etc.) connecting the servers. A subset of components is typically used as the source and/or data sink (or data receiver, sink) of the data from the overall calculation, e.g., to and/or from a data file, a database table, and an external data stream. Then, after, for example, component processes and data flows are established by a coordination process, the data flows through the entire computing system, which implements the calculation of a graph that is typically controlled by the availability of input data at each component and schedules computing resources for each component. Therefore, at least by enabling different components to be executed in parallel by different processes (hosted on the same or different server computers or processor cores), parallelism can be achieved, wherein the parallel execution of different components on different paths through the data flow graph is referred to as component parallelism in this article, and the parallel execution of different components on different parts of the same path through the data flow graph is referred to as pipeline parallelism in this article.

Other forms of parallelism are also supported by this approach. For example, an input data set can be split, for example, based on partitions of the values of fields in the records of the data set, with each portion being sent to a separate copy of a component that processes the records of the data set. Such separate copies (or "instances") of a component can be executed on separate server computers or separate processor cores of a server computer, thereby achieving what is referred to herein as data parallelism. The results of separate components can be merged to form a single data stream or data set again. The number of computers or processor cores used to execute component instances will be specified by the developer when developing the data flow diagram.

Various methods can be used to improve the efficiency of this approach. For example, each instance of a component does not necessarily have to be hosted in its own operating system process, such as using a single operating system process to implement multiple components (e.g., components that form connected subgraphs of a larger graph).

At least some implementations of the above methods are subject to limitations related to the efficiency of executing the resulting processes on the underlying computer servers. For example, these limitations may relate to the difficulty of reconfiguring a running instance of a graph to change the degree of data parallelism, to change the servers hosting various components, and/or to balance the load on different computing resources. Existing graph-based computing systems also suffer from slow startup times, which is often because too many processes are started unnecessarily, wasting large amounts of memory. Typically, a process begins when the execution of the graph is initiated and ends when the execution of the graph is completed.

Other systems for distributed computing have been used in which an overall computation is divided into smaller parts and these parts are distributed from a master computer server to various other (e.g., "slave") computer servers, which each independently perform the computation and return their results to the master server. Some such approaches are referred to as "grid computing." However, such approaches typically rely on the independence of each computation, other than via a master computer server that calls those parts, and do not provide mechanisms for passing data between computation parts or for scheduling and/or sequencing the execution of those parts. Thus, such approaches do not provide a straightforward and effective solution for hosting computations that involve interactions between multiple components.

Another approach to distributed computing of large data sets uses the MapReduce framework, for example, as embodied in the Apache system. Typically, Hadoop has a distributed file system in which parts for each named file are distributed. The user specifies the computation in terms of two functions: a map function that is executed in a distributed manner on all parts of the named input, and a reduce function that is executed on parts of the output of the map function execution. The output of the map function execution is split and stored again in intermediate parts in the distributed file system. The reduce function is then executed in a distributed manner to process these intermediate parts, thereby producing the result of the overall computation. Although computations that can be represented in the MapReduce framework and whose inputs and outputs can be modified for storage in the file system of the MapReduce framework can be efficiently performed, many computations do not match the framework and/or are not easily adapted to have all of their inputs and outputs in a distributed file system.

In general, there is a need to improve the computational efficiency of graph-based computations (e.g., increasing the number of records processed per unit of given computing resources) compared to the aforementioned approaches in which components (or concurrently executing copies of components) are hosted on different servers. Furthermore, it is desirable to be able to adapt to changing computing resources and demands. There is also a need to provide a computational method that allows for adapting to changes in available computing resources during the execution of one or more graph-based computations, and/or to adapting to changes in the computational load of different components of such computations, for example due to the characteristics of the data being processed, or to temporal variations in the load. There is also a need to provide a computational method that can effectively utilize computing resources with different characteristics, such as using servers with different numbers of processors per server, different numbers of processor cores per processor, etc., to effectively support both homogeneous and heterogeneous environments. It is also desirable to enable graph-based computations to start quickly. One aspect of providing this efficiency and adaptability is to provide appropriate separation and abstraction barriers between the choices made by the developer when creating the graph (at design time), the actions taken by the compiler (at compile time), and the actions taken by the runtime system (at runtime).

Summary of the Invention

In one aspect, generally speaking, a method for processing a graph-based program specification includes: receiving the graph-based program specification, the graph-based program specification including: a plurality of components, each component corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each of the one or more links connecting an output port of an upstream component in the plurality of components to an input port of a downstream component in the plurality of components; and processing the graph-based program specification to generate prepared code representing a subset of the plurality of components of the graph-based program specification (corresponding to an "execution set"). As used herein, "prepared code" includes code written in any target language used by a compiler or interpreter when transforming parsed elements of a graph-based program specification, which may include executable code or code that can be further compiled or interpreted into executable code. The processing includes: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of the linked components; forming subsets based on the identified subset boundaries; and generating prepared code for each formed subset, which, when executed by a runtime system, causes processing tasks corresponding to the components in the formed subset to be performed according to information embedded in the prepared code for the formed subset.

Various aspects can include one or more of the following features.

Forming subsets includes traversing components of the graph-based program specification while maintaining a record of traversed subset boundaries, and associating each component of the graph-based program specification with a single subset identifier determined from the record of traversed subset boundaries.

Each subset identifier associated with the identified subset of the plurality of components is unique.

A record of the subset boundaries traversed is maintained as a path of identifier values.

The path of identifier values includes character strings of identifier values separated from each other by separators.

Forming a subset includes associating a first component of the graph-based program specification with a subset identifier; propagating the subset identifier to components downstream of the first component; and modifying the subset identifier during the propagation of the subset identifier based on the identified subset boundaries.

Modifying the subset identifier during the propagation process of the subset identifier includes: changing the value of the subset identifier from the first subset identifier value to the second subset identifier value associated with the first subset boundary when traversing the first subset boundary; and changing the value of the subset identifier to the first subset identifier value when traversing the second subset boundary associated with the first subset boundary.

Identifying one or more subset boundaries based at least in part on characteristics of the linked components includes identifying subset boundaries based on links between ports of a first type on an upstream component and ports of a second type on a downstream component.

Identifying one or more subset boundaries based at least in part on characteristics of the linked components includes identifying subset boundaries based on a determined type of link between the upstream component and the downstream component, wherein the determined type of link is one of a plurality of different types of links between the components.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding information indicating allowed concurrency between processing tasks corresponding to components in the formed subset into the prepared code.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding information indicating a priority relative to other formed subsets into the prepared code.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding information indicating transactionality of one or more processing tasks corresponding to components in the formed subset into the prepared code.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding information in the prepared code indicating at least one resource to be locked during execution of the prepared code.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding information indicating ordering characteristics between data elements processed by one or more processing tasks corresponding to components of the formed subset into the prepared code.

Generating the prepared code for each formed subset includes, for at least one formed subset, embedding in the prepared code information indicating a number of data elements operated on by executing each instance of the formed subset using the prepared code.

In another aspect, in general, a software stored in a non-transitory form on a computer-readable medium for processing a graph-based program specification includes instructions for causing a computing system to: receive the graph-based program specification, the graph-based program specification including: a plurality of components, each component corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each of the one or more links connecting an output port of an upstream component in the plurality of components to an input port of a downstream component in the plurality of components; and process the graph-based program specification to generate prepared code representing subsets of the plurality of components of the graph-based program specification, the processing including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of the linked components; forming subsets based on the identified subset boundaries; and generating prepared code for each formed subset, the prepared code causing, when executed by a runtime system, the processing task corresponding to the component in the formed subset to be performed according to information embedded in the prepared code for the formed subset.

In another aspect, in general, a computing system for processing a graph-based program specification includes: at least one input device or port configured to receive the graph-based program specification, the graph-based program specification including: a plurality of components, each component corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each of the one or more links connecting an output port of an upstream component in the plurality of components to an input port of a downstream component in the plurality of components; and at least one processor configured to process the graph-based program specification to generate prepared code representing subsets of the plurality of components of the graph-based program specification, the processing including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of the linked components; forming subsets based on the identified subset boundaries; and generating prepared code for each formed subset, the prepared code causing, when executed by a runtime system, the processing task corresponding to the component in the formed subset to be performed according to information embedded in the prepared code for the formed subset.

Various aspects can have one or more of the following advantages.

The technology described herein also facilitates the use of unconventional technical features at each level of its hierarchical structure to efficiently process large amounts of data in a computing system. These technical features work together in various operational stages of the computing system (including design time, compile time, and runtime). The programming platform enables a graph-based program specification to specify the desired calculation at design time. The compiler prepares the target program specification at compile time so that fine-grained tasks are effectively distributed among multiple servers of the computing system at runtime. For example, multiple tasks are configured based on any control flow and data flow constraints in the graph-based program specification. The runtime system supports dynamic allocation of these concurrently executed tasks in a manner that improves computing efficiency (for example, in terms of the number of processed records per unit of a given computing resource). Various technical features work together to achieve efficiency gains relative to conventional systems.

For example, a computing system can use tasks corresponding to components of a data processing graph (or other graph-based program specification) to facilitate processing data elements in a flexible runtime execution of these tasks without placing undue burden on the programmer. A graphical user interface allows connections between different types of ports on components that perform the desired data processing computations, and the computing system can automatically identify subsets of one or more components and/or nested subsets of components for later use when processing the program specification. For example, the execution set discovery pre-process can identify a hierarchy of potentially nested component execution sets that is very difficult for a human to identify, and the system can then determine resource allocations in the underlying system architecture to execute those subsets for efficient parallel data processing. By automatically identifying these subsets of components ("execution sets"), the computing system can ensure that the data processing graph meets certain consistency requirements, as described in more detail below, and allows the underlying computing system to operate the execution sets with a highly scalable degree of parallelism, because the parallelism of the execution sets can be determined at runtime and is only limited by the computing resources available at runtime, thereby facilitating efficient execution of the data processing graph. Furthermore, by embedding certain information into the prepared code that identifies the execution sets, these execution sets can ultimately be processed by the underlying computing system as specific tasks, and the computing system can ensure that the processing tasks are performed in a manner that improves the efficiency of the internal functions of the computing system by, for example, parallelizing the tasks.

When executing the methods described herein, these techniques also exhibit further technical effects on the internal functions of the computing system, such as reducing the demand for memory and other computing resources, and reducing the delay of the system when processing individual data elements. In particular, these advantages contribute to the efficient execution of data processing graphs. For example, due to the fact that multiple processes are started by other processes (e.g., Unix processes) when executing the graph, traditional graph-based computing systems may have relatively high latency (e.g., on the order of tens of milliseconds), resulting in cumulative startup time for these processes. However, the techniques described herein facilitate achieving relatively low latency (e.g., on the order of tens of microseconds) and a higher throughput of data processed per second by allowing program code within a single process to directly start other program code without the need for process startup overhead. Other aspects that contribute to the efficient execution of data processing graphs will become apparent in the following description.

Other features and advantages of the invention will be apparent from the following description and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of a task-based computing system.

FIG. 2A is an example of a portion of a data processing graph having control and data ports.

2B-2C are examples of data processing graphs having control ports and data ports.

3A is a data processing diagram including connections of multiple scalar output ports to scalar input ports.

3B is a data processing diagram including connections of multiple aggregate output ports to aggregate input ports.

3C is a data processing diagram including aggregate output port to scalar input port connections and scalar output port to aggregate input port connections.

FIG4A is a scalar port to scalar port connection between two components.

FIG. 4B is a manifold port to manifold port connection between two components.

FIG4C is a collection port to scalar port connection between two components, including an execution set entry point.

FIG4D is a scalar port to set port connection between two components, including an execution set exit point.

FIG5 is a diagram of data processing using a stack-based allocation algorithm.

FIG6 is a diagram showing data processing using a global mapping-based allocation algorithm.

FIG7 is a data processing diagram with a user-defined execution set.

8A and 8B illustrate "same set" relationships in a data processing graph.

FIG. 9 is a data processing diagram with entry points for duplicating data elements.

10A-10C illustrate a user interface workflow.

FIG. 11A is a diagram of data processing with an illegal execution set.

FIG. 11B is a data processing diagram of a loop with an illegal execution set.

12A-12B are diagrams of examples of data processing graphs and corresponding control graphs.

13A-13B are state transition diagrams of an exemplary execution state machine.

FIG14 is a diagram of a set of processing engines.

DETAILED DESCRIPTION

1 , a task-based computing system 100 uses a high-level program specification 110 to control the computing and storage resources of a computing platform 150 to perform the computations specified by the program specification 110. A compiler/interpreter 120 receives the high-level program specification 110 and generates a task-based specification 130 in a form that can be executed by a task-based runtime interface/controller 140. The compiler/interpreter 120 identifies one or more "execution sets" of one or more "components," which can be instantiated individually or as a unit as fine-grained tasks to be applied to each of a plurality of data elements. As described in more detail below, part of the compilation or interpretation process involves identifying these execution sets and preparing them for execution. It should be understood that the compiler/interpreter 120 can use any of a variety of algorithms that include steps such as parsing the high-level program specification 110, validating syntax, type-checking data formats, generating any errors or warnings, and preparing the task-based specification 130, and that, for example, the compiler/interpreter 120 can utilize various techniques to optimize the efficiency of computations executed on the computing platform 150. The target program specification generated by the compiler/interpreter 120 may itself be in an intermediate form to be further processed (e.g., further compiled, interpreted, etc.) by another portion of the system 100 to produce the task-based specification 130. The following discussion outlines one or more examples of such transformations, but of course, other methods for transformation are possible, as will be appreciated by those skilled in compiler design, for example.

Typically, computing platform 150 is composed of multiple computing nodes 152 (e.g., individual server computers that provide both distributed computing resources and distributed storage resources), thereby achieving a high degree of parallelism. As discussed in further detail below, the computations represented in high-level program specifications 110 are executed on computing platform 150 as relatively fine-grained tasks, further enabling efficient parallel execution of the specified computations.

1 Data processing diagram

In some embodiments, the high-level program specification 110 is a graph-based program specification called a "data processing graph" that includes a set of "components," each of which specifies a portion of the overall data processing calculation to be performed on the data. Components are represented as nodes in a graph, for example, in a programming user interface and/or a data representation of a calculation. Unlike some graph-based program specifications (such as the data flow graphs described in the background art above), a data processing graph can include links between nodes representing either or both of data transfer or control transfer. One way to indicate link characteristics is to provide different types of ports on a component. A link is a directed link that is coupled from an output port of an upstream component to an input port of a downstream component. A port has an indicator that indicates how to write and read data elements from a link and/or how to control a component to process data.

These ports can have a number of different characteristics. One characteristic of a port is its directionality as an input port or an output port. A directed link represents data and/or control passing from an output port of an upstream component to an input port of a downstream component. This allows developers to link different types of ports together. Some data processing characteristics of a data processing graph depend on how different types of ports are linked together. For example, as described in more detail below, links between different types of ports can result in nested subsets of components in different "execution sets" that provide a hierarchical form of parallelism. Certain data processing characteristics are implied by the type of port. The different types of ports a component may have include:

A collection input or output port means that an instance of a component will read or write, respectively, all data elements of a collection that will pass through the link coupled to that port. For a pair of components with a single link between their collection ports, this typically allows the downstream component to read data elements while the upstream component is writing them, thereby achieving pipeline parallelism between the upstream and downstream components. As described in more detail below, data elements can also be reordered, which makes parallelization efficient. In some graphical representations, such as programming graphical interfaces, such collection ports are typically indicated by square connectors at the component.

A scalar input or output port means that one instance of the component will read or write at most one data element, respectively, from the link coupled to the port. For a pair of components with a single link between their scalar ports, serial execution of the downstream component is enforced by using the transfer of a single data element as a control transfer after the upstream component has completed execution. In some graphical representations, such as programming graphical interfaces, such scalar ports are typically indicated by a triangle connector at the component.

A control input or output port, similar to a scalar input or output, but requiring no data elements to be sent, is used to communicate the transfer of control between components. For a pair of components with a link between their control ports, serial execution of the downstream component is enforced after the upstream component has completed execution (even if those components also have links between collection ports). In some graphical representations, such as programming graphical interfaces, such control ports are typically indicated by circular connectors at the component.

These different types of ports enable flexible design of data processing graphs, allowing powerful combinations of data flow and control flow with overlapping properties of port types. In particular, there are two types of ports (called "data ports"): collection ports and scalar ports, which are used to transmit some forms of data; there are also two types of ports (called "serial ports"): scalar ports and control ports, which are used to force serial execution. A data processing graph will typically have one or more components that act as "source components" without any connected input data ports, and one or more components that act as "sink components" without any connected output data ports. Some components will have connected input and output data ports. In some embodiments, the graph is not allowed to have cycles and therefore must be a directed acyclic graph (DAG). This feature can be used to exploit certain properties of DAGs, as described in more detail below.

The use of dedicated control ports on components of a data processing graph also enables flexible control of different parts of a computation that is not possible using certain other control flow techniques. For example, job control solutions that are able to apply dependency constraints between data flow graphs cannot provide the fine-grained control enabled by control ports that define dependency constraints between components within a single data flow graph. Furthermore, data flow graphs that assign components to different stages that run sequentially do not allow for flexibility in the ordering of individual components. For example, nested control topologies that are not possible using simple stages can be defined using the control ports and execution sets described in this paper. This greater flexibility can also potentially improve performance by allowing more components to run concurrently as much as possible.

By connecting different types of ports in different ways, developers are able to specify different types of link configurations between ports of components of a data processing graph. For example, one type of link configuration may correspond to a specific type of port connected to ports of the same type (e.g., a scalar to scalar link), and another type of link configuration may correspond to a specific type of port connected to ports of different types (e.g., a collection to scalar link). These different types of link configurations serve both as a way for developers to visually identify the expected behavior associated with a portion of a data processing graph and as a way to indicate to the compiler/interpreter 120 the corresponding type of compilation process required to enable that behavior. While the examples described herein use unique shapes for different types of ports to visually represent different types of link configurations, other implementations of the system may distinguish the behavior of different types of link configurations by providing different types of links and assigning unique visual indicators (e.g., thickness, line type, color, etc.) to each type of link. However, in order to represent the same type of link configuration using link types rather than port types with the three types of ports listed above, there will be more than three types of links (e.g., scalar to scalar, collection to collection, control to control, collection to scalar, scalar to collection, scalar to control, etc.). Other examples may include different types of ports, but without explicitly indicating the port type visually within the data processing diagram.

Compiler/interpreter 120 performs processes to prepare the data processing graph for execution. The first process is an execution set discovery preprocessing process, which is used to identify a hierarchy of potentially nested execution sets of components. The second process is a control graph generation process, which is used to generate a corresponding control graph for each execution set. Compiler/interpreter 120 will use this control graph to generate control code at runtime. This control code will effectively implement a state machine for controlling the execution of components within each execution set. Each of these processes will be described in more detail below.

The component with at least one input data port specifies the processing to be performed to each input data element or set (or the data element and/or set of the tuple on its multiple input ports).A form of this specification is as the process to be performed to one or tuple input data element and/or set.If the component has at least one output data port, then it can produce corresponding one or tuple output data element and/or set.Such process can be specified with a high-level statement-based language (for example, using Java source statements or the data manipulation language (DML) used in, for example, U.S. Patent 8,069,129 "Editing and Compiling Business Rules (editing and compiling business rules)"), or can be provided in some completely or partially compiled forms (for example, as Java Bytecode (Java bytecode)).For example, a component can have a working process, the independent variable of the working process includes its input data element and/or set and its output data element and/or set, or more generally, including a reference to such data element or set or to a process or data object (referred to as "handle" herein), which is used to obtain input and provide output data element or set.

Work processes can be of various types. This document does not wish to limit the types of processes that can be specified, and one type of work process specifies discrete computations on data elements based on a record format. A single data element can be a record from a table (or other type of data set), and a set of records can be all the records in the table. For example, one type of work process for a component with a single scalar input port and a single scalar output port includes receiving an input record, performing a computation on the record, and providing an output record. Another type of work process can specify how a tuple of input records received from multiple scalar input ports is processed to form a tuple of output records emitted on multiple scalar output ports.

The semantic definition of a computation specified by a data processing graph is inherently parallel because it expresses constraints and/or no constraints on the order and concurrency of the processing of the computation defined by the graph. Thus, the definition of a computation does not require that the result be equivalent to some sequential ordering of the computation's steps. On the other hand, the definition of a computation does provide specific constraints requiring the ordering of parts of the computation and restrictions on the parallel execution of parts of the computation.

In the discussion of data processing graphs, it is assumed that multiple instances of a component are implemented as separate "tasks" in the runtime system as a way to express ordering and parallelism constraints. A more specific discussion of implementing a data processing graph as a task-based specification that implements computations consistent with the semantic definition will be presented more fully after discussing the characteristics of the graph-based specification itself.

Typically, each component in a data processing graph will be instantiated multiple times in the computing platform during the graph's execution. The number of instances of each component can depend on which of multiple execution sets the component is assigned to. When multiple instances of a component are instantiated, more than one instance can execute in parallel, and different instances can execute on different computing nodes in the system. The interconnections of the components (including the types of ports) determine the nature of the parallel processing allowed by a given data processing graph.

Although, as described below, state is not generally maintained between executions of different instances of a component, certain provisions are provided in the system for explicitly referencing persistent storage that can span executions of multiple instances of a component.

In an example where the work process specifies how to process a single record to produce a single output record and the port is indicated as a collection port, a single instance of the component can be executed and the work process iterated to process successive records to generate successive output records. In this case, state may be maintained within the component from iteration to iteration.

In examples where the work process specifies how to process a single record to produce a single output record and the port is indicated as a scalar port, multiple instances of the component can be executed and no state is maintained for different input records between multiple executions of the work process.

Furthermore, in some embodiments, the system supports work processes that do not follow the most fine-grained specification described above. For example, a work process may implement iteration internally, for example, accepting a single record through a scalar port and providing multiple output records through a collection port.

As mentioned above, there are two types of data ports: collection ports and scalar ports, which are used to transmit some forms of data; there are also two types of serial ports: scalar ports and control ports, which are used to enforce serial execution. In some cases, one type of port can be connected to another type of port via a link. Some of these cases will be described below. In some cases, one type of port will be linked to a port of the same type. The link between two control ports (called a "control link") imposes serial execution ordering between the linked components without the need to send data over the link. The link between two data ports (called a "data link") provides data flow and also enforces serial execution ordering constraints in the case of scalar ports, and does not require serial execution ordering in the case of collection ports. A typical component usually has at least two types of ports, including input and output data ports (collection ports or scalar ports) and input and output control ports. The control link connects the control port of the upstream component to the control port of the downstream component. Similarly, the data link connects the data port of the upstream component to the data port of the downstream component.

Developers can use a graphical user interface to specify a specific data processing calculation from a set of components, each of which performs a specific task (e.g., a data processing task). Developers make such specifications by assembling a data processing diagram on a canvas area displayed on a display screen. This involves placing components on the canvas, connecting their various ports with appropriate links, and configuring the components appropriately. The following simple example illustrates certain behaviors in the case of a component with a single pair of collection ports and a single pair of control ports.

Figure 2a shows an example of a portion of an assembled data processing graph including a first component 210A having an input control port 212A and an output control port 214A, as well as an input aggregation port 216A and an output aggregation port 218A. Control links 220A, 222A connect the input control port 212A and the output control port 214A to the control ports of other components in the data processing graph. Similarly, data links 224A, 226A connect the input aggregation port 216A and the output aggregation port 218A to the ports of other components in the data processing graph. Aggregation ports 216A, 218A are represented in the figure as rectangles, while control ports 212A, 214A are represented as circles.

Typically, the input collection port 216A receives data to be processed by the component 210A, and the output collection port 214 provides data that has been processed by the component 210A. In the case of a collection port, the data is typically an unordered collection of an unspecified number of data elements. In the specific instance of a holistic computation, a collection may include multiple data elements, or a single data element, or no data elements. In some implementations, a collection is associated with parameters that determine whether the elements in the collection are unordered or ordered (and, if ordered, what determines the order). As will be described in more detail below, for an unordered collection, the order in which the data elements are processed by the components on the receiving side of the data link can be different from the order in which the data elements are provided by the components on the sending side of the data link. Thus, in the case of a collection port, the data link between them acts as a "packet" of data elements from which data elements can be pulled in any order, as opposed to a "conveyor belt" that moves data elements from one component to another in a particular order.

The control link is used to transmit control information between control ports, which determines whether and when a component begins execution. For example, control link 222A indicates that component 210B will begin execution after component 210A has completed (i.e., in serial order), or indicates that component 210B will not begin execution (i.e., is "disabled"). Therefore, although no data is sent over the control link, it can be regarded as sending a signal to the component on the receiving side. The manner in which this signal is sent can vary depending on the implementation, and in some implementations can involve sending control messages between components. Other implementations may not involve sending actual control messages, but may instead involve directly calling a process or calling a function associated with the task represented by the component on the receiving side (or omitting such a call or function call in the case of prohibition).

The ability to link control ports thus enables developers to control the relative ordering between different parts of the data processing computation represented by different components of the data processing graph. Furthermore, using control ports on components to provide this ordering mechanism enables the intermixing of logic associated with data flow and control flow. In effect, this enables data to be used to make decisions about control.

In the example shown in Figure 2A, control ports are connected to other control ports, and data ports are connected to other data ports. However, the data on the data ports inherently carries two different types of information. The first is the data itself, and the second is the presence of data. This second type of information can be used as a control signal. As a result, by enabling scalar data ports to be connected to control ports, additional flexibility can be provided.

FIG. 2B illustrates an exemplary data processing graph 230 that exploits the flexibility afforded by the ability to connect scalar ports to control ports.

Data processing diagram 230 features the following: a first component 231 labeled "Calculate Date Information," a second component 232 labeled "Make Monthly Report?", a third component 233 labeled "Make Weekly Report," a fourth component 234 labeled "Monthly Report," a fifth component 235 labeled "Make Weekly Report?", and a sixth component 236 labeled "Weekly Report." Data processing diagram 230 executes a process that always produces a daily report, a daily and weekly report, or all three. The decision as to which of these outcomes will occur depends on an evaluation of certain date information provided by first component 231. Thus, FIG. 2B illustrates an example of data that effectively controls execution.

Execution begins when the first component 231 provides date information from its output scalar port to the input scalar port of the second component 232 and the input scalar port of the third component 233. The second component 232, which has no connected input control port, operates immediately. All other components, including the third component 233, have connected input control ports and must wait to be activated by the appropriate positive control signal.

The second component 232 examines the date information and determines whether it is appropriate to produce a monthly report. There are two possible results: a monthly report is required, or not. The second component 232 and the third component 233 both have two output scalar ports and are configured to perform a selection function that provides a data element that is used as a positive control signal on one output scalar port (i.e., the selected port) and as a negative control signal on the other output scalar port.

If, based on the date information, the second component 232 determines that monthly reporting is not required, the second component 232 sends a data element from its bottom output scalar port to the input control port of the third component 233. This data element is interpreted as a positive control signal that indicates to the third component 233 that the second component 232 has completed processing the data provided by the first component 231 and that the third component 233 can now begin processing the date information data it receives.

On the other hand, if the second component 232 determines that monthly reporting is required based on the date information provided by the first component 231, the second component 232 instead sends a data element interpreted as a positive control signal from its output scalar port to the input control port of the fourth component 234. Although the data element is more than just a control signal, the fourth component 234 treats it as a positive control signal because it is being provided to its input control port. The fourth component 234 ignores the actual data in the data element and uses only the presence of the data element as a positive control signal.

The fourth component 234 continues to create the monthly report. When completed, the fourth component 234 outputs a control signal from its output control port to the input control port of the third component 233. This tells the third component 233 that it (i.e., the third component 233) can now begin processing the date information provided to it by the first component 231.

Therefore, the third component 233 will always end up processing the data provided by the first component 231 via its input scalar port. The only difference is which component triggers it to start processing: the second component 232 or the fourth component 234. This is because the two input control ports on the third component 233 will be combined using OR logic, so that a positive control signal received at either port (or both) will trigger processing.

The remainder of diagram 230 operates in substantially the same manner, except that the third component 233 takes over the role of the second component 232 and the sixth component 236 takes over the role of the fourth component 234.

When activated by a control signal from the second component 232 or the fourth component 234 at its input control port, the third component 233 checks the date information provided by the first component 231 via a data link connecting the first component 231 to the third component 233. If the third component 233 determines that a weekly report is not required based on the date information, the third component 233 sends a data element interpreted as a positive control signal from one of its output scalar ports to the input control port of the fifth component 235.

On the other hand, if the third component 233 determines that a weekly report is needed, the third component 233 sends the data element interpreted as a positive control signal from its other output scalar port to the input control port of the sixth component 236. The sixth component 236 continues to create the weekly report. When completed, the sixth component 236 sends the data element interpreted as a positive control signal from its output scalar port to the input control port of the fifth component 235.

Therefore, the fifth component 235 will always be executed eventually, the only difference is whether the third component 233 or the sixth component 236 finally triggers the fifth component 235 to start executing. Upon receiving a control signal from the third component 233 or the sixth component 236, the fifth component 235 creates a daily report.

FIG2C also illustrates the use of scalar and aggregate data ports.

2C shows a data processing diagram 240 having a first component 241 labeled "Input File," a second component 242 labeled "Get File Name from Request," a third component 243 labeled "Read File," a fourth component 244 labeled "Is It a Bad Record?", a fifth component 245 labeled "Invalid Record," a sixth component 246 labeled "Generate Bad Record File Name," a seventh component 247 labeled "Any Verification Errors?", and an eighth component 248 labeled "Send Alert." This diagram is intended to write bad records to a file and send an alert when such a bad record is detected.

Components 241 and 243 are examples of components that serve as sources of data, and component 245 is an example of a component that serves as a sink (receiver) of data. Components 241 and 243 use an input file that can be stored in any of a variety of formats in a file system (such as a local file system or a distributed file system) as their source. The input file component reads the contents of the file and generates a collection of records from the file. A scalar input port (as shown in component 243) provides data elements that specify the location (e.g., path or uniform resource locator) of the file to be read and the record format to be used. In some cases, the location and record format can be provided as parameters to the input file component, in which case the input scalar port does not need to be connected to any upstream component and does not need to be displayed (this is also true for component 241). A collection output port (as shown on both components 241 and 243) provides a collection of records. Similarly, an output file component (such as component 245) writes the collection of records received through the input collection port to an output file (whose location and record format can optionally be specified by the input scalar port). An input file component or an output file component may also include a control input or output port that is linked to a control port of another component, such as component 245 .

In the illustrated data processing diagram 240, the components within the larger dashed rectangle are part of an execution set. This execution set contains another execution set nested within it. This nested execution set is also shown within the dashed rectangle and contains only the fourth component 244. Execution sets are discussed in more detail below.

In operation, the first component 241 reads the input file. When the first component 241 is executing, the first component 241 provides the set of records in the input file to the second component via the data link from the output set data port to the input set data port of the second component 242. Different instances of the second component 242 and other downstream components (which are in the same execution set) can be executed for each record in the set, as will be described in more detail below. Since the second component 242 has nothing connected to its control input, the second component 242 starts processing immediately. When completed, the second component 242 provides the file name on its output scalar port. The file name is received by both the third component 243 and the sixth component 246 at the corresponding input scalar port.

The third component 243 immediately reads the file identified by the file name and provides the contents of the file on an output collection port for delivery to an input scalar port of an instance of the fourth component 244. Simultaneously, the sixth component 246 receives the same file name and outputs another file name, which is provided on an output scalar port connected to the corresponding input scalar ports of the fifth component 245 and the seventh component 247.

Upon receiving a file name from the sixth component 246 and a bad record from the fourth component 244 , the fifth component 245 writes the bad record to an output file whose file name is identified by the sixth component 246 .

Seventh component 247 is the only component that is not ready to be started when its data input port receives data. When fifth component 245 completes writing to the output file, fifth component 245 sends a control signal from its control output port to the input control port of seventh component 247. If seventh component 247 determines that an error has occurred, seventh component 247 provides data to the input scalar port of eighth component 248. This causes eighth component 248 to generate an alarm. This provides an example of how control ports can be used to limit the execution of certain components in a data processing graph.

It should be clear that the ability to control processing in one component based on the state of another component enables the component to control processing when a group of multiple upstream components have all reached a specific state. For example, a data processing graph can support multiple control links to or from the same control port. Alternatively, in some implementations, a component may include multiple input and output control ports. Default logic can be applied by the compiler/interpreter 120. Developers can also provide custom logic for determining how to combine control signals. This can be achieved by appropriately arranging the combinational logic to apply to the various control links of the upstream components and triggering the startup of the component only when a certain logical state is reached (for example, in the case of default OR logic, when all upstream components have completed and when at least one upstream component has sent an activation control signal).

Typically, a control signal can be a signal that triggers the start of a process or a signal that triggers the inhibition of a process. The former is a "positive control signal" and the latter is a "negative control signal." However, if combinatorial logic is used to determine whether a task should be called (triggering the start of a process), the logic can "invert" the usual interpretation so that the task is called only when all inputs provide a negative control signal. In general, combinatorial logic can provide an arbitrary "truth table" for determining the next state in a state machine corresponding to a control diagram, as described in more detail below.

Unconnected control ports can be assigned a default state. In one embodiment, the default state corresponds to a positive control signal. As described in more detail below, this can be achieved by using implicit start components and end components in a control diagram representing a data processing diagram.

Different types of data ports on various components allow data to be passed between components over links in different ways, depending on the type of input and output ports linking those components. As described above, a scalar port represents the production (for a scalar output port) or consumption (for a scalar input port) of at most a single data element (i.e., 0 or 1 data element). A collection port, on the other hand, represents the production (for a collection output port) or consumption (for a collection input port) of a group of potentially multiple data elements. By supporting both types of data ports in a single data processing graph, computational resources can be allocated more efficiently, and more complex control and data flows can be generated between tasks, allowing developers to easily indicate desired behavior.

3A , data processing graph 300 includes a series of three connected components: a first component (A1) 302, a second component (B1) 304, and a third component (C1) 306. The first component includes a set input port 308 and a scalar output port 310. The second component 304 includes a scalar input port 312 and a scalar output port 314. The third component includes a scalar input port 316 and a set output port 318.

A first link 320 connects the scalar output port 310 of the first component 302 to the scalar input port 312 of the second component 304, which allows data to be passed between the first component 302 and the second component 304 while enforcing serial execution of the first component 302 and the second component 304. Similarly, a second link 322 connects the scalar output port 314 of the second component 304 to the scalar input port 316 of the third component 306, which allows data to be passed between the second component 304 and the third component 306 and enforces serial execution of the second component 304 and the third component 306.

Due to the interconnection of the scalar ports in Figure 3A, the second component 304 begins execution only after the first component 302 completes (and passes a single data element on the first link 320), and the third component 306 begins execution only after the second component 304 completes (and passes a single data element on the second link 322). In other words, each of the three components in the data processing diagram runs once in the strict order A1/B1/C1.

In some examples, one or more components can be placed in a disabled state, which means that one or more components do not execute and therefore do not pass any data element from its output port. For example, by ensuring that the components that will not perform any useful processing do not need computing resources (e.g., processes or memories) dedicated to them, it is possible to disable components and avoid wasting resources. Any component with a scalar input port that is only connected to the output port of the disabled component will not execute because they do not receive data. For example, if the first component 302 is placed in a disabled state, the scalar input port 312 of the second component 304 does not receive data from the scalar output port 310 of the first component 302 and is therefore not executed. Since the second component 304 does not execute, the scalar input port 316 of the third component 306 does not receive data from the scalar output port 314 of the second component 304 and is also not executed. Therefore, the data transferred between the two scalar ports is also used as a positive control signal, similar to the signal sent between the control ports of the two links.

In the exemplary data processing diagram of Figure 3A, the input port 308 of the first component 302 and the output port of the third component 318 happen to be collective ports, which have no effect on the serial execution behavior of the first component 302, the second component 304 and the third component 306 imposed by the scalar ports connecting them.

Typically, collection ports are used to pass collections of data elements between components, and at the same time, grant the runtime system permission to reorder the data elements in the set. Reordering the data elements of an unordered collection is allowed because there are no dependencies on the state of the computation from one data element to another, or if there is global state that is accessed when processing each data element, the final state is independent of the order in which the data elements are processed. This permission to reorder provides the flexibility to defer decisions about parallelization to the runtime.

3B , data processing graph 324 includes a series of three connected components: a first component (A2) 326, a second component (B2) 328, and a third component (C2) 330. First component 326 includes a collective input port 332 and a collective output port 334. Second component 328 includes a collective input port 336 and a collective output port 338. Third component 330 includes a collective input port 340 and a collective output port 342.

Each of the three components 326, 328, and 330 specifies how to process a set of one or more input elements to generate a set of one or more output elements. There is not necessarily a one-to-one correspondence between specific input elements and specific output elements. For example, the multiple data elements in the first set of data elements 344 between the first component 326 and the second component 328 can be different from the multiple data elements in the second set of data elements 346 between the second component 328 and the third component 330. The only constraint imposed on the connection between the collection ports is that each data element in the set is passed from one collection port to another collection port, while allowing the first component 326 and the second component 328 and the second component 328 and the third component 330 to be arbitrarily reordered relative to the order in which they are processed. Alternatively, in other examples, the collection ports can be optionally configured to maintain order. In this example, the three components 326, 328, and 330 are started together and run concurrently, thereby allowing pipeline parallelism.

The compiler/interpreter 120 described with reference to Figure 1 is configured to identify the collection port to collection port connection, and converts the calculation into executable code in a manner suitable for the calculation being performed. The unordered nature of the collection data link makes the compiler/interpreter 120 flexible in how to achieve this. For example, if for the second component 328, each output element is calculated based on a single input element (that is, there is no state maintained across data elements), then the compiler/interpreter 120 can allow the runtime system to dynamically parallelize the processing of data elements (for example, depending on the computing resources available at runtime) by instantiating up to one component for each data element. Alternatively, in special cases, state can be maintained across data elements in a component with multiple input collection ports. But in general, the runtime system can be allowed to parallelize the tasks of the component. For example, if the runtime system detects a global state that is not maintained, then task parallelization can be allowed. Some components can also be configured to support maintaining state, in which case parallelization can not be allowed. If the collection is unordered, the fact that no order needs to be maintained between the data elements means that each instance of the second component 328 can provide its output data elements to the third component 330 as soon as they are available, and the third component 330 can start processing these data elements before all instances of the second component 328 have completed.

In some examples, a graph developer can explicitly indicate that the processing of data elements in a data set can be dynamically parallelized by connecting the set output port of one component to the scalar input port of another component. This indication also requires that no state needs to be maintained between the processing of different elements of the set. Referring to Figure 3C, a data processing graph 348 includes a series of three connected components: a first component (A3) 350, a second component (B3) 352, and a third component (C3) 354. The first component 350 includes a set input port 356 and a set output port 358. The second component 352 includes a scalar input port 360 and a scalar output port 362. The third component 354 includes a set input port 364 and a set output port 366.

The first component's collective output port 358 is connected to the scalar input port 360 of the second component 352 via a first link 368, and the scalar output port 362 of the second component 352 is connected to the collective input port 364 via a second link 370. As described in more detail below, the link from the collective output port to the scalar input port represents an entry point into the execution set, and the link from the scalar output port to the collective input port represents an exit point from the execution set. It is quite common that, as described in more detail below, multiple components included in an execution set can be dynamically parallelized by the runtime controller to process data elements from a collection of data elements.

In Figure 3C , link 368 between aggregate output port 358 of first component 350 and scalar input port 360 of second component 352 represents the entry point into the execution set. Link 370 between scalar output port 362 of second component 352 and aggregate input port 364 of third component 354 represents the exit point of the execution set. In other words, second component 352 is the only component in the execution set.

Because the second component 352 is included in the execution set, a separate instance of the second component 352 is launched for each data element received from the collection-type output port 358 of the first component 350. At least some of the separate instances can run in parallel, depending on decisions that cannot be made until runtime. In this example, the first component (350) and the third component (354) are launched together and run concurrently, while the second component (352) runs once for each data element in the collection received via link 368. Alternatively, the second component 352 runs once for each tuple of multiple data elements in the collection.

2 Execution Set

As described above with reference to FIG1 , the compiler/interpreter 120 performs an execution set discovery preprocess on the data processing graph to prepare the data processing graph for execution. In a general sense, as used herein, the term “execution set” refers to a collection of one or more components that can be called as a unit and applied to a portion of data (such as a portion of data elements of an output set port). Thus, at most one instance of each component in the execution set is implemented for each input data element (or a tuple of multiple input data elements presented to one or more input ports of the execution set). Within the execution set, ordering constraints are imposed by links to scalars and control ports, allowing parallel execution of components in the execution set as long as the ordering constraints are not violated. The code prepared by the compiler/interpreter 120 for the execution set may include embedded information (e.g., comments or modifiers) indicating how the tasks corresponding to the components (e.g., the degree of parallelism) are to be implemented when the code is executed. In an example where an instance of an execution set is executed on a tuple of multiple data elements in a received set, the tuple may include, for example, a fixed number of data elements or multiple data elements that share some characteristics (e.g., a common key value). In examples where there are at least some components that are allowed to execute in parallel, an execution set can be implemented using multiple tasks, e.g., a task for the execution set as a whole, and one or more subtasks for concurrent execution of instances of one or more components. Thus, the tasks representing different instances of an execution set can themselves be broken down into even finer-grained tasks, e.g., having subtasks that can be executed concurrently. Tasks for different execution sets can typically be executed independently and in parallel. Thus, if a large data set has, for example, a million records, there may be a million independent tasks. Some tasks may be executed on different nodes 152 of a computing platform 150. Tasks may be executed using lightweight threads that can be efficiently executed concurrently, even on a single node 152.

Typically, execution sets identified by the allocation algorithm (i.e., execution sets other than the root execution set) receive data elements via "driven" scalar data ports at the boundaries of the execution set. For each data element received at the driven input scalar data port of the execution set, each component within the execution set executes once (if enabled) or not at all (if disabled). Multiple instances of the execution set can be instantiated and executed in parallel to process multiple data elements available to the execution set from upstream collection ports. The parallelism of the execution set can be determined at runtime (including the possible decision not to parallelize the execution set), and the parallelism is only limited by the computing resources available at runtime. The individual outputs of the independent instances of the execution set are aggregated at the output port(s) of the execution set, regardless of order, and provided to downstream components. Alternatively, in other embodiments, execution sets other than the root execution set that do not require driven input scalar data ports can be identified (in some cases, based on user input). Such execution sets without driven input scalar data ports can be executed in a single instance (e.g., a locked execution set described below) or in parallel in multiple instances, if appropriate, using the processes described herein. For example, parameters may be set that determine the number of times an execution set is to be executed and/or the number of parallel instances of the execution set that are to be executed.

Very commonly, the execution set discovery process uses an allocation algorithm that determines subsets of components within a data processing graph that will be applied as sets of input elements to an unordered set of data elements. The allocation algorithm traverses the data processing graph and assigns each component to a subset based on an allocation rule. As will be apparent in the following example, a given data processing graph can include multiple execution sets nested at different levels of the execution set hierarchy.

In the data processing diagram described herein, there are two types of data ports: scalar data ports and set data ports. Typically, if a pair of linked components (i.e., upstream component A 402 and downstream component B 404 of FIG. 4A to FIG. 4D) are connected by a link between ports of the same type, they will be in the same execution set (unless they are in different execution sets for another reason) by default. In FIG. 4A , component A 402 has an output port 406, which has a scalar type, and component B 404 has an input port 408, which has a scalar type. Since the link 410 between component A 402 and component B 404 connects two scalar ports, component A 402 and component B 404 are in the same execution set in this example. In FIG. 4A , since the link between component A 402 and component B 404 is a scalar to scalar link, 0 data elements or 1 data element are transmitted between upstream component A 402 and downstream component B 404 by link 410. When processing by upstream component A 402 is complete, the data element is passed over link 410 , unless upstream component A 402 is disabled (as described above), in which case no data element is passed over link 410 .

Referring to Figure 4B , component A 402 has an output port 412 of a collection type, and component B 404 has an input port 414 of a collection type. Because link 410 between component A 402 and component B 404 connects the two collection-type ports, in this example, component A 402 and component B 404 are also in the same execution set. In Figure 4B , because link 410 between component A 402 and component B 404 is a collection-to-collection link, a set of data elements is passed between the upstream component and the downstream component via link 410.

When there is a mismatch between the port types at both ends of a link, there is an implicit change in the level of the execution set hierarchy. Specifically, the mismatched port represents the entry point or exit point of the execution set at a particular level of the execution set hierarchy. In some examples, an execution set entry point is defined as a link between a set-type output port and a scalar-type input port. In Figure 4C, an example of an execution group entry point 424 is shown at link 410 between component A 402 and component B 404 because the output port 416 of component A 402 is a set-type port, while the input port 418 of component B 404 is a scalar-type port.

In some examples, an execution group exit point is defined as a link between a scalar output port and a collective input port. Referring to FIG4D , an example of an execution group exit point 426 is shown at link 410 between component A 402 and component B 404 because output port 420 of component A 402 is a scalar port, while input port 422 of component B 404 is a collective port.

The allocation algorithm implemented before compilation and/or interpretation by compiler/interpreter 120 uses execution set entry points and execution set exit points to discover execution sets that exist in the data processing graph.

2.1 Stack-based allocation algorithm

For illustration purposes, in a first example, the data processing graph has a simple one-dimensional graph structure, and a simpler allocation algorithm is shown using a stack-based algorithm. In the stack-based allocation algorithm, each component in the data processing graph is labeled with one or more "ID strings" consisting of integers separated by separator characters '/'. The number of times the separator character '/' appears in the ID string of a given component determines the level of the component in the execution set hierarchy. In some examples, a component can have multiple input links and therefore can have multiple ID strings. In this case, the algorithm has rules, described in more detail below, for determining which ID string to use.

In one example of a stack-based allocation algorithm, compiler/interpreter 120 traverses the data processing graph in an upstream to downstream direction according to the following process: Initially, the most upstream component is marked with an ID string '0', indicating that it is a component at the root level in the execution set hierarchy.

The links and components on the path from the most upstream component to the most downstream component are then traversed. If a link is encountered between a collection-type output port of an upstream component and a collection-type input port of a downstream component, the ID string of the upstream component is propagated to the downstream component. Similarly, if a link is encountered between a scalar-type output port of an upstream component and a scalar-type input port of a downstream component, the ID string of the upstream component is propagated to the downstream component.

If a link is encountered between a collection-type output port of an upstream component and a scalar-type input port of a downstream component, the downstream component is assigned a tag consisting of the tag of the upstream component with '/n' appended to the end, where n is <the largest integer among all existing ID string integers> + 1. If a link is encountered between a scalar-type output port of an upstream component and a collection-type input port of a downstream component, the downstream component is assigned a tag consisting of the tag of the upstream component minus its rightmost ID string integer (and its separator character '/').

In some examples, various conditions may be considered illegal and will cause errors in the algorithm (eg, if a component has two different ID strings at the same level in the execution set hierarchy, or if there is a loop in the execution set).

Referring to FIG5 , applying the stack-based allocation algorithm described above to an exemplary data processing graph 550 results in the discovery of two execution sets (besides the root, a "level 0" execution set 551): a first "level 1" execution set 570 and a second "level 2" execution set 572 nested within the first "level 1" execution set 670. To achieve the discovery of the two execution sets 570 and 572, the stack-based allocation algorithm first marks the upstream component, the first data set 656 having the ID string '0'. The stack-based allocation algorithm then traverses the components of a one-dimensional path through the data processing graph 550. While traversing this path, the stack-based allocation algorithm first traverses the link from the first data set 556 to the first component 558. Because the output port of the first data set 556 is a set-type output port and the input port of the first component 558 is a scalar-type input port, the first component 558 is assigned the ID string '0/1', which is the ID string of the first data set 556 appended with '/1' at the end, where 1 is the maximum value of all existing ID string integers + 1. Typically, appending '/1' to the ID string of the first component 558 is an indication of a transition from the root "level 0" execution set 551 to the "level 1" execution set 570. In some examples, this transition is represented using the first execution set entry point indicator 557.

The allocation algorithm then traverses the link from the first component 558 to the second component 560. Since the output port of the first component 558 is a set-type output port and the input port of the second component 560 is a scalar-type input port, the second component 560 is assigned the ID string '0/1/2', which is the ID string of the first component 558 with '/2' appended to the end, where 2 is the maximum value of all existing ID string integers + 1. Typically, appending '/2' to the ID string of the second component 560 is an indication of a transition from a "level 1" execution set 570 to a "level 2" execution set 572. In some examples, this transition is indicated using a second execution set entry point indicator 559.

The allocation algorithm then traverses the link from the second component 560 to the third component 562. Since the output port of the second component 560 is a scalar output port and the input port of the third component 562 is a scalar input port, the ID string of the second component 560 (i.e., '0/1/2') is propagated to the third component 562.

The allocation algorithm then traverses the link from the third component 562 to the fourth component 564. Since the output port of the third component 562 is a scalar output port and the input port of the fourth component 564 is a set input port, the fourth component is assigned the ID string '0/1', which is the ID string of the third component 562 minus its rightmost ID string '2' (and its separator character '/'). Typically, removing '/2' from the ID string of the third component 562 is an indication of a transition from a "level 2" execution set 572 to a "level 1" execution set 570. In some examples, this transition is represented using a first execution set exit point indicator 563.

The allocation algorithm then traverses the link from the fourth component 564 to the fifth component 566. Since the output port of the fourth component 564 is a scalar output port and the input port of the fifth component 566 is a set input port, the fifth component 566 is assigned an ID string of '0', which is the ID string of the fourth component 564 minus its rightmost ID string integer (and its separator character '/'). Typically, removing '/1' from the ID string of the fourth component 564 is an indication of a transition from the "level 1" execution set 570 to the root "level 0" execution set 551. In some examples, this transition is represented using a second execution set exit point indicator 565.

Finally, the allocation algorithm traverses the link from the fifth component 566 to the second data set 568. Since the output port of the fifth component 566 is a collection-type output port and the input port of the second data set 568 is a collection-type input port, the ID string of the fifth component 566 (i.e., ‘0’) is propagated to the second data set 568.

In some examples, in addition to entry point indicators and exit point indicators, additional visual cues within the user interface can be used to visually represent the flow of a collection of data elements and the changes between individual scalar data elements. For example, a line representing a link can be thicker between a collection port and an indicator and thinner between an indicator and a scalar port.

The result of the stack-based allocation algorithm includes a version of the data processing graph 550 in which each component is labeled with an ID string. In the example of Figure 5, the first data set 556, the second data set 568, and the fifth component 566 are all labeled with the ID string '0'. The first component 558 and the fourth component 564 are labeled with the ID string '0'. The second component 560 and the third component 562 are each labeled with the ID string '0/1/2'.

Each unique ID string represents a unique execution set in the execution set hierarchy. Those components with ID string ‘0’ are grouped into the root “Level 0” execution set 551 in the execution hierarchy. Those components with ID string ‘0/1’ are grouped into the “Level 1” execution set 670 nested within the root execution set 651 (where ‘0/1’ can be read as execution set 1 nested within execution set 0). Those components with ID string ‘0/1/2’ are grouped into the “Level 2” execution set 572, which is nested within the root “Level 0” execution set 551 and the “Level 1” execution set 570.

2.2 Allocation algorithm based on global mapping

In some examples, for more general data processing graphs, a stack-based allocation algorithm may not be sufficient to correctly determine the execution set hierarchy. For example, in a general data processing graph, any given component may have multiple input ports and/or multiple output ports, making general data processing graphs incompatible with a stack-based approach. In such examples, a global map-based allocation algorithm is used to determine the execution set hierarchy.

The global map-based allocation algorithm exploits the fact that the data processing graph is restricted to a directed acyclic graph. The directed acyclic graph can be processed in a topologically sorted order to ensure that each component of the graph is processed only after all components immediately upstream of it have been processed. Since all components immediately upstream of a component are known to have been processed, the ID string of the component can be determined by selecting the ID string of the most deeply nested component immediately upstream of the component (in the execution set hierarchy).

In some examples, the global map-based allocation algorithm uses a standard topological sorting algorithm, such as Kahn's algorithm, to obtain a topologically sorted order for a given data processing graph. The Kahn algorithm is outlined in the following pseudocode:

L←The empty list that will contain the elements to be sorted

S←the set of all nodes without incoming edges

As long as S is non-empty, do the following:

Remove node n from S

Add n to the end of L

For each node m that has an edge e from n to m, do the following:

Remove edge e from the graph

If m has no other incoming edges, then

Insert m into S

If the graph has multiple edges, then

Returns error (graph has at least one cycle)

otherwise

Return L (the order after topological sorting)

After determining the topologically sorted order, a global map-based allocation algorithm traverses the components of the data processing graph in the topologically sorted order to determine the appropriate ID string (or ID number for short) for each component. Specifically, as the components are traversed, each component of the data processing graph copies its ID string to its output port. Components that are directly downstream of an upstream component and have not been separated from the upstream component by an execution set entry point or an execution set exit point read the ID string from the upstream component's output port and use that ID string as their ID string.

For an upstream component that is separated from a downstream component due to an execution set entry point, a new ID string is allocated at the execution set entry point and provided to the downstream component to be used as its ID string. The mapping of the upstream component's ID string to the downstream component's ID string (i.e., the parent/child mapping) is stored in the global mapping data store for later use.

For an upstream component that is separated from a downstream component due to an execution set exit point, the ID string at the output port of the upstream component is read by the execution set exit point. The global mapping data store is then queried to determine the parent ID string of the ID string at the output port. The parent ID string is provided to the downstream component to use as its ID string.

6 , an example of an exemplary general two-dimensional data processing graph 628 is analyzed using the global mapping-based assignment algorithm described above. The data processing graph 628 includes a first data set (D1) 632, a first component (C1) 638, a second component (C2) 640, a third component (C3) 645, a fourth component (C4) 646, a fifth component (C5) 642, a sixth component (C6) 644, and a second data set (D2) 634. Before assigning ID strings to the components of the data processing graph 628, a topological sorting algorithm (e.g., Kahn's algorithm) is applied to the data processing graph, resulting in the following topologically sorted order: D1, C1, C2, C3, C4, C5, C6, D2.

Utilizing the determined topologically sorted order, the global map-based allocation algorithm traverses the components of the data processing graph in the topologically sorted order to determine the appropriate ID string for each component, resulting in the discovery of a "level 1" execution set 630 and a "level 2" execution set 631 (in addition to the root "level 0" execution set). To achieve discovery of the two execution sets 630, 631, the global map-based allocation algorithm first marks the most upstream component, the first data set (D1) 632 with ID string '0'. The stack-based allocation algorithm then traverses the components and links of the data processing graph 628 in the topologically sorted order.

The global mapping-based allocation algorithm first traverses the link from the first data set (D1) 632 to the first component (C1) 638. Since the output port of the first data set (D1) 632 is a set-type output port and the input port of the first component (C1) 638 is a set-type input port, no execution set entry point or exit point is identified, and the ID string of the first data set (D1) 632 (i.e., '0') is read from the output port of the first data set (D1) 632 and the ID string is allocated to the first component (C1) 638.

The allocation algorithm then traverses the link between the first component (C1) 638 and the second component (C2) 640. Since the output port of the first component (C1) 638 is a set-type output port and the input port of the second component (C2) 640 is a scalar-type input port, a first execution set entry point 639 is identified between the two components 638, 640. At the first execution set entry point 639, a new ID string (i.e., '1') is allocated and assigned as the ID string of the second component (C2) 640. A mapping 653 of the parent ID string of the first execution set entry point 639 (i.e., '0') to the child ID string of the first execution set entry point 639 (i.e., '1') is stored in the global mapping data store 649 for later use.

The allocation algorithm then traverses the link from the second component (C2) 640 to the third component (C3) 645. Since the output port of the second component (C2) 640 is a set-type output port and the input port of the third component 645 is a scalar-type input port, a second execution set entry point 641 is identified between the two components 640, 645. At the second execution set entry point 641, a new ID string (i.e., '2') is allocated and assigned as the ID string of the third component (C3) 645. A mapping 651 of the parent ID string of the second execution set entry point 641 (i.e., '1') to the child ID string of the second execution set entry point 641 (i.e., '2') is stored in the global mapping data store 649 for later use.

The allocation algorithm then traverses the link from the third component (C3) 645 to the fourth component (C4) 646. Since the output port of the third component (C3) 645 is a set-type output port and the input port of the fourth component (C4) 646 is a set-type input port, no execution set entry point or exit point is identified, and the ID string (i.e., '2') of the third component (C3) 645 is read from the output port of the third component (C3) 645 and allocated to the fourth component (C4) 646.

The allocation algorithm then traverses the link from the fourth component (C4) 646 to the fifth component (C5) 642. Since the output port of the fourth component (C4) 646 is a scalar output port and the input port of the fifth component (C5) 642 is a set input port, a first execution set exit point 647 is identified between the two components 646, 642. At the first execution set exit point 647, the ID string of the fourth component (C4) 646 is read from the output port of the fourth component (C4) 646 and the ID string is used to query the global mapping data store 649. The global mapping data store 649 returns the parent-child relationship 651 (i.e., '1/2') stored in association with the second execution set entry point 641. The parent ID string (i.e., '1') in the parent/child relationship 651 is assigned as the ID string of the fifth component (C5) 642.

The allocation algorithm then traverses the link from the fifth component (C5) 642 to the sixth component (C6) 644. Since the output port of the fifth component (C5) 642 is a scalar output port and the input port of the sixth component (C6) 644 is a set input port, a second execution set exit point 643 is identified between the two components 642, 644. At the second execution set exit point 643, the ID string of the fifth component (C5) 642 is read from the output port of the fifth component (C5) 642 and the ID string is used to query the global mapping data store 649. The global mapping data store 649 returns the parent/child relationship 653 (i.e., '0/1') stored in association with the first execution set entry point 639. The parent ID string (i.e., '0') in the parent/child relationship 653 is assigned as the ID string of the sixth component (C6) 644.

Finally, the allocation algorithm traverses the link from the sixth component (C6) 644 to the second data set (D2) 634. Since the output port of the sixth component (C6) 644 is a collection-type output port and the input port of the second data set (D2) 634 is a collection-type input port, no execution set entry point or exit point is identified, and the ID string of the sixth component (C6) 644 (i.e., ‘0’) is read from the output port of the sixth component (C6) 644 and allocated to the second data set (D2) 634.

The result of the global mapping-based allocation algorithm includes a version of the data processing graph 628 in which each component is labeled with an ID string. In the example of Figure 6, the first data set (D1) 632, the first component (C1) 638, the sixth component (C6) 644, and the second data set (D2) 634 are all labeled with an ID string of '0'. The second component (C2) 640 and the fifth component (C5) 642 are both labeled with an ID string of '1'. The third component (C3) 645 and the fourth component (C4) 646 are both labeled with an ID string of '2'.

Each unique ID string represents a unique execution set in the execution set hierarchy. Those components with ID string ‘0’ are grouped into the root “Level 0” execution set 629 in the execution hierarchy. Those components with ID string ‘1’ are grouped into the “Level 1” execution set 630 nested within the root execution set 629. Those components with ID string ‘2’ are grouped into the “Level 2” execution set 631, which is nested within the root “Level 0” execution set 629 and further nested within the “Level 1” execution set 630.

2.3 User-defined execution sets

In the above examples, (multiple) allocation algorithms are used to automatically discover execution sets that exist in the data processing graph without any user intervention. However, in some examples, the user may require functionality in addition to that provided by the allocation algorithm. In this case, the user can explicitly add execution set entry points and exit points to explicitly define where the execution sets begin and/or end. Referring to Figure 7, the data processing graph 776 includes a first data set 774, a first component 778, a second component 780, and a second data set 790. Applying the above allocation algorithm to the data processing graph 776 will result in the discovery of a single execution set that includes the first component 778 and the second component 780. However, in this case, the user has explicitly defined two execution sets for the data processing graph 776 (i.e., a first execution set 782 and a second execution set 786). Specifically, the user has inserted the execution set exit point component 784 into the link coming out of the output port of the first component 778, and has inserted the execution set entry point 788 into the link going into the input port of the second component 780. By adding the execution set exit point 784 and the execution set entry point 788 to the link between the first component 778 and the second component 780, the user has essentially broken down what was originally a single execution set into two separate execution sets 782, 786.

In some examples, a user defines all execution set entry points and exit points for a data processing graph. In other examples, a user defines some execution set entry points and exit points and then leaves it to an allocation algorithm to discover the remaining execution set entry points and exit points for the data processing graph.

2.4 Same set relations

In some examples, a user may wish to explicitly specify to which execution set a given component belongs. For example, as shown in Figure 8A, a data processing diagram 892 includes a first execution set 894 that receives data elements from a create data component 896 and a read table component 898. These components are similar to input file components, except that they have different sources for the sets of data elements they provide. For the create data component 896, there is no scalar input port for specifying a file location, but there is an (optional) scalar input port for specifying how multiple record data elements will be generated, and there are also parameters for specifying how each data element will be generated. For the read table component 898, there is no scalar input port for specifying a file location, but there is an (optional) scalar input port for specifying a table in a database. The first execution set 894 includes a first component 891 and a second component 893, which together process the data elements from the create data component 896 and the read table component 898 to generate an output that is provided to a first data set 899.

In Figure 8A, the table read component 898 is external to the first execution set 894, meaning that it runs once and outputs a set of data elements from its set-type output port. The set of data elements traverses the boundary of the first execution set 894 and is provided to the set-type input port on the first component 891. For each parallel instance of the components in the execution set 894, a copy of the set of data elements is created at the set-type input port on the first component 891. In general, whether the link comes from a set port, a scalar port, or a control port, links between components assigned to different execution sets copy the data elements or control elements to all instances of the link flowing into the execution set and aggregate the data elements or control elements from all instances of the link flowing out of the execution set. Data elements are aggregated into sets, and control elements are aggregated into vectors, which can be processed appropriately (including possibly marking them as errors) according to the control logic of the downstream component.

8B , in some examples, a user may request that a table reading component 898 be executed for each parallel instance of a component in an execution set 894. To accomplish this, the user may specify a "same set" relationship between the table reading component 898 and the first component 891. As a result of the user specifying the "same set" relationship, the table reading component 898 is moved to the same execution set as the first component 891 (i.e., the first execution set 894). Because the table reading component 898 is included in the first execution set 894, each parallel instance of the component in the first execution set 894 executes an instance of the table reading component 898.

In some examples, a user can specify a "same set" relationship by selecting a target execution set from a menu associated with a source execution set, or by dragging a component from a source execution set to a target execution set (e.g., via a user interface described in more detail below). In some examples, error checking is implemented to verify that the dragged component can legally be located in the target execution set. For example, one possible requirement that can be imposed on any two components that are to have a "same set" relationship with each other is that there must be at least one path through the data processing graph that includes those components.

2.5 Collection Data Replication

In some examples, multiple components in an execution set can each have a scalar input port connected to a single aggregate output port of an upstream component via an execution set entry point. Similarly, multiple components in an execution set can each have a scalar output port connected to a single aggregate input port of a component downstream of the execution set.

In some examples, to provide the same data from the collection-type output ports of multiple components to the scalar input ports, the execution set entry point creates a copy of each data element from the collection for each scalar input port and provides the copy to their corresponding scalar input ports. Similarly, to merge data output by the scalar output ports of multiple components (from different corresponding iterations of the execution set), the execution set exit point can receive output data elements from the multiple scalar output ports, merge the output data elements, and then provide the merged output data elements to the collection input port of the downstream component. Typically, the collection input port of the downstream component is configured to process the merged data elements.

As shown in FIG9 , data processing graph 923 includes a first data set 924, a second data set 926, and an execution set 928. Execution set 928 includes two components: a first component 930 and a second component 932. First data set 924 has a collection output port 934 that is connected to an execution set entry point 936 of execution set 928 and provides a collection of data elements thereto. Second data set 926 has a collection input port 938 that is connected to an execution set exit point 940 of execution set 928 and receives a collection of data elements therefrom.

Within execution set 928, first component 930 has a first scalar input port 942, and second component 932 has a second scalar input port 944. First scalar input port 942 and second scalar input port 944 are both connected to an execution set entry point 936 and receive respective data elements from the execution set entry point 936. As described above, execution set entry point 936 replicates the data elements received from set output port 934 to provide a copy of each data element of the set of data elements to each scalar input port connected to execution set entry point 936. In Figure 9, execution set entry point 936 creates two copies of each data element and provides one copy to first scalar input port 942 and the other copy to second scalar input port 944. As can be seen in the figure, in some examples, the visual representation of execution set entry point 936 in the graphical user interface provides an indication of how many copies of the data element are created by execution set entry point 936. Furthermore, in other examples, the different entry point indicators representing different copies of the replica may be separated and distributed around the boundary of the execution set as many components as there are copies of each replicated data element within the execution set that need to be provided from the aggregate output port feeding the execution set.

The first component 930 and the second component 932 process their respective data elements and provide their respective processed data elements to the execution set exit point 940 via scalar output ports 946 and 948. In some examples, the execution set exit point 940 groups the processed data elements into pairs and outputs the processed data element pairs to the collection input port 938 of the second data set 926. As can be seen in the figure, in some examples, the visual representation of the execution set exit point 940 in the graphical user interface provides an indication of how many copies of the data elements are grouped by the execution set entry point 936.

2.6 Resource Locking

In some examples, components in a given execution set can run multiple times in parallel instances. In some examples, components of the parallel running instances may need to access shared resources. To prevent race conditions and other issues associated with multiple processes accessing shared resources, a locking mechanism can be used. Typically, a locking mechanism allows an instance of a component in an execution set to obtain a runtime lock on a shared resource for the duration of the instance's execution time. While the instance holds the shared resource locked, only components in the instance can access the shared resource, and components in other instances must wait for the lock to be released. After the instance completes, it releases the runtime lock, allowing other instances to access the shared resource. The locking mechanism must lock and unlock shared resources within a single execution set (e.g., using an explicit lock component at the upstream end and an explicit unlock component at the downstream end). In some embodiments, such "locked execution sets" cannot be nested or overlap with each other.

2.7 Others

Note that although the global map-based assignment algorithm is described with respect to a two-dimensional data processing graph, the algorithm can also be used to discover execution sets for a one-dimensional data processing graph.

In general, execution sets can be nested arbitrarily.

Typically, an execution set has at most one driving data element received from a linked output set port for each instance of the execution set. However, multiple scalar input ports can receive the same data element if the same data element is explicitly or implicitly replicated across execution set boundaries.

Typically, all output scalar ports with links that cross the execution set boundary have all data elements from each of the multiple instances of the execution set aggregated into the same set that is provided to the linked input capture ports. However, if the execution set has only a single instance, then output scalar ports with links that cross the execution set boundary can link to input scalar ports.

In general, a link between two ports of the same type can traverse the execution set boundary, assuming that the traversal of the execution set does not cause any cycles in the data processing graph.

In some examples, each execution set is assigned a unique identifier by default (e.g., '1'). In other examples, each execution set can be assigned an execution set ID path (e.g., '1/3/6'). In some examples, the user explicitly provides an execution set ID string. The execution set ID string is not necessarily unique. In the case where the execution set ID string is not unique, the execution set ID string can be combined with the execution set ID string of its parent, grandparent, etc. to form a unique ID string.

In some examples, the global mapping-based assignment algorithm causes a component to be assigned an ID string corresponding to the most deeply nested execution set. In some examples, when an execution set is assigned an execution set ID path, the execution set ID path is not necessarily unique. To compensate for the situation where the execution set ID path is not unique, a constraint is imposed on the execution set ID path requiring that the execution set ID path upstream of a given execution set must be "compatible", where two execution set ID paths are compatible if and only if they are identical, or one is a proper prefix of the other. For example:

/1/2/3 and /1/2/3 are compatible

/1/2/3 and /1/2 are compatible

/1/2 and /1/2/3 are compatible

1/2/3 and /1 compatible

1/2/3 and 1/4 are incompatible

/1/2/3 and /1/4/5 are incompatible

The above embodiments impose essentially no ordering/concurrency constraints on the execution of instances of a scalar block. However, in some embodiments, additional inputs are provided to control the allowed concurrency and required serialization of subsets of data elements received from the set fed to the execution set. In some embodiments, sequential processing according to a partial ordering may be imposed on some subsets of data elements.

By default, instances of an execution set can run completely in parallel. However, in some cases, the user may desire different behavior. For example, if the data being processed is account-level data, the user may wish to enforce certain restrictions on the data within each account. For example, the user may want to force serial execution. In this case, any degree of parallelism across accounts can be allowed, but two data elements of the same account cannot be processed at the same time (i.e., concurrently). Optionally, an additional restriction can be in-order processing, such that two data elements of the same account may not be processed out of order according to the order defined by the key or, for example, by the order in which they were received.

To achieve this, a serialization key can be provided for the execution set. All data elements with the same serialization key value must be processed serially, and in some cases must be processed in a well-defined order. One way for the runtime system to enforce serial execution of data elements with the same serialization key is to partition the execution set instances by the serialization key: instances whose driving data elements have a specific serialization key (or a hash value of the serialization key) are assigned to execute on a specific compute node 152. At runtime, the system can ensure that work is evenly distributed across multiple compute nodes 152 by scanning the set of data elements to ensure that the queue of runnable tasks remains full. In cases where a well-defined order is not required (for example in a set), the order can be the same order in which they are produced from the output port (even the set output port), or the same order as the order associated with different collation keys that control the processing order within the serialization key group. In some cases, the execution set can be forced to run completely serially by providing a predefined value as the serialization key.

In some embodiments, it may be possible to appear that order is preserved even if processing is not performed strictly according to that order. If the data at both the input and output of an execution set is associated with a particular order (e.g., the order of elements within a vector), then the user may wish to preserve that order. Even if there is no serialization in the processing of the data elements, the output data elements may be sorted, for example using a sort key that was carried with the data elements when they were processed, to restore the order associated with the corresponding set of input data elements. Alternatively, output data elements produced in parallel may be merged in the same order as they entered the execution set, without necessarily requiring an explicit sort operation.

Various computational characteristics associated with the execution code prepared for an execution set can be configured by the compiler/interpreter 120, with or without input from a user. For example, the embedded information described above for indicating how to implement tasks corresponding to components within a particular execution set can include any of the following. The information can include a compiler comment indicating that the tasks are to be executed entirely serially (i.e., without parallelism). The information can include a compiler comment indicating that the tasks are to be implemented with as much parallelism as allowed by the ordering constraints. The information can include a compiler comment indicating that tasks associated with the same key value are to be executed serially and tasks associated with different key values are to be executed in parallel (i.e., serialization by key as described above).

Compiler commentary or modifiers can be used to indicate any of a variety of computational characteristics:

Concurrency (e.g., parallel, serial, key-by-key serial as described above)

precedence between different execution sets (e.g., all tasks in one execution set occur after all tasks in another execution set)

Transactional (for example, executing a set of tasks is handled as a database transaction)

Resource locking (e.g., tasks of an execution set implement locks on specific resources (e.g., shared variables), allowing tasks to access the resource as an atomic unit)

Sorting (e.g., preserving the ordering between data elements)

Tuple size (e.g., the number of data elements that will be operated on by each instance of the execution set)

The compiler/interpreter 120 can determine these characteristics based on automatic analysis of the properties of the execution set or data processing graph as a whole, and/or based on receiving input from a user (e.g., user annotations in the graph). For example, if a key value is referenced in an execution set, a compiler annotation can indicate that the key is serialized. If a resource is used within an execution set, a compiler modifier can allow the resource to be locked/unlocked before/after the execution set. If there are database operations within the execution set, each instance of the execution set can be configured to execute as a database transaction. If the number of available cores can be determined at compile time, a compiler annotation can indicate that each core will execute an instance of the execution set on a tuple of data items consisting of a number of data items equal to the total size of the set divided by the number of cores.

Compiler annotations and modifiers can be added to code prepared in a target language, such as a suitable high-level language (e.g., DML) or low-level executable code, or to a target intermediate form of a data processing graph. For example, compiler/interpreter 120 can insert a component into a data processing graph that explicitly indicates the entry or exit points to an execution set, or a component used to start/end a transaction can be placed at the entry/exit points of a component set used to process a transaction, or a component can be used to lock/unlock resources. Alternatively, compiler/interpreter 120 can add modifiers to modify a type of data flow link.

3. User interface of data processing diagram

In some examples, a user interface allows a user to develop a data processing graph by dragging components onto a canvas and connecting the components' ports together using links. In some examples, the user interface repeatedly applies the above-described allocation algorithm to the data processing graph as the user develops the data processing graph. For example, as the user adds components to the data processing graph being developed, the allocation algorithm can be applied to the graph with the added components. The resulting execution sets discovered by the allocation algorithm can then be displayed, for example, as boxes drawn around the components in the user interface, or as arbitrarily shaped areas enclosing the components, which can be distinguished by unique colors, shading, textures, or markings used to present areas containing multiple components in the same execution set. In some examples, the user can then modify the execution sets discovered by the allocation algorithm by adding components to or removing components from the execution set. In some examples, the allocation algorithm verifies that the modified execution sets are legal. For example, there may be some configurations of components and links between various ports that can potentially be divided into multiple execution sets in any of a variety of legal ways. In such ambiguous cases, the allocation algorithm may default to selecting one allocation of the execution set, but the user may have desired a different allocation of the execution set, in which case the user can modify the allocation (e.g., by inserting an exit point to close the execution set earlier in the component chain). Alternatively, the allocation algorithm may be configured to identify ambiguous configurations in which multiple legal allocations are possible, and prompt the user for input to select one configuration.

10A , a user has dragged three components (a first data set 1022, a first computation component 1024, and a second data set 1026) onto the canvas 1028 of the data processing graph development user interface. The user has not yet connected the ports of the components 1022, 1024, and 1026 together using links, and the allocation algorithm has not yet discovered any execution sets (other than the root execution set) in the data processing graph.

10B , when a user connects the ports of components 1022, 1024, and 1026 using links, the allocation algorithm automatically discovers a first execution set 1030, which includes the first computing component 1024. First execution set 1030 is displayed to the user via the user interface. As the user continues to add components and links to the graph, the allocation algorithm automatically discovers execution sets and displays them via the user interface.

10C , in some examples, the user may need to break a link (e.g., insert another component into the link). In such an example, if the allocation algorithm is allowed to reanalyze the data processing graph, the first execution set 1030 will be removed, potentially causing interruption and loss of the user's work.

To avoid this disruption, when a user removes a flow or component from a data processing graph, the allocation algorithm can be left unexecuted, leaving the remaining components and their execution set associations intact. For example, in FIG10C , even though its input and output ports are disconnected, the first component 1024 is still included in the first execution set 1030. In some examples, when the disconnected component is reconnected, the allocation algorithm is allowed to automatically discover and display any execution sets associated with the reconnected component.

In some examples, if a component of a data processing graph does not have an explicit (e.g., user-defined) execution set designation, the allocation algorithm is allowed to discover which execution set the component belongs to. Otherwise, if the component has an explicit user-defined execution set designation, the allocation algorithm is not allowed to select which execution set the component is included in. For example, if a user manually moves a component to a given execution set, the allocation algorithm is not allowed to include the component in any execution set other than the user-specified execution set. That is, any user modifications to the data processing graph cannot be overridden by the allocation algorithm.

In some examples, the user interface allows a user to promote a component to a given execution set and/or demote a component from a given execution set using gestures or other interactions via an input device. In some examples, the user can use a menu option or other prompt to promote or demote a component. In other examples, the user can simply drag a component to a desired execution set in the user interface.

In some examples, the user interface allows a user to specify one or more constraints for an execution set in a data processing graph. For example, a user can constrain an execution to run no more than N times in parallel at a given time.

In some examples, compiler/interpreter 120 receives a representation of a data processing graph that includes a mix of manually defined execution sets and execution sets discovered by an allocation algorithm.

In some examples, users can use the interface to define another type of execution set, called an enable/disable execution set. For example, a user can draw a box around one or more components they wish to include in an enable/disable execution set. An enable/disable execution set includes one or more components and has scalar input ports. If the scalar output port of an upstream component provides a data element to the scalar input port of the enable/disable execution set, the components in the enable/disable execution set are allowed to execute. If the scalar output port of an upstream component provides a zero data element to the scalar input port of the enable/disable execution set, the components included in the enable/disable execution set are disabled. Any execution set (including an enable/disable execution set) can include control input and output ports, which can be used to determine whether the entire execution set will be executed and whether control signals are propagated to other components or execution sets. If an execution set is parallelized (i.e., has multiple instances), the input control ports must be activated before any instance is executed, and the output control ports must be activated after all instances have completed execution. In some examples, these input and output control ports are provided by placing visual representations of the ports on the boundaries of the execution set. In other examples, these input and output control ports are provided by placing them on an additional component in front of the execution set. For example, this additional "for all components" can be inserted (e.g., automatically by a user interface or manually by a user) between the upstream aggregate output data port and the entry point indicator, or in place of the entry point indicator (i.e., between the upstream aggregate output data port and the driving input scalar data port).

As noted above with reference to FIG. 7 , in some examples, a user may explicitly define execution set entry and exit points by placing execution set entry and exit point components along the flow of a data processing graph.

In some examples, the user interface provides real-time feedback to notify users when their diagrams include illegal operations. For example, if there is a conflict caused by a component in a user-specified execution set, the allocation algorithm can issue a warning to the user through the user interface. To provide real-time feedback, the allocation algorithm applies validation rules to the data processing diagram to notify the user whether the data processing diagram is legal. Referring to Figure 11A, an example of an illegal data processing diagram configuration 1195 includes two data sources: a first data source 1191 that feeds a first set of data elements to a scalar port of a first component 1102 in a first execution set 1197, and a second data source 1198 that feeds a second set of data elements to a scalar port of a second component 1104 in a second execution set 1199. The second execution set 1199 outputs a third set of data elements, which is then input to a scalar data port of a third component 1106 in the first execution set 1197. Because two different sets of data elements are connected to different scalar ports in first execution set 1197, there is no way to know how many parallel instances of the component in first execution set 1197 should be instantiated (because one instance of the component is generated for each data element present at the boundary of first execution set 1197). In some examples, the user is notified of this conflict by displaying an error indicator 1108 on, for example, second component 1104.

11B , another example of an illegal data processing configuration 1110 includes a data source 1112 feeding a set of data elements to a scalar input port of a first component 1114 in a first execution set 1116. The scalar output of the first component 1114 provides its output as a set of data to a collection port of a second component 1118 external to the first execution set 1116. The second component 1118 provides the set of data elements from the collection output port to a scalar data port of a third component 1120 in the first execution set 1116.

An "execution set loop" is defined by passing a set of data elements from a collection output port of a first component 1114 outside a first execution set 1116, processing the set of data elements at a second component 1118, and then passing the processed set of data elements back to a scalar port of a third component 1120.

In general, execution set loops are illegal because they violate execution ordering. For example, it is generally permitted to have additional flows entering or leaving an execution set because, for inputs, input data can be cached before the execution set is executed, and for outputs, output data can be aggregated after the execution set completes execution. However, this is not possible if external components need to run before and after the execution set.

In some examples, the user is notified that a set loop is being executed by displaying an error indicator 1108 on one or more components.

In some examples, a data processing graph is considered illegal if each execution set entry point does not match at least one corresponding execution set exit point. Alternatively, an execution set with entry points but no corresponding exit points may be allowed as a user-defined execution set, even though it will not be automatically recognized by the allocation algorithm. In these cases, the execution set can end (without providing any output data elements) after the (one or more) most downstream components complete execution. In some examples, a data processing graph is considered illegal if each lock operation does not match a corresponding unlock operation. Alternatively, if not explicitly specified, an unlock operation may be inferred and only indicated as illegal if the inferred unlock operation would need to be in a different execution set than the lock operation. In some examples, a data processing graph is considered illegal if a lock operation and its corresponding unlock operation do not exist in the same execution set.

4. State Machine of Control Diagram

In preparing the data processing graph for execution, compiler/interpreter 120 also generates a control graph during a control graph generation process. In some implementations, generating the control graph includes generating executable code for performing tasks corresponding to various components and code corresponding to various inter-component links that determine the flow of data and control between tasks. This includes the transfer of data and control within the hierarchy of execution sets discovered by compiler/interpreter 120.

Part of generating such executable code includes generating a corresponding control graph for each execution set, including any enable/disable execution sets, in some data structure representation. Any nested execution sets within an execution set are treated as a single component representing the nested execution set for the purpose of generating the control graph. The ports of this representative component correspond to the ports of components within the nested execution sets that are connected to links that cross the boundaries of the nested execution sets. The compiler/interpreter 120 will then use the control graph to generate control code. This generated control code effectively implements a state machine that controls execution at runtime. In particular, once execution begins, the generated control code controls when a component or port transitions from one state of the state machine to another.

12A shows an example of how compiler/interpreter 120 combines a first component pair 1202 and a second component pair 1204 of a root execution set into a control graph 1206. In this example, first component pair 1202 includes a first component 1208 and a second component 1210 connected by respective aggregate data ports 1212, 1214. Second component pair 1204 includes a third component 1216 and a fourth component 1218 connected by respective scalar data ports 1220, 1222.

The compiler/interpreter 120 creates a control graph by adding a start component 1224 and a completion component 1226 and connecting the components to the start component 1224 and the completion component 1226 as dictated by the topology of the data processing graph. The start component and the completion component do not perform any computational tasks, but the compiler/interpreter 120 uses the start component and the completion component to manage control signals that are used to start the execution of certain components and to determine when all components in the execution set have completed execution.

To determine whether a particular component needs to be connected to the start component 1224, the compiler/interpreter 120 examines the input to the component to determine whether the component is not specified to start execution based on the existing link to the upstream serial port, which includes a control port and a scalar port as described above.

For example, if a component does not have a link to its control input port, it may never begin execution because there will never be a control signal to instruct it to begin. On the other hand, even if there is no control input, the arrival of data may trigger the execution of the component, depending on the type of data input the component has. For example, if a component has a scalar input port, then even if there is no control signal at its control input port, once the component sees data at its scalar input port, the component will still begin execution. On the other hand, if a component only has a collection data input, this situation will not occur. If such a component does not have a control input or a scalar data input to trigger execution, it will need a connection to the start component 1224.

In the context of Figure 12A, first component 1208 has neither control inputs nor scalar data inputs. Therefore, first component 1208 cannot start execution on its own. Therefore, first component 1208 must be linked to start component 1224. Third component 1216 also has neither control inputs nor scalar data inputs. Therefore, third component 1216 must also be linked to start component 1224.

The fourth component 1218 has no control inputs. However, it is connected to receive scalar data input from the third component 1216. Therefore, it will begin execution when it receives data through its input scalar port 1222. Therefore, the fourth component 1218 does not need to be connected to the start component 1224.

The second component 1210 is configured to receive data from the first component 1208. However, the data is received at the input aggregate port 1214 rather than at the input scalar port. As a result, similar to the first component, the second component 1210 must also be connected to the start component 1224.

The compiler/interpreter 120 also needs to identify which components will need to be connected to the completion component 1226.

Typically, when a component lacks a control output link or a data output link (of any type), it is connected to the completion component 1226. In the diagram on the left side of Figure 12A, only the second component 1210 and the fourth component 1218 meet this condition. Therefore, as shown on the right side of Figure 12A, only these two components are connected to the completion component 1226.

Figure 12B is similar to Figure 12A, except that a control link exists between the first component 1208 and the third component 1216 on the left side of the diagram. Consistent with the rule, it is no longer necessary to connect the third component 1216 to the start component 1224 in the resulting alternative control diagram 1206'.

The control diagram effectively defines a distributed state machine in which a component and its serial port transition from one state to another in response to transitions occurring in upstream components and serial ports. Typically, an upstream component will transition from one state to another, causing its output serial port to transition, which causes the linked serial input ports of downstream components to transition, which causes those downstream components to transition, and so on. An example of a specific type of state machine for implementing this behavior is described in more detail below with reference to a state transition diagram for a component and its serial port.

In order to provide the control to the conversion of state machine, compiler/interpreter 120 transplants extra control code in the code for performing the task represented by specific component.As used herein, " transplant (graft) " means the control code that is attached before, attached after or both attached before and attached after.The control code that is attached before is called " prequel (prologue) " code herein, and the control code that is attached after is called " end (epilogue) " code.The prequel code of component is carried out before component performs its task.The end code of component is carried out after component 610A has finished performing its task.

The ported control code examines stored state information, such as the value of an accumulator (e.g., a counter counting down to a value indicating that input is ready for the calling component) or the state of a flag (e.g., a flag set to a value indicating that the component has been disabled), to determine whether to cause one or more downstream components to perform their corresponding tasks.

In one embodiment, the prologue code monitors the status of the upstream output serial port and updates the status of the component's input serial port and the status of the component, while the epilogue code updates the component's output serial port after the component completes performing its task.

In another embodiment, instead of the preamble code of the downstream component monitoring the upstream output serial port, the end code of the upstream component updates the aggregate state of the downstream input serial ports and monitors the aggregate state to trigger the preamble code of the downstream component to execute at the appropriate time (e.g., when a counter initialized to the number of input serial ports reaches zero). Alternatively, instead of the counter counting down from the number of input ports (or counting up to the number of input ports), another form of accumulator can be used to store state information for triggering the component, such as a bitmap storing bits representing the states of different ports of different components.

As a result of this ported control code, completion of a task automatically results in the automatic execution of other tasks based on the occurrence of a set of one or more upstream logic states at the beginning and end of execution of a particular component, in a manner consistent with the data control dependencies represented by the control diagram, and in a manner that allows concurrent operation of multiple components and allows control using conditional control logic.

Figures 13A and 13B illustrate state transition diagrams for exemplary state machines that may be used for a component (state transition diagram 1300 in Figure 13A) and its serial port (state transition diagram 1310 in Figure 13B). The two state transition diagrams are similar, except that, since the active state 1304 is associated with ongoing execution and since only the component, not the port, performs execution, only the component can be in the active state 1304.

All possible states of the two state transition diagrams will be described, as well as the conditions required for each transition between the states, with reference to Figures 13A and 13B. All input and output ports involved in this description of the state transition diagram are serial ports, because the components in the control diagram only need to link serial ports (rather than collection ports). A specific component in the control diagram can be in one of the four logical states in the state transition diagram 1300. The first state is pending state 1302. This is the state that the component starts when the execution set associated with the control diagram begins execution. If any input port of the component is in pending state 1312, the component remains in pending state 1302. If the component happens to have no input port, it starts in pending state 1302, but is immediately eligible to transition out of pending state 1302.

From the pending state 1302 , the component can transition to the active state 1304 or the disabled state 1306 .

If a component has no input ports in the pending state 1312 and not all of its input ports are in the disabled state 1316 (i.e., at least one input port is in the completed state 1314), the component transitions to the active state 1304. Ports are "required" by default, but can be marked as "optional." An optional port can remain unconnected to another port without causing an error (although there may be a warning). Any optional port that is not connected is automatically in the completed state 1314. As long as the component is still performing its task, the component remains in the active state 1304. When a component is in the active state 1304, its multiple output ports can transition from the pending state 1312 to the completed state 1314 or the disabled state 1316 at different times or together. When the execution of its task is completed, the component transitions from the active state 1304 to the completed state 1308.

If the component's task has completed execution and all of its output ports are “resolved,” ie, no longer pending, the component transitions to the completed state 1308 .

A component is in the disabled state 1306 if its predecessor has triggered a transition to the disabled state 1306 due to custom control logic, or because all of its input ports are disabled, or because at least one of its required input ports is disabled, or because of an unhandled error in the component. All of the component's output ports also transition to the disabled state 1316 to propagate this disablement downstream.

For ports, the state transition rules depend on whether the port is an input port or an output port.

The initial state of a port is the pending state 1312. An input port generally follows the state of the upstream output port to which it is linked. Therefore, when an upstream output port transitions, the input ports linked to that output port in the control graph transition to the same state. An output port remains pending until the component determines what state the output port should resolve to during its active state.

As described above, input ports follow the upstream output ports to which they are linked. Thus, for an input port linked to a single upstream output port, the input port transitions to the completed state 1314 when the upstream output port to which it is linked transitions to the completed state 1314. If the input port is linked to multiple upstream output ports via multiple links, the input port transitions to the completed state 1314 after at least one of its upstream output ports transitions to the completed state 1314. Otherwise, if all upstream output ports transition to the disabled state 1316, the input port transitions to the disabled state 1316. Some embodiments use other logic than the default "OR logic" to determine whether to transition an input port to the completed state 1314 or the disabled state 1316 (e.g., "AND logic," where an input port transitions to the completed state 1314 only when all upstream output ports are in the completed state 1314). If a component's input data port resolves to the completed state 1314, the data element is ready for processing by the component. If the output data port of a component resolves to the done state 1314, then the data element is ready to be sent downstream from the component.

Consistent with the rule that input ports follow the state of the upstream output ports to which they are linked, an input port is determined to be in the disabled state 1316 when the upstream output port to which it is linked is determined to be in the disabled state 1316. An output port is determined to be in the disabled state 1316 either because a result computed by an active component determines that the output port should be disabled, or to propagate the disablement downstream from an upstream disabled component, or if there is an unhandled error in the component. In some embodiments, the compiler can optimize execution by disabling a tree of downstream components rooted at the disabled component, rather than having to propagate the disablement downstream component by component.

In other embodiments, various alternative state machines may be used, in which links between collection ports may also be included in the control diagram. In some such embodiments, the state transition diagram for the collection port may include active states in addition to the pending state, the completed state, and the disabled state, such as in the state transition diagram 1300 for the component. The collection port is in an active state when it produces data (as an output port) or consumes data (as an input port). For an input collection port, for example, once it is determined that not all input ports will be disabled, the active state may be triggered when the first data element is produced upstream. In some embodiments, there is no disabled state for a collection port. The conversion rules followed by the components in the control diagram that include the state transition for the collection port may handle the active state of the input collection port in the same manner as the completed state is handled for the input scalar port or control port.

5Computing Platform

1 , instances of components of a data processing graph are spawned as tasks in the context of executing the data processing graph and are typically executed in a plurality of compute nodes 152 of a computing platform 150. As discussed in more detail below, a controller 140 provides supervisory control aspects of the scheduling and execution trajectory of these tasks in order to achieve performance goals of the system related to, for example, distribution of computational load, reduction of communication or input/output overhead, and use of memory resources.

Typically, after translation by compiler/interpreter 120, the entire computation is expressed as a task-based specification 130 of procedures in the target language that can be executed by computing platform 150. These procedures utilize primitives such as "spawn" and "wait" and can include primitives within the procedure or call work procedures specified by the programmer for components in the high-level (e.g., graph-based) program specification 110.

In many cases, each instance of a component is implemented as a task, with some tasks implementing a single instance of a single component, some tasks implementing a single instance of multiple components that execute a set, and some tasks implementing sequential instances of a component. The specific mapping from components and their instances depends on the specific design of the compiler/interpreter so that the resulting execution remains consistent with the semantic definition of the computation.

Typically, multiple tasks in a runtime environment are arranged hierarchically, for example, a top-level task spawns multiple tasks, for example, one task for each top-level component of a data processing graph. Similarly, a computation that performs a set can have one task for processing the entire set, with multiple (i.e., many) subtasks for processing elements of the set, respectively.

In the runtime environment, each spawned task can be in one of a set of possible states. When first spawned, a task is in the Derived state before it is initially executed. During execution, it is in the Executing state. Tasks may occasionally be in the Paused state. For example, in some implementations, the scheduler may place a task in the Paused state when it has exceeded processor utilization, is waiting for resources, or otherwise. In some implementations, task execution is not preempted, and the task must relinquish control. There are three Pause substates: Runnable, Blocked, and Done. For example, a task is Runnable if it relinquishes control before completing its computation. A task is Done when, for example, it completes its processing before its parent task retrieves its return value. A task is Blocked if it is waiting for an event external to it, such as the completion of another task (e.g., because it has used a "wait" primitive) or the availability of a data record (e.g., blocking an execution of an in.read() or out.write() function).

Referring again to FIG. 1 , each compute node 152 has one or more processing engines 154. In at least some implementations, each processing engine is associated with a single operating system process executing on the compute node 150. Depending on the characteristics of the compute node, it may be efficient to execute multiple processing engines on a single compute node. For example, the compute node may be a server computer with multiple individual processors, or the server computer may have a single processor with multiple processor cores, or it may be a combination of multiple processors with multiple cores. In any case, executing multiple processing engines may be more efficient than using only a single processing engine on the compute node 152.

One example of a processing engine is hosted in the context of a virtual machine. One type of virtual machine is the Java Virtual Machine (JVM), which provides an environment in which tasks specified in compiled Java Bytecode can be executed. However, other forms of processing engines may also be used, which may or may not utilize a virtual machine architecture.

Referring to Figure 14, each processing engine 154 of computing node 152 has one or more runners 1450. Each runner 1450 uses one or more processes or process threads to execute runnable tasks. In some implementations, each runner has an associated process thread, but this association of a runner with a thread is not necessary. At any time, each runner is performing at most one runnable task of a calculation. Each runner has a separate runnable queue 1466. Each runnable task of a calculation is in a runnable queue 1466 of a runner 1450 of the system. Each runner 1450 has a scheduler/interpreter 1460, which monitors the currently running task and, when the task changes its state to being completed, blocked, or suspended, selects another task from the run queue 1466 and executes it. Tasks are associated with runners, and tasks of runners that are not runnable are kept outside the runnable queue 1466, for example, as shown in the figure, in a blocked and completed queue 1468.

For example, when processing engine 154 is initialized, runners 1450 may be created, with a preconfigured number of runners for each engine. As described below, in some implementations, runners may be added to or removed from processing engines, and even processing engines themselves may be added to and removed from computing platform 150 during execution of a data processing graph. However, for the initial description below, we assume that the number of processing engines and the number of runners within each processing engine remain constant.

As an example, the processing of data processing diagram begins at executing main process in top-level task.For example, one of the computing nodes that the controller 140 instructions of task and the monitor 1452 communication of one of processing engine 1450 begins to execute main process.In this example, monitor 1452 will be placed in the runnable queue 1466 of one of processing engine for the task of executing main process.In this example, operative device is idle (that is, other tasks are not in operation at this moment, and in runnable queue, there are not other runnable tasks), so the scheduler/interpreter 1460 of this operative device takes out this task from runnable queue and begins to perform task.When expressing the process in the language that needs to be explained, the continuous statement of scheduler/interpreter 1460 interpretive processes.

In this example, the first statement of the main procedure creates link buffers 1470 (i.e., allocates memory) for links supporting the flow of unordered collections, which in this example include unordered, unbounded buffers, namely, buffer 1, buffer 2, and buffer 3. Various methods can be used to create this type of inter-component link and manage the associated computing resources (including link buffers 1470) for these links, including any link whose upstream port is a collection port. In some examples, link buffers 1470 include a buffer for the output collection port representing the source of the collection and a separate buffer for the input collection port representing the destination of the collection. These buffers can be allocated at runtime before processing of the collection begins and de-allocated (i.e., the memory used for the buffers is released) after processing of the collection ends. In this example, these link buffers 1470 are allocated in the memory of the processing engine 154 in which the runner of the task is executing. Typically, the memory in which the buffers are created is in semiconductor random access memory (RAM), although in some implementations, other storage devices such as disks can be used to store at least some of the cached data. Note that in other methods, the buffer can be local to the runner itself. In practice, if the processing engine 154 is implemented as an operating system process, the buffer is created as the memory area in the address space of this process. Therefore, the access to the buffer directly based on the hardware address is limited to the instruction executed in this process. Note that in this method, if multiple runners will be able to read or write the buffer, then the buffer may need to be at least synchronized and access controlled, such as using a lock or semaphore. In the method where each runner is implemented as a single thread in the operating system process, the buffer can be associated with a specific runner, and all accesses can be limited to this runner, thereby avoiding potential contention from multiple threads. In the following discussion, we assume that the buffer can be accessed from any runner in the processing engine, and implement appropriate access control to allow such shared access.

Subsequent steps of the main process involve forking or calling a forall primitive of the main process. Typically, at least by default, the derivation of tasks or subtasks causes these tasks to initially form in the same runner as the parent process. For example, a derived Work_Read_External_Data task is derived on the same runner. To the extent that a task is accessing external data, the task can utilize an I/O interface 1464 to that external data. For example, the interface can include an open connection to an external database, an endpoint of a network data connection, etc. Such an I/O interface can be bound to a specific runner, so a task using that interface may need to access an interface only from that runner, as further discussed below in the context of potential migration of tasks between runners. In this example, we assume that the tasks fill buffer 1 in a reasonably measured manner and do not "overwhelm" the system, for example, by causing buffer 1 to grow beyond the capacity of the processing engine. Methods for controlling this aspect are also discussed below, for example, to avoid congestion or exhaustion of resources.

Concurrently with the execution of the Work_Read_External_Data task, forall_Work A causes a task to be spawned for each record read from Buffer 1. In particular, the "forall" primitive causes multiple instances of the task identified by the argument to the primitive to be executed, where the number of instances is generally determined by the number of data elements received at runtime, and where they are executed and the order in which they are called can be determined later at runtime without compiler constraints. As described above, by default, these tasks are also created on the same runner 1450 and, in the absence of other controls, are spawned as quickly as data is available from Buffer 1. The Work_B and Work_Read_External_Data tasks are similarly created on the same runner.

Note that the task-based specification uses the "forall" primitive without explicitly specifying how the runtime controller will implement the distribution of tasks to result in all data being processed. As described above, one approach that the runtime controller can use is to spawn separate tasks on the same compute node and then rely on migration features to cause the tasks to be executed on separate nodes, thereby balancing the load. Other approaches can be used, where the "forall" primitive causes multiple tasks to be executed directly on multiple nodes. In the case of a cursor defining an index-based subset of the rows of a table in an in-memory database, the implementation of the cursor "forall" primitive can cause the cursor to be split into multiple parts, each associated with a record stored on a different node, and tasks to be spawned for the separate parts of the cursor on different nodes, thereby resulting in locality of processing and data storage. However, it should be understood that a variety of methods can be implemented in one or more embodiments of the runtime controller and distributed computing platform to implement the "forall" primitive used in the task-based specification 130 as the output of the compiler 120. In some examples, the choice of method can depend on runtime decisions based on, for example, the number of records, the distribution of data on the compute nodes, the load on the nodes, etc. In any case, the method for implementing the "forall" primitive is not necessarily known to the developer of the data processing graph or the designer of the compiler.

The system is characterized in that tasks can be transferred between runners after the runners are created. Very generally, one way of doing this task transfer is through a "steal" or "pull" mechanism, where an idle or at least lightly loaded runner has tasks from another runner transferred to it. Although various criteria can be used, the multiple runnable tasks in a runner's runnable queue 1466 can determine whether the runner should seek tasks stole from other runners based on local criteria (such as whether there are fewer than a threshold number of tasks in its runnable queue). In some implementations, a more global decision-making process can be used to rebalance task queues across multiple runners, but the overall effect is similar.

In at least some embodiments, migrating a task from one runner to another does not necessarily involve transferring all of the task's data. For example, only the data accessible in the current execution "frame" (e.g., data for local and global variables accessible from the current program scope, such as the current subroutine call) is packaged back into the task's "home" runner along with a reference. This data is sufficient to create a runnable copy of the task at the migrated target runner, and the entry in the target runnable queue is ready for execution in that runner.

When the migrated runner completes execution, or when the migrated runner exhausts the data transferred to it by returning from the program scope where local variables are available, the task is transferred back to the main runner, where the data for the task is merged and the task is once again runnable at its main runner.

Note that during the transfer of tasks within a single processing engine, communication between runners can be through local memory (i.e., avoiding disk or network communication), thereby consuming relatively few resources. In implementations that allow for appropriation and migration between processing engines, tasks consume relatively few resources when switching from one runner to another, e.g., primarily consuming communication resources rather than computing resources between the processing engines. In addition, the latency of such communications is relatively insignificant because the master runner and the target runner are assumed to be busy computing during the transfer, the master runner because its runnable queue is heavily filled and therefore cannot be empty, and the target runner because the appropriation is performed in anticipation of the runnable queue at the target being emptied.

In the example of the execution of the task associated with the calculation shown in Figure 2A-Figure 2B, the task misappropriation mechanism distributes the load for calculation on the runners of one or more processing engines. However, it is noted that some data access is limited to a specific runner (or may be limited to a specific processing engine). For example, as described above, the data of buffer 2 can be accessed by a single runner (or possibly a group of runners), but the Work_A task that may need to be written to buffer 2 may have been misappropriated by a runner that cannot write to buffer 2. In such a case, when a task needs to take an action that must be performed on a runner different from the runner where the currently executing task is located, the task is migrated to the appropriate runner in a "migration" or "push" manner.

In at least some examples, computing platform 150 supports a global data storage of a set of global variable pairs (key, value). This data storage can be distributed on the memory (e.g., RAM or disk) on multiple computing nodes (or processing engines). The name space of the key is global because the specification of the key has the same meaning at all computing nodes 152 and their runners 1450. The values of these variables last when the task is instantiated, executed and terminated, thereby providing a way to pass information between tasks without the need for this information to be passed from one task to another via a common parent task. As described below, the access performed according to the key pair is controlled so that the use and updating of the value will not cause conflicts between tasks. In some examples, tasks obtain exclusive access to specific (key, value) pairs for some or all of their executions.

Typically, the storage of (key, value) pairs is distributed, and any particular (key, value) pair is associated with a particular compute node 152. For example, the (key, value) pair is stored in the distributed table storage 1480 at that compute node. In some implementations, the derive primitive allows the specification of a key and a mapping from the associated variable to the task's local variables. When a key is specified, the derive task obtains exclusive access to the key for the duration of its execution. Before execution begins, the value is passed from storage to the task's local context, and after execution is complete, the value in the local context is passed back to the global storage. If a derive primitive specifies a key used by another executing task, the newly derived task will be blocked until it can obtain exclusive access to the key. In some implementations, each compute node can determine the primary node for a particular key, and when a derive task is requested, the request is processed by the compute node where the (key, value) pair resides, and execution of the task will initially begin at that node. In alternative embodiments, other methods for obtaining similar exclusive access to such globally shared (key, value) pairs do not necessarily involve initiating a task at the same location as the storage, such as by transmitting a request for exclusive access and subsequently transmitting a release of exclusive access using an updated key value. A task can create new (key, value) pairs, which by default are stored on the node where the task runs when the new (key, value) pair is created.

One use for global state variables is to aggregate data during the execution of functions on consecutive records of a collection. For example, a global store maintains windows of values assigned to a key, rather than values as individual items. Thus, in the programming model, values can be added to a history maintained in association with a key, and functions on previously added values can be provided. A window of values can be defined by the number of items (e.g., the last 100 items), by a time window (e.g., items added in the last 10 minutes, by the time a value was added, or by an explicit timestamp provided when each value was added). Note that the programming model does not require explicit deletion of old values that fall outside the window; the definition of a window allows the implementation to automatically perform such deletion. The programming model includes several primitives for creating such window-based keyed global variables (e.g., defining the nature and scope of the window), adding values to a key, and computing functions on windows of values (e.g., maximum, average, number of distinct values, etc.). Some primitives combine the addition of a new value to a key with the return of a window function (e.g., adding a new value to a key and returning the average of the last 100 values added).

In at least some examples, global storage also includes record-oriented shared data accessed via identifiers called handles. For example, a handle can identify the source or destination of a data record, or as another example, a handle can identify a specific record in a data set. Typically, handles are typed because they provide a way to access data and also provide a definition of the data structure being accessed. For example, a handle can have a field (column) structure of the data record associated with it.

In at least some examples, global memory (e.g., in the memory of a compute node) includes table storage for one or more tables of multi-row typed data, where the table or a specific record of the table is accessed via an identifier called a handle. The row type of a table can be a hierarchical record type having a vector, a record vector, etc. In some examples, a table can have one or more indexes that provide hash or B-tree (ordered) access to rows, and a cursor can be created based on the table, the index, or the index and the key value. Rows can be inserted, updated, or deleted individually. To support transaction processing, a task can lock one or more rows of one or more tables, for example, for read or update access during processing of a component of a data processing graph. Tables can be viewed as collections for data parallel operations, for example, as sources or targets of data in a data processing graph. Typically, a table is indexed, and a subset of the table's rows can be selected based on the index that generates the cursor, which is then used to provide the selected rows as a data source. In some examples, additional primitives are available to tasks for actions such as splitting a cursor and estimating the number of records associated with a handle. When a cursor is provided as a data source for an execution set, the cursor can be split into multiple parts, each part providing some rows of the table to a corresponding instance of the execution set, thereby providing parallelism and appropriately splitting the cursor to enable execution on the nodes where the rows are stored. The data table can also be accessed by tasks that implement transactions, so that modifications to the data table are maintained so as not to be visible outside the task until the task explicitly commits the modifications. In some examples, such transaction support can be implemented by locking one or more rows of the table, while in other examples, more complex approaches involving multiple versions of the row can be implemented to provide higher potential concurrency than can be provided using locking alone.

Files, data streams, and in-memory tables are all examples of what are called collections. Reader tasks read records from a collection, and writer tasks write records to a collection. Some tasks are both readers and writers.

As described above, streams representing collections can be implemented in the runtime system using in-memory buffers. Alternatively, in various implementations, any form of storage can be used, including tables within a database or a distributed storage system. In some implementations, an in-memory distributed database is used. In some implementations, the compiler implements such streams using in-memory tables in a manner that is not necessarily exposed to the developer of the data processing graph. For example, the compiler can cause an upstream component to populate multiple rows of a table, and a downstream component to read previously populated rows, thereby implementing an unordered data flow. The runtime controller can call multiple instances of the task corresponding to the execution set, thereby processing the driving data elements from the upstream collection port by retrieving these data elements from storage in an order different from the order in which they were received into storage, preventing certain forms of blocking. For example, an instance of a task can be called without blocking any specific other instance from calling any other instance (i.e., until any specific other instance has completed processing one or more data elements).

Typically, a record in a collection can have a handle before the data in that record is first written. For example, a table can be set as the target of an indexed set of records, and the individual records can have handles even before the data for those records is written.

6 Implementation

The above methods can be implemented, for example, using a programmable computing system that executes appropriate software instructions, or can be implemented in suitable hardware such as a field programmable gate array (FPGA) or some hybrid form. For example, in a programming method, the software can include processes in one or more computer programs executed on one or more programmed or programmable computing systems (which can be various architectures such as distributed, client/server, or grid), each computing system including at least one processor, at least one data storage system (including volatile and/or non-volatile memory and/or storage elements), at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software can include, for example, one or more modules of a larger program that provides services related to the design, configuration, and execution of data processing graphs. The modules of the program (e.g., components of the data processing graph) can be implemented as data structures or other organized data that conform to a data model stored in a data warehouse.

The software may be stored in a non-transitory form, such as embodied in a volatile or non-volatile storage medium or any other non-transitory medium, using physical properties of the medium (e.g., surface pits and lands, magnetic domains, or charge, etc.) that persist for a period of time (e.g., the time between refresh cycles of a dynamic memory device (e.g., dynamic RAM)). In preparation for loading instructions, the software may be provided on a tangible, non-transitory medium, such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or on a tangible, non-transitory medium that can be delivered (e.g., encoded into a propagated signal) over a communication medium of a network to a computing system where it is executed. Some or all of the processing may be performed on a dedicated computer or using dedicated hardware such as a coprocessor or field programmable gate array (FPGA) or a specialized application-specific integrated circuit (ASIC). The processing may be implemented in a distributed manner, with different portions of the computation specified by the software being performed by different computing elements. Each such computer program is preferably stored or downloaded onto a computer-readable storage medium (e.g., solid-state memory or media, or magnetic or optical media) of a storage device accessible by a general or special purpose programmable computer for configuring and operating the computer when the computer reads the storage device medium to perform the processes described herein. The system of the present invention may also be considered to be embodied as a tangible, non-transitory medium configured with a computer program, wherein the medium so configured causes the computer to operate in a specific and predefined manner to perform one or more process steps described herein.

Several embodiments of the present invention have been described. However, it should be understood that the foregoing description is intended to illustrate, not to limit, the scope of the present invention, which is defined by the scope of the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be order-independent and, therefore, may be performed in an order different from that described.

Claims (18)

1. A method for processing graph-based program specifications, the method comprising: Receive the graph-based program specification, which includes: Multiple components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and One or more links, each of the one or more links connecting the output port of an upstream component of the plurality of components to the input port of a downstream component of the plurality of components; and Processing the graph-based program specification to generate prepared code representing a subset of the plurality of components of the graph-based program specification, the processing includes: Based at least in part on the characteristics of linked components, multiple subset boundaries between components are identified in different subsets; Subsets are formed based on the identified subset boundaries; and Prepared code is generated for each formed subset. When executed by the runtime system, the prepared code enables the execution of processing tasks corresponding to the components in the formed subset based on information embedded in the prepared code for that subset. The runtime system instantiates a certain number of instances of the formed subset, and the number is dynamically determined during runtime, at least in part, based on the performance of the runtime system when processing sets of multiple data elements. 2. The method of claim 1, wherein forming a subset comprises: traversing the components of the graph-based program specification while maintaining a record of the boundaries of the traversed subsets, and associating each component of the graph-based program specification with a single subset identifier determined from the record of the boundaries of the traversed subsets. 3. The method of claim 2, wherein each subset identifier associated with an identified subset of the plurality of components is unique. 4. The method of claim 2, wherein the records of the traversed subset boundaries are maintained as paths of identifier values. 5. The method of claim 4, wherein the path of the identifier value comprises a string of identifier values separated from each other by a delimiter. 6. The method of claim 1, wherein forming a subset comprises: Associate the first component of the graph-based program specification with a subset of identifiers; Propagate the subset identifier to downstream components of the first component; and Based on the identified subset boundaries, the subset identifier is modified during the propagation of the subset identifier. 7. The method of claim 6, wherein modifying the subset identifier during the propagation of the subset identifier comprises: When traversing the boundary of the first subset, the value of the subset identifier is changed from the first subset identifier value to the second subset identifier value associated with the first subset boundary; and When traversing the second subset boundary associated with the first subset boundary, the value of the subset identifier is changed to the value of the first subset identifier. 8. The method of claim 1, wherein identifying one or more subset boundaries based at least in part on the characteristics of linked components comprises: identifying subset boundaries based on the link between a first type of port on an upstream component and a second type of port on a downstream component. 9. The method of claim 1, wherein identifying one or more subset boundaries based at least in part on the characteristics of the linked components comprises: identifying subset boundaries based on a determined type of link between an upstream component and a downstream component, wherein the determined type of link is one of a variety of different types of links between components. 10. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding information indicating permissible concurrency between processing tasks corresponding to components in the formed subset into the prepared code. 11. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding information indicating priority relative to other formed subsets into the prepared code. 12. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding transactional information indicating one or more processing tasks corresponding to components in the formed subset into the prepared code. 13. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding information indicating at least one resource to be locked during the execution of the prepared code into the prepared code. 14. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding information indicating sorting characteristics between data elements processed by one or more processing tasks corresponding to components of the formed subset into the prepared code. 15. The method of claim 1, wherein generating prepared code for each formed subset comprises: for at least one formed subset, embedding information indicating the number of data elements operated on by each instance of the formed subset using the prepared code into the prepared code. 16. The method of claim 1, wherein if multiple instances exist, each of the multiple instances is applied to a different corresponding subset of the data elements in the set of the multiple data elements. 17. A computer-readable medium storing software in a non-transitory form, the software being used to process graph-based program specifications, the software including instructions for causing a computing system to perform the following operations: Receive the graph-based program specification, which includes: Multiple components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and One or more links, each of the one or more links connecting the output port of an upstream component of the plurality of components to the input port of a downstream component of the plurality of components; and Processing the graph-based program specification to generate prepared code representing a subset of the plurality of components of the graph-based program specification, the processing includes: Based at least in part on the characteristics of linked components, multiple subset boundaries between components are identified in different subsets; Subsets are formed based on the identified subset boundaries; and Prepared code is generated for each formed subset. When executed by the runtime system, the prepared code enables the execution of processing tasks corresponding to the components in the formed subset based on information embedded in the prepared code for that subset. The runtime system instantiates a certain number of instances of the formed subset, and the number is dynamically determined during runtime, at least in part, based on the performance of the runtime system when processing sets of multiple data elements. 18. A computational system for processing graph-based program specifications, the computational system comprising: At least one input device or port is configured to receive the graph-based program specification, which includes: Multiple components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and One or more links, each of the one or more links connecting the output port of an upstream component of the plurality of components to the input port of a downstream component of the plurality of components; and At least one processor is configured to process the graph-based program specification to generate prepared code representing a subset of the plurality of components of the graph-based program specification, the processing comprising: Based at least in part on the characteristics of linked components, multiple subset boundaries between components are identified in different subsets; Subsets are formed based on the identified subset boundaries; and Prepared code is generated for each formed subset. When executed by the runtime system, the prepared code enables the execution of processing tasks corresponding to the components in the formed subset based on information embedded in the prepared code for that subset. The runtime system instantiates a certain number of instances of the formed subset, and the number is dynamically determined during runtime, at least in part, based on the performance of the runtime system when processing sets of multiple data elements.