WO2016107488A1 - 一种流图优化方法及其装置 - Google Patents

一种流图优化方法及其装置 Download PDF

Info

Publication number
WO2016107488A1
WO2016107488A1 PCT/CN2015/098695 CN2015098695W WO2016107488A1 WO 2016107488 A1 WO2016107488 A1 WO 2016107488A1 CN 2015098695 W CN2015098695 W CN 2015098695W WO 2016107488 A1 WO2016107488 A1 WO 2016107488A1
Authority
WO
WIPO (PCT)
Prior art keywords
operator
flow graph
graph
flow
sub
Prior art date
Application number
PCT/CN2015/098695
Other languages
English (en)
French (fr)
Inventor
张松山
夏命榛
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP15875166.9A priority Critical patent/EP3229150B1/en
Publication of WO2016107488A1 publication Critical patent/WO2016107488A1/zh
Priority to US15/640,685 priority patent/US10613909B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24568—Data stream processing; Continuous queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9024—Graphs; Linked lists
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/30003—Arrangements for executing specific machine instructions
    • G06F9/3005—Arrangements for executing specific machine instructions to perform operations for flow control
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system

Definitions

  • Embodiments of the present invention relate to the field of stream processing, and in particular, to a stream graph optimization method and apparatus thereof.
  • Stream Computing that is, real-time stream processing, refers to processing the data to be processed as a data stream.
  • a data stream is a collection of a series of data records that are infinite in time distribution and number; a data tuple is the smallest component of a data stream.
  • the most important feature of stream computing is the ability to achieve real-time response to calculation results. The value of the data decreases with the passage of time. After the data appears, it must be processed as soon as possible. It is best to process the data as soon as it appears, and generate a data for processing, instead of caching it into a batch. It is the meaning of flow calculation.
  • the flow calculation model diagram referred to as the Streaming Graph, can be represented by FIG.
  • the operators op1-op7 in the figure represent operators
  • the edges s1-s7 in the figure represent data streams.
  • the operator is the data processing unit that carries the business logic, and is the smallest unit of the distributed scheduling execution of the stream computing platform
  • the data stream is the data exchanged between the operators.
  • the user writes the flow graph, submits the prepared flow graph to the flow computing platform, and the flow computing platform parses the flow graph, and then dispatches it to different physical machines for distributed execution to implement different business scenarios.
  • the mainstream streaming computing platform in the current industry is IBM InfoSphere Streams.
  • the user constructs a flow graph by dragging and dropping the operator and data stream, and then submits the deployment.
  • the quality of the flow graph directly affects the performance of the application after deployment.
  • Most of the users who develop applications based on the stream computing platform are market personnel or engineers who lack experience in streaming application development. As a result, the developed flow graphs have computational logic imbalance and redundant traffic transmission, resulting in system resources not being fully utilized. Streaming application execution performance is poor.
  • the embodiment of the invention provides a flow graph optimization method and device, which improves the utilization of the stream processing platform resources and the execution performance of the flow application by optimizing and redistributing the flow graph.
  • a first aspect of the embodiments of the present invention provides a flow graph optimization method for a stream computing system, where the stream computing system includes a plurality of working nodes and a master node, the method is performed by a master node, and the method is include:
  • the flow graph type includes: a pulse flow graph, a split graph, and an iterative graph;
  • the pulsed flow graph includes a data stream flowing sequentially according to a flow graph operator arrangement order; wherein the pulse flow graph includes a pulse operator, the pulse operator includes a window buffer, and the data output has a batch processing feature ;
  • the split-type flow graph includes a data stream splitting at an entry operator and converge at an exit operator; wherein the split-stream graph includes a split-type operator, and the split-type operator is the score An entrance operator or an exit operator of a union flow graph;
  • the iterative flow graph including the output data stream of the successor operator, is returned to the previous operator as an input stream, and iterative processing is performed; wherein the iterative flow graph includes an iterative operator, and the iterative operator is the iteration The entry operator or exit operator of the flow graph.
  • the flow graph is disassembled according to a maximum atom division rule of a flow graph type to obtain at least one sub- Flow diagram, including:
  • An operator used as a boundary operator is selected from the plurality of types of operators in order of priority from highest to lowest; wherein the order of priority from high to low is an iterative operator, a split operator, and a pulse. operator;
  • the operator in the flow graph by using a maximum atomic partitioning rule of the flow graph type to obtain at least one subflow graph; wherein the maximum atom partitioning rule includes: splitting The subflow graph contains as many operators as possible, and any subflow graph belongs to only one of an iterative flow graph, a split flow graph, and a pulse flow graph.
  • Processing logic complexity of the flow graph performing a contiguous operator merging on the at least one sub-flow graph to obtain an optimized flow graph, so that processing logic complexity of each sub-flow graph after merging by the contiguous operator is balanced, include:
  • the merging algorithm performs merging operator merging on the at least one sub-flow graph in the determined merging range according to the merging algorithm, To get an optimized flow graph, specifically:
  • the contiguous operator to be merged is a system operator
  • the at least one substream graph is merged by the contiguous operator according to the merging priority list to obtain an optimized stream graph
  • the adjacency operator to be merged is a user-defined operator, calculating a relative complexity of the pre-merger operator of the user-defined operator and a relative complexity of the merged operator, performing a neighboring calculation on the at least one sub-flow graph Sub-merging to obtain an optimized flow graph such that the relative complexity of the merged operator is less than or equal to N times the relative complexity of the pre-merger operator, where N is greater than or equal to one.
  • the method further includes:
  • the optimized flow graph is deployed to the work node for execution.
  • a second aspect of the embodiments of the present invention provides a flow graph optimization apparatus for a flow computing system, where the flow computing system includes a plurality of working nodes and a master control node, and the master control node includes:
  • a receiving module configured to receive a flow graph
  • a parsing module configured to parse the stream graph, construct a stream graph operator and a data stream edge description structure
  • a disassembly module configured to disassemble the flow graph according to a maximum atom division rule of the flow graph type to obtain at least one subflow graph
  • a merging module configured to: according to processing logic complexity of each sub-flow graph in the at least one sub-flow graph,
  • a sub-flow graph is merged by the contiguous operator to obtain an optimized flow graph such that the sub-flows are merged by the contiguous operator
  • the processing logic complexity of the graph is balanced.
  • the flow graph type includes: a pulse flow graph, a split graph, and an iterative graph;
  • the pulsed flow graph includes a data stream flowing sequentially according to a flow graph operator arrangement order; wherein the pulse flow graph includes a pulse operator, the pulse operator includes a window buffer, and the data output has a batch processing feature ;
  • the split-type flow graph includes a data stream splitting at an entry operator and converge at an exit operator; wherein the split-stream graph includes a split-type operator, and the split-type operator is the score An entrance operator or an exit operator of a union flow graph;
  • the iterative flow graph including the output data stream of the successor operator, is returned to the previous operator as an input stream, and iterative processing is performed; wherein the iterative flow graph includes an iterative operator, and the iterative operator is the iteration The entry operator or exit operator of the flow graph.
  • the disassembling module includes: a query unit, a selecting unit, and a disassembling unit;
  • the query unit is configured to traverse the operator in the flow graph to find a plurality of types of operators; wherein the plurality of types of operators include an iterative operator, a split-type operator, and a pulse operator;
  • the selecting unit is configured to select an operator used as a boundary operator in order of priority from the plurality of types of operators; wherein the order of priority from high to low is an iterative operator , split operator, pulse operator;
  • the disassembling unit is configured to divide an operator in the flow graph according to the selected boundary operator by using a maximum atomic partitioning rule of the flow graph type to obtain at least one substream graph; wherein
  • the maximum atomic division rule includes: the split subflow graph contains as many operators as possible, and any subflow graph belongs to only one of an iterative flow graph, a split flow graph, and a pulse flow graph.
  • the merging module is specifically used according to the cluster system resource and
  • the operator logic complexity determines a merge range according to the merge algorithm to perform a contiguous operator merging on the at least one sub-flow graph within the determined merge range to obtain an optimized flow graph, so that each sub-flow graph is merged by the contiguous operator
  • the processing logic complexity is balanced. .
  • the contiguous operator merging the at least one sub-flow graph in the determined merging range according to the merging algorithm To obtain an optimized flow graph, so that the processing logic complexity of each sub-flow graph after the merging of the adjacent operators is equalized, and the merging module is specifically used for
  • the contiguous operator to be merged is a system operator
  • the at least one substream graph is merged by the contiguous operator according to the merging priority list to obtain an optimized stream graph
  • the adjacency operator to be merged is a user-defined operator, calculating a relative complexity of the pre-merger operator of the user-defined operator and a relative complexity of the merged operator, performing a neighboring calculation on the at least one sub-flow graph Sub-merging to obtain an optimized flow graph such that the relative complexity of the merged operator is less than or equal to N times the relative complexity of the pre-merger operator, where N is greater than or equal to one.
  • the device further includes a deployment module, where the deployment module is configured to deploy the optimized flow graph to the working node for execution.
  • the flow graph optimization method and apparatus analyzes the flow graph by accepting the flow application flow graph provided by the user, constructs the flow graph operator and the data flow description structure, and convects according to the maximum atomic division rule of the flow graph type. Performing disassembly to obtain at least one sub-flow graph, and performing contiguous operator merging on the at least one sub-flow graph according to processing logic complexity of each sub-flow graph in the at least one sub-flow graph to obtain an optimized stream
  • the graph makes the processing logic complexity of each sub-flow graph after the merging of the adjacent operators is equalized.
  • Figure 1 is a schematic diagram of a flow calculation model
  • FIG. 2A is a schematic structural diagram of a stream computing system provided by the present invention.
  • FIG. 2B is a schematic structural diagram of a stream processing platform according to an embodiment of the present invention.
  • FIG. 3 is a flow chart of a flow chart optimization method in an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a flow diagram type in an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a flow diagram disassembly according to an embodiment of the present invention.
  • FIG. 6 is a flow chart of another flow chart optimization method in an embodiment of the present invention.
  • FIG. 7 is a schematic diagram of a flow graph optimization apparatus according to an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a disassembly module in an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of another flow chart optimization apparatus according to an embodiment of the present invention.
  • FIG. 10 is a schematic diagram showing the physical structure of a flow graph optimization apparatus according to an embodiment of the present invention.
  • FIG. 2A depicts the basic structure of the flow computing system, including a master node (Master) and multiple working nodes, and the master node is mainly responsible for scheduling and managing individual work nodes, which are logical entities that carry actual data processing operations.
  • Master master node
  • work nodes which are logical entities that carry actual data processing operations.
  • Flow calculation is based on a streaming data processing model.
  • the business processing logic usually needs to be converted into a Directed Acyclic Graph (DAG), as shown in Figure 4 or 5, where the operator ( Operator) is the business logic carrier, which is the smallest unit that can be scheduled and executed by the stream computing system; stream represents the data transmission between each operator.
  • DAG Directed Acyclic Graph
  • Operator the business logic carrier
  • stream represents the data transmission between each operator.
  • FIG. 2B is a schematic structural diagram of a stream processing platform according to an embodiment of the present invention.
  • a stream processing platform may include an underlying resource management module, a task scheduling module, a state/resource monitoring module, a data transmission and storage module, and various operator processing units operable on the upper layer, and users of the suite development. Interface and optional operator libraries.
  • the architecture of the stream processing platform is not limited to the architecture shown in FIG. 2B, and some subsystems or modules may be added or deleted on the basis of FIG. 2B.
  • an embodiment of the present invention provides a flow graph optimization method and apparatus therefor.
  • FIG. 3 is a flow chart of a flow chart optimization method according to an embodiment of the present invention.
  • S301 the user writes a streaming application, develops a streaming application flow graph, and submits the application flow graph to the streaming computing platform.
  • the flow computing platform parses the application flow graph and constructs a flow graph data description structure.
  • the flow computing platform parses the received user flow graph, constructs a flow graph data description structure, and describes a relationship between the flow graph operator and the data stream edge.
  • the flow graph types include: a pulse flow graph, a split flow graph, and an iterative flow graph.
  • the pulsed flow graph flows through the sequence of flow graph operators in order, which is the most basic processing type of the stream application; the pulsed flow graph contains the pulse operator, and the operator Operator1 is the pulse.
  • the operator has a window cache and the data output has a batch feature.
  • the data stream is shunted at the ingress operator and aggregated at the exit operator.
  • the shunting method includes multiple methods, such as copying and distributing, and the splitting flow graph includes points.
  • the joint operator such as the operator Operator2, Operator3; the operator Operator2 is the entry operator of the split-flow graph, the flow of the operator Operator2 is increased, and the operator Operator3 is the export operator of the split-flow graph, the operator The flow rate of Operator3 is reduced, and the operators Operator2 and Operator3 can be used as boundary operators of the split flow graph.
  • the iterative flow graph as shown in Fig. 4(c), the output data stream of the successor operator is returned to the previous operator as an input stream for iterative processing; the iterative flow graph contains iterative operators, such as operator Operator4 and Operator5;
  • the operator Operator4 is the entry operator of the iterative flow graph, and the operator Operator5 is the exit operator of the iterative flow graph.
  • the intermediate result of the iterative calculation is repeatedly transmitted inside the subflow graph, and the data transmission amount is large, which will generate a large amount of data traffic. Focused on the subflow graph, the operators Operator4 and Operator5 can be used as the boundary operator of the iterative flow graph.
  • the boundary operator is selected from the above-mentioned plurality of types of operators in descending order of priority; wherein the order of priority from high to low is an iterative operator, a split operator, and a pulse operator;
  • the operator as shown in Figure 5, operator Operator1, is both a pulse operator and a split operator, and acts as a high-priority split operator.
  • the operator in the flow graph is divided by the maximum atomic partitioning rule of the flow graph type to obtain at least one subflow graph; wherein the maximum atomic partitioning rule refers to the split subflow.
  • the graph contains as many operators as possible, but obeys the single type constraint, there is no ambiguity of the flow graph type, that is, the split subflow graph belongs only to the iterative flow graph, the split flow graph and the pulse flow graph. One of them.
  • a finite adjacency operator is merged between the sub-flow graphs, and the contiguous operator is merged with the at least one sub-flow graph according to the processing logic complexity of each sub-flow graph to obtain an optimized flow graph, so that After the merging operator is merged, the processing logic complexity of each sub-flow graph is balanced.
  • the merging range is determined according to the cluster system resource and the operator logic complexity; and the at least one sub-flow graph is merged by the contiguous operator in the determined merging range according to the merging algorithm to obtain an optimized stream graph, so that After processing the adjacent operators, the processing logic complexity of each sub-flow graph is balanced.
  • the adjacency operator is merged based on the flow graph disassembly result, and two elements need to be calculated: the available CPU cores of the cluster and the operator logic complexity.
  • the occupancy rate is lower than M is an available core, and the operator combination range is determined according to the user configuration; wherein 0 ⁇ M ⁇ 1, which may be set according to the cluster system resource and the operator logic complexity, for example, may be 50%;
  • the adjacency operator to be merged is a system operator
  • the at least one substream graph is merged according to the merge priority list; wherein, for the system operator, there is a fixed merge priority list;
  • the contiguous operator to be merged the relative complexity of the pre-merger operator of the user-defined operator and the relative complexity of the merged operator are calculated, and the at least one sub-flow graph is merged by the contiguous operator to obtain an optimization.
  • the flow graph is such that the relative complexity of the combined operator is less than or equal to N times the relative complexity of the pre-merger operator, where N is greater than or equal to 1, and can be set according to the cluster system resource and the operator logic complexity, for example, 120%. .
  • the available CPU cores are: 3, 2, 3, so the total available CPU core is 8, and the operator consolidation upper limit is 8, that is, at least 8 are reserved after the merge.
  • the actual operator; the lower limit is 0, that is, no merging is performed, especially when the system resources are sufficient, the merging operator is completely unnecessary, and the user can adjust the combined value of the upper and lower limits according to the actual situation to control the merging range.
  • Operators are divided into system operators and user-defined operators.
  • the system operator provides the library operator for the platform to use directly, and the operator logic complexity is fixed, so there is a fixed merge priority list;
  • the user-defined operator is the processing operation realized by the development interface provided by the user inheritance platform.
  • the complexity of such operators needs to be determined after calculation, and the computational complexity method These include, but are not limited to, common methods of calculating code complexity, such as operator execution time characterization operator complexity.
  • Operator merging can be implemented in a variety of ways, including but not limited to: process communication, thread communication, function calls, etc., the complexity of the merged operator can not exceed the complexity of the contiguous operator, otherwise the computational load will not be Equilibrium, for example, the complexity of the combined operator does not exceed 120% of the complexity of the pre-merger operator.
  • FIG. 6 is a flow chart of another flow chart optimization method according to an embodiment of the present invention.
  • the user writes a flow graph
  • the flow computing platform receives the flow graph submitted by the user.
  • the flow computing platform parses the received flow graph, constructs a flow graph data description structure, and describes a relationship between the flow graph operator and the data flow.
  • the disassembly process can refer to the disassembly method in S303.
  • the operator merging process may refer to the operator merging method in S304.
  • FIG. 7 is a schematic diagram of a flow graph optimization apparatus according to an embodiment of the present invention.
  • the flow graph optimization apparatus 100 includes: a receiving module 101, a parsing module 102, a disassembling module 103, and a merging module 104;
  • the receiving module 101 is configured to receive a flow graph.
  • the user writes a flow graph
  • the receiving module 101 receives the flow graph submitted by the user.
  • the parsing module 102 is configured to parse the flow graph, construct a flow graph operator and a data stream edge description structure.
  • the stream computing platform parsing module 102 parses the received stream graph, constructs a stream graph data description structure, and describes a relationship between the stream graph operator and the data stream.
  • the disassembly module 103 is configured to disassemble the flow graph according to a maximum atom division rule of the flow graph type to obtain at least one subflow graph.
  • the flow graph types include: a pulse flow graph, a split flow graph, and an iterative flow graph.
  • the pulsed flow graph flows through the flow graph operator in order of flow and performs corresponding processing, which is the most basic processing type of the stream application;
  • the pulsed flow graph contains the pulse operator, the operator Operator1 is a pulse operator with a window buffer and the data output has a batch feature.
  • the data stream is shunted at the ingress operator and aggregated at the exit operator.
  • the shunting method includes multiple methods, such as copying and distributing, and the splitting flow graph includes points.
  • Associative operator, Operator2 is the entry operator of the split-flow graph, the flow of the operator Operator2 is increased, the operator Operator3 is the export operator of the split-flow graph, and the flow of the operator Operator3 is reduced.
  • the sub-operator 2 and the operator 3 can be used as boundary operators of the split-type flow graph.
  • the iterative flow graph as shown in Fig. 4(c), the output data stream of the successor operator is returned to the previous operator as an input stream for iterative processing; the iterative flow graph contains an iterative operator, and the operator Operator4 is an iterative stream.
  • the entrance operator of the graph, operator Operator5 is the exit operator of the iterative flow graph.
  • the intermediate result of the iterative calculation is repeatedly transmitted inside the subflow graph. The data transmission amount is large, and a large amount of data traffic is concentrated in the subflow graph. Operators Operator4 and Operator5 can be used as boundary operators for iterative flow graphs.
  • the disassembly module 103 includes a query unit 1031, a selection unit 1032, and a disassembly unit 1033.
  • the query unit 1031 is configured to traverse the operators in the flow graph to find all the above multiple types of operators as an alternative split boundary set.
  • the selecting unit 1032 is configured to select a boundary operator from the high-to-low priority order among the plurality of types of operators; wherein the order of priority from high to low is an iterative operator, a split operator, and a pulse. Operator; for multi-type operators, as shown in Fig. 5, the operator op1, which is both a pulse operator and a splitting operator, acts as a high-priority splitting operator.
  • the disassembling unit 1033 is configured to divide the operator in the flow graph according to the selected boundary operator by using a maximum atomic partitioning rule of the flow graph type to obtain at least one subflow graph; wherein, the maximum atom partitioning rule, It means that the split subflow graph contains as many operators as possible, but obeys the single type constraint, there is no ambiguity of the flow graph type, that is, the split subflow graph belongs only to the iterative flow graph and the split graph.
  • a flow graph and a pulse flow graph is a pulse flow graph.
  • the merging module 104 is configured to perform contiguous operator merging on the at least one sub-flow graph according to processing logic complexity of each sub-flow graph in the at least one sub-flow graph to obtain an optimized flow graph, so that the contiguous operator Each substream described after merging
  • the processing logic complexity of the graph is balanced.
  • Determining the merge range according to the cluster system resource and the operator logic complexity and performing the contiguous operator merging on the at least one sub-flow graph in the determined merge range according to the merging algorithm to obtain the optimized flow graph, so that the contiguous operator After processing, the processing logic complexity of each sub-flow graph is balanced.
  • the merging module 104 determines the merging range according to the cluster system resource and the operator logic complexity, and performs contiguous operator merging.
  • the adjacency operator is merged based on the flow graph disassembly result, and two elements need to be calculated: the number of CPU cores available for the cluster and the operator logic complexity.
  • the occupancy rate is lower than M is an available core, and the operator combination range is determined according to the user configuration; wherein 0 ⁇ M ⁇ 1, which may be set according to the cluster system resource and the operator logic complexity, for example, may be 50%;
  • the contiguous operator to be merged is a system operator
  • the at least one substream graph is merged by the contiguous operator according to the merging priority list to obtain an optimized stream graph
  • the adjacency operator to be merged is a user-defined operator, calculating a relative complexity of the pre-merger operator of the user-defined operator and a relative complexity of the merged operator, performing a neighboring calculation on the at least one sub-flow graph Sub-merging to obtain an optimized flow graph such that the relative complexity of the merged operator is less than or equal to N times the relative complexity of the pre-merger operator, where N is greater than or equal to one.
  • the available CPU cores are: 3, 2, 3, so the total available CPU core is 8, and the operator consolidation upper limit is 8, that is, at least 8 are reserved after the merge.
  • the actual operator; the lower limit is 0, that is, no merging is performed, especially when the system resources are sufficient, the merging operator is completely unnecessary, and the user can adjust the combined value of the upper and lower limits according to the actual situation to control the merging range.
  • Operators are divided into system operators and user-defined operators.
  • the system operator provides the library operator for the platform to use directly, and the operator logic complexity is fixed, so there is a fixed merge priority list; the user-defined operator is the processing operation realized by the development interface provided by the user inheritance platform.
  • Such operator complexity needs to be determined after calculation.
  • Computational complexity methods include, but are not limited to, common methods of calculating code complexity.
  • Operator merging can be implemented in a variety of ways, including but not limited to: process communication, thread communication, function calls, etc., the complexity of the merged operator can not exceed the adjacent operator is too high, otherwise it will cause the calculation load imbalance.
  • the flow graph optimization apparatus 100 further includes a deployment module 105, configured to deploy the optimized flow graph to the work node for execution.
  • a physical device diagram of a flow graph optimization apparatus includes at least one processor 1001, a communication bus 1002, a memory 1003, and at least one communication interface 1004.
  • the communication bus 1002 is used to implement the connection and communication between the above components, and the communication interface 1004 is used to connect and communicate with external devices.
  • the memory 1003 is configured to store program code that needs to be executed.
  • the processor 1001 executes the program code in the memory 1003, the following functions are implemented:
  • Any sub-flow graph in at least one sub-flow graph contains as many operators as possible, and any one of the sub-flow graphs belongs to only the flow
  • a flow graph optimization method and apparatus are provided.
  • a flow application flow diagram provided by a user
  • a user flow graph is parsed, a flow graph operator node and a data stream edge description structure are constructed, and the flow graph is decomposed.
  • Solving realizing the maximum atomic division of the flow graph type to obtain at least one sub-flow graph, and merging the adjacent operators on the at least one sub-flow graph according to the merging algorithm to obtain an optimized flow graph.
  • the flow graph calculation logic is balanced and the redundant traffic transmission is reduced, and the utilization of the stream system resources and the execution performance of the stream application are improved.
  • the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory include instructions.
  • the manufacture of the device, the instruction device being implemented in a block or blocks of a flow or a flow and/or a block diagram of the flowchart.
  • These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device.
  • the instructions provide steps for implementing the functions in one or more blocks of the flowchart or in a flow or block of the flowchart.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Information Transfer Between Computers (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

一种流图优化方法和装置，涉及流处理领域。通过接受用户提供的流应用流图（S601），解析该流图，构造算子节点与数据流边描述的流图（S602），按最大原子划分规则对流图进行拆解，以得到至少一个子流图（S603），根据合并算法对所述至少一个子流图，进行邻接算子合并，以得到优化流图（S604）。通过对用户提供的流图进行拆解和合并，实现流图计算逻辑的均衡以及减少冗余流量传输，提高了流系统资源的利用率与流应用的执行性能。

Description

一种流图优化方法及其装置 技术领域
本发明实施例涉及流处理领域，更具体地，涉及一种流图优化方法及其装置。
背景技术
流计算(Stream Computing)，即实时流处理，是指将待处理数据看成数据流的形式来处理。数据流是在时间分布和数量上无限的一系列数据记录的集合体；数据元组是数据流的最小组成单元。流计算最重要的特性是能够实现计算结果的实时响应。数据的价值随着时间的流逝而降低，数据出现后必须尽快地对其进行处理，最好数据出现时便立刻对其进行处理，产生一个数据进行一次处理，而不是缓存起来成一批处理，这就是流计算的意义所在。
流计算模型图，简称流图(Streaming Graph)，可用图1来表示。如图1所示，图中算子op1-op7表示算子(operator)，图中边s1-s7表示数据流(stream)。其中，算子是承载业务逻辑的数据处理单元，为流计算平台分布式调度执行的最小单元，而数据流则是算子间交互的数据。
用户编写流图，将编写好的流图提交到流计算平台上，流计算平台解析流图，然后调度到不同的物理机器上分布式执行，实现不同的业务场景。当前业界主流的流计算平台为IBM InfoSphere Streams，用户在InfoSphere Streams的集成开发环境下，通过拖拽编辑，将算子和数据流边连线组成流图，然后提交部署。流图编写的好坏直接影响应用部署后的执行性能。而基于流计算平台开发应用的用户大多为市场人员，或是缺乏流式应用开发经验的工程人员，导致开发出来的流图存在计算逻辑不均衡和冗余流量传输,导致系统资源不能充分利用，流应用执行性能差。
发明内容
本发明实施例提供一种流图优化方法及装置，通过对流图进行优化再部署，提高了流处理平台资源的利用率与流应用的执行性能。
本发明实施例第一方面，提供了一种用于流计算系统的流图优化方法,所述流计算系统包括多个工作节点和主控节点，所述方法由主控节点执行，所述方法包括：
接收用户流图；
解析所述用户流图，构造算子节点与数据流边描述的流图；
按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，其中所述至少一个子流图中的任一子流图包含尽可能多的算子，且所述任一子流图只属于所述流图类型中的一种。
根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
结合第一方面所述的流图优化方法，在第一方面第一种可能的实现方式中，所述流图类型包括：脉冲式流图、分联式流图和迭代式流图；
所述脉冲式流图，包括数据流按流图算子排列顺序依次流过；其中，脉冲式流图包含脉冲式算子，所述脉冲式算子包含窗口缓存，且数据输出具有批处理特征；
所述分联式流图，包括数据流在入口算子处分流，在出口算子处汇聚；其中，分联式流图包含分联式算子，所述分联式算子为所述分联式流图的入口算子或出口算子；
所述迭代式流图，包括后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；其中，迭代式流图包含迭代式算子，所述迭代式算子为所述迭代式流图的入口算子或出口算子。
结合第一方面或第一方面第一种可能的实现方式，在第一方面第二种可能实现方式中，所述按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，包括：
遍历流图中算子，找出多种类型算子；其中，多种类型的算子包括迭代式算子、分联式算子和脉冲式算子；
从所述多种类型算子中按优先级从高到低的顺序选取用作边界算子的算子；其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；
根据所述选取的边界算子采用所述流图类型的最大原子划分规则对所述流图中算子进行划分，以得到至少一个子流图；其中，所述最大原子划分规则，包括：分裂出的子流图包含尽可能多的算子，且任一子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
结合第一方面或第一方面第一种可能实现方式或第一方面第二种可能实现方式，在第一方面第三种可能实现方式中，所述根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流图的处理逻辑复杂度达到均衡，具体包括：
根据集群系统资源和算子逻辑复杂度确定合并范围；
根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
结合第一方面第三种可能实现方式，在第一方面第四种可能实现方式中，所述根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，具体包括：
若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并，以得到优化流图；
若待合并的邻接算子为用户自定义算子，计算所述用户自定义算子的合并前算子相对复杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1。
结合结合第一方面或第一方面第一种可能实现方式或第一方面第二种可能实现方式或第一方面第三种可能实现方式或第一方面第四种可能实现方式，在第一方面第五种可能实现方式中，所述方法还包括：
将优化后的流图部署到工作节点中执行。
本发明实施例第二方面提供了一种用于流计算系统的流图优化装置，所述流计算系统包括多个工作节点和主控节点，所述主控节点包括：
接收模块，用于接收流图；
解析模块，用于解析所述流图，构造流图算子与数据流边描述结构；
拆解模块，用于按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图；
合并模块，用于根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少
一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流
图的处理逻辑复杂度达到均衡。
结合第二方面，在第二方面第一种实现方式中，所述流图类型包括：脉冲式流图、分联式流图和迭代式流图；
所述脉冲式流图，包括数据流按流图算子排列顺序依次流过；其中，脉冲式流图包含脉冲式算子，所述脉冲式算子包含窗口缓存，且数据输出具有批处理特征；
所述分联式流图，包括数据流在入口算子处分流，在出口算子处汇聚；其中，分联式流图包含分联式算子，所述分联式算子为所述分联式流图的入口算子或出口算子；
所述迭代式流图，包括后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；其中，迭代式流图包含迭代式算子，所述迭代式算子为所述迭代式流图的入口算子或出口算子。
结合第二方面或第二方面第一种实现方式中，在第二方面第二种实现方式中，所述拆解模块包括：查询单元、选择单元和拆解单元；
所述查询单元，用于遍历流图中算子，找出多种类型算子；其中，多种类型的算子包括迭代式算子、分联式算子和脉冲式算子；
所述选择单元，用于从所述多种类型算子中按优先级从高到低的顺序选取用作边界算子的算子；其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；
所述拆解单元，用于根据所述选取的边界算子采用所述流图类型的最大原子划分规则对所述流图中算子进行划分，以得到至少一个子流图；其中，所述最大原子划分规则，包括：分裂出的子流图包含尽可能多的算子，且任一子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
结合第二方面或第二方面第一种可能实现方式或第二方面第二种可能实现方式，在第二方面第三种可能实现方式中，所述合并模块，具体用于根据集群系统资源和算子逻辑复杂度确定合并范围根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。。
结合第二方面第三种可能实现方式，在第二方面第四种可能实现方式中，在所述根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡，所述合并模块具体用于，
若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并，以得到优化流图；
若待合并的邻接算子为用户自定义算子，计算所述用户自定义算子的合并前算子相对复杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1。
结合第二方面或第二方面第一种可能实现方式或第二方面第二种可能实现方式或第二方面第三种可能实现方式或第二方面第四种可能实现方式，在或第二方面第五种可能实现方式中，所述装置还包括部署模块，所述部署模块用于将优化后的流图下发部署到工作节点中执行。
根据本发明实施例提供的流图优化方法和装置，通过接受用户提供的流应用流图，解析该流图，构造流图算子与数据流描述结构，按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。通过对用户提供的流图进行拆解和合并，实现流图计算逻辑的均衡以及减少了冗余流量传输，提高了流系统资源的利用率与流应用的执行性能。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是流计算模型示意图；
图2A为本发明提供的流计算系统架构示意图；
图2B是本发明实施例中一种流处理平台架构示意图；
图3是本发明实施例中一种流图优化方法流程图；
图4是本发明实施例中一种流图类型示意图；
图5是本发明实施例中一种流图拆解示意图；
图6是本发明实施例中另一种流图优化方法流程图；
图7是本发明实施例中一种流图优化装置示意图；
图8是本发明实施例中一种拆解模块示意图；
图9是本发明实施例中另一种流图优化装置示意图；
图10为本发明实施例中流图优化装置的实体结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
应当理解，下面描述的实施方式是本发明优选的实施方式，此处所描述的优选实施例仅用于说明和解释本发明，并不用于限定本发明，并且在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
本发明实施例提供的技术方案可典型地应用于流计算系统中，图2A描述了流计算系统的基本结构，包括一个主控节点(Master)和多个工作节点(worker)，主控节点主要负责调度和管理各个工作节点，而工作节点是承载实际的数据处理操作的逻辑实体，
流计算是基于流式数据处理模型进行的，在流计算系统中，业务处理逻辑通常需要转化为无回路有向图(Directed Acyclic Graph，DAG)，如图4或5所示，其中算子(Operator)是业务逻辑载体，是可被流计算系统调度执行的最小单元；stream代表各Operator间的数据传输。需要说明的是，流计算系统只是本发明技术方案的一个典型应 用场景，并不对本发明的应用场景构成限制，其它涉及流应用优化的应用场景，本发明实施例的技术方案均适用。
图2B是本发明实施例一种流处理平台架构示意图。
如图2B所示，一个流处理平台可包括底层的资源管理模块、任务调度模块、状态/资源监控模块、数据传输和存储模块，以及上层可运行的各种算子处理单元，套件开发的用户界面和可选用的算子库等。当然，流处理平台的架构并不仅限于图2B所示的架构，还可以在图2B的基础上添加或删减部分子系统或模块。用户在流系统上开发应用场景时，通过用户界面完成应用定制再将应用部署在流处理平台上实现应用的流式处理。
为了解决由缺乏流式应用开发经验的用户编写的流图存在计算逻辑不均衡和冗余流量传输的问题，本发明实施例提出了一种流图优化方法及其装置。
图3是本发明实施例一种流图优化方法流程图。S301，用户编写流式应用，开发流式应用流图，提交应用流图到流计算平台。
S302，流计算平台解析应用流图，构造流图数据描述结构。
具体的，流计算平台解析接收到的用户流图，构造流图数据描述结构，描述流图算子与数据流边的关系。
S303，拆解流图，对流图进行最大原子划分以得到子流图。
具体的，如图4所示，流图类型包括：脉冲式流图、分联式流图和迭代式流图。
脉冲式流图，如图4(a)，数据流按流图算子排列顺序依次流过，是流应用最基本的处理类型；脉冲式流图中包含脉冲式算子，算子Operator1为脉冲式算子，含有窗口缓存，且数据输出具有批处理特征。
分联式流图，如图4(b)，数据流在入口算子处分流，在出口算子处汇聚，分流方式包括多种方式，如复制、分发等；分联式流图中包含分联式算子，如算子Operator2、Operator3；算子Operator2为分联式流图的入口算子，算子Operator2的出流量增加，算子Operator3为分联式流图的出口算子，算子Operator3的出流量减少，算子Operator2、Operator3可作为分联式流图的边界算子。
迭代式流图，如图4(c)，后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；迭代式流图中包含迭代式算子，如算子Operator4、Operator5；算子Operator4为迭代式流图的入口算子，算子Operator5为迭代式流图的出口算子，迭代式计算的中间结果在子流图内部反复传递，数据传输量大，将大量的数据流量集中在子流图内部，算子Operator4、Operator5可作为迭代式流图的边界算子。
拆解流图，对流图进行最大原子划分得到子流图，具体实现为：
如图5所示，遍历流图中算子，找出所有多种类型算子，作为备选的分裂边界集合。
从上述多种类型算子中按优先级从高到低的顺序选取边界算子；其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；对于多类型算子，如图5中算子Operator1，既是脉冲算子，又是分联算子，则作为高优先级的分联算子。
根据选取的边界算子，采用所述流图类型的最大原子划分规则对所述流图中算子进行划分，得到至少一个子流图；其中，最大原子划分规则，是指分裂出的子流图包含尽可能多的算子，但是遵守类型单一的约束，不存在流图类型的二义性，即分裂出的子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
S304，子流图之间进行有限的邻接算子合并。
子流图之间进行有限的邻接算子合并，具体实现为：根据各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流图的处理逻辑复杂度达到均衡。
具体的，根据集群系统资源和算子逻辑复杂度确定合并范围；根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
具体的，邻接算子合并基于流图拆解结果进行，需要计算两个要素：集群可用CPU核数与算子逻辑复杂度。
占用率低于M为可用核，结合用户配置，确定算子合并范围；其中，0≤M≤1，可根据集群系统资源以及算子逻辑复杂度设置，例如可以为50％；
若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并；其中，对于系统算子，存在固定的合并优先级列表；
若待合并的邻接算子，计算所述用户自定义算子的合并前算子相对复杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1，可根据集群系统资源以及算子逻辑复杂度设置，例如可以为120％。
例如，集群中有3个工作算子，可用CPU核数分别为：3、2、3，所以总可用CPU核数为8，算子合并上限值为8，即合并后至少要保留8个实际算子；下限值为0，即不进行任何合并，尤其当系统资源足够充裕时，合并算子完全无必要，用户可根据实际情况调整上下限的合并值，以控制合并范围。
算子分为系统算子和用户自定义算子。系统算子为平台提供给用户直接使用的库算子，算子逻辑复杂度固定，所以存在固定的合并优先级列表；用户自定义算子为用户继承平台提供的开发接口自己实现的处理操作，这类算子复杂度需要经过计算后确定，计算复杂度方法 包括但不限于常见的各种代码复杂度计算方法，如可用算子执行时间表征算子复杂度。
算子合并可以通过多种方式实现，包括但不限于：进程通信、线程通信、函数调用等，合并后的算子复杂度不能超邻接算子的复杂度过高，否则会造成计算负载的不均衡，例如合并后算子复杂度不超过合并前算子复杂度的120％。
算子合并为实现如下目标：
将算子间的数据传输转换成更高性能的消息交互，例如：跨网络通信变成进程间通信，进程间通信变成线程间消息传递，线程消息传递变成普通函数调用，从而减小数据传输时延导致的算子悬停与计算资源浪费的问题；
对于极高复杂度的算子，如Join、Agg、CEP等系统算子，以及复杂度很高的用户自定义算子，合并难度大，计算复杂，且优化效果很难保证，所以除非用户设定算子最小优化值，否则不优先对此类算子进行合并处理。
S305，将优化后流图部署到工作节点执行。
图6是本发明实施例另一流图优化方法流程图。
S601，接收用户流图。
具体的，用户编写流图，流计算平台接收用户提交的流图。
S602，解析用户流图，构造算子节点与数据流边描述的流图。
具体的，流计算平台解析接收到的流图，构造流图数据描述结构，描述流图算子与数据流的关系。
S603，按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，其中所述至少一个子流图中的任一子流图包含尽可能多的算子，且所述任一子流图只属于所述流图类型中的一种。
具体的，拆解过程可参照S303中拆解方法。
S604，根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
具体的，算子合并过程可参照S304中的算子合并方法。
图7是本发明实施例一种流图优化装置示意图。
如图7所示，流图优化装置100包括：接收模块101、解析模块102、拆解模块103和合并模块104；其中，
接收模块101，用于接收流图。
具体的，用户编写流图，接收模块101接收用户提交的流图。
解析模块102，用于解析所述流图，构造流图算子与数据流边描述结构。
具体的，流计算平台解析模块102解析接收到的流图，构造流图数据描述结构，描述流图算子与数据流的关系。
拆解模块103，用于按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图。
具体的，如图4所示，流图类型包括：脉冲式流图、分联式流图和迭代式流图。
脉冲式流图，如图4(a)，数据流按流图算子排列顺序依次流过进行相应处理，是流应用最基本的处理类型；脉冲式流图中包含脉冲式算子，算子Operator1为脉冲式算子，含有窗口缓存，且数据输出具有批处理特征。
分联式流图，如图4(b)，数据流在入口算子处分流，在出口算子处汇聚，分流方式包括多种方式，如复制、分发等；分联式流图中包含分联式算子，算子Operator2为分联式流图的入口算子，算子Operator2的出流量增加，算子Operator3为分联式流图的出口算子，算子Operator3的出流量减少，算子Operator2、Operator3可作为分联式流图的边界算子。
迭代式流图，如图4(c)，后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；迭代式流图中包含迭代式算子，算子Operator4为迭代式流图的入口算子，算子Operator5为迭代式流图的出口算子，迭代式计算的中间结果在子流图内部反复传递，数据传输量大，将大量的数据流量集中在子流图内部，算子Operator4、Operator5可作为迭代式流图的边界算子。
具体的，如图8所示，拆解模块103包括查询单元1031、选择单元1032和拆解单元1033。
查询单元1031，用于遍历流图中算子，找出所有上述多种类型算子，作为备选的分裂边界集合。
选择单元1032，用于从上述多种类型算子中按优先级从高到低的顺序选取边界算子；其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；对于多类型算子，如图5中算子op1，既是脉冲算子，又是分联算子，则作为高优先级的分联算子。
拆解单元1033，用于根据选取的边界算子采用所述流图类型的最大原子划分规则对所述流图中算子进行划分，以得到至少一个子流图；其中，最大原子划分规则，是指分裂出的子流图包含尽可能多的算子，但是遵守类型单一的约束，不存在流图类型的二义性，即分裂出的子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
合并模块104，用于根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流 图的处理逻辑复杂度达到均衡。具体实现：
根据集群系统资源和算子逻辑复杂度确定合并范围，根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
根据合并算法对所述至少一个子流图进行邻接算子合并，得到优化流图。
具体的，合并模块104根据集群系统资源和算子逻辑复杂度确定合并范围，进行邻接算子合并。
更具体的，邻接算子合并基于流图拆解结果进行，需要计算两个要素：集群可用CPU核数与算子逻辑复杂度。
占用率低于M为可用核，结合用户配置，确定算子合并范围；其中，0≤M≤1，可根据集群系统资源以及算子逻辑复杂度设置，例如可以为50％；
若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并，以得到优化流图；
若待合并的邻接算子为用户自定义算子，计算所述用户自定义算子的合并前算子相对复杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1。
例如，集群中有3个工作算子，可用CPU核数分别为：3、2、3，所以总可用CPU核数为8，算子合并上限值为8，即合并后至少要保留8个实际算子；下限值为0，即不进行任何合并，尤其当系统资源足够充裕时，合并算子完全无必要，用户可根据实际情况调整上下限的合并值，以控制合并范围。
算子分为系统算子和用户自定义算子。系统算子为平台提供给用户直接使用的库算子，算子逻辑复杂度固定，所以存在固定的合并优先级列表；用户自定义算子为用户继承平台提供的开发接口自己实现的处理操作，这类算子复杂度需要经过计算后确定，计算复杂度方法包括但不限于常见的各种代码复杂度计算方法。
算子合并可以通过多种方式实现，包括但不限于：进程通信、线程通信、函数调用等，合并后的算子不能超邻接算子的复杂度过高，否则会造成计算负载的不均衡。
算子合并为实现如下目标：
将算子间的数据传输转换成更高性能的消息交互，例如：跨网络通信变成进程间通信，进程间通信变成线程间消息传递，线程消息传递变成普通函数调用，从而消除数据传输时延导致的算子悬停与计算资源浪费的问题；
对于极高复杂度的算子，如Join、Agg、CEP等系统算子，以及复杂度很高的用户自定义算子，合并难度大，计算复杂，且优化效果很难保证，所以除非用户设定算子最小优化值， 否则不优先对此类算子进行合并处理。
可选的，如图9所示，流图优化装置100还包括部署模块105，用于将优化后流图部署到工作节点执行。
如图10所示，为本发明提供的流图优化装置的实体装置图，流图优化装置包括至少一个处理器1001，通信总线1002，存储器1003以及至少一个通信接口1004。
其中，通信总线1002用于实现上述组件之间的连接并通信，通信接口1004用于与外部设备连接并通信。
其中，存储器1003用于存储需要执行的程序代码，当处理器1001执行存储器1003中的程序代码时，实现如下功能：
接收用户流图；
解析所述用户流图，构造算子节点与数据流边描述的流图；
按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，其中所述
至少一个子流图中的任一子流图包含尽可能多的算子，且所述任一子流图只属于所述流
图类型中的一种。
根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡
在本发明实施例中，提供了一种流图优化方法和装置，通过接受用户提供的流应用流图，解析用户流图，构造流图算子节点与数据流边描述结构，对流图进行拆解，实现流图类型的最大原子划分，以得到至少一个子流图，根据合并算法对至少一个子流图进行邻接算子合并，得到优化流图。通过对用户提供的流图进行拆解和合并，实现流图计算逻辑的均衡以及减少了冗余流量传输，提高了流系统资源的利用率与流应用的执行性能。
本发明是参照根据本发明实施例的方法、装置的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令 装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中的功能的步骤。
尽管已描述了本发明的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。
显然，本领域的技术人员可以对本发明实施例进行各种改动和变型而不脱离本发明实施例的精神和范围。这样，倘若本发明实施例的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。

Claims (12)

  1. 一种用于流计算系统的流图优化方法，其特征在于，所述流计算系统包括多个工作节点和主控节点，所述方法由主控节点执行，所述方法包括：
    接收用户流图；
    解析所述用户流图，构造算子节点与数据流边描述的流图；
    按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，其中所述至少一个子流图中的任一子流图包含尽可能多的算子，且所述任一子流图只属于所述流图类型中的一种。
    根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
  2. 根据权利要求1所述的流图优化方法，其特征在于，所述流图类型包括：脉冲式流图、分联式流图和迭代式流图；
    所述脉冲式流图，包括数据流按流图算子排列顺序依次流过；其中，脉冲式流图包含脉冲式算子，所述脉冲式算子包含窗口缓存，且数据输出具有批处理特征；
    所述分联式流图，包括数据流在入口算子处分流，在出口算子处汇聚；其中，分联式流图包含分联式算子，所述分联式算子为所述分联式流图的入口算子或出口算子；
    所述迭代式流图，包括后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；其中，迭代式流图包含迭代式算子，所述迭代式算子为所述迭代式流图的入口算子或出口算子。
  3. 根据权利要求1或2所述的流图优化方法，其特征在于，所述按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图，包括：
    遍历流图中算子，找出多种类型算子；其中，多种类型的算子包括迭代式算子、分联式算子和脉冲式算子；
    从所述多种类型算子中按优先级从高到低的顺序选取用作边界算子的算子，其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；
    根据所述选取的边界算子采用所述流图类型的最大原子划分规则对所述流图中算子进行 划分，以得到至少一个子流图；其中，所述最大原子划分规则，包括：分裂出的子流图包含尽可能多的算子，且任一子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
  4. 根据权利要求1至3中任一项所述的流图优化方法，其特征在于，所述根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流图的处理逻辑复杂度达到均衡，具体包括：
    根据集群系统资源和算子逻辑复杂度确定合并范围；
    根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
  5. 根据权利要求4所述的流图优化方法，其特征在于，所述根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，具体包括：
    若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并，以得到优化流图；
    若待合并的邻接算子为用户自定义算子，计算所述用户自定义算子的合并前算子相对复杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1。
  6. 根据权利要求1至5中任一项所述的流图优化方法，其特征在于，所述方法还包括：
    将优化后的流图部署到工作节点中执行。
  7. 一种用于流计算系统的流图优化装置，其特征在于，所述流计算系统包括多个工作节点和主控节点，所述主控节点包括：
    接收模块，用于接收流图；
    解析模块，用于解析所述流图，构造流图算子与数据流边描述结构；
    拆解模块，用于按照流图类型的最大原子划分规则对流图进行拆解，以得到至少一个子流图；
    合并模块，用于根据所述至少一个子流图中各个子流图的处理逻辑复杂度，对所述至少
    一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后所述各个子流
    图的处理逻辑复杂度达到均衡。
  8. 根据权利要求7所述的流图优化装置，其特征在于，所述流图类型包括：脉冲式流图、分联式流图和迭代式流图；
    所述脉冲式流图，包括数据流按流图算子排列顺序依次流过；其中，脉冲式流图包含脉冲式算子，所述脉冲式算子包含窗口缓存，且数据输出具有批处理特征；
    所述分联式流图，包括数据流在入口算子处分流，在出口算子处汇聚；其中，分联式流图包含分联式算子，所述分联式算子为所述分联式流图的入口算子或出口算子；
    所述迭代式流图，包括后继算子的输出数据流返回至前面算子作为输入流，进行迭代处理；其中，迭代式流图包含迭代式算子，所述迭代式算子为所述迭代式流图的入口算子或出口算子。
  9. 根据权利要求7或8所述的流图优化装置，其特征在于，所述拆解模块包括：查询单元、选择单元和拆解单元；
    所述查询单元，用于遍历流图中算子，找出多种类型算子；其中，多种类型的算子包括迭代式算子、分联式算子和脉冲式算子；
    所述选择单元，用于从所述多种类型算子中按优先级从高到低的顺序选取用作边界算子的算子；其中，优先级从高到低的顺序依次为迭代算子、分联算子、脉冲算子；
    所述拆解单元，用于根据所述选取的边界算子采用所述流图类型的最大原子划分规则对所述流图中算子进行划分，以得到至少一个子流图；其中，所述最大原子划分规则，包括：分裂出的子流图包含尽可能多的算子，且任一子流图只属于迭代式流图、分联式流图和脉冲式流图中的一种。
  10. 根据权利要求7至9中任一项所述的流图优化装置，其特征在于，
    所述合并模块，具体用于根据集群系统资源和算子逻辑复杂度确定合并范围，根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡。
  11. 根据权利要求10所述的流图优化装置，其特征在于，在所述根据合并算法在所述确定的合并范围内对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得经邻接算子合并后各个子流图的处理逻辑复杂度达到均衡，所述合并模块具体用于，
    若待合并的邻接算子为系统算子，根据合并优先级列表对所述至少一个子流图进行邻接算子合并，以得到优化流图；
    若待合并的邻接算子为用户自定义算子，计算所述用户自定义算子的合并前算子相对复 杂度及合并后算子相对复杂度，对所述至少一个子流图进行邻接算子合并，以得到优化流图，使得合并后算子相对复杂度小于或等于合并前算子相对复杂度的N倍，其中N大于或等于1。
  12. 根据权利要求7至11任一项所述的流图优化装置，其特征在于，所述装置还包括部署模块，所述部署模块用于将优化后的流图部署到工作节点中执行。
PCT/CN2015/098695 2015-01-04 2015-12-24 一种流图优化方法及其装置 WO2016107488A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP15875166.9A EP3229150B1 (en) 2015-01-04 2015-12-24 Streaming graph optimization method and apparatus
US15/640,685 US10613909B2 (en) 2015-01-04 2017-07-03 Method and apparatus for generating an optimized streaming graph using an adjacency operator combination on at least one streaming subgraph

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510003823.4A CN104504143B (zh) 2015-01-04 2015-01-04 一种流图优化方法及其装置
CN201510003823.4 2015-01-04

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/640,685 Continuation US10613909B2 (en) 2015-01-04 2017-07-03 Method and apparatus for generating an optimized streaming graph using an adjacency operator combination on at least one streaming subgraph

Publications (1)

Publication Number Publication Date
WO2016107488A1 true WO2016107488A1 (zh) 2016-07-07

Family

ID=52945540

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/098695 WO2016107488A1 (zh) 2015-01-04 2015-12-24 一种流图优化方法及其装置

Country Status (4)

Country Link
US (1) US10613909B2 (zh)
EP (1) EP3229150B1 (zh)
CN (1) CN104504143B (zh)
WO (1) WO2016107488A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10025827B1 (en) 2017-01-17 2018-07-17 International Business Machines Corporation Operator fusion management in a stream computing environment
US10432462B2 (en) 2018-03-05 2019-10-01 International Business Machines Corporation Automatic selection of cut-point connections for dynamically-cut stream processing systems

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9100904B2 (en) * 2012-09-13 2015-08-04 First Principles, Inc. Data stream division to increase data transmission rates
CN104504143B (zh) 2015-01-04 2017-12-29 华为技术有限公司 一种流图优化方法及其装置
CN106547522B (zh) * 2015-09-17 2020-02-14 华为技术有限公司 一种流应用优化的方法及装置
CN105354089B (zh) * 2015-10-15 2019-02-01 北京航空航天大学 支持迭代计算的流式数据处理单元及系统
CN105610992A (zh) * 2016-03-31 2016-05-25 浪潮通信信息系统有限公司 一种分布式流计算系统任务分配负载均衡方法
CN107678790B (zh) * 2016-07-29 2020-05-08 华为技术有限公司 流计算方法、装置及系统
FI130232B (en) * 2018-04-18 2023-05-03 Meeshkan Oy Procedure for distributed processing of information and distributed information processing system
CN108628605A (zh) * 2018-04-28 2018-10-09 百度在线网络技术（北京）有限公司 流式数据处理方法、装置、服务器和介质
US11102258B2 (en) 2018-07-10 2021-08-24 International Business Machines Corporation Stream processing without central transportation planning
CN109508412B (zh) * 2018-11-20 2019-12-20 中科驭数（北京）科技有限公司 一种时间序列处理的计算流图构建方法和装置
CN110020720B (zh) * 2019-04-01 2021-05-11 中科寒武纪科技股份有限公司 算子拼接方法及装置
CN110795265B (zh) * 2019-10-25 2021-04-02 东北大学 一种基于乐观容错方法的迭代器
CN111597211B (zh) * 2020-05-14 2023-10-20 腾讯科技（深圳）有限公司 一种数据流图处理方法、装置、设备以及可读存储介质
CN114879943A (zh) * 2022-06-30 2022-08-09 浙江大华技术股份有限公司 一种算法方案生成方法、装置以及计算机可读存储介质
CN117056068B (zh) * 2023-08-08 2024-03-19 杭州观远数据有限公司 ETL中JobEngine任务拆分方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101630275A (zh) * 2009-07-31 2010-01-20 清华大学 一种实现生成循环任务配置信息的方法和装置
US20110131554A1 (en) * 2009-11-30 2011-06-02 International Business Machines Corporation Application generation system, method, and program product
CN104239141A (zh) * 2014-09-05 2014-12-24 北京邮电大学 数据中心中基于工作流关键路径的任务优化调度方法
CN104504143A (zh) * 2015-01-04 2015-04-08 华为技术有限公司 一种流图优化方法及其装置

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7774189B2 (en) * 2006-12-01 2010-08-10 International Business Machines Corporation System and method for simulating data flow using dataflow computing system
US8291006B2 (en) * 2008-05-30 2012-10-16 International Business Machines Corporation Method for generating a distributed stream processing application
US7856544B2 (en) * 2008-08-18 2010-12-21 International Business Machines Corporation Stream processing in super node clusters of processors assigned with stream computation graph kernels and coupled by stream traffic optical links
US8490072B2 (en) * 2009-06-23 2013-07-16 International Business Machines Corporation Partitioning operator flow graphs
JP5395565B2 (ja) * 2009-08-12 2014-01-22 株式会社日立製作所 ストリームデータ処理方法及び装置
JP4967014B2 (ja) * 2009-12-16 2012-07-04 株式会社日立製作所 ストリームデータ処理装置及び方法
US8856060B2 (en) * 2011-03-09 2014-10-07 International Business Machines Corporation Creating stream processing flows from sets of rules
US8874751B2 (en) * 2011-12-01 2014-10-28 International Business Machines Corporation Candidate set solver with user advice
US9146775B2 (en) 2012-04-26 2015-09-29 International Business Machines Corporation Operator graph changes in response to dynamic connections in stream computing applications
US8887056B2 (en) * 2012-08-07 2014-11-11 Advanced Micro Devices, Inc. System and method for configuring cloud computing systems
US8843524B2 (en) * 2012-09-25 2014-09-23 International Business Machines Corporation Handling out-of-sequence data in a streaming environment
US9930081B2 (en) * 2012-11-13 2018-03-27 International Business Machines Corporation Streams optional execution paths depending upon data rates
WO2014145092A2 (en) * 2013-03-15 2014-09-18 Akuda Labs Llc Hierarchical, parallel models for extracting in real time high-value information from data streams and system and method for creation of same
CN104052811B (zh) * 2014-06-17 2018-01-02 华为技术有限公司 一种业务调度的方法、装置及系统
CN104050543B (zh) * 2014-06-30 2017-06-20 华为技术有限公司 流处理系统中的事件处理方法及流处理系统
US9483332B2 (en) 2014-06-30 2016-11-01 Huawei Technologies Co., Ltd. Event processing method in stream processing system and stream processing system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101630275A (zh) * 2009-07-31 2010-01-20 清华大学 一种实现生成循环任务配置信息的方法和装置
US20110131554A1 (en) * 2009-11-30 2011-06-02 International Business Machines Corporation Application generation system, method, and program product
CN104239141A (zh) * 2014-09-05 2014-12-24 北京邮电大学 数据中心中基于工作流关键路径的任务优化调度方法
CN104504143A (zh) * 2015-01-04 2015-04-08 华为技术有限公司 一种流图优化方法及其装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10025827B1 (en) 2017-01-17 2018-07-17 International Business Machines Corporation Operator fusion management in a stream computing environment
US10432462B2 (en) 2018-03-05 2019-10-01 International Business Machines Corporation Automatic selection of cut-point connections for dynamically-cut stream processing systems
US11153166B2 (en) 2018-03-05 2021-10-19 International Business Machines Corporation Automatic selection of cut-point connections for dynamically-cut stream processing systems

Also Published As

Publication number Publication date
EP3229150A4 (en) 2017-12-06
CN104504143B (zh) 2017-12-29
US10613909B2 (en) 2020-04-07
CN104504143A (zh) 2015-04-08
EP3229150A1 (en) 2017-10-11
US20170300367A1 (en) 2017-10-19
EP3229150B1 (en) 2020-06-17

Similar Documents

Publication Publication Date Title
WO2016107488A1 (zh) 一种流图优化方法及其装置
CN107678790B (zh) 流计算方法、装置及系统
EP3001646B1 (en) Service scheduling method, apparatus, and system
US9628541B2 (en) Runtime grouping of tuples in a streaming application
US9256640B2 (en) Streaming delay patterns in a streaming environment
CN103309738B (zh) 用户作业调度方法及装置
US9477512B2 (en) Task-based modeling for parallel data integration
US20100293535A1 (en) Profile-Driven Data Stream Processing
US9122725B2 (en) Tuple routing in a streaming application
US20170060536A1 (en) Fusion recommendation for performance management in streams
CN106547522B (zh) 一种流应用优化的方法及装置
CN106681820A (zh) 基于消息组合的可扩展大数据计算方法
CN112905317B (zh) 快速可重构信号处理异构平台下任务调度方法和系统
US9720802B2 (en) Breakpoint for predicted tuple processing time in a streaming environment
Chen et al. Sapipe: Staleness-aware pipeline for data parallel dnn training
US9600617B1 (en) Automated timing analysis
Jiang et al. AMS: Adaptive multiget scheduling algorithm for distributed key-value stores
US20160299953A1 (en) Limiting tuple creation at the tuple level
CN110362397B (zh) 一种具有延迟约束功能的气泡执行方法
CN114240632A (zh) 批量作业执行方法、装置、设备、介质和产品
Hanczewski et al. High-Accuracy Analytical Model for Heterogeneous Cloud Systems with Limited Availability of Physical Machine Resources Based on Markov Chain
CN113986596A (zh) 一种数据处理方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15875166

Country of ref document: EP

Kind code of ref document: A1

REEP Request for entry into the european phase

Ref document number: 2015875166

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE