WO2016062087A1 - 一种流式应用升级方法、主控节点及流计算系统 - Google Patents

一种流式应用升级方法、主控节点及流计算系统 Download PDF

Info

Publication number
WO2016062087A1
WO2016062087A1 PCT/CN2015/079944 CN2015079944W WO2016062087A1 WO 2016062087 A1 WO2016062087 A1 WO 2016062087A1 CN 2015079944 W CN2015079944 W CN 2015079944W WO 2016062087 A1 WO2016062087 A1 WO 2016062087A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
working node
adjusted
node
streaming application
Prior art date
Application number
PCT/CN2015/079944
Other languages
English (en)
French (fr)
Inventor
洪斯宝
夏命榛
张松山
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP15853083.2A priority Critical patent/EP3200095B1/en
Priority to EP22188019.8A priority patent/EP4148558A1/en
Publication of WO2016062087A1 publication Critical patent/WO2016062087A1/zh
Priority to US15/492,392 priority patent/US10785272B2/en
Priority to US17/014,388 priority patent/US20200404032A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/60—Software deployment
    • G06F8/65—Updates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network

Definitions

  • the present invention relates to the field of computer technologies, and in particular, to a streaming application upgrade method, a master control node, and a flow computing system.
  • the flow computing system generally includes a master node (Master) and a plurality of worker nodes.
  • the master node is mainly responsible for scheduling and managing each working node, and the working node is carrying the actual data processing operation.
  • the logical entity, the working node specifically processes the data by calling a plurality of execution units (PE, Process Element), and the PE is a physical execution unit of the business logic.
  • an application or service deployed in a flow computing system is called a streaming application.
  • a logical model of the streaming application needs to be predefined, and the streaming application is required.
  • the logical model is usually represented by a Directed Acyclic Graph (DAG), as shown in Figure 2, where PE is the physical carrier that carries the actual data processing operations and is the smallest unit that can be scheduled to be executed by the streaming computing system. ;stream represents the data flow of the transmission between the PEs, and the arrow indicates the direction of the data flow; the PE can dynamically load and execute the business logic to process the data of the streaming application in real time.
  • DAG Directed Acyclic Graph
  • the flow computing system deploys each PE on different working nodes according to the logical model.
  • Each PE performs operations according to its own logic and forwards the operation result to the downstream PE.
  • the streaming application needs to be updated or upgraded, and the initial logical model is no longer applicable. Therefore, it is first necessary to complete the update of the streaming application offline, define a new logical model, then stop the old application, and then redeploy the updated streaming application to the streaming computing system according to the new logical model, and finally start the update.
  • Streaming application It can be seen that the update of the streaming application in the prior art requires the interruption of the original service and the inability to perform online application of the streaming application. Level, resulting in business loss.
  • the embodiment of the invention provides a streaming application upgrade method, a main control node and a flow computing system, which are used to implement online upgrade of a streaming application in a flow computing system without interrupting services.
  • an embodiment of the present invention provides a streaming application upgrade method, which is applied to a main control node in a flow computing system, where the flow computing system includes the main control node and at least one working node, where multiple An execution unit PE is distributed on one or more working nodes of the at least one working node for processing data of a streaming application deployed on the flow computing system, wherein an initial logical model of the streaming application a data flow direction between the plurality of PEs and the plurality of PEs for processing data of the streaming application; the method comprising:
  • the master node acquires the updated logical model of the streaming application, and determines by comparing the initial logical model of the streaming application with the updated logical model.
  • the first working node is a working node where the PE related to the data flow to be adjusted is located, and the upgrade instruction is used to indicate that the first working node adjusts The data flow between the PEs distributed on it.
  • the comparing the initial logical model of the streaming application and the updated logical model, determining the data flow to be adjusted includes:
  • the determining, by comparing the initial logical model of the streaming application and the updated logical model, determining the data flow to be adjusted includes:
  • the upgrade instruction includes:
  • the sending the upgrade command to the first working node includes:
  • the first upgrade instruction is used to instruct the first working node to adjust a data flow direction between PEs distributed thereon, and the second upgrade instruction is used to instruct the second working node to adjust a PE distributed on the second working node. number.
  • the method further includes:
  • the master node determines, according to the dependency relationship between the input stream and the output stream of the PE related to the data stream to be adjusted, the target PE that needs to perform data recovery and the checkpoint checkpoint that performs data recovery on the target PE;
  • the master node After determining that the first working node completes the adjustment, and the PE distributed on the first working node is ready, the master node triggers the target PE to input the restored data into the downstream PE of the target PE for processing. .
  • the data flow to be adjusted includes: a data flow to be updated and a data flow to be deleted;
  • the PE's input stream and the output stream depend on each other to determine the target PE that needs to perform data recovery.
  • the status data of each PE is backed up by the PE under the trigger of the output event, and is used to indicate the status of the PE processing data. .
  • the PE to be adjusted includes a PE to be added, and the second working node is a The working node selected by the master node according to the load status of each working node in the flow computing system; the second upgrade instruction is used to instruct the second working node to create the PE to be added.
  • the PE to be adjusted includes a PE to be deleted; the second working node is a working node where the PE to be deleted is located; and the second upgrade instruction is used to indicate that the second working node is deleted.
  • the PE to be deleted includes a PE to be deleted; the second working node is a working node where the PE to be deleted is located; and the second upgrade instruction is used to indicate that the second working node is deleted.
  • the method further includes:
  • the plurality of PEs are configured to process data of the streaming application according to an initial logical model of the streaming application.
  • the initial logical model of the streaming application is a directed acyclic graph DAG Expressed.
  • an embodiment of the present invention provides a master control node in a flow computing system, where the flow computing system includes the master control node and at least one working node; wherein a plurality of execution units PE are distributed in the Processing data of a streaming application deployed on the flow computing system on one or more working nodes of the at least one working node, wherein an initial logical model of the streaming application is used to represent processing the flow
  • the master node includes:
  • an obtaining and comparing module configured to obtain, after the streaming application update, the updated logical model of the streaming application, and compare the initial logical model of the streaming application with the updated logical model Determining the data stream to be adjusted;
  • An upgrade instruction generating module configured to generate an upgrade instruction according to the data stream to be adjusted
  • a sending module configured to send the upgrade instruction to the first working node, where the first working node is a working node where the PE related to the data flow to be adjusted is located, and the upgrade instruction is used to indicate the The first working node adjusts the data flow direction between the PEs distributed thereon.
  • the acquiring and comparing module is specifically configured to:
  • the PE represented by the initial logical model of the streaming application is not identical to the PE represented by the updated logical model;
  • the upgrade instruction generating module is specifically configured to generate a first liter according to the data stream to be adjusted. a level instruction; generating a second upgrade instruction according to the PE to be adjusted;
  • the sending module is specifically configured to send the first upgrade command to the first working node, and send the second upgrade command to a second working node, where the second working node includes the The working node where the adjusted PE is located; the first upgrade instruction is used to instruct the first working node to adjust a data flow direction between PEs distributed thereon, and the second upgrade instruction is used to indicate the second work
  • the node adjusts the number of PEs distributed over it.
  • the method further includes:
  • a data recovery module configured to determine, according to a dependency relationship between the input stream and the output stream of the PE related to the data stream to be adjusted, a target PE that needs to perform data recovery and a checkpoint checkpoint that performs data recovery on the target PE;
  • the sending module is further configured to: send a data recovery instruction to the working node where the target PE is located, where the data recovery instruction is used to instruct the target PE to recover data according to the checkpoint;
  • the master node further includes: an input triggering module, configured to trigger the data input by the target PE to be restored after determining that the first working node completes the adjustment, and the PE distributed on the first working node is ready The downstream PE of the target PE is processed.
  • an input triggering module configured to trigger the data input by the target PE to be restored after determining that the first working node completes the adjustment, and the PE distributed on the first working node is ready The downstream PE of the target PE is processed.
  • the data stream to be adjusted includes: a data stream to be updated and a data stream to be deleted; to:
  • the PE's input stream and the output stream depend on each other to determine the target PE that needs to perform data recovery.
  • the status data of each PE is backed up by the PE under the trigger of the output event, and is used to indicate the status of the PE processing data. .
  • the PE to be adjusted includes a PE to be deleted, and the second working node is the The working node where the PE to be deleted is located; the second upgrade instruction is used to instruct the second working node to delete the PE to be deleted.
  • the PE to be adjusted includes a PE to be added, and the second working node is a Master
  • the control node selects the working node according to the load status of each working node in the flow computing system; the second upgrade instruction is used to instruct the second working node to create the PE to be added.
  • the method further includes: a configuration module, configured according to the initial logic of the streaming application The model configures the plurality of PEs to process data of the streaming application.
  • an embodiment of the present invention provides a flow computing system, including: a master node and at least one working node, wherein a plurality of execution units PE are distributed to one or more of the at least one working node. And processing, on the node, data for the streaming application deployed on the flow computing system, wherein the initial logical model of the streaming application is used to represent the plurality of PEs that process data of the streaming application and Data flow between the plurality of PEs;
  • the master node is configured to: obtain, after the streaming application update, acquire the updated logical model of the streaming application, and compare the initial logical model of the streaming application with the updated logic a data stream to be adjusted is determined by the model, and an upgrade instruction is generated according to the data stream to be adjusted; and the upgrade instruction is sent to the first working node, where the first working node is the data stream to be adjusted.
  • the upgrade instruction is used to instruct the first working node to adjust a data flow direction between PEs distributed thereon;
  • the first working node is configured to receive an upgrade instruction sent by the master control node, and adjust a data flow direction between PEs distributed on the first working node according to the indication of the upgrade instruction.
  • the PE represented by the initial logical model of the streaming application is the same as the PE represented by the updated logical model.
  • the PE represented by the initial logical model of the streaming application is not exactly the same as the PE represented by the updated logical model; a data flow, generating a first upgrade instruction; generating a second upgrade instruction according to the PE to be adjusted;
  • the first working node is configured to receive the first upgrade instruction sent by the master control node, and adjust data between PEs distributed on the first working node according to the indication of the first upgrade instruction.
  • the second working node is specifically configured to receive the second upgrade finger sent by the master control node And, according to the indication of the second upgrade instruction, adjusting the number of PEs distributed on the second working node.
  • the master control node is further configured to: according to the data to be adjusted Dependent relationship between the input stream and the output stream of the flow-related PE, the target PE that needs to perform data recovery, and the checkpoint checkpoint that performs data recovery on the target PE; and the data recovery instruction is sent to the working node where the target PE is located.
  • the data recovery instruction is used to instruct the target PE to recover data according to the checkpoint; when it is determined that the first working node completes the adjustment, and the PE distributed on the first working node is ready, the target PE is triggered.
  • the recovered data is input to the downstream PE of the target PE for processing.
  • the streaming application upgrade method and the flow computing system provided by the embodiments of the present invention dynamically determine the data flow to be adjusted by comparing the difference between the logical models before and after the update of the streaming application, and according to the data flow to be adjusted.
  • the corresponding upgrade command is generated and sent to the working node, so that the online upgrade of the streaming application in the flow computing system is implemented without interrupting the service.
  • FIG. 1 is a schematic structural diagram of a stream computing system provided by the present invention.
  • FIG. 2 is a schematic diagram of a logical model of a streaming application according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a streaming application deployment according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a working principle of a stream computing system according to an embodiment of the present invention.
  • FIG. 5 is a flowchart of a streaming application upgrade method according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a change of a logical model before and after a streaming application update according to an embodiment of the present invention
  • FIG. 7 is a schematic diagram of a change of a logical model before and after a streaming application update according to an embodiment of the present invention.
  • FIG. 8 is a flowchart of a streaming application upgrade method according to an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of a streaming application logic model according to an embodiment of the present invention.
  • FIG. 10 is a schematic diagram of adjusting a logical model of a streaming application according to an embodiment of the present invention.
  • FIG. 11 is a schematic diagram of a PE deployment after a streaming application upgrade according to an embodiment of the present disclosure
  • FIG. 12 is a schematic diagram of a dependency relationship between an input stream and an output stream of a PE according to an embodiment of the present disclosure
  • FIG. 13 is a schematic diagram of a dependency relationship between an input stream and an output stream of a PE according to an embodiment of the present disclosure
  • FIG. 14 is a schematic diagram of backup of PE state data according to an embodiment of the present invention.
  • FIG. 15 is a schematic diagram of a master control node according to an embodiment of the present disclosure.
  • FIG. 16 is a schematic diagram of a flow computing system according to an embodiment of the present invention.
  • FIG. 17 is a schematic diagram of a master node according to an embodiment of the present invention.
  • FIG. 4 describes a basic structure of a flow computing system, where the flow computing system includes: a master node (Master) and a plurality of working nodes (workers).
  • the master node and the working node may have one or more.
  • the master node may be a physical node separated from the working node; in a stand-alone deployment, the master node and the working node may be deployed in the same physics.
  • the master node is responsible for scheduling the data stream to the working node for processing.
  • a physical node is a working node.
  • one physical node can correspond to multiple working nodes.
  • the number of working nodes corresponding to one physical node depends on the hardware physics of the physical node. Resources.
  • a working node can be understood as a hardware physical resource. The working nodes corresponding to the same physical node communicate by means of process communication, and the working nodes corresponding to different physical nodes communicate by means of network communication.
  • the flow computing system includes a master node, a work node 1, a work node 2, and a work node 3.
  • the master node deploys the streaming application to the three working nodes of the working node 1, the working node 2, and the working node 3 according to the logical model of the streaming application for processing.
  • the logical model shown in Figure 3 is a logical relationship diagram consisting of nine execution units (PEs): PE1 to PE9, and the flow of data between the nine PEs.
  • the direction of the data flow between PEs also reflects Input flow and output of each PE Outflow dependencies.
  • the data stream is also simply referred to as a stream.
  • the master node configures PE1, PE2, and PE3 on the working node 1, PE4, PE7, and PE9 on the working node 2, and PE5, PE6, and PE8 on the working node 3 to convenely apply according to the logical model of the streaming application.
  • the data stream is processed. It can be seen that after configuration, the data flow direction between the PEs on the working nodes 1, 2, and 3 is matched with the logical model of the streaming application.
  • the logical model of the streaming application in the embodiment of the present invention may be a directed acyclic graph (DAG), a tree diagram, or a graph containing a ring.
  • the logical model of the streaming application can be understood by referring to FIG. 2.
  • the flow computing application diagram shown in FIG. 2 includes 7 operators, from PE1 to PE7, and has 8 data streams, respectively from s1 to s8.
  • Figure 2 clearly shows the direction of the data flow.
  • the data stream s1 is from PE1 to the operator PE5, indicating that PE5 processes the stream output by PE1, that is, the output of PE5 depends on the input of PE1, which is also called PE5.
  • the downstream PE of PE1, and PE1 is the upstream PE of PE5.
  • the upstream PE and the downstream PE are determined according to the data flow direction between PEs. For a data flow, there is only one associated with it.
  • Two PEs the source PE that outputs the data stream, and the destination PE to which the data stream flows (that is, the PE that receives the data stream). From the direction of the data stream, the source PE is the upstream PE of the destination PE, and the destination is The PE is the downstream PE of the source PE.
  • the data stream S2 is input to the PE2 and processed by the PE2 logic to generate two data streams S3 and S4, which are respectively entered into the PE3 and the PE4 for logical processing.
  • the PE2 is also the PE1.
  • the downstream PE, and PE1 is the upstream PE of PE2; the data stream S4 output by PE4 and the data stream S7 output by PE3 are both input to PE6, that is, the output of PE6 depends on the input of PE3 and PE4.
  • a PE that outputs an input that depends on a single PE is defined as a stateless PE, such as PE5, PE3, and PE4; and a PE that outputs an input that depends on multiple PEs is defined as a stateful PE, such as PE6, PE7.
  • a data stream contains a single piece of data, called a tuple, which can be structured or unstructured. Generally, a tuple can represent the state of something at a specific point in time.
  • the PE in the stream computing system processes the data stream generated by the streaming application in units of tuples, and can also consider the tuple to be the smallest data in the stream computing system. Granularity and representation.
  • the flow computing system is only a typical application scenario of the technical solution of the present invention, and does not constitute a limitation of the application scenario of the present invention.
  • the technical solutions of the examples are applicable.
  • the embodiment of the present invention provides a streaming application upgrade method, which can be applied to the flow computing system shown in FIG. 1 and FIG. 4, and assumes that the streaming application A is deployed on the flow computing system, and the flow is
  • the initial logical model of the application A is D1
  • the main control node of the flow computing system processes a plurality of execution units (PEs) according to the initial logical model D1 to process the data flow of the streaming application A, wherein the multiple execution units
  • the PE is distributed on one or more working nodes of the flow computing system; as shown in Figure 6, when the streaming application A is upgraded or updated, its logical model is updated accordingly (assuming the updated logical model is D2)
  • the update of the logical model is generally completed by a developer, or is performed by a developer by means of a development tool, and the present invention is not particularly limited.
  • the main process of the streaming application upgrade method is described as follows:
  • the master node determines the data stream to be adjusted by comparing the updated logical model D2 with the initial logical model D1.
  • S503 The master control node generates an upgrade instruction according to the data stream to be adjusted.
  • the master node sends the generated upgrade command to the first working node, where the first working node is a working node where the PE related to the data stream to be adjusted is located, and the upgrade instruction is used to indicate that the first working node adjusts The data flow between the PEs distributed on it.
  • the data flow to be adjusted in the embodiment of the present invention may be one or multiple, depending on the specific situation, and for each data flow to be adjusted, the PE related thereto is specific.
  • the streaming application upgrade method and the flow computing system provided by the embodiments of the present invention dynamically determine the data flow to be adjusted by comparing the difference between the logical models before and after the flow application is updated, and generate corresponding upgrade instructions according to the data flow to be adjusted.
  • the work node is sent to implement online upgrade of the streaming application in the stream computing system without interrupting the service.
  • the logical model of the streaming application is used to represent multiple PEs that process data of the streaming application and the data flow direction between the multiple PEs; when the streaming application is upgraded or updated, The logical model is updated accordingly.
  • the difference between the updated logical model and the initial logical model is mainly divided into two types: (1) the PE represented by the initial logical model and the updated logical model.
  • the PE is exactly the same, only the data flow direction between the PEs changes; (2) the PE represented by the initial logical model is not exactly the same as the PE represented by the updated logical model.
  • the data flow direction between PEs also changes; for the above two types of differences, the corresponding processing flow is introduced below.
  • the PE represented by the initial logical model of the streaming application is identical to the PE represented by the updated logical model of the streaming application, and the data flow between the PEs is oriented.
  • the PEs in the logical model are all PE1-PE7, which are identical, but the data flow direction changes, that is, the data flow from PE4 to PE6 becomes the flow from PE4.
  • the data stream S11 of the PE7 is simultaneously added with the data stream S12 flowing from the PE2 to the PE6.
  • the main flow of the streaming application upgrade method is as follows:
  • Step 1 Determine the data stream to be adjusted by comparing the initial logical model of the streaming application with the updated logical model of the streaming application; wherein the data stream to be adjusted includes one or more data streams; specifically,
  • the data stream to be adjusted may include at least one of a data stream to be added, a data stream to be deleted, or a data stream to be updated, where the data stream to be updated refers to a streaming application.
  • the destination node or one of the source nodes changes data flow; specifically, to the situation shown in FIG. 6, the data stream to be adjusted includes: a data stream S12 to be added, and a data stream S11 to be updated.
  • Step 2 Generate an upgrade instruction according to the data stream to be adjusted, where the upgrade instruction may include one or more instructions, and the upgrade instruction is related to the type of the data stream to be adjusted, for example, if the data flow to be adjusted includes The newly added data stream and the data stream to be updated generate upgrade instructions including: instructions for adding a new data stream and instructions for updating the data stream; wherein different types of upgrade instructions may be separate instructions, It can be integrated into one instruction, and the invention is not particularly limited.
  • the generated upgrade instruction includes an instruction to add the data stream S12 and an instruction to update the data stream S6 to S11.
  • Step 3 The generated upgrade instruction is sent to the first working node, where the first working node is a working node where the PE related to the data stream to be adjusted is located; it can be understood that the first working node may be one or more
  • the first working node performs an operation indicated by the upgrade instruction, such as adding a new data stream S12, and updating the data stream S6 to S11, so that the PEs distributed on the first working node are The data flow direction is adjusted, and the adjusted data flow direction matches the updated logical model.
  • the master node determines, according to the dependence of the input stream and the output stream of the PE related to the data stream to be adjusted.
  • the master node After the control node determines that the first working node completes the adjustment, and the PE distributed on the first working node is ready, the master node triggers the target PE to input the restored data into the downstream PE of the target PE for processing.
  • the master node can sense the state of the PEs on each working node in the flow computing system by sending the query message actively, or the working node can report the status of each PE distributed by the working node to the master node.
  • the status includes: running status, ready status, and stop status.
  • the PE establishes a successful channel with the upstream and downstream PEs, it indicates that the PE is in the ready state and can receive the data stream and process it.
  • the master node may further configure, by using the initial logical model of the streaming application, data of the plurality of PEs to process the application.
  • the streaming application upgrade method dynamically determines the data flow to be adjusted by comparing the difference between the logical models before and after the flow application is updated, and generates a corresponding upgrade instruction and sends the corresponding upgrade instruction to the working node to complete the streaming application.
  • Online upgrades ensure that the application does not need to be interrupted during the upgrade process. Further, data is restored during the upgrade process to ensure that critical data is not lost and affects business operations.
  • the PE represented by the initial logical model of the streaming application is not exactly the same as the PE represented by the updated logical model of the streaming application, and the data between the PEs is The flow direction has also changed.
  • the number of PEs in the logical model has changed (PE2, PE3, PE4, and PE6 are deleted, PE9-PE13 has been added), and the data flow direction is also changed.
  • a change has occurred, that is, the original data streams S4, S5, S6, and S7 are deleted, and the data stream S11-S16 is newly added.
  • the destination PE of the original data stream S3 is updated, and the source PE of the original data stream S9 is generated. Updated.
  • the main flow of the streaming application upgrade method is as follows:
  • the master node determines the PE to be adjusted and the data stream to be adjusted by comparing the initial logical model of the streaming application with the updated logical model of the streaming application.
  • the PE to be adjusted includes one or more PEs.
  • the data stream to be adjusted includes one or more data streams; specifically, in one
  • the PE to be adjusted includes at least one of a PE to be added or a PE to be deleted, and the data stream to be adjusted may include: a data stream to be added, and a data stream to be deleted. Or at least one of the data streams to be updated.
  • the master control node can determine that the sub-logic model consisting of PE2, PE3, PE4, and PE6 in the original logical model needs to be replaced by PE9-PE13 by comparing the logical model before and after the streaming application update.
  • the sub-logical model can be the same as the updated logical model. Therefore, it is determined that PE2, PE3, PE4, PE6, and PE9-PE13 are PEs to be adjusted.
  • PE2, PE3, PE4, and PE6 are PEs to be deleted.
  • PE9-PE13 is the PE to be added, and determines the data flow related to the PE to be adjusted. That is, all the input streams and output streams of the PE to be adjusted are the flows to be adjusted, as indicated by the dotted line in FIG.
  • the stream is the data stream to be deleted, the stream indicated by the black bold portion is the data stream to be added, and the stream indicated by the light bold portion is the data stream to be updated.
  • the main control node generates a first upgrade instruction according to the data flow to be adjusted, and generates a second upgrade instruction according to the PE to be adjusted, where the first upgrade instruction and the second upgrade instruction may respectively include one or more instructions, and The first upgrade command is related to the type of the data stream to be adjusted, and the second upgrade command is related to the type of the PE to be adjusted.
  • the generated first upgrade instruction includes: an instruction for adding a data stream and an instruction for updating the data stream; if the PE to be adjusted includes a PE to be added, the generated second upgrade instruction includes: The instruction of the PE; wherein the first upgrade instruction and the second upgrade instruction may be separate instructions, or may be integrated into one instruction, and the invention is not particularly limited.
  • the generated first upgrade instruction includes: an instruction to delete a data stream, an instruction to add a data stream, and an instruction to update the data stream, where the second upgrade instruction includes: adding a PE instruction, deleting the PE Instructions.
  • the adjustment policy may be further determined, that is, how to The PE and the flow are adjusted to match the deployment of the PE after the adjustment (including the number of PEs and the dependency of the data flow between the PEs) and the updated logical model of the streaming application.
  • the adjustment strategy consists of two parts: (1) the PE number adjustment policy, that is, which PEs need to be added and/or which PEs need to be deleted; (2) the adjustment strategy of the data flow direction between PEs, that is, which PE data flows toward Need to update, which data streams need to be added, and which data streams need to be deleted.
  • the adjustment strategy mainly includes at least one of the following situations:
  • Update flow a change occurs in one of the destination node or the source node of the data stream
  • the master node may generate an upgrade instruction, where the upgrade command is used to indicate the working node (specifically, the working node where the PE to be adjusted is located and the data flow to be adjusted)
  • the working node where the PE is located implements the determined adjustment strategy.
  • the upgrade instruction includes at least one of a new PE instruction, a delete PE instruction, an update flow instruction, a delete flow instruction, or a new flow instruction.
  • the upgrade instruction specifically includes:
  • the master node sends the generated first upgrade command to the first working node, and sends the generated second upgrade command to the second working node, where the first working node is related to the data stream to be adjusted.
  • the working nodes may overlap, that is, one working node may belong to the first working node and also belong to the second working node; the first upgrade instruction is used to instruct the first working node to adjust the data flow direction between the PEs distributed thereon, The second upgrade instruction is used to instruct the second working node to adjust the number of PEs distributed thereon; after receiving the upgrade instruction, the first working node and the second working node perform an operation indicated by the upgrade instruction, so that the first The data flow direction between the PE and the PE distributed on the working node and the second working node is adjusted
  • the second working node includes the working node where the PE to be deleted is located, and the second upgrade instruction is used to indicate the second work.
  • the node deletes the PE to be deleted.
  • the second working node may select, according to the load status of each working node in the flow computing system, the working node.
  • the working node that is randomly selected by the master node may be used.
  • the second upgrade command is used to instruct the second working node to create the PE to be added.
  • the main control node sends a new PE9 command to the worker 2, and sends a new PE 10 command to the worker 3, and sends the new PE 11 and PE 11 commands.
  • the instruction to add PE13 is sent to worker6; the instruction to delete PE2 and PE3 is sent to worker3, the instruction to delete PE4 and PE6 is sent to worker4; the instruction to delete the flow between PE2 and PE3 is sent to PE2 and PE3.
  • each working node maintains the data flow configuration information of all the PEs on the working node, and the data flow configuration information of each PE includes the source address, the destination address, the port number, and the like, so the data stream is deleted. And updates, essentially by modifying the data flow configuration information.
  • PE9 As shown in Figure 11, according to the upgrade command issued by the master node, worker 2 has added PE9, worker3 has deleted PE2 and PE3, PE10 has been added, worker 4 has deleted PE6 and PE4, and PE11 has been added. PE12 and worker 6 have added PE13.
  • worker1-6 adjusts the data flow direction between PEs by deleting, adding, and updating flows. Specifically, PE9-PE13 is added. The flow between PE2, PE3, PE4, and PE6 is deleted. The destination PE of the output stream of PE1 is changed from PE2 to PE9. The source PE of the input stream of PE7 is changed from PE6 to PE13.
  • the deployment situation of the PE after the adjustment matches the updated logical model of the streaming application A.
  • the streaming application upgrade method further includes:
  • the master node determines, according to the dependency relationship between the input stream and the output stream of the PE related to the data stream to be adjusted, the target PE that needs to perform data recovery and the checkpoint checkpoint that performs data recovery on the target PE;
  • the working node sends a data recovery instruction, where the data recovery instruction is used to indicate that the target PE recovers data according to the checkpoint; when the main control node determines that the first working node and the second working node complete the adjustment, and the first working node and the second work After the PEs distributed on the node are ready, the master node triggers the target PE to input the recovered data into the downstream PE of the target PE for processing.
  • the master node can sense the state of the PEs on each working node in the flow computing system by sending the query message actively, or the working node can report the status of each PE distributed by the working node to the master node.
  • the status includes: running status, ready status, and stop status.
  • the PE establishes a successful channel with the upstream and downstream PEs, it indicates that the PE is in the ready state and can receive the data stream and process it.
  • the adjustment of the PE needs to involve the adjustment of the data flow.
  • some data may be processed.
  • Adjusting the dependency of the original input stream and the output stream of the PE related to the data stream determining the target PE that needs to perform data recovery and the checkpoint checkpoint of the target PE to perform data recovery, so as to ensure that the PE is not processed by the PE before the application is upgraded.
  • the completed data can continue to be reprocessed after the upgrade is completed; among them, the data that needs to be recovered here is usually referred to as a tuple.
  • the input/output relationship of the sub-logic models composed of ⁇ PE1, PE2, PE3, PE4, PE6, and PE7 ⁇ related to the data stream to be adjusted is as follows: tuple i 1 , i 2, i 3 and i 4
  • PE1 input PE2 was then PE2 treated to obtain the tuple k 1, k 2, k 3, and j 1, then tuple k 1, k 2, k 3 is input PE4 treated
  • Obtaining m 1 the tuple j 1 is input to PE3 for processing, and l 1 is obtained, PE6 is treated with m 1 to obtain O 2 , and after l 1 is treated, O 1 is obtained.
  • the dependence of the input stream and the output stream of the PE to be adjusted can be analyzed, as shown in FIG.
  • O 1 depends on input PE6 l 1, and l 1 is dependent on j 1, j 1 are also dependent on i 2, so the logic for the entire sub-model, the output O 1 of PE6 dependent on PE2 input i 2;
  • O 2 depends on the input m 1 of PE6 , while m 1 depends on the inputs k 1 , k 2 and k 3 of PE4 , k 1 , k 2 and k 3 are in turn dependent on i 1 , i 3 and i 4 , so for the whole
  • the output O 2 of PE6 depends on the inputs i 1 , i 3 and i 4 of PE2.
  • the latest state data of the PE backup related to the data flow to be adjusted may determine that the input is to be adjusted. Whether the data of the PE related to the data stream has been processed and output to the downstream PE, and then the checkpoint of the checkpoint for performing data recovery by the target PE can be determined.
  • the status data of the PE is used to indicate the processing data status of the PE, and the specific content thereof is well known to those skilled in the art.
  • the status data may include: buffer data in the tuple receiving queue, and in the message channel.
  • the data stream to be adjusted includes: a data stream to be updated, a data stream to be deleted, and a data stream to be added, only the data stream to be updated and the data to be deleted need to be
  • the status data of the PE-related PE can determine the checkpoint for performing data recovery, and determine the target PE that needs to perform data recovery according to the data stream to be updated and the dependence of the input stream and the output stream of the PE related to the data stream to be deleted.
  • the data stream to be adjusted includes: the data stream to be updated and the data stream to be added, only the state data of the PE related to the data stream to be updated, and the data stream to be updated are related.
  • the dependency of the input stream and the output stream of the PE can determine the checkpoint for performing data recovery, and The target PE for data recovery.
  • the status data of the PE is periodically backed up, that is, the flow computing system periodically triggers each PE to back up its own state data, and the PE receives the checkpoint (checkpoint). After the event, it will back up its current state data, record the checkpoint, and clean up the expired data.
  • checkpoint can be understood as a record point of data backup or an index of backup data, and a checkpoint corresponds to a data backup operation. The data backed up at different times has different checkpoints, and can pass checkpoint. Query and get the data backed up under this checkpoint.
  • the output triggering mechanism (triggered by the PE output) may be used to perform backup of the state data, as shown in FIG. 14, when the PE completes processing of the input stream Input_Stream1-5, and outputs the processing result.
  • the trigger module triggers the state data processing module, and the state data processing module starts a new checkpoint to record the latest state data of the PE to the memory or the disk.
  • the triggering method is accurate and effective, and the efficiency is better than the periodic triggering method.
  • the state data processing module can also clean up the historical data of the previous checkpoint record, thereby reducing the intermediate data and effectively saving storage space.
  • the PE1 of i 3 and i 4 completes the data recovery, that is, the target PE that needs to recover the data is PE1, so that it can be determined that the checkpoints of i 1 , i 3 and i 4 can be recovered.
  • the target PE may restore the data i 1 , i 3 and i 4 according to the determined checkpoint, and when the first working node and the second After the working node completes the adjustment, and the PEs distributed on the first working node and the second working node are ready, the target PE re-processes the recovered data i 1 , i 3 , and i 4 downstream PEs to ensure that the upgrade process does not Data loss will occur and the purpose of non-destructive upgrade will be achieved.
  • the master node may further configure, by using the initial logical model of the streaming application, data of the plurality of PEs to process the application.
  • the streaming application upgrade method dynamically determines the data flow to be adjusted by comparing the difference between the logical models before and after the flow application is updated, and generates a corresponding upgrade instruction and sends the corresponding upgrade instruction to the working node to complete the streaming application.
  • Online upgrades ensure that the application does not need to be interrupted during the upgrade process. Further, data is restored during the upgrade process to ensure that critical data is not lost and affects business operations.
  • the embodiment of the present invention further provides a master control node in a flow computing system, where the master control node may be a computer or a server; the flow computing system further includes at least one working node; A streaming application A is deployed on the flow computing system, and a plurality of execution units (PEs) are distributed on one or more working nodes of the at least one working node for processing data of the streaming application A, wherein the streaming application
  • the logical model of A is used to represent the multiple PEs of the data processing the streaming application and the data flow direction between the multiple PEs; it is assumed that the initial logical model of the streaming application A is D1, when the streaming application A is upgraded or After the update, its logical model is updated accordingly (assuming the updated logical model is D2).
  • the master node 30 includes:
  • the obtaining and determining module 301 is configured to obtain the updated logical model D2 of the streaming application A in the case that the streaming application A is updated, and compare the updated logical model D2 and the initial logical model D1 to determine the to-be-adjusted Data flow
  • the upgrade instruction generating module 302 is configured to generate an upgrade instruction according to the flow to be adjusted
  • the sending module 303 is configured to send the generated upgrade command to the first working node, so that the first working node adjusts the data flow direction between the PEs distributed on the first working node according to the indication of the upgrade instruction;
  • a working node is one or more working nodes in a working node included in the flow computing system, and the first working node is a working node where the PE related to the data flow to be adjusted is located.
  • the data flow to be adjusted in the embodiment of the present invention may be one or multiple, depending on the specific situation, and for each data flow to be adjusted, the PE related thereto is specific.
  • the main control node in the flow computing system provided by the embodiment of the present invention dynamically determines the data flow to be adjusted by comparing the difference between the logical models before and after the update of the streaming application, and generates a corresponding upgrade command according to the data flow to be adjusted.
  • the working node is implemented to implement an online upgrade of the streaming application in the streaming computing system without interrupting the service.
  • the acquisition and determination module 301 also has a certain difference in specific processing details.
  • the acquisition and comparison module 301 is specifically configured to:
  • the obtaining and comparing module 301 is specifically configured to: compare the initial logical model D1 and the updated logical model D2 to determine the PE to be adjusted and the data stream to be adjusted, where the streaming application A
  • the PE represented by the initial logical model D1 is not exactly the same as the PE represented by the updated logical model D2.
  • the upgrade instruction generating module 302 is specifically configured to generate a first upgrade instruction according to the data flow to be adjusted determined by the obtaining and comparing module 301; and the PE to be adjusted determined according to the obtaining and comparing module 301
  • the second upgrade instruction is generated, and the sending module 303 is specifically configured to: send the first upgrade command to the first working node, and send the second upgrade command to the second working node, where the second working node includes the PE to be adjusted.
  • the first upgrade instruction is used to instruct the first working node to adjust the data flow direction between the PEs distributed on the working node
  • the second upgrade instruction is used to instruct the second working node to adjust the number of PEs distributed thereon.
  • the master node 30 further includes:
  • the data recovery module 304 is configured to determine, according to a dependency relationship between the input stream and the output stream of the PE related to the data stream to be adjusted, a target PE that needs to perform data recovery and a checkpoint that performs data recovery on the target PE;
  • the sending module 303 is further configured to: after the data recovery module 304 determines the target PE and the checkpoint, send a data recovery instruction to the working node where the target PE is located, where the data recovery instruction is used to indicate that the target PE is according to the checkpoint The data is recovered; it can be understood that the data recovery instruction is constructed according to the target PE and the checkpoint determined by the data recovery module 304, and includes information indicating the checkpoint.
  • the master node 30 further includes: an input triggering module 305, configured to trigger the target PE determined by the data recovery module 304 after determining that the first working node completes the adjustment, and the PEs distributed on the first working node are all ready.
  • the recovered data is input to the downstream PE of the target PE for processing.
  • the main control node in the flow computing system provided by the embodiment of the present invention dynamically determines the data flow to be adjusted by comparing the difference between the logical models before and after the flow application is updated, and generates a corresponding upgrade instruction.
  • the main control node in the flow computing system provided by the present invention is used to implement the streaming application upgrade method in the foregoing method embodiment.
  • the main control node in the flow computing system provided by the present invention is used to implement the streaming application upgrade method in the foregoing method embodiment.
  • the embodiment of the present invention further provides a flow computing system, which is used to implement the streaming application upgrade method provided by the embodiment of the present invention.
  • the flow computing system includes: a master node 30 and at least one working node (eg, The working node 31-34 in FIG. 16; the master node 30 configures a plurality of execution units (PEs) according to the initial logical model of the streaming application to process the data stream of the streaming application, and the initial logical model of the streaming application a plurality of PEs for processing data of the streaming application and a data flow direction between the plurality of PEs; wherein, as shown in FIG.
  • PEs execution units
  • the configured plurality of PEs are distributed on one or more working nodes;
  • the master control node 30 is configured to: after the flow application A is updated, obtain the updated logical model of the streaming application, and compare the updated logical model and the initial logical model to determine the data flow to be adjusted;
  • the upgraded instruction is generated, and the generated upgrade instruction is sent to the first working node, where the first working node is one or more working nodes in the working node included in the flow computing system, and A working node is a node associated with the working data stream to be adjusted in the PE resides.
  • the first working node is configured to receive an upgrade instruction sent by the master control node 30, and adjust a data flow direction between the PEs distributed on the first working node according to the indication of the upgrade instruction.
  • the flow computing system dynamically determines the data flow to be adjusted by comparing the difference between the logical models before and after the update of the streaming application, and generates a corresponding upgrade instruction to be sent to the working node to complete the online application of the streaming application. Upgrade to ensure that the original business does not need to be interrupted during the application upgrade process.
  • the upgrade instruction includes at least one of a new PE instruction, a delete PE instruction, an update flow instruction, a delete flow instruction, or a new flow instruction.
  • the first working node after receiving the upgrade instruction, performs at least one of a new execution unit, a delete execution unit, an update flow, a delete flow, or a new flow, so that the deployment of the PE after the foregoing operation is performed ( The number of PEs and the dependency of the data flow between the PEs are matched with the updated logical model of the streaming application A.
  • the master node is specifically configured to: compare initial logic of the streaming application The model and the updated logical model determine a data stream to be adjusted, wherein the PE represented by the initial logical model of the streaming application is the same as the PE represented by the updated logical model.
  • the master node is specifically configured to: compare the initial logical model of the streaming application with the updated logical model to determine the PE to be adjusted and the data stream to be adjusted, where the streaming application
  • the PE represented by the initial logical model is not exactly the same as the PE represented by the updated logical model; the first upgrade instruction is generated according to the data flow to be adjusted; the second upgrade instruction is generated according to the PE to be adjusted; and the first upgrade instruction is
  • the second working node is sent to the second working node, where the first working node is the working node where the PE related to the data stream to be adjusted is located, and the second working node includes the working node to be adjusted.
  • the first working node is specifically configured to receive the first upgrade instruction sent by the master control node 30, and adjust the data flow direction between the PEs distributed on the first working node according to the indication of the first upgrade instruction;
  • the node is configured to receive the second upgrade instruction sent by the master node 30, and adjust the number of PEs distributed on the second working node according to the indication of the second upgrade instruction.
  • the master node 30 is further configured to determine, according to the dependence of the input stream and the output stream of the PE related to the data stream to be adjusted, the target PE and the target PE that need to perform data recovery. Performing a checkpoint checkpoint of the data recovery; sending a data recovery instruction to the working node where the target PE is located, the data recovery instruction is used to indicate that the target PE recovers data according to the checkpoint; when it is determined that the first working node completes the adjustment, and the first work After the PEs distributed on the nodes are all ready, the target PE is triggered to input the recovered data into the downstream PE of the target PE for processing.
  • the execution unit (PE) in the embodiment of the present invention may exist in a software form, such as a process, a thread, or a software function module, or may exist in the form of hardware, such as a processor core or a logic circuit having data processing capability.
  • the functions described in the embodiments of the present invention are implemented by reading the executable code or the service processing logic in the memory, and the present invention is not particularly limited.
  • FIG. 17 is a schematic structural diagram of a master node 40 according to an embodiment of the present invention.
  • the master node 40 can include an input device 410, an output device 420, a processor 430, and a memory 440.
  • the main control node 40 provided by the embodiment of the present invention is applied to a flow computing system, where the flow computing system further includes a working node, and the flow computing system is deployed with a streaming application.
  • Memory 440 can include read only memory and random access memory and provides instructions and data to processor 430. A portion of memory 440 may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • Memory 440 stores operational instructions, operating systems (including various system programs for implementing various basic services and processing hardware-based tasks), executable modules or data structures, or a subset thereof, or an extended set thereof.
  • the processor 430 performs the following operations by calling an operation instruction stored in the memory 440 (the operation instruction can be stored in the operating system):
  • the updated logical model of the streaming application comparing the updated logical model and the initial logical model to determine a data flow to be adjusted; generating an upgrade instruction according to the flow to be adjusted; and generating the upgrade instruction And being sent to the first working node, where the first working node is one or more working nodes in the working node included in the flow computing system, and the first working node is a working node where the PE related to the data flow to be adjusted is located .
  • the main control node provided by the embodiment of the present invention compares the difference between the logical model before and after the update of the streaming application, dynamically determines the data flow to be adjusted, and generates a corresponding upgrade instruction to be sent to the working node to complete the online upgrade of the streaming application. In order to ensure that the original business does not need to be interrupted during the application upgrade process.
  • the processor 430 controls the operation of the device 40 for service processing, and the processor 430 may also be referred to as a CPU (Central Processing Unit).
  • Memory 440 can include read only memory and random access memory and provides instructions and data to processor 430. A portion of memory 440 may also include non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the various components of the service processing device 40 are coupled together by a bus system 450.
  • the bus system 450 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. However, for clarity of description, various buses are labeled as bus system 450 in the figure.
  • Processor 430 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 430 or an instruction in a form of software.
  • the processor 430 described above may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or discrete hardware. Component.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA off-the-shelf programmable gate array
  • the methods, steps, and logical block diagrams disclosed in the embodiments of the present invention may be implemented or carried out.
  • General purpose processor can be micro The processor or the processor can also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present invention may be directly implemented by the hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like.
  • the storage medium is located in memory 440, and processor 430 reads the information in memory 440 and, in conjunction with its hardware, performs the steps of the above method.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in the network device provided by each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium.
  • the technical solution of the present invention which is essential or contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium.
  • a number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Security & Cryptography (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)
  • Stored Programmes (AREA)

Abstract

一种流式应用升级方法及流计算系统，该方法包括：主控节点获取流式应用更新后的逻辑模型（S501），通过对比更新后的逻辑模型以及初始逻辑模型，以确定待调整的流（S502）；根据待调整的流，生成升级指令（S503）；将生成的升级指令下发给第一工作节点，指示第一工作节点调整其上分布的PE之间的数据流走向（S504）。所述方法能够实现流式应用的在线升级，不中断业务。

Description

一种流式应用升级方法、主控节点及流计算系统 技术领域
本发明涉及计算机技术领域，尤其涉及一种流式应用升级方法、一种主控节点以及一种流计算系统。
背景技术
随着大数据时代的到来，对海量数据进行实时处理、分析、决策的市场需求不断扩大，如：电信领域的精准广告推送，金融领域的交易动态实时分析，工业领域的实时监控等等。在此背景下，数据密集型应用，比如金融服务、网络监控、电信数据管理等，已经得到越来越广泛的应用，同时适应于数据密集型应用的流计算系统也应运而生。数据密集型应用产生的数据具有数据量大、快速、时变的特点，当数据密集型应用部署到流计算系统之后，流计算系统可以在接收到应用的数据时就立即对其进行处理，以保证实时性。如图1所示，流计算系统通常包括一个主控节点(Master)和多个工作节点(worker)，主控节点主要负责调度和管理各个工作节点，而工作节点是承载实际的数据处理操作的逻辑实体，工作节点具体通过调用若干个执行单元(PE，Process Element)来对数据进行处理，PE是业务逻辑的物理执行单元。
通常，流计算系统中部署的应用程序或业务被称为流式应用，现有技术中，当部署一个流式应用到流计算系统时，需要预先定义该流式应用的逻辑模型，流式应用的逻辑模型通常以无回路有向图(Directed Acyclic Graph，DAG)来表示，如图2所示，其中PE是承载实际的数据处理操作的物理载体，也是可被流计算系统调度执行的最小单元；stream代表各PE间的传输的数据流，箭头表示了数据流的走向；PE可以动态加载并执行业务逻辑，对流式应用的数据进行实时处理。如图3所示，流计算系统根据逻辑模型将各个PE部署在不同的工作节点上执行，各个PE依照自己的逻辑进行运算，并把运算结果转发给下游PE。而当用户需求或业务场景发生变化时，流式应用就需要更新或升级，初始的逻辑模型已经不再适用。因此首先需要离线完成流式应用的更新，并定义新的逻辑模型，然后停止旧的应用，再根据新的逻辑模型将更新后的流式应用重新部署到流计算系统中，最后再启动更新后的流式应用。可以看出，现有技术中进行流式应用的更新，需要中断原先业务，无法进行流式应用的在线升 级，导致业务损失。
发明内容
本发明实施例提供一种流式应用升级方法、主控节点及流计算系统，用以在不需要中断业务的情况下，实现流计算系统中流式应用的在线升级。
第一方面，本发明实施例提供了一种流式应用升级方法，应用于流计算系统中的主控节点，所述流计算系统包括所述主控节点和至少一个工作节点，其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；所述方法包括：
在所述流式应用更新的情况下，所述主控节点获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；
根据所述待调整的数据流，生成升级指令；
将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整其上分布的PE之间的数据流走向。
在第一方面的第一种可能的实现方式中，所述比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流包括：
比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
在第一方面的第二种可能的实现方式中，所述通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流包括：
比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定待调整的PE以及所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；
所述根据所述待调整的数据流，生成升级指令包括：
根据所述待调整的数据流，生成第一升级指令；根据所述待调整的PE生成第二升级指令；
所述将所述升级指令下发给第一工作节点包括：
将所述第一升级指令下发给所述第一工作节点，将所述第二升级指令下发给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；所述第一升级指令用于指示所述第一工作节点调整其上分布的PE之间的数据流走向，所述第二升级指令用于指示所述第二工作节点调整其上分布的PE的数目。
结合第一方面，或者第一方面第一至第二种任意一种可能的实现方式，在第三种可能的实现方式中，还包括：
所述主控节点根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint；
向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；
当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，所述主控节点触发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
结合第一方面第三种可能的实现方式，在第四种可能的实现方式中，所述待调整的数据流包括：待更新的数据流和待删除的数据流；所述主控节点根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint，包括：
所述主控节点根据与所述待更新的数据流和待删除的数据流相关的PE的状态数据，确定执行数据恢复的checkpoint；根据与所述待更新的数据流和待删除的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE；其中，每个PE的状态数据是该PE在输出事件的触发下备份的，用于指示该PE处理数据的状态。
结合第一方面第二到第四种中任意一种可能的实现方式，在第五种可能的实现方式中，所述待调整的PE包括待新增的PE；所述第二工作节点为所述主控节点根据所述流计算系统中各个工作节点的负载状况选择的工作节点；所述第二升级指令用于指示所述第二工作节点创建所述待新增的PE。
结合第一方面第二到第五种中任意一种可能的实现方式，在第六种可能的 实现方式中，所述待调整的PE包括待删除的PE；所述第二工作节点为所述待删除的PE所在的工作节点；所述第二升级指令用于指示所述第二工作节点删除所述待删除的PE。
结合第一方面，或者第一方面第一至第六种任意一种可能的实现方式，在第七种可能的实现方式中，还包括：
根据所述流式应用的初始逻辑模型配置所述多个PE对所述流式应用的数据进行处理。
结合第一方面，或者第一方面第一至第七种任意一种可能的实现方式，在第八种可能的实现方式中，所述流式应用的初始逻辑模型是用有向无环图DAG表示的。
第二方面，本发明实施例提供了一种流计算系统中的主控节点，，所述流计算系统包括所述主控节点和至少一个工作节点；其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；所述主控节点包括：
获取及比较模块，用于在所述流式应用更新的情况下，获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；
升级指令生成模块，用于根据所述待调整的数据流，生成升级指令；
发送模块，用于将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整其上分布的PE之间的数据流走向。
在第二方面的第一种可能的实现方式中，所述获取及比较模块具体用于：
比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
在第二方面的第二种可能的实现方式中，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；
所述升级指令生成模块具体用于，根据所述待调整的数据流，生成第一升 级指令；根据所述待调整的PE生成第二升级指令；
所述发送模块具体用于，将所述第一升级指令下发给所述第一工作节点，将所述第二升级指令下发给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；所述第一升级指令用于指示所述第一工作节点调整其上分布的PE之间的数据流走向，所述第二升级指令用于指示所述第二工作节点调整其上分布的PE的数目。
结合第二方面，或者第二方面第一至第三种任意一种可能的实现方式，在第三种可能的实现方式中，还包括：
数据恢复模块，用于根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint；
所述发送模块还用于，向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；
所述主控节点还包括：输入触发模块，用于当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，触发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
结合第二方面第三种可能的实现方式，在第四种可能的实现方式中，所述待调整的数据流包括：待更新的数据流和待删除的数据流；所述数据恢复模块具体用于：
所述主控节点根据与所述待更新的数据流和待删除的数据流相关的PE的状态数据，确定执行数据恢复的checkpoint；根据与所述待更新的数据流和待删除的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE；其中，每个PE的状态数据是该PE在输出事件的触发下备份的，用于指示该PE处理数据的状态。
结合第二方面第二到第四种中任意一种可能的实现方式，在第五种可能的实现方式中，所述待调整的PE包括待删除的PE；所述第二工作节点为所述待删除的PE所在的工作节点；所述第二升级指令用于指示所述第二工作节点删除所述待删除的PE。
结合第二方面第二到第五种中任意一种可能的实现方式，在第六种可能的实现方式中，所述待调整的PE包括待新增的PE；所述第二工作节点为所述主 控节点根据所述流计算系统中各个工作节点的负载状况选择的工作节点；所述第二升级指令用于指示所述第二工作节点创建所述待新增的PE。
结合第二方面，或者第二方面第一至第六种任意一种可能的实现方式，在第七种可能的实现方式中，还包括：配置模块，用于根据所述流式应用的初始逻辑模型配置所述多个PE对所述流式应用的数据进行处理。
第三方面，本发明实施例提供了一种流计算系统，，包括：主控节点和至少一个工作节点，其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；
所述主控节点用于：在所述流式应用更新的情况下，获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；根据所述待调整的数据流，生成升级指令；将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整其上分布的PE之间的数据流走向；
所述第一工作节点，用于接收所述主控节点发送的升级指令，根据所述升级指令的指示，调整所述第一工作节点上分布的PE之间的数据流走向。
在第三方面的第一种可能的实现方式中，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
在第三方面的第二种可能的实现方式中，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；根据所述待调整的数据流，生成第一升级指令；根据所述待调整的PE生成第二升级指令；
将所述第一升级指令下发给所述第一工作节点，将所述第二升级指令下发给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；
所述第一工作节点具体用于，接收所述主控节点发送的所述第一升级指令，根据所述第一升级指令的指示，调整所述第一工作节点上分布的PE之间的数据流走向；
所述第二工作节点具体用于，接收所述主控节点发送的所述第二升级指 令，根据所述第二升级指令的指示，调整所述第二工作节点上分布的PE的数目。
结合第三方面，或者第三方面第一至第二种任意一种可能的实现方式，在第三种可能的实现方式中，所述主控节点还用于，根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint；向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，触发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
由上述技术方案可知，本发明实施例提供的流式应用升级方法及流计算系统，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并根据待调整的数据流生成相应的升级指令下发给工作节点，从而在不需要中断业务的情况下，实现流计算系统中流式应用的在线升级。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明提供的流计算系统架构示意图；
图2为本发明实施例提供的流式应用的逻辑模型示意图；
图3为本发明实施例提供的流式应用部署示意图；
图4为本发明实施例提供的流计算系统工作原理图；
图5为本发明实施例提供的流式应用升级方法的流程图；
图6为本发明实施例提供的流式应用更新前后逻辑模型变化示意图；
图7为本发明实施例提供的流式应用更新前后逻辑模型变化示意图；
图8为本发明实施例提供的流式应用升级方法的流程图；
图9为本发明实施例提供的流式应用逻辑模型示意图
图10为本发明实施例提供的流式应用的逻辑模型调整示意图；
图11为本发明实施例提供的流式应用升级之后的PE部署示意图；
图12为本发明实施例提供的PE的输入流和输出流的依赖关系示意图；
图13为本发明实施例提供的PE的输入流和输出流的依赖关系示意图；
图14为本发明实施例提供的PE状态数据备份示意图；
图15为本发明实施例提供的主控节点示意图；
图16为本发明实施例提供的流计算系统示意图；
图17为本发明实施例提供的主控节点示意图。
具体实施方式
为使本发明的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明的技术方案进行清楚地描述。显然，下述的各个实施例都只是本发明一部分的实施例。基于本发明下述的各个实施例，本领域普通技术人员即使没有作出创造性劳动，也可以通过等效变换部分甚至全部的技术特征，而获得能够解决本发明技术问题，实现本发明技术效果的其它实施例，而这些变换而来的各个实施例显然并不脱离本发明所公开的范围。
本发明实施例提供的技术方案可典型地应用于流计算系统中，图4描述了流计算系统的基本结构，该流计算系统包括：主控节点(Master)和多个工作节点(worker)，在集群部署时，主控节点和工作节点都可以有一个或多个，主控节点可以是与工作节点分离的物理节点；在单机部署时，主控节点和工作节点可以是部署在同一个物理节点上的逻辑单元；其中，物理节点具体可以为计算机或服务器。主控节点负责将数据流调度到工作节点上进行处理。通常情况下，一台物理节点即为一个工作节点，在某些情形下，一台物理节点可以对应于多个工作节点，一台物理节点对应的工作节点的数量取决于该物理节点的硬件物理资源。可以将一个工作节点理解为是一份硬件物理资源。对应于同一物理节点的工作节点间采用进程通信的方式进行通信，对应于不同物理节点的工作节点间采用网络通信的方式进行通信。
如图4所示，流计算系统包括主控节点、工作节点1、工作节点2和工作节点3。
主控节点根据流式应用的逻辑模型将流式应用部署到到工作节点1、工作节点2和工作节点3这三个工作节点上进行处理。图3所示的逻辑模型为包含九个执行单元(PE，Process Element)：PE1至PE9，以及这九个PE之间的数据流走向的逻辑关系图，PE之间数据流的走向也体现了各个PE的输入流和输 出流的依赖关系。需要说明的是，本发明实施例中数据流也简称为流(stream)。
主控节点根据流式应用的逻辑模型，配置工作节点1上的PE1、PE2和PE3，工作节点2上的PE4、PE7和PE9，以及工作节点3上的PE5、PE6、PE8来对流式应用的数据流进行处理。可以看出，经过配置之后，工作节点1、2、3上的PE之间的数据流走向与流式应用的逻辑模型是匹配的。
本发明实施例中的流式应用的逻辑模型可以为有向无环图(Directed Acyclic Graph，DAG)、树状图或者含有环的图。流式应用的逻辑模型可以参阅图2进行理解，如图2所示的流式计算应用图包括7个算子，分别从PE1到PE7，有8条数据流，分别从s1到s8。图2中明确标出了数据流的走向，例如：数据流s1是从PE1到算子PE5，表示PE5处理的是PE1输出的流，即PE5的输出依赖于PE1的输入，通常也称PE5为PE1的下游PE，而PE1为PE5的上游PE；可以理解的是，上游PE和下游PE是根据PE之间的数据流走向来确定的，对于一条数据流来说，与其相关的有且仅有两个PE：输出该数据流的源PE，以及该数据流流向的目的PE(也即接收该数据流的PE)，从数据流的方向来看，源PE为目的PE的上游PE，而目的PE为源PE的下游PE；进一步地，数据流S2输入PE2，并经PE2逻辑处理后，产生两条数据流S3和S4，分别进入PE3和PE4进行逻辑处理，同理，PE2也为PE1的下游PE，而PE1为PE2的上游PE；PE4输出的数据流S4和PE3输出的数据流S7均作为PE6的输入，即PE6的输出依赖于PE3和PE4的输入。需要说明的是，在本发明实施例中，将输出依赖于单一PE的输入的PE定义为无状态PE，比如PE5，PE3，PE4；将输出依赖于多个PE的输入的PE定义为有状态PE，比如PE6，PE7。数据流中包含单个数据段，称为元组，元组可以是结构化或非结构化数据。通常，元组中可表示特定时间点某事物的状态，流计算系统中的PE以元组为单位对流式应用产生的数据流进行处理，也可以认为元组是流计算系统中的数据的最小粒度划分和表示。
还需要说明的是，流计算系统只是本发明技术方案的一个典型应用场景，并不对本发明的应用场景构成限制，其它涉及分布式系统或云计算系统应用部署与升级的应用场景，本发明实施例的技术方案均适用。
本发明实施例提供一种流式应用升级方法，该方法可典型地应用与图1及图4所示的流计算系统中，假设该流计算系统上部署有流式应用A，且流式 应用A的初始逻辑模型为D1，则该流计算系统的主控节点根据初始逻辑模型D1，部署多个执行单元(PE)来对流式应用A的数据流进行处理，其中，该多个执行单元PE分布于流计算系统的一个或多个工作节点上；如图6所示，当流式应用A发生升级或更新后，其逻辑模型也相应地发生更新(假设更新后的逻辑模型为D2)，通常逻辑模型的更新一般由开发人员来完成，或者由开发人员借助开发工具来完成，本发明不做特别限定。如图5所示，该流式应用升级方法的主要流程描述如下：
S501：在流式应用A更新的情况下，流计算系统的主控节点获取流式应用A更新后的逻辑模型D2；
S502：主控节点通过对比更新后的逻辑模型D2以及初始逻辑模型D1，以确定待调整的数据流；
S503：主控节点根据待调整的数据流，生成升级指令；
S504：主控节点将生成的升级指令下发给第一工作节点，其中，第一工作节点是与待调整的数据流相关的PE所在的工作节点，该升级指令用于指示第一工作节点调整其上分布的PE之间的数据流走向。
需要说明的是，本发明实施例中的待调整的数据流可以为一条，也可以为多条，视具体的情况而定，而对于每条待调整的数据流来说，与其相关的PE具体是指该待调整的数据流的源PE和目的PE，其中，该待调整的数据流的源PE为输出该待调整的数据流的PE,待调整的数据流的目的PE为接收该待调整的数据流的PE或该待调整的数据流的源PE的下游PE。
本发明实施例提供的流式应用升级方法及流计算系统，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并根据待调整的数据流生成相应的升级指令下发给工作节点，从而在不需要中断业务的情况下，实现流计算系统中流式应用的在线升级。
在本发明实施例中，流式应用的逻辑模型用于表示处理该流式应用的数据的多个PE以及该多个PE之间的数据流走向；当流式应用发生升级或更新后，其逻辑模型也相应地发生更新，通常情况下，更新后的逻辑模型与初始的逻辑模型的差异主要分为两种类型：(1)初始逻辑模型所表示的PE与更新后的逻辑模型所表示的PE完全相同，仅仅是PE之间的数据流走向发生变化；(2)初始逻辑模型所表示的PE与更新后的逻辑模型所表示的PE不完全相同，同 时PE之间的数据流走向也发生变化；针对上述两种类型的差异，下面分别介绍相应的处理流程。
在一个具体的实施例中，如图6所示，流式应用的初始逻辑模型所表示的PE与该流式应用更新后的逻辑模型所表示的PE完全相同，而PE之间的数据流走向发生变化，根据图6，流式应用更新前后，其逻辑模型中的PE均为PE1-PE7，完全相同，但是数据流走向发生了变化，即原来由PE4流向PE6的数据流变为由PE4流向PE7的数据流S11，同时新增了从PE2流向PE6的数据流S12。在这种情况下，流式应用升级方法的主要流程如下：
步骤1：通过比较流式应用的初始逻辑模型以及该流式应用更新后的逻辑模型，确定待调整的数据流；其中，该待调整的数据流包括一条或多条数据流；具体地，在一个优选的实施例中，待调整的数据流可以包括：待新增的数据流、待删除的数据流或待更新的数据流中的至少一种，其中待更新的数据流是指流式应用的逻辑模型更新后目的节点或者源节点之一发生变化的数据流；具体到图6所示的情形，待调整的数据流包括：待新增的数据流S12、待更新的数据流S11。
步骤2：根据待调整的数据流，生成升级指令；其中，升级指令可以包括一条或多条指令，且升级指令与待调整的数据流的类型相关，例如，若待待调整的数据流包括待新增的数据流和待更新的数据流，则生成的升级指令包括：用于新增数据流的指令和用于更新数据流的指令；其中不同类型的升级指令可以可以为单独的指令，也可以集成到一条指令当中，也本发明不做特别限定。具体到图6所示的情形，生成的升级指令包括：新增数据流S12的指令和将数据流S6更新为S11的指令。
步骤3：将生成的升级指令下发给第一工作节点，第一工作节点是与待调整的数据流相关的PE所在的工作节点；可以理解的是，第一工作节点可能为一个或多个，第一工作节点在接收到该升级指令后，执行升级指令所指示的操作，比如新增数据流S12，以及将数据流S6更新为S11，从而使得第一工作节点上分布的PE之间的数据流走向发生调整，并且调整之后的数据流走向与更新后的逻辑模型相匹配。
进一步的，由于在第一工作节点对其上分布的PE之间的数据流进行调整时，可能会使正在处理的数据发生丢失，因此需要对数据进行恢复，具体地， 在一个实施例中，在第一工作节点对其上分布的PE之间的数据流进行调整之前，主控节点根据与待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和该目标PE执行数据恢复的检查点checkpoint；向该目标PE所在工作节点下发数据恢复指令，该数据恢复指令用于指示该目标PE根据该checkpoint恢复数据；当主控节点确定第一工作节点完成调整，且第一工作节点上分布的PE准备就绪之后，主控节点触发该目标PE将恢复的数据输入该目标PE的下游PE进行处理。
需要说明的是，主控节点可以通过主动发送询问消息的方式感知流计算系统中各个工作节点上PE的状态，也可以由工作节点将自身分布的各个PE的状态上报给主控节点，PE的状态包括：运行状态、就绪状态和停止状态；当PE与上下游的PE的通道建立成功，就表示PE出于就绪状态，可以接收数据流并进行处理了。
可选地，主控节点在执行上述流式应用升级方法步骤之前，还可以根据上述流式应用的初始逻辑模型配置多个PE对流式应用的数据进行处理。
本发明实施例提供的流式应用升级方法，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并生成相应的升级指令下发给工作节点，以完成流式应用的在线升级，从而可以保证应用升级过程中不需要中断原先业务；进一步的，通过在升级的过程中对数据进行恢复，以确保不会造成关键数据丢失，影响业务运行。
在另一个具体的实施例中，如图7所示，流式应用的初始逻辑模型所表示的PE与该流式应用更新后的逻辑模型所表示的PE不完全相同，且PE之间的数据流走向也发生了变化，根据图7，流式应用更新前后，其逻辑模型中的PE数目发生了变化(PE2、PE3、PE4和PE6被删除，新增了PE9-PE13)，数据流走向也发生了变化，即原来的数据流S4、S5、S6和S7被删除，同时新增了数据流S11-S16，原来的数据流S3的目的PE发生了更新，原来的数据流S9的源PE发生了更新。在这种情况下，如图8所示，流式应用升级方法的主要流程如下：
S801：主控节点通过比较流式应用的初始逻辑模型以及该流式应用更新后的逻辑模型，确定待调整的PE和待调整的数据流；其中，该待调整的PE包括一个或多个PE，该待调整的数据流包括一条或多条数据流；具体地，在一 个优选的实施例中，该待调整的PE包括：待新增的PE或待删除的PE中的至少一种，待调整的数据流可以包括：待新增的数据流、待删除的数据流或待更新的数据流中的至少一种。
具体地，如图9所示，主控节点通过对比流式应用更新前后的逻辑模型，可以确定需要将原始逻辑模型中PE2、PE3、PE4和PE6组成的子逻辑模型替换成由PE9-PE13组成的子逻辑模型，才能与更新后的逻辑模型相同；因此，确定PE2、PE3、PE4、PE6，以及PE9-PE13为待调整的PE(其中，PE2、PE3、PE4和PE6为待删除的PE，PE9-PE13为待新增的PE)，以及确定与待调整的PE相关的数据流，即待调整的PE的所有输入流和输出流为待调整的流，如图9中的虚线部分所指示的流为待删除的数据流，黑色加粗部分所指示的流为待新增的数据流，浅色加粗部分所指示的流为待更新的数据流。
S802：主控节点根据待调整的数据流，生成第一升级指令；根据待调整的PE生成第二升级指令；其中，第一升级指令和第二升级指令分别可以包括一条或多条指令，且第一升级指令与待调整的数据流的类型相关，第二升级指令与待调整的PE的类型相关，例如，若待调整的数据流包括待新增的数据流和待更新的数据流，则生成的第一升级指令包括：用于新增数据流的指令和用于更新数据流的指令；若待调整的PE包括待新增的PE，则生成的第二升级指令包括：用于新增PE的指令；其中，第一升级指令和第二升级指令可以为单独的指令，也可以集成到一条指令中，也本发明不做特别限定。具体到图7所示的情形，生成的第一升级指令包括：删除数据流的指令、新增数据流的指令和更新数据流的指令，第二升级指令包括：新增PE的指令，删除PE的指令。
在一个具体的实施例中，如图9所示，主控节点通过对比流式应用更新前后的逻辑模型，确定出待调整的PE和待调整的流之后，可以进一步确定调整策略，即如何对PE和流进行调整，才能使得调整之后PE的部署情况(包括PE的个数以及PE之间数据流的依赖关系)与流式应用更新后的逻辑模型相匹配。调整策略包含两部分内容：(1)PE数目的调整策略，即需要新增哪些PE和/或需要删除哪些PE；(2)PE之间数据流走向的调整策略，即哪些PE的数据流走向需要更新，哪些数据流需要增加，以及哪些数据流需要删除。
在一个优选的实施例中，调整策略主要包括如下情形中的至少一种：
(1)更新流:数据流的目的节点或者源节点之一发生变化；
(2)删除流:数据流流在应用更新后需废弃；
(3)新增流:数据流原先不存在，应用更新后新增的流；
(4)删除PE:应用更新后需废弃的PE；
(5)新增PE:应用更新后新增的PE。
具体到图7和图9所示的逻辑模型，结合图10可以看出，需要新增5个PE(PE9-PE13)，以及新增PE9-PE13之间的数据流；需要删除PE2、PE3、PE4和PE6，以及删除PE2、PE3、PE4和PE6之间的数据流；同时，由于PE1的输出流的目的PE发生了变化(由PE2变为PE9)，PE7的输入流也发生了变化(由PE6的输出流变为PE13的输出流，即流的源节点发生变化)，因此，需要对PE1的输出流和PE7的输入流进行更新。基于上述分析，可以得出调整策略为：
(1)新增PE9-PE13；
(2)新增PE9-PE13之间的流；其中，PE9-PE13之间的数据流走向是由更新后的逻辑模型决定的；
(3)删除PE2、PE3、PE4和PE6；
(4)删除PE2、PE3、PE4和PE6之间的流；
(5)将PE1的输出流的目的PE由PE2改为PE9；将PE7的输入流的源PE由PE6改为PE13。
调整策略确定之后，基于确定出的调整策略，主控节点即可生成升级指令，升级指令用于指示工作节点(具体而言是待调整的PE所在的工作节点和待调整的数据流所涉及的PE所在的工作节点)实施确定出的调整策略。与调整策略相对应，升级指令包括：新增PE指令、删除PE指令、更新流指令、删除流指令或新增流指令中的至少一个。具体到图7和图9所示的逻辑模型，升级指令具体包括：
(1)新增PE9-PE13的指令；
(2)新增PE9-PE13之间的流的指令；
(3)删除PE2、PE3、PE4和PE6的指令；
(4)删除PE2、PE3、PE4和PE6之间的流的指令；
(5)将PE1的输出流的目的PE由PE2改为PE9的指令，以及将PE7的输入流的源PE由PE6改为PE13的指令。
S803：主控节点将生成的第一升级指令下发给第一工作节点，将生成的第二升级指令下发给第二工作节点，其中，第一工作节点是与待调整的数据流相关的PE所在的工作节点，第二工作节点包括待调整的PE所在的工作节点；可以理解的是，第一工作节点和第二工作节点均可能为一个或多个，并且第一工作节点和第二工作节点可以存在重叠，即一个工作节点可能即属于第一工作节点，也同时属于第二工作节点；第一升级指令用于指示第一工作节点调整其上分布的PE之间的数据流走向，第二升级指令用于指示所述第二工作节点调整其上分布的PE的数目；第一工作节点和第二工作节点在接收到升级指令后，执行升级指令所指示的操作，从而使得第一工作节点和第二工作节点上分布的PE以及PE之间的数据流走向得到调整。可以理解的是，第二工作节点调整其上分布的PE的数目具体可以是创建新的PE和/或删除已创建的PE。
可选地，在一个具体的实施例中，如果待调整的PE包括待删除的PE；则第二工作节点包括该待删除的PE所在的工作节点；第二升级指令用于指示该第二工作节点删除该待删除的PE。
可选地，在另一个具体的实施例中，如果待调整的PE包括待新增的PE；则第二工作节点可以为主控节点根据流计算系统中各个工作节点的负载状况选择的工作节点，也可以为主控节点随机选择的工作节点；第二升级指令用于指示第二工作节点创建该待新增的PE。
具体到图7和图9所示的逻辑模型，如图11所示，主控节点将新增PE9指令发送给worker2，将新增PE10的指令发送给worker3，将新增PE11和PE11的指令发送给worker4，将新增PE13的指令发送给worker6；将删除PE2、PE3的指令发送给worker3，将删除PE4、PE6的指令发送给worker4；将删除PE2和PE3之间流的指令发送给PE2和PE3初始所在的工作节点worker3，将删除PE3和PE6之间数据流的指令发送给PE3所在的工作节点worker3和PE6所在的工作节点worker4；其余的指令以此类推，此处不再赘述。需要说明的是，每个工作节点都维护了该工作节点上所有PE的数据流配置信息，每个PE的数据流配置信息包括源地址、目的地址、端口号等信息，因此对数据流的删除和更新，实质上是通过修改数据流配置信息来实现的。
如图11所示，根据主控节点下发的升级指令，worker 2新增了PE9，worker3删除了PE2、PE3，新增了PE10，worker 4删除了PE6、PE4，新增了PE11， PE12，worker 6新增了PE13；同时，worker1-6通过删除流、新增流、更新流等操作，对PE之间的数据流走向也进行了调整，具体而言：新增了PE9-PE13之间的流，删除了PE2、PE3、PE4和PE6之间的流，将PE1的输出流的目的PE由PE2改为PE9；将PE7的输入流的源PE由PE6改为PE13。从图11可以看出，调整之后PE的部署情况(包括PE的个数以及PE之间数据流的依赖关系)与流式应用A更新后的逻辑模型相匹配。
进一步的，由于在第一工作节点和第二工作节点对其上分布的PE以及PE之间的数据流进行调整时，可能会使正在处理的数据发生丢失，因此需要对数据进行恢复，具体地，在一个实施例中，该流式应用升级方法还包括：
S804：主控节点根据与待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和该目标PE执行数据恢复的检查点checkpoint；向该目标PE所在工作节点下发数据恢复指令，该数据恢复指令用于指示该目标PE根据该checkpoint恢复数据；当主控节点确定第一工作节点和第二工作节点完成调整，且第一工作节点和第二工作节点上分布的PE准备就绪之后，主控节点触发该目标PE将恢复的数据输入该目标PE的下游PE进行处理。需要说明的是，主控节点可以通过主动发送询问消息的方式感知流计算系统中各个工作节点上PE的状态，也可以由工作节点将自身分布的各个PE的状态上报给主控节点，PE的状态包括：运行状态、就绪状态和停止状态；当PE与上下游的PE的通道建立成功，就表示PE出于就绪状态，可以接收数据流并进行处理了。
在流式应用更新或升级过程中，因为调整PE的部署需要涉及数据流的调整，而在对PE的部署进行调整时，可能有些数据正在处理，为了保证升级过程中数据不丢失，需要根据待调整的数据流相关的PE的的原始输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint，以保证在应用升级之前尚未被PE处理完毕的数据在升级完成之后能够继续被重新处理；其中，这里所说的需要恢复的数据，通常是指元组。
在一个具体的实施例中，如图12所示，待调整的数据流相关的{PE1，PE2,PE3,PE4,PE6,PE7}组成的子逻辑模型的输入/输出关系如下:元组i1、i2、i3和i4由PE1输入PE2之后，经PE2处理之后得到元组k1、k2、k3和j1，然后元 组k1、k2、k3被输入PE4进行处理，得到m1，元组j1被输入PE3进行处理，得到l1，PE6对m1处理之后得到O2，对l1处理之后得到O1。基于上述输入/输出关系，可分析得到待调整的PE的输入流和输出流的依赖关系，如图13所示：
O1依赖于PE6的输入l1，而l1依赖于j1，j1又依赖于i2，所以对于整个子逻辑模型而言，PE6的输出O1依赖于PE2的输入i2；
O2依赖于PE6的输入m1，而m1依赖于PE4的输入k1、k2和k3，k1、k2和k3又依赖于i1、i3和i4，所以对于整个子逻辑模型而言，PE6的输出O2依赖于PE2的输入i1、i3和i4。通过分析得出的上述依赖关系可知，PE2,PE3,PE4和PE6均依赖于PE1的输出，因此，当第一工作节点和第二工作节点对其上分布的PE以及PE之间的数据流进行调整时，PE2,PE3,PE4和PE6中尚有数据未被处理完毕，则PE1需要对数据恢复，即PE1为目标PE。
进一步地，根据第一工作节点和第二工作节点对其上分布的PE以及PE之间的数据流进行调整时，待调整的数据流相关的PE备份的最新的状态数据，可以确定输入待调整的数据流相关的PE的数据当前是否已经处理完毕并输出给下游PE，进而可以确定目标PE执行数据恢复的检查点checkpoint。需要说明的是，PE的状态数据用于表示PE的处理数据状态，其具体包含的内容是本领域技术人员熟知的，例如状态数据可包括：元组接收队列中的缓存数据、消息通道中的缓存数据、PE在处理自身接收队列中的一个或多个普通元组的过程中产生的数据(比如当前处理的普通元组的处理结果及中间过程数据)中的一种或多种数据。需要说明的是，由于新增数据流并不需要进行数据恢复，因此在确定执行数据恢复的checkpoint，以及需要进行数据恢复的目标PE时，不需要用到待新增数据流相关的PE的状态信息，以及待新增数据流相关的PE的输入流和输出流的依赖关系。例如在一个实施例中，因此如果待调整的数据流中包括：待更新的数据流、待删除的数据流和待新增的数据流，则只需要根据待更新的数据流以及待删除的数据流相关的PE的状态数据，就可以确定执行数据恢复的checkpoint，根据待更新的数据流以及待删除的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE；类似的，如果待调整的数据流中包括：待更新的数据流和待新增的数据流，则只需要根据待更新的数据流相关的PE的状态数据，以及待更新的数据流相关的PE的输入流和输出流的依赖关系，就可以确定执行数据恢复的checkpoint，以及需 要进行数据恢复的目标PE。
需要说明的是，在本发明的一个实施例中，PE的状态数据是周期性备份的，即流计算系统周期性地触发各个PE对自身的状态数据进行备份，PE在收到检查点(checkpoint)事件之后，进行将自身当前的状态数据进行备份，并记录checkpoint，同时清理过期的数据。本领域技术人员可以理解的是，checkpoint可以理解为数据备份的记录点或者是备份数据的索引，一个checkpoint对应于一次数据备份操作，在不同的时候备份的数据具有不同的checkpoint，且通过checkpoint可以查询并得到在该checkpoint下备份的数据。在本发明的另一个实施例中，可以使用输出触发机制(由PE输出触发)来进行状态数据的备份，如图14所示，当PE完成对输入流Input_Stream1-5的处理，并输出处理结果Output_Stream1时，触发模块触发状态数据处理模块，状态数据处理模块进而启动新的checkpoint记录PE最新的状态数据至内存或磁盘，这样的触发方式精准有效，相比于周期性触发的方式，效率更好，可避免过多的资源消耗；进一步地，状态数据处理模块还可以清理上一个checkpoint记录的历史数据，从而减少中间数据，有效节省存储空间。
下面以图12所示的情形为例，详细说明根据PE的输入流和输出流的依赖关系和状态数据来确定需要进行数据恢复的目标PE和该目标PE执行数据恢复的检查点checkpoint的过程：若根据待调整的数据流相关的{PE1，PE2,PE3,PE4,PE6,PE7}的状态数据判断出输入PE6尚未完成对元组m1的处理，或者对元组m1处理之后得到的O2尚未送达PE6的下游PE：PE7，则根据上述输入流和输出流的依赖关系可以确定出：O2所依赖的i1、i3和i4需要被恢复，且应该由输出i1、i3和i4的PE1来完成数据恢复，即需要恢复数据的目标PE为PE1，从而可以确定出可以恢复出i1、i3和i4的checkpoint。这样，在第一工作节点和第二工作节点调整自身PE的部署情况之前，目标PE可以根据确定出的checkpoint恢复数据i1、i3和i4，且当第一工作节点和所述第二工作节点完成调整，且第一工作节点和第二工作节点上分布的PE准备就绪之后，目标PE重新将恢复的数据i1、i3和i4其下游PE进行处理，从而确保升级过程中不会发生数据的丢失，达到无损升级的目的。
可选地，主控节点在执行上述流式应用升级方法步骤之前，还可以根据上述流式应用的初始逻辑模型配置多个PE对流式应用的数据进行处理。
本发明实施例提供的流式应用升级方法，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并生成相应的升级指令下发给工作节点，以完成流式应用的在线升级，从而可以保证应用升级过程中不需要中断原先业务；进一步的，通过在升级的过程中对数据进行恢复，以确保不会造成关键数据丢失，影响业务运行。
基于上述方法及系统实施例，本发明实施例还提供一种流计算系统中的主控节点，该主控节点可以为一台计算机或服务器；该流计算系统还包括至少一个工作节点；假设该流计算系统上部署有流式应用A，多个执行单元(PE)分布于上述至少一个工作节点中的一个或多个工作节点上，用于对流式应用A的数据进行处理，其中流式应用A的逻辑模型用于表示处理该流式应用的数据的多个PE以及该多个PE之间的数据流走向；假设流式应用A的初始逻辑模型为D1，当流式应用A发生升级或更新后，其逻辑模型也相应地发生更新(假设更新后的逻辑模型为D2)，如图15所示，主控节点30包括：
获取及确定模块301，用于在流式应用A发生更新的情况下，获取流式应用A更新后的逻辑模型D2，并通过对比更新后的逻辑模型D2以及初始逻辑模型D1，以确定待调整的数据流；
升级指令生成模块302，用于根据待调整的流，生成升级指令；
发送模块303，用于将生成的升级指令下发给第一工作节点，以使第一工作节点根据该升级指令的指示，调整第一工作节点上分布的PE之间的数据流走向；其中第一工作节点是流计算系统包含的工作节点中的一个或多个工作节点，且第一工作节点是与待调整的数据流相关的PE所在的工作节点。
需要说明的是，本发明实施例中的待调整的数据流可以为一条，也可以为多条，视具体的情况而定，而对于每条待调整的数据流来说，与其相关的PE具体是指该待调整的数据流的源PE和目的PE，其中，该待调整的数据流的源PE为输出该待调整的数据流的PE,待调整的数据流的目的PE为接收该待调整的数据流的PE或该待调整的数据流的源PE的下游PE。
本发明实施例提供的流计算系统中的主控节点，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并根据待调整的数据流生成相应的升级指令下发给工作节点，从而在不需要中断业务的情况下，实现流计算系统中流式应用的在线升级。
进一步地，根据更新后的逻辑模型与初始的逻辑模型的差异的类型，获取及确定模块301的在具体的处理细节上也存在一定差异。例如，在一个优选的实施例中，获取及比较模块301具体用于：
比较流式应用A的初始逻辑模型以及所述更新后的逻辑模型，以确定待调整的数据流，其中流式应用A的初始逻辑模型D1所表示的PE与更新后的逻辑模型D2所表示的PE相同。
在另一个优选的实施例中，获取及比较模块301具体用于：比较初始逻辑模型D1以及更新后的逻辑模型D2，以确定待调整的PE以及待调整的数据流，其中流式应用A的初始逻辑模型D1所表示的PE与更新后的逻辑模型D2所表示的PE不完全相同。相应地，在这种情况下，升级指令生成模块302具体用于，根据获取及比较模块301确定的待调整的数据流，生成第一升级指令；根据获取及比较模块301确定的待调整的PE生成第二升级指令；发送模块303具体用于：将:第一升级指令下发给第一工作节点，将第二升级指令下发给第二工作节点，其中第二工作节点包括待调整的PE所在的工作节点，第一升级指令用于指示第一工作节点调整其上分布的PE之间的数据流走向，第二升级指令用于指示第二工作节点调整其上分布的PE的数目。
进一步地，在一个优选的实施例中，主控节点30还包括：
数据恢复模块304，用于根据与待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和该目标PE执行数据恢复的checkpoint；
发送模块303还用于，在数据恢复模块304确定出目标PE和checkpoint之后，向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；可以理解的是，该数据恢复指令是根据数据恢复模块304确定出的目标PE和checkpoint中来构造的，其包含有指示该checkpoint的信息。
相应地，主控节点30还包括：输入触发模块305，用于当确定第一工作节点完成调整，且第一工作节点上分布的PE均准备就绪之后，触发数据恢复模块304确定出的目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
本发明实施例提供的流计算系统中的主控节点，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并生成相应的升级指令下 发给工作节点，以完成流式应用的在线升级，从而可以保证应用升级过程中不需要中断原先业务；进一步的，由于在升级的过程中对数据进行了恢复，因此不会造成关键数据丢失，影响业务运行。
本发明提供的流计算系统中的主控节点用于实施上述方法实施例中的流式应用升级方法，其具体实现细节，可以参照上述方法实施例，此处不再赘述。
本发明实施例还提供一种流计算系统，用于实现本发明实施例提供的流式应用升级方法，如图16所示，该流计算系统包括：主控节点30和至少一个工作节点(如图16中的工作节点31-34)；主控节点30根据流式应用的初始逻辑模型配置多个执行单元(PE)来对该流式应用的数据流进行处理，流式应用的初始逻辑模型用于表示处理所述流式应用的数据的多个PE以及该多个PE之间的数据流走向；其中，如图16所示，配置的多个PE分布于一个或多个工作节点上；主控节点30用于：在流式应用A发生更新的情况下，获取流式应用更新后的逻辑模型，并通过对比更新后的逻辑模型以及初始逻辑模型，以确定待调整的数据流；根据待调整的流，生成升级指令；将生成的升级指令下发给第一工作节点，其中第一工作节点是该流计算系统包含的工作节点中的一个或多个工作节点，且第一工作节点是与待调整的数据流相关的PE所在的工作节点。
第一工作节点，用于接收主控节点30发送的升级指令，根据该升级指令的指示，调整第一工作节点上分布的PE之间的数据流走向。
本发明实施例提供的流计算系统，通过对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并生成相应的升级指令下发给工作节点，以完成流式应用的在线升级，从而可以保证应用升级过程中不需要中断原先业务。
具体地，在一个实施例中，升级指令包括：新增PE指令、删除PE指令、更新流指令、删除流指令或新增流指令中的至少一个。相应地，第一工作节点在接收到升级指令后执行：新增执行单元、删除执行单元、更新流、删除流或新增流中的至少一项操作，使得执行上述操作之后PE的部署情况(包括PE的个数以及PE之间数据流的依赖关系)与流式应用A更新后的逻辑模型相匹配。
优选地，在一个实施例中，主控节点具体用于：比较流式应用的初始逻辑 模型以及所述更新后的逻辑模型，以确定待调整的数据流，其中流式应用的初始逻辑模型所表示的PE与更新后的逻辑模型所表示的PE相同。
优选地，在另一个实施例中，主控节点具体用于：比较流式应用的初始逻辑模型以及更新后的逻辑模型，以确定待调整的PE以及待调整的数据流，其中流式应用的初始逻辑模型所表示的PE与更新后的逻辑模型所表示的PE不完全相同；根据待调整的数据流，生成第一升级指令；根据待调整的PE生成第二升级指令；将第一升级指令下发给第一工作节点，将第二升级指令下发给第二工作节点；其中，第一工作节点是与待调整的数据流相关的PE所在的工作节点，第二工作节点包括待调整的PE所在的工作节点。相应地，第一工作节点具体用于，接收主控节点30发送的第一升级指令，根据第一升级指令的指示，调整第一工作节点上分布的PE之间的数据流走向；第二工作节点用于，接收主控节点30发送的第二升级指令，根据第二升级指令的指示，调整第二工作节点上分布的PE的数目。
优选地，在另一个实施例中，主控节点30还用于，根据与待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和该目标PE执行数据恢复的检查点checkpoint；向该目标PE所在工作节点下发数据恢复指令，该数据恢复指令用于指示该目标PE根据该checkpoint恢复数据；当确定第一工作节点完成调整，且第一工作节点上分布的PE均准备就绪之后，触发该目标PE将恢复的数据输入该目标PE的下游PE进行处理。
需要说明的是，本发明提供的流计算系统用于实施上述方法实施例这种的流式应用升级方法，其具体实现细节，可以参照上述方法实施例，此处不再赘述。本发明实施例中的执行单元(PE)可以以软件形态存在，例如进程、线程或软件功能模块，也可以以硬件的形态存在，比如处理器核，或具有数据处理能力的逻辑电路等，通过读取存储器中的可执行代码或业务处理逻辑，实现本发明实施例所描述的功能，本发明不做特别限定。
本发明实施例还提供一种流计算系统中的主控节点，该主控节点可以为一台计算机或服务器。图17是本发明实施例提供的主控节点40的结构示意图。主控节点40可包括输入设备410、输出设备420、处理器430和存储器440。
本发明实施例提供的主控节点40应用于流计算系统，所述流计算系统还包括工作节点，所述流计算系统部署有流式应用。
存储器440可以包括只读存储器和随机存取存储器，并向处理器430提供指令和数据。存储器440的一部分还可以包括非易失性随机存取存储器(NVRAM)。
存储器440存储了操作指令、操作系统(包括各种系统程序，用于实现各种基础业务以及处理基于硬件的任务)、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集。
在本发明实施例中，当该流式应用发生更新后，处理器430通过调用存储器440存储的操作指令(该操作指令可存储在操作系统中)，执行如下操作：
通过输入设备410获取该流式应用更新后的逻辑模型，通过对比更新后的逻辑模型以及初始逻辑模型，以确定待调整的数据流；根据待调整的流，生成升级指令；将生成的升级指令下发给第一工作节点，其中第一工作节点是该流计算系统包含的工作节点中的一个或多个工作节点，且第一工作节点是与待调整的数据流相关的PE所在的工作节点。
本发明实施例提供的主控节点，对比流式应用发生更新前后逻辑模型的差异，动态确定待调整的数据流，并生成相应的升级指令下发给工作节点，以完成流式应用的在线升级，从而可以保证应用升级过程中不需要中断原先业务。
处理器430控制业务处理的装置40的操作，处理器430还可以称为CPU(Central Processing Unit，中央处理单元)。存储器440可以包括只读存储器和随机存取存储器，并向处理器430提供指令和数据。存储器440的一部分还可以包括非易失性随机存取存储器(NVRAM)。具体的应用中，业务处理的装置40的各个组件通过总线系统450耦合在一起，其中总线系统450除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都标为总线系统450。
上述本发明实施例揭示的方法可以应用于处理器430中，或者由处理器430实现。处理器430可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器430中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器430可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本发明实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处 理器或者该处理器也可以是任何常规的处理器等。结合本发明实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器440，处理器430读取存储器440中的信息，结合其硬件完成上述方法的步骤。
应该理解，在本申请所提供的几个实施例所揭露数据备份和流计算系统还可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例提供的网络设备中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (20)

  1. 一种流式应用升级方法，其特征在于，应用于流计算系统中的主控节点，所述流计算系统包括所述主控节点和至少一个工作节点，其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；所述方法包括：
    在所述流式应用更新的情况下，所述主控节点获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；
    根据所述待调整的数据流，生成升级指令；
    将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整所述第一工作节点上分布的PE之间的数据流走向。
  2. 根据权利要求1所述的升级方法，其特征在于，所述比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流包括：
    比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
  3. 根据权利要求1所述的升级方法，其特征在于，所述通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流包括：
    比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定待调整的PE以及所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；
    所述根据所述待调整的数据流，生成升级指令包括：
    根据所述待调整的数据流，生成第一升级指令；根据所述待调整的PE生成第二升级指令；
    所述将所述升级指令下发给第一工作节点包括：
    将所述第一升级指令下发给所述第一工作节点，将所述第二升级指令下发 给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；所述第一升级指令用于指示所述第一工作节点调整所述第一工作节点上分布的PE之间的数据流走向，所述第二升级指令用于指示所述第二工作节点调整所述第二工作节点上分布的PE的数目。
  4. 根据权利要求1-3任一项所述的升级方法，其特征在于，还包括：
    所述主控节点根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint；
    向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；
    当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，所述主控节点触发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
  5. 根据权利要求4所述的升级方法，其特征在于，所述待调整的数据流包括：待更新的数据流和待删除的数据流；所述主控节点根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint，包括：
    所述主控节点根据与所述待更新的数据流和待删除的数据流相关的PE的状态数据，确定执行数据恢复的checkpoint；根据与所述待更新的数据流和待删除的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE；其中，每个PE的状态数据是该PE在输出事件的触发下备份的，用于指示该PE处理数据的状态。
  6. 根据权利要求3至5任一项所述的升级方法，其特征在于，所述待调整的PE包括待新增的PE；所述第二工作节点为所述主控节点根据所述流计算系统中各个工作节点的负载状况选择的工作节点；所述第二升级指令用于指示所述第二工作节点创建所述待新增的PE。
  7. 根据权利要求3至6任一项所述的升级方法，其特征在于，所述待调整的PE包括待删除的PE；所述第二工作节点为所述待删除的PE所在的工作节点；所述第二升级指令用于指示所述第二工作节点删除所述待删除的PE。
  8. 根据权利要求1至7任一项所述的升级方法，其特征在于，还包括：
    根据所述流式应用的初始逻辑模型配置所述多个PE对所述流式应用的数据进行处理。
  9. 根据权利要求1至8任一项所述的升级方法，其特征在于，所述流式应用的初始逻辑模型是用有向无环图DAG表示的。
  10. 一种流计算系统中的主控节点，其特征在于，所述流计算系统包括所述主控节点和至少一个工作节点；其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；所述主控节点包括：
    获取及比较模块，用于在所述流式应用更新的情况下，获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；
    升级指令生成模块，用于根据所述待调整的数据流，生成升级指令；
    发送模块，用于将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整所述第一工作节点上分布的PE之间的数据流走向。
  11. 根据权利要求10所述的主控节点，其特征在于，所述获取及比较模块具体用于：
    比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
  12. 根据权利要求10所述的主控节点，其特征在于，所述获取及比较模块具体用于：比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定待调整的PE以及所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；
    所述升级指令生成模块具体用于，根据所述待调整的数据流，生成第一升级指令；根据所述待调整的PE生成第二升级指令；
    所述发送模块具体用于，将所述第一升级指令下发给所述第一工作节点， 将所述第二升级指令下发给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；所述第一升级指令用于指示所述第一工作节点调整所述第一工作节点上分布的PE之间的数据流走向，所述第二升级指令用于指示所述第二工作节点调整所述第二工作节点上分布的PE的数目。
  13. 根据权利要求10至12任一项所述的主控节点，其特征在于，还包括：
    数据恢复模块，用于根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的检查点checkpoint；
    所述发送模块还用于，向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；
    所述主控节点还包括：输入触发模块，用于当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，触发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
  14. 根据权利要求13所述的主控节点，其特征在于，所述待调整的数据流包括：待更新的数据流和待删除的数据流；所述数据恢复模块具体用于：
    所述主控节点根据与所述待更新的数据流和待删除的数据流相关的PE的状态数据，确定执行数据恢复的checkpoint；根据与所述待更新的数据流和待删除的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE；其中，每个PE的状态数据是该PE在输出事件的触发下备份的，用于指示该PE处理数据的状态。
  15. 根据权利要求12至14任一项所述的主控节点，其特征在于，所述待调整的PE包括待删除的PE；所述第二工作节点为所述待删除的PE所在的工作节点；所述第二升级指令用于指示所述第二工作节点删除所述待删除的PE。
  16. 根据权利要求12至15任一项所述的主控节点，其特征在于，所述待调整的PE包括待新增的PE；所述第二工作节点为所述主控节点根据所述流计算系统中各个工作节点的负载状况选择的工作节点；所述第二升级指令用于指示所述第二工作节点创建所述待新增的PE。
  17. 一种流计算系统，其特征在于，包括：主控节点和至少一个工作节点，其中，多个执行单元PE分布于所述至少一个工作节点中的一个或多个工作节点上，用于对所述流计算系统上部署的流式应用的数据进行处理，其中所述流 式应用的初始逻辑模型用于表示处理所述流式应用的数据的所述多个PE以及所述多个PE之间的数据流走向；
    所述主控节点用于：在所述流式应用更新的情况下，获取所述流式应用更新后的逻辑模型，并通过比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流；根据所述待调整的数据流，生成升级指令；将所述升级指令下发给第一工作节点，所述第一工作节点是与所述待调整的数据流相关的PE所在的工作节点，所述升级指令用于指示所述第一工作节点调整所述第一工作节点上分布的PE之间的数据流走向；
    所述第一工作节点，用于接收所述主控节点发送的升级指令，根据所述升级指令的指示，调整所述第一工作节点上分布的PE之间的数据流走向。
  18. 根据权利要求17所述的流计算系统，其特征在于，在比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，确定待调整的数据流的方面，所述主控节点具体用于：比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE相同。
  19. 根据权利要求17所述的流计算系统，其特征在于，所述主控节点具体用于，比较所述流式应用的初始逻辑模型以及所述更新后的逻辑模型，以确定待调整的PE以及所述待调整的数据流，其中所述流式应用的初始逻辑模型所表示的PE与所述更新后的逻辑模型所表示的PE不完全相同；根据所述待调整的数据流，生成第一升级指令；根据所述待调整的PE生成第二升级指令；
    将所述第一升级指令下发给所述第一工作节点，将所述第二升级指令下发给第二工作节点，所述第二工作节点包括所述待调整的PE所在的工作节点；
    所述第一工作节点具体用于，接收所述主控节点发送的所述第一升级指令，根据所述第一升级指令的指示，调整所述第一工作节点上分布的PE之间的数据流走向；
    所述第二工作节点具体用于，接收所述主控节点发送的所述第二升级指令，根据所述第二升级指令的指示调整所述第二工作节点上分布的PE的数目。
  20. 根据权利要求17至19任一项所述的流计算系统，其特征在于，所述主控节点还用于，根据与所述待调整的数据流相关的PE的输入流和输出流的依赖关系，确定需要进行数据恢复的目标PE和所述目标PE执行数据恢复的 检查点checkpoint；向所述目标PE所在工作节点下发数据恢复指令，所述数据恢复指令用于指示所述目标PE根据所述checkpoint恢复数据；当确定所述第一工作节点完成调整，且所述第一工作节点上分布的PE准备就绪之后，触
    发所述目标PE将恢复的数据输入所述目标PE的下游PE进行处理。
PCT/CN2015/079944 2014-10-22 2015-05-27 一种流式应用升级方法、主控节点及流计算系统 WO2016062087A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP15853083.2A EP3200095B1 (en) 2014-10-22 2015-05-27 Streaming application update method, master node, and stream computing system
EP22188019.8A EP4148558A1 (en) 2014-10-22 2015-05-27 Streaming application upgrading method, master node, and stream computing system
US15/492,392 US10785272B2 (en) 2014-10-22 2017-04-20 Streaming application upgrading method, master node, and stream computing system
US17/014,388 US20200404032A1 (en) 2014-10-22 2020-09-08 Streaming Application Upgrading Method, Master Node, and Stream Computing System

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201410568236.5A CN104317556B (zh) 2014-10-22 2014-10-22 一种流式应用升级方法、主控节点及流计算系统
CN201410568236.5 2014-10-22

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/492,392 Continuation US10785272B2 (en) 2014-10-22 2017-04-20 Streaming application upgrading method, master node, and stream computing system

Publications (1)

Publication Number Publication Date
WO2016062087A1 true WO2016062087A1 (zh) 2016-04-28

Family

ID=52372792

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/079944 WO2016062087A1 (zh) 2014-10-22 2015-05-27 一种流式应用升级方法、主控节点及流计算系统

Country Status (4)

Country Link
US (2) US10785272B2 (zh)
EP (2) EP4148558A1 (zh)
CN (1) CN104317556B (zh)
WO (1) WO2016062087A1 (zh)

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10678225B2 (en) 2013-03-04 2020-06-09 Fisher-Rosemount Systems, Inc. Data analytic services for distributed industrial performance monitoring
US10866952B2 (en) 2013-03-04 2020-12-15 Fisher-Rosemount Systems, Inc. Source-independent queries in distributed industrial system
US10649424B2 (en) * 2013-03-04 2020-05-12 Fisher-Rosemount Systems, Inc. Distributed industrial performance monitoring and analytics
US10649449B2 (en) 2013-03-04 2020-05-12 Fisher-Rosemount Systems, Inc. Distributed industrial performance monitoring and analytics
US9665088B2 (en) 2014-01-31 2017-05-30 Fisher-Rosemount Systems, Inc. Managing big data in process control systems
US9558220B2 (en) 2013-03-04 2017-01-31 Fisher-Rosemount Systems, Inc. Big data in process control systems
US10909137B2 (en) 2014-10-06 2021-02-02 Fisher-Rosemount Systems, Inc. Streaming data for analytics in process control systems
US10152031B2 (en) 2013-03-15 2018-12-11 Fisher-Rosemount Systems, Inc. Generating checklists in a process control environment
CN104317556B (zh) * 2014-10-22 2018-03-16 华为技术有限公司 一种流式应用升级方法、主控节点及流计算系统
JP6468356B2 (ja) * 2015-06-25 2019-02-13 富士通株式会社 プログラム生成装置、プログラム生成方法および生成プログラム
WO2017106379A1 (en) * 2015-12-14 2017-06-22 Pivotal Software, Inc. Workload management in distributed database systems
US10503483B2 (en) 2016-02-12 2019-12-10 Fisher-Rosemount Systems, Inc. Rule builder in a process control network
CN107229815A (zh) * 2016-03-25 2017-10-03 阿里巴巴集团控股有限公司 数据分析方法及装置
US10122788B2 (en) * 2016-03-29 2018-11-06 Amazon Technologies, Inc. Managed function execution for processing data streams in real time
US10901998B2 (en) 2016-05-31 2021-01-26 International Business Machines Corporation Managing a modification to a consistent region of a stream computing environment
US10572276B2 (en) 2016-09-12 2020-02-25 International Business Machines Corporation Window management based on a set of computing resources in a stream computing environment
US10439917B2 (en) * 2016-11-15 2019-10-08 At&T Intellectual Property I, L.P. Recovering a replica in an operator in a data streaming processing system
US10320652B2 (en) * 2017-01-09 2019-06-11 Cisco Technology, Inc. Dynamic installation of bypass path by intercepting node in storing mode tree-based network
CN108363545B (zh) 2017-01-26 2021-12-03 华为技术有限公司 一种数据配置方法及数据配置装置
CN107391136B (zh) * 2017-07-21 2020-11-06 众安信息技术服务有限公司 一种基于流式的编程系统和方法
CN109960570B (zh) * 2017-12-14 2021-09-03 北京图森智途科技有限公司 一种多模块调度方法、装置及系统
TWI669614B (zh) * 2018-03-07 2019-08-21 宏碁股份有限公司 電子裝置與其離線更新程式的方法
US10977089B2 (en) * 2018-04-16 2021-04-13 International Business Machines Corporation Fusing and unfusing operators during a job overlay
CN110213073B (zh) * 2018-04-20 2021-10-22 腾讯科技（深圳）有限公司 数据流向变更方法、电子设备、计算节点及存储介质
US10795734B2 (en) * 2018-06-22 2020-10-06 International Business Machines Corporation Processing element restart precedence in a job overlay environment
US10965750B2 (en) * 2018-09-27 2021-03-30 International Business Machines Corporation Distributed management of dynamic processing element connections in streaming applications
CN111510468B (zh) * 2019-01-30 2023-05-02 杭州海康威视数字技术股份有限公司 计算任务的调度方法、装置、服务器及计算系统
US11182145B2 (en) 2019-03-15 2021-11-23 International Business Machines Corporation Optimizing updates to distributed data processing applications
US11275573B1 (en) * 2019-07-30 2022-03-15 Intuit Inc. Intelligent rolling update of a cluster of servers via container orchestration
CN112000453B (zh) * 2020-08-25 2024-06-28 支付宝(杭州)信息技术有限公司 一种流计算系统的调度方法和系统
CN114489572B (zh) * 2020-11-13 2023-12-29 华为技术有限公司 业务需求调整方法和相关设备
US20220283876A1 (en) * 2021-03-03 2022-09-08 NEC Laboratories Europe GmbH Dynamic resource allocation for efficient parallel processing of data stream slices
CN113162992B (zh) * 2021-04-01 2022-05-27 杭州数跑科技有限公司 系统的升级不断流控制方法、装置、设备和存储介质
CN113626192A (zh) * 2021-08-10 2021-11-09 支付宝(杭州)信息技术有限公司 用于对算子节点进行扩缩容调整的方法、装置及系统
CN114443441B (zh) * 2022-04-08 2022-07-08 苏州浪潮智能科技有限公司 一种存储系统管理方法、装置、设备及可读存储介质
CN118519688A (zh) * 2024-07-22 2024-08-20 深圳市汇顶科技股份有限公司 芯片、组网系统和电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100287204A1 (en) * 2007-11-13 2010-11-11 Lisa Amini Systems and methods for using provenance information for data retention in stream-processing
CN103763378A (zh) * 2014-01-24 2014-04-30 中国联合网络通信集团有限公司 基于分布式流式计算系统的任务处理方法、系统及节点
CN103870340A (zh) * 2014-03-06 2014-06-18 华为技术有限公司 流计算系统中的数据处理方法、控制节点及流计算系统
CN104050261A (zh) * 2014-06-16 2014-09-17 深圳先进技术研究院 基于Storm的可变逻辑的通用数据处理系统及方法
CN104317556A (zh) * 2014-10-22 2015-01-28 华为技术有限公司 一种流式应用升级方法、主控节点及流计算系统

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5815707A (en) * 1995-10-19 1998-09-29 Hewlett-Packard Company Dynamic function replacement for streams framework
US7185075B1 (en) * 1999-05-26 2007-02-27 Fujitsu Limited Element management system with dynamic database updates based on parsed snooping
US20110238855A1 (en) * 2000-09-25 2011-09-29 Yevgeny Korsunsky Processing data flows with a data flow processor
US7346480B1 (en) * 2002-06-29 2008-03-18 Applied Wave Research, Inc. Impedance mismatch modeling in a data flow or discrete time based system simulation
US7765255B2 (en) * 2002-07-30 2010-07-27 Sap Ag Extended web infrastructure for business applications
US6973654B1 (en) 2003-05-27 2005-12-06 Microsoft Corporation Systems and methods for the repartitioning of data
US20070261041A1 (en) * 2005-08-23 2007-11-08 Lisa Amini Method and system for dynamic application composition in streaming systems
US8127130B2 (en) * 2006-04-18 2012-02-28 Advanced Communication Concepts, Inc. Method and system for securing data utilizing reconfigurable logic
US7613848B2 (en) * 2006-06-13 2009-11-03 International Business Machines Corporation Dynamic stabilization for a stream processing system
US7493406B2 (en) * 2006-06-13 2009-02-17 International Business Machines Corporation Maximal flow scheduling for a stream processing system
EP1890230A1 (en) 2006-08-15 2008-02-20 Alcatel Lucent Software replacement in a stream processing system
US20090210631A1 (en) * 2006-09-22 2009-08-20 Bea Systems, Inc. Mobile application cache system
US7889651B2 (en) * 2007-06-06 2011-02-15 International Business Machines Corporation Distributed joint admission control and dynamic resource allocation in stream processing networks
CN100492290C (zh) * 2007-11-22 2009-05-27 上海交通大学 软件动态演化中构件运行时状态的获取和转换方法
US8041378B2 (en) * 2008-12-19 2011-10-18 Cisco Technology, Inc. System and method for providing channel configurations in a communications environment
US8429287B2 (en) * 2009-04-29 2013-04-23 Rangecast Technologies, Llc Network audio distribution system and method
KR101777347B1 (ko) * 2009-11-13 2017-09-11 삼성전자주식회사 부분화에 기초한 적응적인 스트리밍 방법 및 장치
US8595234B2 (en) * 2010-05-17 2013-11-26 Wal-Mart Stores, Inc. Processing data feeds
CN101984405A (zh) 2010-10-11 2011-03-09 中兴通讯股份有限公司 一种软件版本升级方法、终端及系统
US8862743B1 (en) * 2011-01-13 2014-10-14 Google Inc. Resource management
US8959313B2 (en) * 2011-07-26 2015-02-17 International Business Machines Corporation Using predictive determinism within a streaming environment
US8898505B2 (en) * 2011-12-01 2014-11-25 International Business Machines Corporation Dynamically configureable placement engine
CN102750330A (zh) * 2012-05-31 2012-10-24 国网电力科学研究院 一种与业务无关的拓扑图形增量更新方法
US9020785B2 (en) * 2012-11-09 2015-04-28 International Business Machines Corporation Identifying and routing poison tuples in a streaming application
US9195559B2 (en) * 2012-12-12 2015-11-24 International Business Machines Corporation Management of stream operators with dynamic connections
WO2014117353A1 (en) * 2013-01-31 2014-08-07 Hewlett-Packard Development Company, L.P. Incremental update of a shape graph
WO2014209260A1 (en) * 2013-06-24 2014-12-31 Hewlett-Packard Development Company, L.P. Processing a data flow graph of a hybrid flow
CN103796041A (zh) * 2013-11-01 2014-05-14 中兴通讯股份有限公司 一种iptv系统的更新方法及装置
US10635644B2 (en) * 2013-11-11 2020-04-28 Amazon Technologies, Inc. Partition-based data stream processing framework
US10264071B2 (en) * 2014-03-31 2019-04-16 Amazon Technologies, Inc. Session management in distributed storage systems

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100287204A1 (en) * 2007-11-13 2010-11-11 Lisa Amini Systems and methods for using provenance information for data retention in stream-processing
CN103763378A (zh) * 2014-01-24 2014-04-30 中国联合网络通信集团有限公司 基于分布式流式计算系统的任务处理方法、系统及节点
CN103870340A (zh) * 2014-03-06 2014-06-18 华为技术有限公司 流计算系统中的数据处理方法、控制节点及流计算系统
CN104050261A (zh) * 2014-06-16 2014-09-17 深圳先进技术研究院 基于Storm的可变逻辑的通用数据处理系统及方法
CN104317556A (zh) * 2014-10-22 2015-01-28 华为技术有限公司 一种流式应用升级方法、主控节点及流计算系统

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of EP3200095A4 *
SUN, DAWEI ET AL.: "Big Data Stream Computing: Technologies and Instances", JOURNAL OF SOFTWARE, vol. 25, no. 4, 30 April 2014 (2014-04-30), pages 847 - 849, XP008185459, ISSN: 1000-9825 *

Also Published As

Publication number Publication date
US20200404032A1 (en) 2020-12-24
EP3200095B1 (en) 2022-08-24
EP4148558A1 (en) 2023-03-15
EP3200095A4 (en) 2017-11-08
EP3200095A1 (en) 2017-08-02
US20170223075A1 (en) 2017-08-03
US10785272B2 (en) 2020-09-22
CN104317556A (zh) 2015-01-28
CN104317556B (zh) 2018-03-16

Similar Documents

Publication Publication Date Title
WO2016062087A1 (zh) 一种流式应用升级方法、主控节点及流计算系统
US12079091B2 (en) Application consistent snapshots as a sidecar of a containerized application
US9015164B2 (en) High availability for cloud servers
US20140095503A1 (en) Compile-time grouping of tuples in a streaming application
US10148719B2 (en) Using anchors for reliable stream processing
US11010203B2 (en) Fusing and unfusing operators during a job overlay
Xu et al. Efficient fault-tolerance for iterative graph processing on distributed dataflow systems
Liu et al. Optimizing shuffle in wide-area data analytics
CN111147541B (zh) 基于参数服务器的节点处理方法、装置、设备及存储介质
US10303558B2 (en) Checkpointing higher order query operators
US7979743B2 (en) Determining correctness of job plans in a stream processing application
CN109725916B (zh) 流处理的拓扑结构更新系统和方法
WO2024139011A1 (zh) 信息处理方法
CN110365809B (zh) 分布式服务器地址配置系统及方法
US10621163B2 (en) Tracking and reusing function results
US11861356B1 (en) Continuous deployment and orchestration of feature processing units in a managed cloud and a provider network for consistent data processing
US10311158B2 (en) Streamlining tuple processing by delivering tuple attributes to associated operators
JP6897866B2 (ja) システム更新装置およびシステム更新方法
US20160103742A1 (en) Buffered cloned operators in a streaming application
US20190370408A1 (en) Dataflow execution graph modification using intermediate graph
JPWO2019187975A1 (ja) システム更新装置およびシステム更新方法
US20230103795A1 (en) Method and system for predicting batch processes
US10747729B2 (en) Device specific chunked hash size tuning
CN113986596A (zh) 一种数据处理方法、装置、电子设备及存储介质
US20210011625A1 (en) System and method for backup of logical layer virtual systems

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15853083

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2015853083

Country of ref document: EP