CN116302576A

CN116302576A - Method and system for parallelizing elastically-telescopic stream application operator

Info

Publication number: CN116302576A
Application number: CN202310594752.4A
Authority: CN
Inventors: 孙大为; 吴明辉
Original assignee: China University of Geosciences Beijing
Current assignee: China University of Geosciences Beijing
Priority date: 2023-05-25
Filing date: 2023-05-25
Publication date: 2023-06-23
Anticipated expiration: 2043-05-25
Also published as: CN116302576B

Abstract

The invention discloses a parallelization method and a parallelization system for an elastically telescopic stream application operator, which are applied to the technical field of big data and comprise the following steps: s101: inputting stream application data into an M/M/K mathematical model, acquiring system information, and storing the system information in a database, wherein the system information comprises calculation nodes in an operator cluster, CPU information of tasks, I/O information of the tasks and memory resource consumption information of the tasks, data transmission rate between the tasks in a topological structure and running state information of a distributed stream calculation system; s102: optimizing the number of instances of each operation in the topology according to the system information; s103: selecting a target node to deploy or recycle an example of an operator according to the system information and the number of examples; s104: the task is notified and the state of the backup is repartitioned. The method and the device can solve the problems that the parallelism ratio between operators in the stream application program can not be adjusted and the response time of the system is minimized when the stream application program occupies fixed computing resources.

Description

Method and system for parallelizing elastically-telescopic stream application operator

Technical Field

The invention belongs to the technical field of big data, and particularly relates to an elastically telescopic stream application operator parallelization method and system.

Background

The ability to process continuous data streams in an extensible and timely manner becomes critical to the internet of things, traffic monitoring, telecommunications, healthcare, and the like. These streaming applications require rapid analysis of a large number of sequential data streams in order to produce predictable and operational results in a high performance computing environment. However, the performance of a streaming computing system is highly dependent on various aspects, including computing resources, operator parallelism, memory settings, and buffer size. Among other things, automatically adjusting the parallelism of operators to optimize system performance has become a critical challenge. Thus, proper operator parallelism has a very important role in the performance of the system.

Real-time applications running on streaming computing systems are modeled as a Directed Acyclic Graph (DAG) to describe the dependencies between tasks. Each DAG is submitted to a cluster deploying the streaming computing system, and each task in the DAG is scheduled to run on a computing node in the cluster. If there is no manual kill or failure, the streaming application deployed in the cluster will always run. Thus, the operator parallelism in the DAGs submitted to the clusters is static and cannot accommodate fluctuating data flow rates. This has two negative effects:

(1) When the arrival rate of a data stream breaks through the bottleneck of the system to process data tuples, a large amount of data will be deposited in the system unrestrictedly, resulting in excessive system response time and further system crashes.

(2) When the arrival rate of the data stream is continuously low, the computing resources occupied by the operators in the stream application cannot be dynamically recovered, so that the system generates additional idle resource consumption. Therefore, it is necessary to allocate resources used by different operators in the streaming application by adjusting parallelism of the operators to minimize overhead. Improper parallelism configuration of operators can lead to low resource utilization and instability of the overall system.

To ensure that the data stream responds at a speed of the order of milliseconds, a flexible operator scaling mechanism is always required in the stream computing system. It is desirable to dynamically adjust the ratio of parallelism between operators in a streaming application to meet the goal of low latency. However, most of the most advanced works do not provide an appropriate operator extension that can know how to coordinate the resource duty cycle between operators in a streaming application, dynamically allocate and release according to the current data stream. The representative working DRS realizes a dynamic resource scheduling module to meet the constraint of real-time performance, and the strategy gradually increases the parallelism of operators in a stream application program in a stable data stream environment so as to minimize the response time of the system. However, they ignore the fact that in a real streaming computing environment, the data stream is unstable and variable, which results in dynamic changes in the computational resources required by operators in the streaming application. The latest streaming applications working for limited buffers implement a method of optimizing operator parallelism that can optimize operator parallelism by modeling the relationship between the number of operator instances and the average residence time of the data tuples. However, it adjusts the parallelism of operators by a greedy algorithm, thus creating more overhead, while also ignoring the fact that stateful operators may become a key to the system bottleneck.

Changing the parallelism of streaming applications can improve system performance, however, it also presents new challenges for task state management, making the auto-expansion mechanism more complex. Changes in the stream application may result in data dependency inconsistencies between the backups and states of the operators before and after auto-expansion. Thus, the state backup of the operator should be repartitioned according to the extended operator, which may create additional overhead.

Disclosure of Invention

The embodiment of the invention aims to solve the problems, and provides an elastically telescopic flow application operator parallelization method and system, which can solve the problems that the parallelization degree between operators in a flow application program can not be adjusted and the response time of a system is minimized when the flow application program occupies fixed computing resources.

In order to solve the technical problems, the invention is realized as follows:

in a first aspect, an embodiment of the present invention provides a method for parallelizing an elastically telescopic stream application operator, including:

s101: inputting stream application data into an M/M/K mathematical model, acquiring system information, and storing the system information in a database, wherein the system information comprises calculation nodes in operator clusters, CPU information of tasks, I/O information of tasks and memory resource consumption information of tasks, data transmission rate between tasks in a topological structure and running state information of a distributed stream calculation system;

s102: optimizing the number of instances of each operation in the topology according to the system information;

s103: selecting a target node to deploy or recycle an operator instance according to the system information and the instance number;

s104: the task is notified and the state of the backup is repartitioned.

Optionally, the step S102 specifically includes:

s1021: acquiring a first operator in the operator cluster

；

S1022: acquiring the input queue rate of the first operator and the rate of processing the data tuples by an operator;

s1023: and adjusting the number of instances of the first operator according to the input queue rate and the processing data tuple rate.

Optionally, the input queue rate of the first operator adopts a formula:

；

wherein,,

is the instance rate of the first operator.

Optionally, the step S102 further includes: preempting the computing node to extend the computing resources used by the streaming application.

Optionally, the stream application data includes a stateless operator and a stateful operator, and the S103 further includes:

in the case that the operator is a stateless operator, when changing the number of instances of the stateless operator, the redirected data stream does not affect the instance processing data tuples of the stateless operator;

in the case where the operator is a stateful operator, backing up the state and caching the data tuples when changing the number of instances of the stateful operator, the redirected data stream may cause the data tuples to miss the state of the stateful operator.

In a second aspect, an embodiment of the present invention provides an elastically telescopic stream application operator parallelization system, including:

the system information comprises calculation nodes in an operator cluster, CPU information of tasks, I/O information of the tasks and memory resource consumption information of the tasks, data transmission rate among the tasks in a topological structure and running state information of a distributed stream calculation system;

the topology analysis module is used for optimizing the number of instances of each operation in the topology structure according to the system information;

the resource analysis module is used for selecting a target node to deploy or recycle the operator instance according to the system information and the instance number;

and the state notification management module is used for notifying tasks and repartitioning the backup state.

Optionally, the topology analysis module specifically includes:

a first obtaining module, configured to obtain a first operator in the operator cluster

；

A second obtaining module, configured to obtain an input queue rate of the first operator and a rate at which an operator processes a data tuple;

and the adjusting module is used for adjusting the number of instances of the first operator according to the input queue rate and the processing data tuple rate.

Optionally, the input queue rate of the first operator adopts a formula:

；

wherein,,

is the instance rate of the first operator.

Optionally, the topology analysis module is further configured to: preempting the computing node to extend the computing resources used by the streaming application.

Optionally, the stream application data includes stateless operators and stateful operators, and the resource analysis module is further configured to:

In the embodiment of the invention, firstly, a data tuple queuing model based on an M/M/k system is constructed to optimize the parallelism of operators and realize the trade-off between the system delay and the resource consumption. Resource scaling is performed by using a bias distribution model to evaluate the resources consumed by the streaming application. Secondly, upstream backup of operator states and caching of data tuples at dynamic time intervals are realized to reduce the cost of state recovery. In addition, the backup time interval is dynamically changed according to the resource load of the node, so that the system overhead is reduced.

Drawings

FIG. 1 is a schematic flow chart of a method for parallelizing elastically telescopic stream application operators according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of an elastically scalable stream application operator parallelization system according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a logic diagram of an operator provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of Es-Stream state management provided by an embodiment of the present invention;

the achievement of the object, functional features and advantages of the present invention will be further described with reference to the embodiments, referring to the accompanying drawings.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present invention may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type, and are not limited to the number of objects, such as the first object may be one or more. It should be understood that, in various embodiments of the present disclosure, the size of the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

It should be understood that in this disclosure, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in this disclosure, "plurality" means two or more. "and/or" is merely an association relationship describing an association object, and means that three relationships may exist, for example, and/or B may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. "comprising A, B and C", "comprising A, B, C" means that all three of A, B, C comprise, "comprising A, B or C" means that one of the three comprises A, B, C, and "comprising A, B and/or C" means that any 1 or any 2 or 3 of the three comprises A, B, C.

It should be understood that in this disclosure, "B corresponding to a", "a corresponding to B", or "B corresponding to a" means that B is associated with a from which B may be determined. Determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information. The matching of A and B is that the similarity of A and B is larger than or equal to a preset threshold value.

As used herein, "if" may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection" depending on the context.

Poor resource allocation between operators can affect system performance or waste computing resources. Attention is first paid to an operator. With two operators

And->

The number of examples is +.>

And->

. Hypothesis operator->

And->

The example rate of processing data clusters is +.>

And->

。

Then, operator

The rate of the input queue of (c) may be calculated by the following formula.

Operator

The rate at which the data tuples are processed can be calculated by the following formula.

Obviously, operators

There must be enough instances to process the data to keep up with the input rate. If->

Then operator->

Will not keep pace with the input speed. Operator->

Will be continuously longer, resulting in an increase in the dwell time of the data tuples in the operator, further resulting in an instance of the operator being continuously started down. Furthermore, if->

Operator->

May create additional idle time to wait for an instance of (a) to uploadThe input data of the operator results in a waste of computational resources. Thus, the operator can be adjusted +.>

To balance ∈>

And->

The dwell time of the data tuples in the operator is minimized.

In a streaming application, an instance processes data tuples

Intermediate results (named states) may be generated and are typically stored in memory at runtime. To improve the reliability of the system, intermediate results in memory are typically backed up to remote storage through a checkpointing mechanism. Example->

At time->

Direction operator->

Downstream instance of (2)

And->

Four data tuples are transmitted +.>

And->

. Example->

Receive data tuple->

And

and performing logic computation to generate states (k 1, v 1) and (k 2, v 2), where v1 is the instance +.>

Treatment->

Intermediate results of the calculation. The process of grouping data tuples can be described by the following formula:

wherein the method comprises the steps of

Representation operator->

Is (are) example number->

Representing processing data tuples->

Examples of (2)

. Furthermore, examples->

Periodically, the status of (c) is backed up to the remote storage system.

At time, adding an instance of an operation results in the instance redirecting the data stream, which can be described by the following formula:

wherein the method comprises the steps of

Representing the number of instance changes.

Thus, if

Then->

. As shown, example->

Receive data tuple->

But data tuple->

The state of (2) is at the time->

Stored at instance->

. Similarly, instance->

Receive data tuple->

But data tuple->

The state of (2) is stored in instance->

Is a kind of medium. Thus, after the data stream is redirected, the data tuples processed by the instance are inconsistent with the stored state.

In an unbounded data stream environment, tasks in the stream application inevitably fail, which makes it difficult to guarantee the reliability of data processing and resource expansion. Checkpointing mechanisms are commonly used to cope with faulty tasks. However, it also incurs some additional overhead:

(1) The calculation is repeated. If one task fails, the system will lose part of the state data and roll back to recalculate the lost state. The latest state of the system is

. When a task in the streaming application fails, the whole system needs to roll back to the state +.>

. In this process +.>

The state of (2) will be lost and the data tuple will be never +.>

And (5) re-inputting.

(2) Global rollback. Once one task in the streaming application fails, the entire topology needs to roll back to the previous state.

In summary, after a series of experiments have been constructed, it can be demonstrated that system delays can be affected by varying parallelism and fluctuating data flows of the streaming application. Additional resource consumption may result from over-configured operator parallelism. And, a stream application model, a communication mode, a Makespan model and a resource constraint model are constructed to quantify indexes of the system, and meanwhile, the problems of double overhead including expandability of operator parallelism, state consistency, state recovery and the like are formalized.

The following describes in detail a flowchart of an elastically scalable stream application operator parallelization method provided by the embodiment of the present invention through a specific embodiment and an application scenario thereof with reference to fig. 1. The embodiment of the invention provides an elastically telescopic stream application operator parallelization method, which comprises the following steps:

s101: inputting stream application data into an M/M/K mathematical model, acquiring system information, and storing the system information in a database, wherein the system information comprises calculation nodes in operator clusters, CPU information of tasks, I/O information of tasks and memory resource consumption information of tasks, data transmission rate between tasks in a topological structure and running state information of a distributed stream calculation system.

S102: and optimizing the number of instances of each operation in the topology structure according to the system information.

Specifically, the emphasis of S102 is on optimizing the number of instances per operator in the topology. For an operator, its instance number is the most important parameter in initializing the topology. A reasonable operator instance number can effectively improve the throughput of the system and reduce the response time of the system. Otherwise, the burden of processing data by the system may be increased, further affecting the system performance. Thus, the topology analysis module extracts data about the topology information stored in the database and analyzes and models the data to obtain the optimal number of instances for each operator in the topology.

Optionally, S102 specifically includes:

s1021: acquiring a first operator in the operator cluster

。

Alternatively, the streaming application may be viewed as a directed acyclic graph. Thus, the data tuples processed by the streaming application are directional, i.e. the upstream instance transmits the processed data tuples into the downstream instance. When expanding or shrinking operators

Operator +.>

The processing of the data tuples by the upstream operator is not affected. However, it may cause downstream operator instances to pile up data tuples or generate idle resources. Thus, the number of instances of the scaling operator should be hierarchical, prioritizing the increase/decrease of the number of instances of the upstream operator. Based on the above analysis we use a topological ordering algorithm to determine the order of the scaling operators. As shown in the figure3, the logical graph of operators is described as a directed acyclic graph. The order of the scaling operators in the streaming application should be +.>

And->

。

The DAG is reconstructed based on the M/K mathematical model to ensure timely processing of the data tuples. If the number of instances of the runtime operator is static, a large number of data tuples may pile up on the system in the face of high data flow rates, or there may be additional idle resources in the system in the face of low data flow rates. Such algorithms may reconfigure the DAG at run-time to accommodate unstable data flows.

Optionally, the input data for the algorithm includes G,

,/>

output is +.>

。

If G is empty, the algorithm stops cycling;

if each is

In G, the following operations are performed: />

。

If each is

In the telescopic queue, the following operations are performed: if->

In the case of empty, will

Assigned to the expansion queue and emptied +.>

。

If it is

The method comprises the steps of carrying out a first treatment on the surface of the Assigning the value of k to +.>

。

S1022: an input queue rate for the first operator and an operator processing data tuple rate are obtained.

Alternatively, when the streaming media application uses insufficient fixed resources, the performance bottleneck of the system cannot be solved by simply increasing the number of instances. Thus, the computing resources used by the streaming application can be extended by preempting more computing nodes to improve the bottleneck problem of the system. If the data flow continues to be in a low rate state, the resources used by the flow application will be curtailed to free up more compute nodes.

S103: and selecting a target node to deploy or recycle the operator examples according to the system information and the number of examples.

Alternatively, a streaming application submitted by a user mainly includes stateless operators and stateful operators. For stateless operators, only each independent event is concerned, the input data is directly converted into an output result, and the calculation is carried out independently of other data. The data tuples are processed independently by the operator instance. Thus, when changing the number of instances of stateless operators, the redirected data stream cannot affect the stateless instance processing data tuples. For stateful operators, the computation of the output result requires some additional data, including mainly the early input data tuples and some results of early data tuple computation, in addition to the input data tuples. The state of the data tuples is stored in the corresponding instance. Thus, when changing the instance number of a stateful operator, the redirected data stream may cause the data tuple issued by the upstream instance to be processed by the wrong downstream instance, causing the tuple to miss its state.

In order to realize a low-overhead state repair mechanism, to expand the parallelism of stateful operators, a mechanism for backing up state and caching data tuples is designed in an upstream operator. It includes two main aspects:

(1) The instance state of the stateful operator is backed up to the upstream instance, which is more efficient because it reuses the existing communication connections. All operators in a stream application form a logical loop. Each instance in the stateful operator manages its own state. Furthermore, if the node's resource load is less than the instance will synchronize its state with the upstream periodically. Otherwise, the synchronization interval will be set to indefinite.

(2) For a stateful operator, the output of an upstream instance processing a data tuple is backed up locally for a time interval. If the parallelism of the stateful operator is changed, the upstream instance thereof will re-partition the backup state and send the partitioned state to the corresponding downstream instance. Secondly, backup output results of the upstream instance in the current time interval are transmitted to the downstream instance again, so that global rollback state of the whole topological structure is avoided, and system overhead is reduced. As shown in FIG. 4, es-Stream state management is schematically illustrated. Operator

The state of the instance of (2) is backed up to operator +.>

Upstream instance of (a). Furthermore, by instance->

Downstream instance->

And->

The outgoing data tuples are buffered locally at the same time during a time interval.

S104: the task is notified and the state of the backup is repartitioned.

In the embodiment of the application, the following four advantages are achieved:

(1) A series of experiments were constructed to demonstrate that system delay can be affected by varying degrees of parallelism and fluctuating data flows of the streaming application. Additional resource consumption may result from over-configured operator parallelism.

(2) The method comprises the steps of constructing a stream application model, a communication mode, a Makespan model and a resource constraint model to quantify indexes of a system, and formalizing the problems of expandability, state consistency, double overheads of state recovery and the like including operator parallelism.

(3) A data tuple queuing model based on an M/M/k system is constructed to optimize the parallelism of operators and realize the trade-off between system delay and resource consumption. Resource scaling is performed by using a bias distribution model to evaluate the resources consumed by the streaming application.

(4) Upstream backups of operator states and buffering of data tuples for dynamic time intervals are implemented to reduce the cost of state recovery. In addition, the backup time interval is dynamically changed according to the resource load of the node, so that the system overhead is reduced.

(5) The experimental evaluation is performed on a real stream application program, and a system adopting the elastic telescopic stream application operator parallelization method evaluates through the indexes, such as system response time and state recovery cost.

Example two

Referring to fig. 2, a schematic structural diagram of a stream application operator parallelization system 30 with elastic scalability according to an embodiment of the present invention is shown.

The elastically telescopic stream application operator parallelization system 30 provided by the embodiment of the invention comprises:

the monitoring module 301 is configured to obtain system information after inputting stream application data into the M/K mathematical model, and store the system information in a database, where the system information includes computing nodes in an operator cluster, CPU information of a task, I/O information of the task, memory resource consumption information of the task, a data transmission rate between tasks in a topology structure, and running state information of a distributed stream computing system.

Optionally, the information collected by the monitoring module is stored in a database.

The topology analysis module 302 is configured to optimize the number of instances of each operation in the topology according to the system information.

Optionally, the focus of the topology analysis module is to optimize the number of instances of each operator in the topology. For an operator, its instance number is the most important parameter in initializing the topology. A reasonable operator instance number can effectively improve the throughput of the system and reduce the response time of the system. Otherwise, the burden of processing data by the system may be increased, further affecting the system performance. Thus, the topology analysis module extracts data about the topology information stored in the database and analyzes and models the data to obtain the optimal number of instances for each operator in the topology.

And the resource analysis module 303 is configured to select an instance of the deployment or reclamation operator for the target node according to the system information and the number of instances.

Optionally, the resource analysis module is responsible for analyzing the resource load of the cluster, and selecting an appropriate node to deploy or reclaim the operator instance according to the result of the topology analysis.

Optionally, the state notification module is responsible for notifying tasks and repartitioning the state of the backup. From the topology analysis module, it can be known that the number of instances of the stateful operator has changed. If the number of operator instances is elastically extended at runtime, the state of the task should be repartitioned to ensure that the buffered data tuples are mapped into the correct partitions.

The state notification management module 304 is configured to notify tasks and repartition the backup state.

This system may be referred to as the system architecture of Es-Stream. Compared to conventional solutions, es-Stream has the following advantages:

(1) Conventional solutions configure the number of operator instances in the topology according to the steady rate of data flow, the configuration topology running in the cluster is static. However, es-Stream can sense the change of the data Stream rate and dynamically adjust the resource allocation weight of operators in the topology. In addition, es-Stream can effectively reduce the resources used by the topology in the cluster in the face of low data Stream rates.

(2) Conventional solutions adjust the number of instances for stateless operators, which makes the bottleneck of the system difficult to improve. However, es-Stream can extend the number of instances of stateless and stateful operators to improve the throughput of the system. In addition, es-Stream backs up the state of a task to an upstream instance and caches data tuples during dynamic time intervals to provide a flexible and low-overhead state recovery mechanism.

Optionally, the topology analysis module specifically includes:

；

Optionally, the input queue rate of the first operator adopts a formula:

wherein,,

is the instance rate of the first operator.

(5) Experimental evaluation is performed on real streaming applications, and systems employing Es-Stream evaluate through these metrics, such as system response time and state recovery costs.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Note that all features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic set of equivalent or similar features. Where used, further, preferably, still further and preferably, the brief description of the other embodiment is provided on the basis of the foregoing embodiment, and further, preferably, further or more preferably, the combination of the contents of the rear band with the foregoing embodiment is provided as a complete construct of the other embodiment. A further embodiment is composed of several further, preferably, still further or preferably arrangements of the strips after the same embodiment, which may be combined arbitrarily.

It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are by way of example only and are not limiting. The objects of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been shown and described in the examples and embodiments of the invention may be modified or practiced without departing from the principles described.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present disclosure, and not for limiting the same; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for parallelizing elastically telescoping stream application operators, comprising:

s104: the task is notified and the state of the backup is repartitioned.

2. The method for parallelizing the elastically telescoping stream application operators in accordance with claim 1, wherein said S102 specifically comprises:

s1021: acquiring a first operator in the operator cluster

；

3. The method for parallelizing elastically telescoping stream application operators in accordance with claim 2, wherein the input queue rate of the first operator employs the formula:

；

wherein,,

is the instance rate of the first operator.

4. The flexible stream application operator parallelization method of claim 1, wherein S102 further comprises: preempting computing nodes to extend computing resources used by the streaming application.

5. The flexible stream application operator parallelization method of claim 1, wherein the stream application data comprises stateless operators and stateful operators, and wherein S103 further comprises: in the case that the operator is a stateless operator, when changing the number of instances of the stateless operator, the redirected data stream does not affect the instance processing data tuples of the stateless operator; in the case where the operator is a stateful operator, backing up the state and caching the data tuples when changing the number of instances of the stateful operator, the redirected data stream may cause the data tuples to miss the state of the stateful operator.

6. An elastically telescoping stream application operator parallelization system, comprising: the system information comprises calculation nodes in an operator cluster, CPU information of tasks, I/O information of the tasks and memory resource consumption information of the tasks, data transmission rate among the tasks in a topological structure and running state information of a distributed stream calculation system; the topology analysis module is used for optimizing the number of instances of each operation in the topology structure according to the system information; the resource analysis module is used for selecting a target node to deploy or recycle the operator instance according to the system information and the instance number; and the state notification management module is used for notifying tasks and repartitioning the backup state.

7. The flexible stream application operator parallelization system of claim 6, wherein the topology analysis module specifically comprises:

；

8. The flexible stream application operator parallelization system of claim 7, wherein the input queue rate of the first operator uses the formula:

；

wherein,,

is the instance rate of the first operator.

9. The flexible stream application operator parallelization system of claim 8, wherein the topology analysis module is further to: preempting the computing node to extend the computing resources used by the streaming application.

10. The flexible stream application operator parallelization system of claim 9, wherein the stream application data comprises stateless operators and stateful operators, the resource analysis module further to: in the case that the operator is a stateless operator, when changing the number of instances of the stateless operator, the redirected data stream does not affect the instance processing data tuples of the stateless operator; in the case where the operator is a stateful operator, backing up the state and caching the data tuples when changing the number of instances of the stateful operator, the redirected data stream may cause the data tuples to miss the state of the stateful operator.