CN114489985A

CN114489985A - Data processing method, device and storage medium

Info

Publication number: CN114489985A
Application number: CN202210096168.1A
Authority: CN
Inventors: 陈壮豪
Original assignee: Alibaba Cloud Computing Ltd
Current assignee: Alibaba Cloud Computing Ltd
Priority date: 2022-01-26
Filing date: 2022-01-26
Publication date: 2022-05-13

Abstract

The embodiment of the application provides a data processing method, data processing equipment and a storage medium. In the data processing method, the created target job instance can dynamically load metadata of one or more data processing models, and run one or more newly added data processing models according to the dynamically loaded metadata. After the newly added data processing model is operated, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In the implementation mode, on one hand, the data processing model can be dynamically loaded by the job instance, and the data processing model does not need to be subjected to customized development when the job is submitted, so that the development cost is reduced; on the other hand, one job instance can support the operation of one or more data processing models, so that the resource sharing is realized, and the resource utilization rate is greatly improved.

Description

Data processing method, device and storage medium

Technical Field

The present application relates to the field of data computing technologies, and in particular, to a data processing method, device, and storage medium.

Background

Streaming processing (streaming processing) is a continuous processing method for unbounded data or events, and the process of streaming processing can be described by using a Directed Acyclic Graph (DAG). In a DAG, each edge may represent a data or event stream and each vertex may represent a data processing model. The data processing model can process data according to set rules.

In the prior art, when a streaming data processing framework is used for processing streaming data, a job task corresponding to a data processing model can be submitted to the streaming data processing framework. The streaming data processing framework can create corresponding job instances by utilizing the resources so as to execute data processing tasks corresponding to the data processing model. However, as the number of data processing models grows, so does the number of job instances, which occupy a large amount of resources. Therefore, a new solution is yet to be proposed.

Disclosure of Invention

Aspects of the present application provide a data processing method, device and storage medium, so as to implement sharing of a plurality of models to a job instance, and improve utilization rate of resources.

An embodiment of the present application provides a data processing method, including: loading metadata of at least one data processing model from a target storage space via a target job instance in response to a model loading event; running the at least one data processing model according to the metadata of the at least one data processing model respectively through the target job instance; and respectively processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model by the target operation example.

Further optionally, before loading, by the target job instance, the model metadata of the at least one data processing model from the specified storage space, the method further includes: receiving a model submission request; the model submitting request carries model metadata of the newly added data processing model and the instance identifier of the target operation instance; and storing the model metadata of the newly added data processing model in the target storage space corresponding to the instance identifier.

Further optionally, before processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model, the method further includes: determining a second data source corresponding to any one of the at least one data processing model; and synchronizing the data in the second data source and the processing progress information of the data to the first data source in real time so as to be read by the target operation instance.

Further optionally, running, by the target job instance, the at least one data processing model according to the metadata of the at least one data processing model respectively includes: determining a plurality of computing units contained in the data processing model according to the metadata of the data processing model; identifying respective calculation types of the plurality of calculation units; dividing the operation process of the data processing model into at least one operation stage according to the respective calculation type of the plurality of calculation units; and respectively operating the computing units respectively contained in the at least one operating stage by adopting different resources corresponding to the target operation instance.

Further optionally, identifying the respective computation types of the plurality of computation units includes: for any computing unit in the plurality of computing units, acquiring an execution script of the computing unit from metadata of the data processing model; analyzing the execution script, and judging whether the computing unit comprises state read-write operation according to an analysis result; and if the computing unit comprises state read-write operation, determining that the computing type of the computing unit is stateful computing.

Further optionally, the target job instance is deployed in a distributed computing system, the distributed computing system comprising: a first class of compute nodes for performing stateless computations and a second class of compute nodes for performing stateful computations; adopting different resources corresponding to the target operation instance to respectively run the computing units respectively contained in the at least one running stage, wherein the method comprises the following steps: and running a computing unit in a stateless computing stage by adopting a first target node in the first class of computing nodes and/or running a computing unit in a stateful computing stage by adopting a second target node in the second class of computing nodes through the target operation instance.

Further optionally, respectively processing, by the target job instance, data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model, including: performing stateless calculation according to the received data through a calculating unit operated on the first target node to obtain a stateless calculation result; and if the data processing model does not contain a state calculation rule, outputting the stateless calculation result through the first target node.

Further optionally, the method further comprises: if the data processing model contains a state calculation rule, the stateless calculation result is sent to the second target node through the first target node according to the identification of the data processing model; and performing stateful calculation according to the received stateless calculation result through a calculation unit operated on the second target node to obtain a stateful calculation result, and outputting the stateful calculation result.

An embodiment of the present application further provides a server, including: a memory and a processor; the memory is to store one or more computer instructions; the processor is to execute the one or more computer instructions to: the steps in the method provided by the embodiments of the present application are performed.

Embodiments of the present application further provide a computer-readable storage medium storing a computer program, where the computer program can implement the steps in the method provided in the embodiments of the present application when executed by a processor.

In the embodiment of the application, the created target job instance may dynamically load metadata of one or more data processing models, and run one or more newly added data processing models according to the dynamically loaded metadata. After the newly added data processing model is operated, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In the implementation mode, on one hand, the data processing model can be dynamically loaded by the job instance, and the data processing model does not need to be subjected to customized development when the job is submitted, so that the development cost is reduced; on the other hand, one job instance can support the operation of one or more data processing models, so that the resource sharing is realized, and the resource utilization rate is greatly improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a flowchart of a streaming data processing method according to an exemplary embodiment of the present application;

FIG. 2 is a processing logic diagram of a data processing method provided by an exemplary embodiment of the present application;

fig. 3 is a schematic structural diagram of a server according to an exemplary embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the prior art, when performing streaming data processing based on a streaming data processing framework, a server may submit a job instance corresponding to a data processing model to the streaming data processing framework. The streaming data processing framework can create corresponding job instances by utilizing the resources so as to execute data processing tasks corresponding to the data processing model. However, as the number of data processing models grows, so does the number of job instances, which occupy a large amount of resources.

In view of the above technical problems, in some embodiments of the present application, a solution is provided, and the technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

Fig. 1 is a schematic flowchart of a data processing method according to an exemplary embodiment of the present application, and as shown in fig. 1, the method includes:

step 101, in response to a model loading event, loading metadata of at least one data processing model from a target storage space through a target job instance.

And 102, running the at least one data processing model according to the metadata of the at least one data processing model respectively through the target operation instance.

And 103, respectively processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model through the target operation instance.

The execution subject of this embodiment may be one or more servers (hereinafter referred to as servers), and any server may be implemented as a conventional server device, a virtual machine, a cloud host, or an elastic computing instance on the cloud. Wherein the one or more servers have a streaming computing application running thereon based on a streaming computing framework (e.g., Blink or Flink) that can be used to provide streaming computing services.

In a streaming computing framework, a resident computing task may be referred to as a Job (Job). When there is a need to perform a job task, the server may run the streaming application to create a job instance. In the application program, the core components for executing the job task may include: a Job Manager component (Job Master), a Task execution component (Task execution), and a Resource Manager component (Resource Manager). Wherein the job manager component is used to manage jobs; the task execution component is used for executing each task; the resource manager component is used to manage resources and service resource requests of the job management component.

When creating the job instance, the server may allocate certain resources for the job instance, including but not limited to CPU resources, memory resources, network resources, and the like.

The model loading event may be a timing event or a user event, and this embodiment is not limited. In some cases, the server may periodically trigger a model loading event to cause the target job instance to periodically check for an updated data processing model. In other cases, the server may trigger a model loading event based on a user-side operation (e.g., a user submitting a new data processing model) to cause the target job instance to load the user-submitted new data processing model.

The data processing model can be integrated by various calculation rules and is used for expressing the flow and the steps of data processing and specific algorithms in each step. A data processing model may be represented by a directed acyclic graph composed of a number of nodes (or called compute units). A node, referring to a basic computational unit in a data processing model, is used to express a computational rule, which can be edited based on a script. In the data processing model, the results produced by the upstream node may be used as input to the downstream node.

In this embodiment, the data processing model may be dynamically configured. When a new data processing model needs to be added in the streaming computing framework, the new data processing model can be submitted to the server, and the server adds the new data processing model to the specified storage space. When the model loading event is triggered, the created target job instance on the server can dynamically load the newly added data processing model.

It is worth noting that the target job instance may load a new data processing model in response to a model loading event while a certain data processing model has been run; a new data processing model may also be loaded in the idle state in response to a model loading event. That is, the target job instance, may run one or more data processing models simultaneously, which may share resources of the target job instance.

When the target operation instance loads the data processing model, the metadata of the data processing model can be loaded from the target storage space. The target storage space may be located locally in the server or in a remote database, which is not limited in this embodiment. In some embodiments, different job instances may correspond to different target storage spaces to facilitate submission of the data processing model to different job instances. Where metadata of a data processing model refers to data used to describe the data processing logic of the data processing model, this may include, but is not limited to: at least one of a calculation rule, a calculation step, a script file, an input format, and an output format of the data processing model.

For any data processing model, after the target operation instance acquires the metadata of the data processing model, the server can run the data processing model according to the metadata of the data processing model through the target operation instance. When the target operation example is loaded to the plurality of data processing models from the target storage space, the server can start a plurality of threads through the target operation example, and run the data processing models respectively according to the metadata of the plurality of data processing models by adopting the plurality of threads. Wherein any thread can be used to run one data processing model, thereby realizing the parallel execution of data processing operation based on a plurality of data processing models.

When the target operation instance runs any data processing model, the data processing model can be loaded into the memory, the calculation rule of the data processing model is analyzed, and data is waited to arrive.

The first data source monitored by the target job instance may be a data source specified for the target job instance when the target job instance is created. The target job instance may fixedly obtain data from the first data source. When data arrives, the target job instance may perform data processing operations according to the computational rules of the data processing model.

In this embodiment, the created target job instance may dynamically load metadata of one or more data processing models, and run the newly added one or more data processing models according to the dynamically loaded metadata. After the newly added data processing model is operated, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In the implementation mode, on one hand, the data processing model can be dynamically loaded by the job instance, and the data processing model does not need to be subjected to customized development when the job is submitted, so that the development cost is reduced; on the other hand, one job instance can support the operation of one or more data processing models, so that the resource sharing is realized, and the resource utilization rate is greatly improved.

In some exemplary embodiments, the at least one data processing model loaded by the target job instance may be previously submitted to the server by the user through the client. Optionally, when the user has a requirement for adding a new data processing model, if the user does not need to specify a job instance, a model submission request may be initiated to the server through the client according to the metadata of the new data processing model. After receiving the model submission request, the server may automatically randomly allocate a job instance to the newly added data processing model, or allocate a job instance with a lower current resource utilization rate to the newly added data processing model, or newly establish a job instance for the newly added data processing model, which is not limited in this embodiment.

Alternatively, the user may specify that a particular job instance run the data processing model when submitting the newly added data processing model. If the user needs to specify the job instance, a model submission request can be sent to the server through the client according to the metadata of the data processing model and the identification of the specified job instance. The instance identifier may be an ID (Identity), a process number, a name, and the like of the job instance, which is not limited in this embodiment.

When multiple job instances are created on the server, the storage space corresponding to each job instance may be different. When different job instances are created, the server can respectively designate storage spaces for storing the data processing models for the different job instances, so that the data processing models can be dynamically loaded by the job instances. If the model submission request carries the instance identifier of the target operation instance specified by the user, after receiving the model submission request sent by the client, the server can store the model metadata of the newly added data processing model in the target storage space corresponding to the instance identifier according to the model metadata of the newly added data processing model and the instance identifier of the target operation instance carried by the submission request. Therefore, the target job instance can read the newly added data processing model from the target storage space so as to meet the requirement of the user for specifying the job instance.

Of course, in some cases, if the current resource occupancy rate of the target job instance specified by the user is high, the server may switch to another job instance for the data processing model newly added by the user, so as to ensure the operating efficiency of the data processing model. After switching the job instance, the server can send a switching message of the job instance to the user through the client so as to prompt the user.

In the embodiment, the data processing model can be dynamically configured by a user through the client, and customized development is not required to be carried out according to the calculation rule of the data processing model when the job instance is created, so that the development cost is greatly reduced. For each operation example, the running data processing model can be loaded in a dynamic mode, and the flexibility of resource sharing of the operation example is greatly improved.

Continuing with the target job example. In some exemplary embodiments, the server, when creating the target job instance, may specify a data source (i.e., the first data source) for the target job instance. Wherein, the data source refers to a database for receiving external data in the streaming computation framework. And the target operation instance can monitor the data source in real time in the running process so as to acquire the data to be processed. In some cases, the data sources of different data processing models are different. Therefore, in order to enable the target job instance to process the data in the data source corresponding to any data processing model after running the data processing model, the data source of the data processing model and the first data source can be synchronously operated in real time.

Taking a data processing model operated by a target operation example as an example, the server can determine a data source corresponding to the data processing model through the target operation example to serve as a second data source, and synchronize data in the second data source and processing progress information of the data to the first data source in real time for reading by the target operation example. The data processing progress information includes an offset (offset) of data, and is used to avoid repeatedly processing the same data. The following description will be made with reference to specific examples.

For example, job instance A1 listens to data source S1 in real time, and job instance A2 listens to data source S2 in real time. The data processing model B1 is used to calculate the user' S registration data, which is deposited in the data source S3. The data processing model B2 is used to calculate the user' S login data, which is deposited in the data source S4. The data processing model B3 is used to calculate transaction data for the user, the login data being deposited in the data source S5. The client submits data processing models B1, B2, carrying instance identification of job instance A1. The client submits data processing model B3, carrying the instance identification of job instance A2. After receiving the submission request from the client, the server may synchronize the data in the data source S3, the data source S4, and the offset of the data to the data source S1 in real time, so as to be read by the operation instance a 1; and data in data source S5 and offsets of the data can be synchronized in real time to data source S2 for job instance A2 to read.

In the embodiment, when the created target job instance has a data source for fixed monitoring, the data source of the newly added data processing model is synchronized into the data source for monitoring the target job instance, so that the data readable by the target job instance can be flexibly updated, on one hand, the operation of the target job instance is not required to be suspended to modify the data source for monitoring, the influence on the executed data processing model is reduced, on the other hand, the data to be processed of the newly added data processing model is updated in real time, and the accuracy and reliability of the output result of the data processing model are ensured.

In some exemplary embodiments, when multiple data processing models share the same job instance, to further improve resource utilization, the operation processes of the multiple data processing models may be divided into stages, and different operation stages use different resources, so as to reduce mutual influence of different operation stages in terms of resource usage. As will be exemplified below.

Continuing with any data processing model as an example, optionally, when the server runs the data processing model according to the metadata of the data processing model, the server may determine a plurality of computing units included in the data processing model according to the metadata of the data processing model, and identify respective computing types of the plurality of computing units. Wherein, one computing unit can correspond to one operator (operator) in the streaming computing framework. The calculation type refers to the type of an operator corresponding to the calculation unit.

In a streaming computing framework, operators include stateful operators as well as stateless operators. The stateless operator is mainly used for performing stateless calculation, and the stateless calculation means that the operator only needs to process current data to obtain a calculation result. The stateful operator is mainly used for performing stateful calculation, which means that the operator needs to perform correlation calculation in combination with some historical states (e.g., historical calculation results, intermediate calculation results, etc.) to obtain a calculation result. The calculation speed of the operator with the state is low, and the demand on resources is large.

Based on the above characteristics of the stateful operator, the server acquires, for any one of the plurality of computing units, an execution script of the computing unit from the metadata of the data processing model when identifying the computing type of each of the plurality of computing units through the job instance; analyzing the execution script, and judging whether the computing unit comprises a state (state) read-write operation according to an analysis result; and if the computing unit comprises state read-write operation, determining that the computing type of the computing unit is stateful computing.

After determining the respective computing type of the plurality of computing units, the server may divide the operation process of the data processing model into at least one operation phase according to the respective computing type of the plurality of computing units. For example, if the data processing model includes only stateless types of computing units, the server may divide the operation of the data processing model into only stateless operational phases. If the data processing model includes a stateless type of computing unit and a stateful type of computing unit, the server may divide the operation process of the data processing model into a stateless computing stage and a stateful computing stage.

After the operation stages are divided, the server can respectively operate the computing units contained in the at least one divided operation stage by using different resources through the target operation example. The resource refers to a resource allocated to the target instance by the server when the target instance is created, and includes but is not limited to resources such as a CPU, a memory, and the like.

Optionally, in this embodiment, the server may allocate different resources to different operation stages respectively, the different computing tasks in the same operation stage may share the allocated resources, and the resources between the computing tasks in different operation stages are in a relatively isolated state, so as to reduce the mutual influence on the resources in different operation stages and improve the computing efficiency. For example, 2 stateless computing units may share a 2-core CPU, 20G memory; 2 stateful computing units may share 4-core CPU, 60G memory.

Optionally, the resources allocated to different operation stages may include resources such as a CPU and a memory on one server, and may also include resources such as a CPU and a memory on different servers in a server cluster. For example, in some embodiments, multiple virtual machines on a server may be assigned to different compute stages, respectively. In other embodiments, multiple server devices in the server cluster may be respectively allocated to different operation phases, which is not limited in this embodiment.

In some exemplary embodiments, the target job instance may be deployed in a distributed computing system comprising: a first class of compute nodes for performing stateless computations, and a second class of compute nodes for performing stateful computations.

Based on the above, if the data processing model only includes the computing unit whose computation type is stateless computation, the server may deploy the computing unit whose computation type is stateless computation to a first target node in the first class of computing nodes. The first target node may be one or more computing nodes in the first class of computing nodes. Correspondingly, when the server runs the computing units respectively contained in the at least one running stage, the computing units in the stateless computing stage can be run by adopting a first target node in the first class of computing nodes through the target job instance.

If the data processing model only includes computing units whose compute types are stateful computations, the server may deploy the computing units whose compute types are stateful computations to a second target node in the second class of computing nodes. The second target node may be one or more computing nodes in the second class of computing nodes. Correspondingly, when the server runs the computing units respectively contained in the at least one running stage, the computing units in the state computing stage can be run by adopting a second target node in the second type of computing nodes through the target job instance.

If the data processing model includes both the computing units whose computing types are stateless computing and stateful computing, the server may deploy the computing unit whose computing type is stateless computing to a first target node in the first class of computing nodes, and deploy the computing unit whose computing type is stateful computing to a second target node in the second class of computing nodes. Correspondingly, when the server runs the computing units respectively contained in the at least one running stage, the computing units in the stateless computing stage can be run by adopting a first target node in the first class of computing nodes and the computing units in the stateful computing stage can be run by adopting a second target node in the second class of computing nodes through the target job instance.

Accordingly, in some exemplary embodiments, when the server processes the currently monitored data in the first data source according to the calculation rule of the data processing model through the target job instance, the server may perform stateless calculation according to the received data through a calculation unit running on the first target node, so as to obtain a stateless calculation result. If the data processing model does not include a state computation rule, the target job instance may output the stateless computation result via the first target node.

Optionally, if the data processing model includes a state calculation, the second target node needs to perform further calculation according to the stateless calculation result of the first target node. It should be noted that, when the computing task is executed in a distributed manner, for stateful computing, the correctness of the intermediate computing process or the accumulated value received by the stateful computing needs to be ensured. That is, the stateless computation result of the first target node needs to be accurately sent to the second target node for stateful computation. When the second type of computing node comprises a plurality of nodes, the first target node needs to accurately determine a node matched with the second type of computing node from the second type of computing node to be used as a second target node.

Optionally, in this embodiment, when different computing units in the data processing model are deployed to different nodes, an identifier of the data processing model may be added to the different computing units in the data processing model. Based on this, if the data processing model includes a state calculation rule, the stateless calculation result may be sent to the second target node via the first target node according to the identifier of the data processing model. Therefore, the server can perform stateful calculation according to the received stateless calculation result through the calculation unit running on the second target node to obtain a stateful calculation result, and output the stateful calculation result.

In the embodiment, the data processing model is operated in a stage splitting mode, and the stateful calculation with high time consumption and the stateless calculation with low time consumption can be operated by using different resources, so that the allocation of the resources is further rationalized, and the calculation efficiency is improved.

The embodiments of the present application will be further illustrated with reference to fig. 2.

As shown in FIG. 2, a user may implement dynamic configuration of a data processing model by way of configuration. The Job instance (Blink Job or Flink Job) can load the data processing model which needs to be processed by the Job instance in a timing and directional mode and perform calculation. For the same job instance, multiple different data processing models can be loaded, thereby achieving model sharing computation. Model 1 and model 2 are shown. When the calculation of the data processing model is more complex, the job instance can judge to independently calculate the data processing model or to run the data processing model and other data processing models in parallel according to the available surplus of resources.

In the streaming framework, the job has set up the data source to be listened to at startup, e.g. the topic (topic) in the database of Kafaka, DataHub, etc. The data processing model is dynamically loaded into the operation after the operation is started, and the data source which needs to be monitored by the operation cannot be determined before the operation is started. Therefore, as shown in fig. 2, it is possible to set a fixed data source for each job (i.e., unified input) to listen, and synchronize the data and basic data information (such as offset) listened by the data processing model into the fixed data source when the data processing model is loaded. The monitoring and distribution of data are automatically realized in the operation process, so that the problem that the data sources cannot be dynamically increased or decreased in the operation in the streaming computation framework is solved.

Each job is mainly used for completing the loading of model metadata, the analysis of calculation rules and the real-time calculation of data. The real-time calculation comprises two types of calculation, wherein one type of calculation is stateless calculation, and the other type of calculation is stateful calculation. Stateful computations (e.g., aggregate computations) require the retention of historical information (e.g., counts, cumulative values of the history that need to be retained), and such computations tend to be complex. In operation, the computation of the data processing model can be split into two phases: a MAP (mapping) phase for completing stateless computations and an AGG (aggregation) phase for completing stateful computations. As shown in fig. 2, in the AGG stage, the storage of the aggregated data may be realized through a state variable provided by Blink or Flink bottom layer.

As shown in fig. 2, when the data processing model only includes stateless computation, after the MAP stage computation, the stateless computation result, such as model 1, can be directly output. When the data processing model includes state calculation, the model can be divided into two stages for calculation from an operator of the existing state calculation, and finally, a state calculation result, such as a model 2, is output. When the data processing model includes state calculation, because the operation is distributed, in order to ensure the correctness of the accumulated value, the nodes in the MAP stage can distribute data according to the ID of the data processing model, so as to ensure that the data of the same model falls on the same node of the AGG, and further description is omitted.

It should be noted that the execution subjects of the steps of the methods provided in the above embodiments may be the same device, or different devices may be used as the execution subjects of the methods. For example, the execution subjects of steps 101 to 104 may be device a; for another example, the execution subject of

steps

101 and 102 may be device a, and the execution subject of step 103 may be device b; and so on.

In addition, in some of the flows described in the above embodiments and the drawings, a plurality of operations are included in a specific order, but it should be clearly understood that the operations may be executed out of the order presented herein or in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are merely used for distinguishing different operations, and the sequence numbers do not represent any execution order per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel.

It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor limit the types of "first" and "second" to be different.

Fig. 3 is a schematic structural diagram of a server according to an exemplary embodiment of the present application, where the server may be used to execute the data processing method described in the foregoing embodiment. As shown in fig. 3, the server includes: a memory 301 and a processor 302.

The memory 301 is used for storing computer programs and may be configured to store other various data to support operations on the server. Examples of such data include instructions for any application or method operating on the server, contact data, phonebook data, messages, pictures, videos, and so forth.

The memory 301 may be implemented, among other things, by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

Wherein the processor 302 is coupled to the memory 301 for executing the computer program in the memory 301 for: in response to a model load event, loading metadata for at least one data processing model from a target storage space via a target job instance; running the at least one data processing model according to the metadata of the at least one data processing model respectively through the target job instance; and respectively processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model by the target operation example.

Further optionally, processor 302, prior to loading model metadata for the at least one data processing model from the specified storage space via the target job instance, is further configured to: receiving a model submission request; the model submitting request carries model metadata of the newly added data processing model and the instance identifier of the target operation instance; and storing the model metadata of the newly added data processing model in the target storage space corresponding to the instance identifier.

Further optionally, before processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model, the processor 302 is further configured to: determining a second data source corresponding to any one of the at least one data processing model; and synchronizing the data in the second data source and the processing progress information of the data to the first data source in real time so as to be read by the target operation instance.

Further optionally, when the processor 302 runs the at least one data processing model according to the metadata of the at least one data processing model respectively through the target job instance, specifically, the processor is configured to: determining a plurality of computing units contained in the data processing model according to the metadata of the data processing model; identifying respective calculation types of the plurality of calculation units; dividing the operation process of the data processing model into at least one operation stage according to the respective calculation type of the plurality of calculation units; and respectively operating the computing units respectively contained in the at least one operating stage by adopting different resources corresponding to the target operation instance.

Further optionally, when identifying the respective calculation types of the plurality of calculation units, the processor 302 is specifically configured to: for any computing unit in the plurality of computing units, acquiring an execution script of the computing unit from metadata of the data processing model; analyzing the execution script, and judging whether the computing unit comprises state read-write operation according to an analysis result; and if the computing unit comprises state read-write operation, determining that the computing type of the computing unit is stateful computing.

Further optionally, the target job instance is deployed in a distributed computing system, the distributed computing system comprising: a first class of compute nodes for performing stateless computations and a second class of compute nodes for performing stateful computations; when the processor 302 respectively runs the computing units included in the at least one running stage by using the different resources corresponding to the target job instance, the processor is specifically configured to: and running a computing unit in a stateless computing stage by adopting a first target node in the first class of computing nodes and/or running a computing unit in a stateful computing stage by adopting a second target node in the second class of computing nodes through the target operation instance.

Further optionally, when the processor 302 processes, by using the target job instance, the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model, specifically: performing stateless calculation according to the received data by using a calculation unit running on the first target node to obtain a stateless calculation result; and if the data processing model does not contain a state calculation rule, outputting the stateless calculation result through the first target node.

Further optionally, the processor 302 is further configured to: if the data processing model contains a state calculation rule, the stateless calculation result is sent to the second target node through the first target node according to the identification of the data processing model; and performing stateful calculation according to the received stateless calculation result through a calculation unit operated on the second target node to obtain a stateful calculation result, and outputting the stateful calculation result.

Further, as shown in fig. 3, the server further includes: communication components 303, and power components 304. Only some of the components are schematically shown in fig. 3, and it is not meant that the server includes only the components shown in fig. 3.

Wherein the communication component 303 is configured to facilitate communication between the device in which the communication component is located and other devices in a wired or wireless manner. The device in which the communication component is located may access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

The power supply component 304 provides power to various components of the device in which the power supply component is located. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power component is located.

In this embodiment, the target job instance created on the server may dynamically load metadata of one or more data processing models, and run one or more newly added data processing models according to the dynamically loaded metadata. After the newly added data processing model is operated, the data in the currently monitored data source can be processed according to the calculation rule of the newly added data processing model. In the implementation mode, on one hand, the data processing model can be dynamically loaded by the job instance, and the data processing model does not need to be subjected to customized development when the job is submitted, so that the development cost is reduced; on the other hand, one job instance can support the operation of one or more data processing models, so that the resource sharing is realized, and the resource utilization rate is greatly improved.

Accordingly, the present application further provides a computer-readable storage medium storing a computer program, where the computer program can implement the steps that can be executed by the server in the foregoing method embodiments when executed.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, comprising:

loading metadata of at least one data processing model from a target storage space via a target job instance in response to a model loading event;

running the at least one data processing model according to the metadata of the at least one data processing model respectively through the target job instance;

and respectively processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model by the target operation example.

2. The method of claim 1, wherein prior to loading, by the target job instance, model metadata for the at least one data processing model from the specified storage space, further comprising:

receiving a model submission request; the model submitting request carries model metadata of the newly added data processing model and the instance identifier of the target operation instance;

and storing the model metadata of the newly added data processing model in the target storage space corresponding to the instance identifier.

3. The method according to claim 1, wherein before processing the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model, the method further comprises:

determining a second data source corresponding to any one of the at least one data processing model;

and synchronizing the data in the second data source and the processing progress information of the data to the first data source in real time so as to be read by the target operation instance.

4. The method of claim 3, wherein executing, by the target job instance, the at least one data processing model according to the metadata of the at least one data processing model, respectively, comprises:

determining a plurality of computing units contained in the data processing model according to the metadata of the data processing model;

identifying respective calculation types of the plurality of calculation units;

dividing the operation process of the data processing model into at least one operation stage according to the respective calculation type of the plurality of calculation units;

and respectively operating the computing units respectively contained in the at least one operating stage by adopting different resources corresponding to the target operation instance.

5. The method of claim 4, wherein identifying the respective computation types of the plurality of computation units comprises:

for any computing unit in the plurality of computing units, acquiring an execution script of the computing unit from metadata of the data processing model;

analyzing the execution script, and judging whether the computing unit comprises state read-write operation according to an analysis result;

and if the computing unit comprises state read-write operation, determining that the computing type of the computing unit is stateful computing.

6. The method of claim 4, wherein the target job instance is deployed in a distributed computing system, the distributed computing system comprising: a first class of compute nodes for performing stateless computations and a second class of compute nodes for performing stateful computations;

adopting different resources corresponding to the target operation instance to respectively run the computing units respectively contained in the at least one running stage, wherein the method comprises the following steps:

and running a computing unit in a stateless computing stage by adopting a first target node in the first class of computing nodes and/or running a computing unit in a stateful computing stage by adopting a second target node in the second class of computing nodes through the target operation instance.

7. The method according to claim 6, wherein the processing, by the target job instance, the data in the currently monitored first data source according to the respective calculation rule of the at least one data processing model respectively comprises:

performing stateless calculation according to the received data through a calculating unit operated on the first target node to obtain a stateless calculation result;

and if the data processing model does not contain a state calculation rule, outputting the stateless calculation result through the first target node.

8. The method of claim 7, further comprising:

if the data processing model contains a state calculation rule, the stateless calculation result is sent to the second target node through the first target node according to the identification of the data processing model;

and performing stateful computation according to the received stateless computation result through a computation unit running on the second target node to obtain a stateful computation result, and outputting the stateful computation result.

9. A server, comprising: a memory and a processor;

the memory is to store one or more computer instructions;

the processor is to execute the one or more computer instructions to: performing the steps of the method of any one of claims 1-8.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, is adapted to carry out the steps of the method of any one of claims 1 to 8.