CN114064142A - Batch-flow integrated data processing system and processing method - Google Patents
Batch-flow integrated data processing system and processing method Download PDFInfo
- Publication number
- CN114064142A CN114064142A CN202111254417.7A CN202111254417A CN114064142A CN 114064142 A CN114064142 A CN 114064142A CN 202111254417 A CN202111254417 A CN 202111254417A CN 114064142 A CN114064142 A CN 114064142A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- batch
- processing
- component model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4482—Procedural
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
Abstract
The invention discloses a batch-flow integrated data processing system and a batch-flow integrated data processing method, belongs to the technical field of big data processing analysis, and aims to solve the technical problem of how to provide a distributed batch-flow integrated data processing mode and improve the real-time performance and convenience of data processing calculation. The method comprises the following steps: the front-end interaction subsystem is used for supporting an administrator to configure a component model, the component model is used for defining rules of data reading, data processing and data writing, and is used for supporting a user to configure tasks and submit task requests based on the component model; the configuration library is used for storing the component models and the configured tasks; the back-end service subsystem is used for executing tasks based on the task requests, monitoring the task operation, generating novel task operation and abnormal alarm, and returning task operation information and abnormal alarm, and the processing logic of the back-end service subsystem is as follows: a special case exists in which stream processing is used as processing logic, and a batch is used as a stream, and when batch data flows in, the batch data is converted into stream data through a time window.
Description
Technical Field
The invention relates to the technical field of big data processing analysis, in particular to a batch-flow integrated data processing system and a batch-flow integrated data processing method.
Background
With the continuous development of computer technology and the continuous improvement of informatization degree, the data resources are rapidly increased, huge and complicated, and great challenges are brought to the traditional data analysis and processing technology. The conventional data processing software has some defects, which mainly include: the method adopts a pseudo-distributed architecture, batch data and stream data need to be processed independently, the processing efficiency is low, and the distributed acquisition, calculation, filtration and analysis of complex, relational and non-relational data cannot be met. Therefore, the distributed data processing system is receiving more and more attention, and among the technologies related to big data computing, a distributed data batch-flow unified computing engine is one of the technologies.
How to provide a distributed batch-flow integrated data processing mode and improve the real-time performance and convenience of data processing calculation is a technical problem to be solved.
Disclosure of Invention
The technical task of the invention is to provide a batch-flow integrated data processing system and a batch-flow integrated data processing method aiming at the defects, so as to solve the technical problems of how to provide a distributed batch-flow integrated data processing mode and improve the real-time performance and convenience of data processing calculation.
In a first aspect, the present invention provides a batch-flow unified data processing system, comprising:
the front-end interaction subsystem interacts with an administrator and a user through a front-end interaction interface, is used for supporting the administrator to configure a component model, and is used for defining rules of data reading, data processing and data writing, and is used for supporting the user to configure a task based on the component model and submit a task request;
the configuration library is interacted with the front-end interaction module and is used for storing the component model and the configured task;
the back-end service subsystem interacts with the front-end interaction subsystem, is used for executing tasks and monitoring task operation based on task requests, generates novel task operation and abnormal alarm, is used for returning task operation information and abnormal alarm based on task query requests of the front-end interaction module, and has the following processing logics: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
Preferably, the front-end interaction subsystem includes:
the system comprises a registration login module, a front-end interface and a server, wherein the registration login module is interacted with the front-end interface and is used for supporting registration login of a user and an administrator in a user name and password mode;
the component configuration module is interacted with an administrator through a front-end interaction interface and is used for supporting the administrator to configure a component model and carry out editing operation on the component model, and the editing operation comprises loading, removing and updating operation;
the task configuration module interacts with a user through a front-end interaction interface, and is used for supporting the user to create task logic according to the component model and select the component model configuration task in a dragging mode, wherein the task logic is as follows: data source-processing components and target libraries;
the task processing module is matched with the front-end interactive interface, a user submits a task request and a task query request by triggering a request button on a front-end page, and the task running information and the abnormal alarm are checked through the front-end interactive interface;
the front-end interaction module interacts with the configuration library and the back-end service subsystem through the interfaces, is used for storing the component models and the tasks in the configuration library, submitting task requests and task query requests to the back-end service subsystem, and acquiring task operation information and abnormal alarms from the back-end service subsystem.
Preferably, the front-end interaction subsystem is a BS framework constructed based on SpringBoot, integrates a jsplimb dragging component to realize dragging of the component model, and realizes interaction between the front-end interaction interface and a user and an administrator by combining a Layui framework.
Preferably, the component model includes:
the data source component model is used for defining a data reading mode and a source database;
a process component model for defining the logic of data processing including data screening, data cleansing and data transformation;
the target library component model is used for defining a data writing mode and a target database.
Preferably, the component parameters of the data source component model and the target library component model comprise type, address, URL string, user name and password;
the component parameters of the processing component model comprise an association type, a processing mode and processing parameters.
Preferably, the back-end service module is a batch processing engine, and includes:
the back-end service subsystem interacts with the front-end interaction subsystem through the interfaces, is used for acquiring task requests and task query requests from the front-end interaction subsystem, and is used for returning task operation information and abnormal alarms to the front-end interaction subsystem;
the resource scheduling unit interacts with the interface and is used for scheduling resources to concurrently execute continuous tasks;
the task management system comprises a task manager, a task management unit and a task management unit, wherein the task manager is provided with a plurality of task slots, each task slot runs a task flow, and the task flows are used for executing tasks and monitoring the tasks to form task running information and abnormal alarms;
and the storage unit is interacted with the task manager and the interface and is used for storing task running information to give an abnormal alarm.
In a second aspect, the batch-flow integrated data processing method of the present invention processes batch data and flow data by the batch-flow integrated data processing system according to any one of the first aspect, and includes the steps of:
configuring a component model for defining rules of data reading, data processing and data writing;
creating a task based on the component model and submitting a task request;
executing the task based on the task request, monitoring the task operation, generating task operation information and an abnormal alarm, returning the task operation information and the abnormal alarm based on the task query request, and executing the task according to the following processing logic: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
Preferably, the component model includes:
the data source component model is used for defining a data reading mode and a source database, and component parameters comprise types, addresses, URL (uniform resource locator) strings, user names and passwords;
the system comprises a processing component model, a data processing module and a data processing module, wherein the processing component model is used for defining the logic of data processing, the data processing comprises data screening, data cleaning and data conversion, and component parameters comprise association types, processing modes and processing parameters;
the target library component model is used for defining a data writing mode and a target database, and the component parameters comprise types, addresses, URL strings, user names and passwords.
Preferably, when a task is created based on the component model, creating task logic according to rules of the component model, where the task logic is: data source-processing component-target repository.
The batch-flow integrated data processing system and the batch-flow integrated data processing method have the following advantages that:
1. the method comprises the steps that a component model is configured through a front-end interaction subsystem, rules for data reading, processing and writing are defined through the component model, a task is created based on the rules of the component model, then the task is executed through a rear-end service subsystem, stream processing is taken as processing logic during the task execution, and batches are taken as a special case of the stream;
2. the system generates the data source, the data assembly and the target library code into componentization, and a user can drag an assembly model to configure a task, so that the task establishment time is effectively reduced, and the task extraction time is shortened;
3. the back-end service subsystem carries out resource scheduling through the resource scheduling unit and can reasonably distribute task operation pressure.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed for the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a schematic block diagram of resource scheduling in a data processing system according to embodiment 1;
fig. 2 is a flow chart diagram of a batch-flow integrated data processing method in embodiment 2.
Detailed Description
The present invention is further described in the following with reference to the drawings and the specific embodiments so that those skilled in the art can better understand the present invention and can implement the present invention, but the embodiments are not to be construed as limiting the present invention, and the embodiments and the technical features of the embodiments can be combined with each other without conflict.
The embodiment of the invention provides a batch-flow integrated data processing system and a batch-flow integrated data processing method, which are used for solving the technical problems of how to provide a distributed batch-flow integrated data processing mode and improving the real-time performance and convenience of data processing calculation.
Example 1:
the invention discloses a batch-flow integrated data processing system, which comprises a front-end interaction subsystem, a configuration library and a back-end service subsystem, wherein the front-end interaction subsystem interacts with an administrator and a user through a front-end interaction interface and is used for supporting the administrator to configure a component model, and the component model is used for defining rules of data reading, data processing and data writing and is used for supporting the user to configure a task and submit a task request based on the component model; the configuration library interacts with the front-end interaction module and is used for storing the component model and the configured task; the back-end service subsystem interacts with the front-end interaction subsystem, is used for executing tasks and monitoring task operation based on task requests, generates novel and abnormal task operation alarms, and is used for returning task operation information and abnormal alarms based on task query requests of the front-end interaction module, and the processing logic of the back-end service subsystem is as follows: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
In this embodiment, the front-end interaction subsystem includes a registration login module, an assembly configuration module, a task processing module, and an interface, and the registration login module interacts with the front-end interface and is used for supporting a user and an administrator to register and login in a user name and password manner; the component configuration module interacts with an administrator through a front-end interaction interface and is used for supporting the administrator to configure a component model and carry out editing operation on the component model, wherein the editing operation comprises loading, removing and updating operation; the task configuration module interacts with a user through a front-end interactive interface, and is used for supporting the user to create task logic according to the component model and select the component model configuration task in a dragging mode, wherein the task logic is as follows: data source-processing components and target libraries; the task processing module is matched with the front-end interactive interface, a user submits a task request and a task query request by triggering a request button on a front-end page, and the task operation information and the abnormal alarm are checked through the front-end interactive interface; the front-end interaction module interacts with the configuration library and the back-end service subsystem through the interfaces, is used for storing the component models and the tasks in the configuration library, submitting task requests and task query requests to the back-end service subsystem, and acquiring task operation information and abnormal alarms from the back-end service subsystem.
The component model comprises a data source component model, a processing component model and a target database component model, wherein the data source component model is used for defining a data reading mode and a source database, the processing component model is used for defining data processing logic, data processing comprises data screening, data cleaning, data conversion and the like, and the target database component model is used for defining a data writing mode and a target database. The data source component and the target library component parameters comprise: type, address, URL string, user name, password, etc., and the processing components include processing target, association type, processing mode, processing parameter, etc. The component configuration module supports fast customization of components.
In this embodiment, the front-end terminal system adopts a BS software architecture of spring boot, integrates a jsplimb dragging component, can realize dragging of a component model so as to facilitate a user to rapidly and efficiently configure a task, and provides a good visual operation experience for the user by combining with a Layui framework.
Through the front-end interaction subsystem, an administrator establishes different data source models, target library models and component processing models according to different service scene requirements so as to establish data source reading, data analysis and data writing rules, and the administrator can customize the loading, removing and updating operations of the models. A user can create a task logic of 'data source-processing component-target library' according to the model rule, and can freely drag the component model to build a new task so as to realize the construction of task data. After a user creates a task, a task request can be submitted through a corresponding task request button, so that the back-end service subsystem can execute the task based on the task request and monitor the running of the task; and the user can submit the task query request through the corresponding task query request button, so that the back-end service subsystem returns the task running information and the abnormal alarm to the front-end interaction subsystem, and the user can check the task running information and the abnormal alarm conveniently.
The back-end service module is a batch processing engine, the batch-flow integrated engine core takes flow processing as processing logic, and a special case of batch as flow exists.
The batch processing engine comprises a plurality of interfaces, a resource scheduling unit, a task manager and a storage unit, wherein the interfaces are API interfaces, and a rear-end service subsystem interacts with a front-end interaction subsystem through the interfaces of the rear-end service subsystem, is used for acquiring a task request and a task query request from the front-end interaction subsystem and is used for returning task operation information and abnormal alarms to the front-end interaction subsystem; the resource scheduling unit interacts with the interface and is used for scheduling resources to concurrently execute continuous tasks, particularly relating to scheduling of a CPU (central processing unit), a memory and the like, a plurality of task slots are configured in the task manager, a task flow runs in each task slot, and the task flow is used for executing the tasks and monitoring the tasks to form task running information and abnormal alarms; the storage unit interacts with the task manager and the interface and is used for storing task running information to give an abnormal alarm.
Distributed resource scheduling defines execution resources by task slots, each task manager has one or more task slots, each task slot can run a task stream, a channel comprises a plurality of continuous task streams, such as a continuous task stream of an nth parallel instance of a conversion operation and an nth parallel instance of a filtering operation, the resource scheduling usually executes the continuous tasks concurrently, and the execution is performed in all cases for streaming data; this is also the case for batch processes. As shown in fig. 1, the first task manager (TaskManage1) has a running graph composed of 2 parallel task flows, each occupying a task slot, and the second task manager (TaskManage2) has a running graph composed of 2 parallel task flows, each occupying a task slot; the 4 task streams running on the 2 task managers are executed in parallel.
In this embodiment, the interface is a distributed API interface, which realizes a multithreading data extraction method. The system can load task JSON data, and then create an execution task according to JSON rules, so as to gradually complete the construction of the task; the distributed system automatically selects batch processing and stream processing according to the loaded data.
The processing system can load a task model, and then creates a task logic of 'data source-processing component-target library' according to the model rule, so as to complete the construction of task data and uniformly process batch data and stream data. The task is distributed to different task operation nodes by using mathematical algorithms such as Hash, modulus and the like, meanwhile, the system provides two tasks of one-time and timing starting, and supports the functions of insertion, updating and the like, and a user can drag and autonomously establish the task according to the self requirement. The task establishing time is effectively reduced, and the task extracting time is shortened.
Example 2:
the batch-flow integrated data processing method processes batch data and flow data through the batch-flow integrated data processing system disclosed in embodiment 1, and the method includes the following steps:
s100, configuring a component model, wherein the component model is used for defining rules of data reading, data processing and data writing;
s200, creating a task based on the component model and submitting a task request;
s300, executing the task based on the task request, monitoring the task operation, generating task operation information and an abnormal alarm, returning the task operation information and the abnormal alarm based on the task query request, and executing the task according to the following processing logic: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
The component model comprises a data source component model, a processing component model and a target library component model, wherein the data source component model is used for defining a data reading mode and a source database, and the component parameters comprise types, addresses, URL (uniform resource locator) strings, user names and passwords; the processing component model is used for defining the logic of data processing, the data processing comprises data screening, data cleaning and data conversion, and the component parameters comprise correlation types, processing modes and processing parameters; the target library component model is used for defining a data writing mode and a target database, and the component parameters comprise types, addresses, URL strings, user names and passwords.
When a task is created based on the component model, creating task logic according to the rule of the component model, wherein the task logic is as follows: data source-processing component-target repository.
According to different data sources, intermediate processing processes and target libraries, different data reading, writing and processing component models are established, and parameters required by data reading, writing and processing are stored in a configuration library; a user creates a task logic of 'data source-processing component-target library' according to the component model, task JSON data is loaded in, and the user establishes a task according to JSON rules in a mode of dragging the component model; during task execution, the batch processing engine automatically selects batch processing and stream processing according to the loaded data, during processing, a source schedule defines execution resources through task slots, each task manager is provided with one or more task slots, each task slot can run a task stream, a channel comprises a plurality of continuous task streams, for example, the continuous task streams of the nth parallel instance of a conversion operation and the nth parallel instance of a filtering operation, the resource schedule usually executes the continuous tasks concurrently, and for stream data, the execution is performed in all cases; this is also the case for batch processes.
While the invention has been shown and described in detail in the drawings and in the preferred embodiments, it is not intended to limit the invention to the embodiments disclosed, and it will be apparent to those skilled in the art that many more embodiments of the invention are possible that combine the features of the different embodiments described above and still fall within the scope of the invention.
Claims (9)
1. Batch-flow unified data processing system characterized by comprising:
the front-end interaction subsystem interacts with an administrator and a user through a front-end interaction interface, is used for supporting the administrator to configure a component model, and is used for defining rules of data reading, data processing and data writing, and is used for supporting the user to configure a task based on the component model and submit a task request;
the configuration library is interacted with the front-end interaction module and is used for storing the component model and the configured task;
the back-end service subsystem interacts with the front-end interaction subsystem, is used for executing tasks and monitoring task operation based on task requests, generates novel task operation and abnormal alarm, is used for returning task operation information and abnormal alarm based on task query requests of the front-end interaction module, and has the following processing logics: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
2. The batch-flow unified data processing system according to claim 1, wherein said front-end interaction subsystem comprises:
the system comprises a registration login module, a front-end interface and a server, wherein the registration login module is interacted with the front-end interface and is used for supporting registration login of a user and an administrator in a user name and password mode;
the component configuration module is interacted with an administrator through a front-end interaction interface and is used for supporting the administrator to configure a component model and carry out editing operation on the component model, and the editing operation comprises loading, removing and updating operation;
the task configuration module interacts with a user through a front-end interaction interface, and is used for supporting the user to create task logic according to the component model and select the component model configuration task in a dragging mode, wherein the task logic is as follows: data source-processing components and target libraries;
the task processing module is matched with the front-end interactive interface, a user submits a task request and a task query request by triggering a request button on a front-end page, and the task running information and the abnormal alarm are checked through the front-end interactive interface;
the front-end interaction module interacts with the configuration library and the back-end service subsystem through the interfaces, is used for storing the component models and the tasks in the configuration library, submitting task requests and task query requests to the back-end service subsystem, and acquiring task operation information and abnormal alarms from the back-end service subsystem.
3. The batch-flow integrated data processing system according to claim 2, wherein the front-end interaction subsystem is a BS framework constructed based on SpringBoot, integrates jsplumb dragging components to realize dragging of component models, and realizes interaction between the front-end interaction interface and users and administrators by combining a Layui framework.
4. The batch-flow unified data processing system according to claim 1, 2 or 3, characterized in that said component model comprises:
the data source component model is used for defining a data reading mode and a source database;
a process component model for defining the logic of data processing including data screening, data cleansing and data transformation;
the target library component model is used for defining a data writing mode and a target database.
5. The batch-and-stream-integrated data processing system as claimed in claim 4, wherein the component parameters of the data source component model and the target library component model include type, address, URL string, username and password;
the component parameters of the processing component model comprise an association type, a processing mode and processing parameters.
6. The batch-flow unified data processing system according to claim 1, 2 or 3, wherein said back-end service module is a batch processing engine comprising:
the back-end service subsystem interacts with the front-end interaction subsystem through the interfaces, is used for acquiring task requests and task query requests from the front-end interaction subsystem, and is used for returning task operation information and abnormal alarms to the front-end interaction subsystem;
the resource scheduling unit interacts with the interface and is used for scheduling resources to concurrently execute continuous tasks;
the task management system comprises a task manager, a task management unit and a task management unit, wherein the task manager is provided with a plurality of task slots, each task slot runs a task flow, and the task flows are used for executing tasks and monitoring the tasks to form task running information and abnormal alarms;
and the storage unit is interacted with the task manager and the interface and is used for storing task running information to give an abnormal alarm.
7. A batch-flow unified data processing method, characterized in that batch data and stream data are processed by a batch-flow unified data processing system according to any of claims 1 to 6, the method comprising the steps of:
configuring a component model for defining rules of data reading, data processing and data writing;
creating a task based on the component model and submitting a task request;
executing the task based on the task request, monitoring the task operation, generating task operation information and an abnormal alarm, returning the task operation information and the abnormal alarm based on the task query request, and executing the task according to the following processing logic: the special case that stream processing is used as processing logic and batches are used as streams exists, when batch data flow in, the batch data are converted into stream data through a time window, and the batch stream data are unified.
8. The batch-flow integrated data processing method according to claim 7, wherein the component model includes:
the data source component model is used for defining a data reading mode and a source database, and component parameters comprise types, addresses, URL (uniform resource locator) strings, user names and passwords;
the system comprises a processing component model, a data processing module and a data processing module, wherein the processing component model is used for defining the logic of data processing, the data processing comprises data screening, data cleaning and data conversion, and component parameters comprise association types, processing modes and processing parameters;
the target library component model is used for defining a data writing mode and a target database, and the component parameters comprise types, addresses, URL strings, user names and passwords.
9. The batch-flow integrated data processing method according to claim 7 or 8, wherein when a task is created based on the component model, a task logic is created according to rules of the component model, and the task logic is: data source-processing component-target repository.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111254417.7A CN114064142A (en) | 2021-10-27 | 2021-10-27 | Batch-flow integrated data processing system and processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111254417.7A CN114064142A (en) | 2021-10-27 | 2021-10-27 | Batch-flow integrated data processing system and processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114064142A true CN114064142A (en) | 2022-02-18 |
Family
ID=80235589
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111254417.7A Withdrawn CN114064142A (en) | 2021-10-27 | 2021-10-27 | Batch-flow integrated data processing system and processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114064142A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117435596A (en) * | 2023-12-20 | 2024-01-23 | 杭州网易云音乐科技有限公司 | Streaming batch task integration method and device, storage medium and electronic equipment |
-
2021
- 2021-10-27 CN CN202111254417.7A patent/CN114064142A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117435596A (en) * | 2023-12-20 | 2024-01-23 | 杭州网易云音乐科技有限公司 | Streaming batch task integration method and device, storage medium and electronic equipment |
| CN117435596B (en) * | 2023-12-20 | 2024-04-02 | 杭州网易云音乐科技有限公司 | Streaming batch task integration method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11475007B2 (en) | Dynamic self-reconfiguration of nodes in a processing pipeline | |
| US11392654B2 (en) | Data fabric service system | |
| US20210149751A1 (en) | Efficient message queuing service using multiplexing | |
| US10509794B2 (en) | Dynamically-generated files for visualization sharing | |
| US8510720B2 (en) | System landscape trace | |
| US11334538B2 (en) | System and method for cardinality estimation feedback loops in query processing | |
| US20080177564A1 (en) | Method and apparatus of supporting business performance management with active shared data spaces | |
| JP2010524060A (en) | Data merging in distributed computing | |
| CN105786603B (en) | Distributed high-concurrency service processing system and method | |
| WO2019200984A1 (en) | Life cycle management method for distributed application, managers, device and medium | |
| EP4024228A1 (en) | System and method for batch and real-time feature calculation | |
| CN114265680A (en) | Mass data processing method and device, electronic equipment and storage medium | |
| WO2018052814A1 (en) | Data integration job conversion | |
| CN108073582B (en) | Computing framework selection method and device | |
| Balliu et al. | A big data analyzer for large trace logs | |
| CN114064142A (en) | Batch-flow integrated data processing system and processing method | |
| CN112948467B (en) | Data processing method and device, computer equipment and storage medium | |
| US10868793B2 (en) | Dynamic query hints in LDAP search operations | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| US11915062B2 (en) | Server instance introspection for shared resource collisions using call stack dependencies | |
| Dharmadasa et al. | Co-Tuning of Cloud Infrastructure and Distributed Data Processing Platforms | |
| CN116643809A (en) | Task management method, device, computer equipment and storage medium | |
| CN117632395A (en) | Workflow processing method, device, apparatus, storage medium and program product | |
| CN115269628A (en) | Cluster data sharing method and device based on stream computing | |
| CN115689196A (en) | Method, device, equipment, medium and program for processing automatic driving data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20220218 |