CN112328220A - Stream data processing system based on dragging arrangement mode and processing method thereof - Google Patents
Stream data processing system based on dragging arrangement mode and processing method thereof Download PDFInfo
- Publication number
- CN112328220A CN112328220A CN202011228187.2A CN202011228187A CN112328220A CN 112328220 A CN112328220 A CN 112328220A CN 202011228187 A CN202011228187 A CN 202011228187A CN 112328220 A CN112328220 A CN 112328220A
- Authority
- CN
- China
- Prior art keywords
- data
- component
- assembly
- field
- dragging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 77
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 22
- 230000008569 process Effects 0.000 claims abstract description 18
- 238000005111 flow chemistry technique Methods 0.000 claims abstract description 15
- 238000004458 analytical method Methods 0.000 claims abstract description 5
- 230000000712 assembly Effects 0.000 claims abstract 3
- 238000000429 assembly Methods 0.000 claims abstract 3
- 238000001914 filtration Methods 0.000 claims description 21
- 238000006243 chemical reaction Methods 0.000 claims description 20
- 238000013515 script Methods 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 claims description 2
- 238000011161 development Methods 0.000 abstract description 10
- 230000003993 interaction Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 5
- 238000013461 design Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000012141 concentrate Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention relates to a stream data processing system based on a dragging arrangement mode and a processing method thereof, which comprises a component dragging arrangement tool for enabling a user to process a business process, a component process analysis module for analyzing the stream processing process constructed by the user by using the component dragging arrangement tool, a task scheduling management module for managing the operation of a stream processing task, and a data source management module for inputting data source management. Dragging the data input assembly, the data processing assembly and the data output assembly to the layout panel to construct a stream processing flow, generating configuration metadata of a stream processing task flow, analyzing the configuration metadata of the stream processing flow constructed by a user, analyzing the configuration of each assembly, the relation among the assemblies and the input and output format of the assembly, and realizing the business logic of each assembly through the tableapi of the flink. Therefore, the assembly flow processing flow is dragged by using the components, and the interaction is friendly. Low code development, low use threshold, decoupling of bottom layer code and upper layer service.
Description
Technical Field
The present invention relates to a data processing system and a processing method thereof, and in particular, to a stream data processing system based on a drag arrangement mode and a processing method thereof.
Background
At present, streaming data processing is widely applied to large data platforms, and is specially used for solving service problems with high real-time requirements. Streaming platforms have become a sub-platform in large data platforms. There are three main categories of streaming data processing frameworks: apache Storm, Spark Streaming, and Apache Flink. Apache Flink is the framework in which the performance is best and the architecture design is best.
To date, the stream data processing systems provided on the market generally provide two ways to develop the business logic of stream data processing: uploading a code file or editing the SQL script online. In the aspect of stream data processing system construction, the main developed products are: 'Dayu' an intelligent data lake operation platform, a 'real-time computing Flink edition' of Aliyun and a 'data center of Yong science and technology'.
The bottom layer of the stream processing system is provided with powerful hardware resources for supporting, functions of managing and running scheduling of different stream processing services are provided, developers only need to concentrate on development and realization of stream processing service logic, and scheduling and running of stream processing service codes are completed by the stream processing system. These systems are all technical platforms that are difficult for users without substantial technical capabilities to use.
The stream processing system on the market at present often has the following disadvantages:
1. the flow processing framework uses a threshold height. Developers must be able to use the flow processing framework skillfully, and need to understand the principles of the flow processing framework, the code structure, the use of api, the framework deployment; it is also necessary to know how to perform performance tuning, etc.
2. The learning cost of the stream processing framework is high. Learning of the stream processing framework requires a considerable period of time.
3. Non-technical personnel are difficult to use the platform and cannot realize the business function by separating codes.
4. The development cost is high, and the development period is long.
5. Code reuse efficiency is low.
6. Code maintenance is difficult.
In view of the above-mentioned drawbacks, the present designer is actively making research and innovation to create a streaming data processing system based on a dragging and editing method and a processing method thereof, so that the streaming data processing system has industrial utility value.
Disclosure of Invention
In order to solve the above technical problems, an object of the present invention is to provide a streaming data processing system based on a dragging and editing manner and a processing method thereof.
The invention relates to a stream data processing system based on a drag arrangement mode, wherein: the system comprises a component dragging and arranging tool for a user to process a business process, a component process analyzing module for analyzing a stream processing process constructed by the user by using the component dragging and arranging tool, a task scheduling management module for managing the operation of a stream processing task, and a data source management module for inputting data source management, wherein the dragging and arranging tool is at least provided with a data input component, a data processing component and a data output component, and data transmission among the data input component, the data conversion component and the data output component is transmitted in a dynamic data table mode.
Further, the streaming data processing system based on the drag orchestration manner, wherein the data input component is used for a user to select a source of streaming data, and comprises a kafka input component and a rabbitMQ input component;
the data conversion assembly includes: the system comprises a data filtering component, an increasing field component, a field selecting component, a field format converting component, a character string replacing component, a data set connecting component and a data counting component;
the data output component stores data into a database, and comprises: the kafka output module, the mysql output module, the postgresql output module and the human fund bin output module.
Furthermore, in the streaming data processing system based on the dragging and arranging manner, the data filtering component is used for filtering data and filtering out data which do not meet the conditions, and a user writes a script which meets the service according to the data filtering script specification provided by the system;
the field adding component is used for adding a new field, and setting the name, the type and the value of the new field, wherein the value of the new field is one or more of a constant value, a random value and a sequence;
the field selection component is used for selecting a specified field list, filtering out the fields which are not needed, selecting the fields by using a select field selection tool of the flash table api, and filtering out the fields which are not needed by reverse selection;
the field format conversion component is used for converting the data format of the field and converting the field format by using a data type conversion tool of the flash table api;
the character string replacing component is used for replacing the value of the character string field; realizing character string replacement by using a character string replacement method carried by jdk or a character string replacement tool provided by flinktableapi;
the data set connecting component is used for connecting a plurality of data sets and connecting the data sets into one data set, and the data sets refer to data sets from different input components;
and the data statistics component is used for performing statistics operation on the data set, and comprises total quantity statistics and average value calculation.
The streaming data processing method based on the dragging arrangement mode comprises the following steps:
dragging a data input assembly, a data processing assembly and a data output assembly to an arrangement panel to construct a stream processing flow, and generating configuration metadata of the stream processing task flow;
and step two, analyzing the configuration metadata of the stream processing flow constructed by the user, analyzing the configuration of each component, the relation among the components and the input and output format of the component, and realizing the business logic of each component through the table api of the flash.
Further, in the streaming data processing method based on the dragging arrangement mode, in the first step, a user selects a required data input component, one or more data conversion components and a data output component, drags the data input component, the one or more data conversion components and the data output component to an arrangement panel, and connects the components by using a one-way arrow to form a directed acyclic graph;
and setting required component parameters for the data input component, the data processing component and the data output component, setting operating parameters for task operation, and finally constructing and finishing a complete flow processing flow.
Furthermore, in the streaming data processing method based on the dragging and editing manner, the metadata is saved in a JSON format.
Furthermore, in the streaming data processing method based on the dragging and arranging manner, the parsing process includes parsing a directed acyclic graph executed by a task from a streaming processing flow diagram constructed by a user through a topological sorting algorithm to obtain an operator queue executed by Flink, and encapsulating the task flow into a processing logic of flinktableapi according to the sequence of the operator queue.
Furthermore, in the streaming data processing method based on the dragging and editing mode, during the parsing period, a function of starting and stopping task execution is provided, the running state of the task is monitored in real time through a heartbeat mechanism, and the log viewing capability is provided externally through an api mode.
By the scheme, the invention at least has the following advantages:
1. and the assembly flow processing flow is dragged by using the components, so that the interaction is friendly.
2. Low code development, low usage threshold. A user does not need to pay attention to how the code is realized, and only needs to design a flow processing flow according to needs; the stream processing system will automatically translate the stream processing flow configuration information into code logic.
3. The underlying code is decoupled from the upper level services, and the underlying code has high reusability.
4. The flow processing flow can be multiplexed, the development efficiency is high, and the iteration speed is high.
5. The development cost is low.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings.
Drawings
Fig. 1 is a schematic flow chart of a simple implementation of a streaming data processing method based on a drag-and-drop arrangement mode.
Detailed Description
The following detailed description of embodiments of the present invention is provided in connection with the accompanying drawings and examples. The following examples are intended to illustrate the invention but are not intended to limit the scope of the invention.
The streaming data processing system based on the drag-and-drop arrangement mode as shown in fig. 1 is distinguished in that: the system comprises a component dragging and arranging tool for a user to process a business process and a component process analyzing module for analyzing a stream processing process constructed by the user by using the component dragging and arranging tool. During implementation, the component flow parsing module parses the configuration of each component, the relationship between the components, and the execution sequence of the components, and uses the tableapi of the flink to implement the business logic of the stream processing flow. The system also comprises a task scheduling management module for managing the running of the stream processing tasks and a data source management module for inputting data source management. And the data source management module can be used for managing the running of the stream processing tasks, running the stream processing tasks needing to be run on the yann cluster in a JAR packet mode, and monitoring the stream processing tasks by using the yann cluster. The dragging and arranging tool is at least provided with a data input component, a data processing component and a data output component, and data transmission among the data input component, the data conversion component and the data output component is transmitted in a dynamic data table mode. Taking the field adding component as an example, the input data is a dynamic table with three fields a, b and c, and the output data is a dynamic table with four fields a, b, c and d. The output data of each component is used as input data for the next component. Meanwhile, the information of the input data source is well defined in advance, and the system is specially provided with a data source management module for managing the connection information of different data sources. When editing the input component, the user only needs to select the data source information already existing in the data source management module. The desired message topic is then selected and the data structure of the message is given.
In view of a preferred embodiment of the present invention, the present specification employs data input components for user selection of a source of streaming data, including a kafka input component, and a rabbitMQ input component. Currently, the main sources of streaming data are message middleware such as kafka and rabbitMQ, and users need to select a specific data source, the message subject of the data source.
Examples of messages entered by kafka and rabbitMQ are as follows:
meanwhile, the data conversion assembly includes: the system comprises a data filtering component, an adding field component, a field selecting component, a field format converting component, a character string replacing component, a data set connecting component and a data statistic component. And, the data is saved to the database through the data output component. Specifically, the data output assembly employed includes: the kafka output module, the mysql output module, the postgresql output module and the human fund bin output module. Furthermore, in order to support more output data sources, the output data sources supported by the flink need to be expanded, and a new data source output tool is realized through java.
Specifically, the data filtering component is used for filtering data and filtering data which do not meet conditions, and a user writes scripts which meet services according to data filtering script specifications provided by the system. After the system obtains the script written by the user, the script is analyzed and converted into code logic.
And the field adding component is used for adding a new field, setting the name, the type and the value of the new field, wherein the value of the new field is one or more of a constant value, a random value and a sequence, and can also be other newly added values.
And the field selection component is used for selecting a specified field list, filtering out the fields which are not needed, selecting the fields by using a select field selection tool of the flash table api, and filtering out the fields which are not needed by reverse selection.
And the field format conversion component is used for converting the data format of the field and converting the field format by using a data type conversion tool of the flash table api.
A string replacement component for replacing a value of a string field. The character string replacement is realized by using a character string replacement method carried by jdk or a character string replacement tool provided by flinktableapi.
And the data set connecting component is used for connecting a plurality of data sets, and connecting the data sets into one data set, wherein the data sets refer to the data sets from different input components. During actual implementation, a plurality of data sets are defined as a plurality of dynamic tables, and the plurality of dynamic tables are connected together by using a table connecting tool.
And the data statistics component is used for performing statistical operations on the data set, including total statistics, average calculation and the like. These statistical functions can be implemented using the capabilities provided by flinktableapi.
In order to better implement the present invention, a streaming data processing method based on a drag-and-drop arrangement mode is provided, which includes the following steps:
dragging a data input assembly, a data processing assembly and a data output assembly to an arrangement panel to construct a stream processing flow, and generating configuration metadata of the stream processing task flow;
and step two, analyzing the configuration metadata of the stream processing flow constructed by the user, analyzing the configuration of each component, the relation among the components and the input and output format of the component, and realizing the business logic of each component through the table api of the flash.
In combination with the actual implementation, in step one, the user selects a desired data input component, one or more data conversion components and a data output component, drags the selected data input component, the one or more data conversion components and the data output component to the layout panel, and connects the selected data input component, the one or more data conversion components and the data output component by using a one-way arrow to form a directed acyclic graph. And setting required component parameters for the data input component, the data processing component and the data output component. Meanwhile, the running parameters of the task running, such as the size of a memory and the running parallelism of the task, can be set during the implementation, and finally a complete flow processing flow is constructed and completed. And the adopted metadata is saved through a JSON format. Therefore, JSON as a lightweight data exchange format has good readability, cross-platform support and high compatibility.
In consideration of the convenience of data processing, the component parameters related to each component of the present invention are roughly as follows: for the kafka input component, the connection information of kafka, the subject of the message in kafka, the grouping name of the message, the format of the message, the message reading mode and the like need to be configured. For the add field component, the name of the add field, the type of the field, and the way of taking the value of the field need to be configured. For the mysql output component, the mapping relation between the connection information of the mysql database and the database field needs to be configured. Others, not specifically mentioned, may be used in accordance with default configurations of the component as is conventional in the art.
The configuration parameters of the kafka input component are as follows:
in order to better practice the present invention, the analytical procedure employed is roughly as follows: analyzing a flow processing flow chart constructed by a user into a directed acyclic graph executed by a task through a topological sorting algorithm to obtain an operator queue executed by the Flink, and packaging the task flow into a processing logic of the flinktableapi according to the sequence of the operator queue. Meanwhile, the function of starting and stopping the execution of the task is provided during the analysis, the running state of the task is monitored in real time through a heartbeat mechanism, and the log viewing capability is provided externally through an api mode.
Meanwhile, in order to enable a single stream processing task to have higher performance and reliability, the task running mode is designed to be that each stream processing task runs independently, and resource parameters are freely configured according to the service characteristics of the tasks.
And the invention encapsulates the Flink stream processing logic into a visual and draggable component, the user can customize the stream processing flow according to the own business requirement, the user drags the needed component to the panel, then connects the needed component according to the data processing flow, clicks to store and operate after the setting is finished, and then the operation state of the task can be checked at any time, the whole flow is simple and convenient to operate and is efficient.
The invention has the following advantages by the aid of the character expression and the accompanying drawings:
1. and the assembly flow processing flow is dragged by using the components, so that the interaction is friendly.
2. Low code development, low usage threshold. A user does not need to pay attention to how the code is realized, and only needs to design a flow processing flow according to needs; the stream processing system will automatically translate the stream processing flow configuration information into code logic.
3. The underlying code is decoupled from the upper level services, and the underlying code has high reusability.
4. The flow processing flow can be multiplexed, the development efficiency is high, and the iteration speed is high.
5. The development cost is low.
Furthermore, the indication of the orientation or the positional relationship described in the present invention is based on the orientation or the positional relationship shown in the drawings, and is only for convenience of describing the present invention and simplifying the description, but does not indicate or imply that the indicated device or configuration must have a specific orientation or be operated in a specific orientation configuration, and thus, should not be construed as limiting the present invention.
The terms "primary" and "secondary" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "primary" or "secondary" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically limited otherwise.
Also, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "connected" and "disposed" are to be construed broadly, e.g., as meaning fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; the two components can be directly connected or indirectly connected through an intermediate medium, and the two components can be communicated with each other or mutually interacted. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations. And it may be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element.
It will be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like, refer to an orientation or positional relationship illustrated in the drawings, which are used for convenience in describing the invention and to simplify the description, and do not indicate or imply that the device or component being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, it should be noted that, for those skilled in the art, many modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (8)
1. The stream data processing system based on the dragging arrangement mode is characterized in that: the system comprises a component dragging and arranging tool for a user to process a business process, a component process analyzing module for analyzing a stream processing process constructed by the user by using the component dragging and arranging tool, a task scheduling management module for managing the operation of a stream processing task, and a data source management module for inputting data source management, wherein the dragging and arranging tool is at least provided with a data input component, a data processing component and a data output component, and data transmission among the data input component, the data conversion component and the data output component is transmitted in a dynamic data table mode.
2. The streaming data processing system based on drag orchestration according to claim 1, wherein: the data input component is used for selecting a source of streaming data by a user and comprises a kafka input component and a rabbitMQ input component;
the data conversion assembly includes: the system comprises a data filtering component, an increasing field component, a field selecting component, a field format converting component, a character string replacing component, a data set connecting component and a data counting component;
the data output component stores data into a database, and comprises: the kafka output module, the mysql output module, the postgresql output module and the human fund bin output module.
3. The streaming data processing system based on drag orchestration according to claim 2, wherein: the data filtering component is used for filtering data and filtering data which do not meet conditions, and a user compiles a script which meets the service according to the data filtering script specification provided by the system;
the field adding component is used for adding a new field, and setting the name, the type and the value of the new field, wherein the value of the new field is one or more of a constant value, a random value and a sequence;
the field selection component is used for selecting a specified field list, filtering out the fields which are not needed, selecting the fields by using a select field selection tool of the flash table api, and filtering out the fields which are not needed by reverse selection;
the field format conversion component is used for converting the data format of the field and converting the field format by using a data type conversion tool of the flash table api;
the character string replacing component is used for replacing the value of the character string field; realizing character string replacement by using a character string replacement method carried by jdk or a character string replacement tool provided by flinktableapi;
the data set connecting component is used for connecting a plurality of data sets and connecting the data sets into one data set, and the data sets refer to data sets from different input components;
and the data statistics component is used for performing statistics operation on the data set, and comprises total quantity statistics and average value calculation.
4. The streaming data processing method based on the dragging arrangement mode is characterized by comprising the following steps:
dragging a data input assembly, a data processing assembly and a data output assembly to an arrangement panel to construct a stream processing flow, and generating configuration metadata of the stream processing task flow;
and step two, analyzing the configuration metadata of the stream processing flow constructed by the user, analyzing the configuration of each component, the relation among the components and the input and output format of the component, and realizing the business logic of each component through the table api of the flash.
5. The streaming data processing method based on the drag editing mode as claimed in claim 4, wherein: in the first step, a user selects a required data input assembly, one or more data conversion assemblies and a data output assembly, drags the data input assembly, the one or more data conversion assemblies and the data output assembly to a layout panel, and connects the data input assembly, the data conversion assembly and the data output assembly by using a one-way arrow to form a directed acyclic graph;
and setting required component parameters for the data input component, the data processing component and the data output component, setting operating parameters for task operation, and finally constructing and finishing a complete flow processing flow.
6. The streaming data processing method based on the drag editing mode as claimed in claim 4, wherein: and the metadata is stored through a JSON format.
7. The streaming data processing method based on the drag editing mode as claimed in claim 4, wherein: the analysis process is to analyze a flow processing flow diagram constructed by a user through a topological sorting algorithm to obtain a directed acyclic graph executed by the task, obtain an operator queue executed by the Flink, and package the task flow into a processing logic of the flinktableapi according to the sequence of the operator queue.
8. The streaming data processing system based on drag orchestration according to claim 7, wherein: the analysis period provides a function of starting and stopping the execution of the task, the running state of the task is monitored in real time through a heartbeat mechanism, and the log viewing capability is provided externally through an api mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011228187.2A CN112328220A (en) | 2020-11-06 | 2020-11-06 | Stream data processing system based on dragging arrangement mode and processing method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011228187.2A CN112328220A (en) | 2020-11-06 | 2020-11-06 | Stream data processing system based on dragging arrangement mode and processing method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112328220A true CN112328220A (en) | 2021-02-05 |
Family
ID=74317151
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011228187.2A Pending CN112328220A (en) | 2020-11-06 | 2020-11-06 | Stream data processing system based on dragging arrangement mode and processing method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112328220A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112882803A (en) * | 2021-03-08 | 2021-06-01 | 万维云网(北京)数据科技有限公司 | Data processing method and system |
| CN113065779A (en) * | 2021-04-07 | 2021-07-02 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113157517A (en) * | 2021-02-19 | 2021-07-23 | 中国工商银行股份有限公司 | Batch-flow integrated index data anomaly detection method and device |
| CN113821538A (en) * | 2021-08-27 | 2021-12-21 | 中通服公众信息产业股份有限公司 | Streaming data processing system based on metadata |
| CN114625356A (en) * | 2022-03-29 | 2022-06-14 | 南京四维智联科技有限公司 | Self-service data processing system and method and computer equipment |
| CN114816368A (en) * | 2022-06-23 | 2022-07-29 | 深圳市瓴码云计算有限公司 | Object-oriented business process development system, method, device and storage medium |
| CN114925241A (en) * | 2022-04-24 | 2022-08-19 | 杭州悦数科技有限公司 | Method, system, electronic device and storage medium for processing graph data |
| CN114936245A (en) * | 2022-04-28 | 2022-08-23 | 北京远舢智能科技有限公司 | Method and device for integrating and processing multi-source heterogeneous data |
| CN115098567A (en) * | 2022-06-20 | 2022-09-23 | 上海纽酷信息科技有限公司 | Low code platform data transmission method based on BI platform |
| CN115794064A (en) * | 2022-10-25 | 2023-03-14 | 中电金信软件有限公司 | Configuration method and device of task processing flow, electronic equipment and storage medium |
| CN116860227A (en) * | 2023-07-12 | 2023-10-10 | 北京东方金信科技股份有限公司 | Data development system and method based on big data ETL script arrangement |
| CN117289924A (en) * | 2023-10-13 | 2023-12-26 | 河北云在信息技术服务有限公司 | Visual task scheduling system and method based on Flink |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100121740A1 (en) * | 2008-11-13 | 2010-05-13 | Oracle International Corporation | Data driven orchestration of business processes |
| CN107480893A (en) * | 2017-08-18 | 2017-12-15 | 浪潮软件股份有限公司 | A kind of flow method of combination and device |
| CN110764753A (en) * | 2019-09-18 | 2020-02-07 | 亚信创新技术(南京)有限公司 | Business logic code generation method, device, equipment and storage medium |
| CN110909039A (en) * | 2019-10-25 | 2020-03-24 | 北京华如科技股份有限公司 | Big data mining tool and method based on drag type process |
| CN111240662A (en) * | 2020-01-16 | 2020-06-05 | 同方知网(北京)技术有限公司 | Spark machine learning system and learning method based on task visual dragging |
| CN111857659A (en) * | 2020-06-30 | 2020-10-30 | 太极计算机股份有限公司 | Data visualization design platform for dragging heterogeneous data source |
-
2020
- 2020-11-06 CN CN202011228187.2A patent/CN112328220A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100121740A1 (en) * | 2008-11-13 | 2010-05-13 | Oracle International Corporation | Data driven orchestration of business processes |
| CN107480893A (en) * | 2017-08-18 | 2017-12-15 | 浪潮软件股份有限公司 | A kind of flow method of combination and device |
| CN110764753A (en) * | 2019-09-18 | 2020-02-07 | 亚信创新技术(南京)有限公司 | Business logic code generation method, device, equipment and storage medium |
| CN110909039A (en) * | 2019-10-25 | 2020-03-24 | 北京华如科技股份有限公司 | Big data mining tool and method based on drag type process |
| CN111240662A (en) * | 2020-01-16 | 2020-06-05 | 同方知网(北京)技术有限公司 | Spark machine learning system and learning method based on task visual dragging |
| CN111857659A (en) * | 2020-06-30 | 2020-10-30 | 太极计算机股份有限公司 | Data visualization design platform for dragging heterogeneous data source |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157517A (en) * | 2021-02-19 | 2021-07-23 | 中国工商银行股份有限公司 | Batch-flow integrated index data anomaly detection method and device |
| CN112882803B (en) * | 2021-03-08 | 2024-05-14 | 万维云网(北京)数据科技有限公司 | Data processing method and system |
| CN112882803A (en) * | 2021-03-08 | 2021-06-01 | 万维云网(北京)数据科技有限公司 | Data processing method and system |
| CN113065779B (en) * | 2021-04-07 | 2023-08-11 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113065779A (en) * | 2021-04-07 | 2021-07-02 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113821538A (en) * | 2021-08-27 | 2021-12-21 | 中通服公众信息产业股份有限公司 | Streaming data processing system based on metadata |
| CN113821538B (en) * | 2021-08-27 | 2024-01-30 | 中通服公众信息产业股份有限公司 | Stream data processing system based on metadata |
| CN114625356A (en) * | 2022-03-29 | 2022-06-14 | 南京四维智联科技有限公司 | Self-service data processing system and method and computer equipment |
| CN114925241A (en) * | 2022-04-24 | 2022-08-19 | 杭州悦数科技有限公司 | Method, system, electronic device and storage medium for processing graph data |
| CN114936245A (en) * | 2022-04-28 | 2022-08-23 | 北京远舢智能科技有限公司 | Method and device for integrating and processing multi-source heterogeneous data |
| CN115098567A (en) * | 2022-06-20 | 2022-09-23 | 上海纽酷信息科技有限公司 | Low code platform data transmission method based on BI platform |
| CN115098567B (en) * | 2022-06-20 | 2024-04-12 | 上海纽酷信息科技有限公司 | Low-code platform data transmission method based on BI platform |
| CN114816368A (en) * | 2022-06-23 | 2022-07-29 | 深圳市瓴码云计算有限公司 | Object-oriented business process development system, method, device and storage medium |
| CN115794064A (en) * | 2022-10-25 | 2023-03-14 | 中电金信软件有限公司 | Configuration method and device of task processing flow, electronic equipment and storage medium |
| CN115794064B (en) * | 2022-10-25 | 2024-02-06 | 中电金信软件有限公司 | Configuration method and device of task processing flow, electronic equipment and storage medium |
| CN116860227A (en) * | 2023-07-12 | 2023-10-10 | 北京东方金信科技股份有限公司 | Data development system and method based on big data ETL script arrangement |
| CN116860227B (en) * | 2023-07-12 | 2024-02-09 | 北京东方金信科技股份有限公司 | Data development system and method based on big data ETL script arrangement |
| CN117289924A (en) * | 2023-10-13 | 2023-12-26 | 河北云在信息技术服务有限公司 | Visual task scheduling system and method based on Flink |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112328220A (en) | Stream data processing system based on dragging arrangement mode and processing method thereof | |
| US11288142B2 (en) | Recovery strategy for a stream processing system | |
| US11086687B2 (en) | Managing resource allocation in a stream processing framework | |
| US20210119892A1 (en) | Online computer system with methodologies for distributed trace aggregation and for targeted distributed tracing | |
| US9842000B2 (en) | Managing processing of long tail task sequences in a stream processing framework | |
| US20180253335A1 (en) | Maintaining throughput of a stream processing framework while increasing processing load | |
| CN110245078A (en) | A kind of method for testing pressure of software, device, storage medium and server | |
| US20080312986A1 (en) | Implementing Key Performance Indicators in a Service Model | |
| Baresi et al. | Event-based multi-level service monitoring | |
| US20140067836A1 (en) | Visualizing reporting data using system models | |
| CN103268327B (en) | Mixing method for visualizing towards high-dimensional service data | |
| CN110519100A (en) | A kind of more cluster management methods, terminal and computer readable storage medium | |
| CN105095329B (en) | A kind of demographic data check method | |
| CN108845798A (en) | A kind of visualization big data task cradle and processing method | |
| KR20080081937A (en) | Multi-dimensional aggregation on event streams | |
| CN114490268A (en) | Full link monitoring method, device, equipment, storage medium and program product | |
| CN114372084A (en) | Real-time processing system for sensing stream data | |
| CN111045911A (en) | Performance test method, performance test device, storage medium and electronic equipment | |
| US9514027B2 (en) | Context-aware model-driven hierarchical monitoring metadata | |
| CN114816375A (en) | Service arranging method, device, equipment and storage medium | |
| Bielefeld | Online performance anomaly detection for large-scale software systems | |
| CN113312242B (en) | Interface information management method, device, equipment and storage medium | |
| CN114756301A (en) | Log processing method, device and system | |
| KR20090073061A (en) | A system and method for managing the business process model which mapped the logical process and the physical process model | |
| Zheng et al. | Research and Application of Configuration System in Core Network Based on Digital Twin |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |