CN110020021B - Visualization-based data stream processing method - Google Patents
Visualization-based data stream processing method Download PDFInfo
- Publication number
- CN110020021B CN110020021B CN201811017986.8A CN201811017986A CN110020021B CN 110020021 B CN110020021 B CN 110020021B CN 201811017986 A CN201811017986 A CN 201811017986A CN 110020021 B CN110020021 B CN 110020021B
- Authority
- CN
- China
- Prior art keywords
- stream processing
- stream
- data
- processing task
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/904—Browsing; Visualisation therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a visualization-based data stream processing method, which comprises the following steps: 1. configuring stream data source information and structure information of the stream data source; 2. selecting a stream processing component through a visual interface to construct a stream processing task and fill in input information, and then configuring execution parameters and submitting the execution parameters to a stream processing task construction engine; 3. analyzing the input information, and converting the stream processing task into SQL; 4. storing the execution parameters, the stream processing tasks and the storage information parameters into an execution schedule; 5. an execution engine scans an execution plan table, acquires execution parameters and adds an execution plan; 6. acquiring stream data, formatting the stream data, creating a virtual table by an execution engine, adding the stream data into the virtual table, inquiring the virtual table, and executing a stream processing task at regular time according to an execution plan to obtain a stream processing result; 7. and saving the stream processing result to a storage medium. The invention creates the stream processing task in a visual mode, does not need programming and is convenient for observing the stream processing result and the log.
Description
Technical Field
The invention relates to the technical field of communication, in particular to a visualization-based data stream processing method.
Background
To address the increasing demand for real-time data processing, many companies are transforming portions of their big data infrastructure into a streaming processing architecture. The traditional stream data processing mainly creates tasks in a programming mode, and a new task usually needs to go through a tedious process of encoding, compiling and packaging. The mode has low working efficiency and low reusability, and is difficult for non-programming personnel.
Compared with the prior art, the stream processing task is established in a visualization mode, and the method and the device have the advantages that the stream processing task can be completed without programming, the use threshold is reduced, and the use and the management of a user are facilitated, so that the technical scheme is generated.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a data stream processing method based on visualization, for different stream data sources, first register the stream data source to the stream processing platform, and then add a data structure definition to the stream data in the stream data source. The method adopts the SQL mode, the user configures and submits the tasks in the visual interface, the system automatically constructs the stream processing tasks through the configuration submitted by the user, and the user can check the stream processing results and logs in real time in the task management and visual interface.
The problem of the invention is realized as follows:
a visualization-based data stream processing method comprises the following steps:
step 1, configuring stream data source information and structure information of a stream data source by a user, and submitting the configured stream data source information and the configured structure information of the stream data source to a database through a stream data source management page;
step 2, when a user creates a stream processing task on a visual interface, selecting a corresponding stream processing component through the visual interface to construct the stream processing task, well filling input information of each stream processing component, configuring corresponding execution parameters of the stream processing task, and submitting the stream processing task, the input information of each stream processing component and the execution parameters to a stream processing task construction engine;
step 3, the stream processing task construction engine analyzes the input information of each stream processing component, so that the stream processing task is converted into an SQL statement;
step 4, storing the execution parameters, the stream processing tasks converted into SQL statements and the storage information parameters in the input information into an execution plan table of a database to wait for the scheduling of an execution engine;
step 5, the execution engine starts the monitoring thread to scan the execution schedule at regular time, acquires the execution parameters of the stream processing task, and adds an execution plan for the stream processing task to be executed according to the execution parameters;
step 6, the flow processing task acquires flow data according to the flow data source information, formats the flow data according to the structure information of the flow data source, the execution engine creates a virtual table for the formatted flow data in the memory according to the defined structure of the flow data source, adds the formatted flow data into the virtual table, queries the virtual table through SQL statements, and executes the flow processing task on the flow data in the virtual table according to the execution plan timing to obtain a flow processing result;
and 7, storing the stream processing result into a corresponding storage medium according to the storage information parameters.
Further, the stream processing component comprises an SQL editor, a source selector, a data filter, a data manipulator, a data calculator, and a storage medium selector;
the SQL editor is used for editing and converting the stream processing task into an SQL statement;
the source selector is used for selecting a certain stream data source according to the stream data source information to acquire the stream data when the stream data is acquired;
the data filter is used for filtering the streaming data in the process of executing the streaming processing task;
the data manipulator is used for grouping, projecting and associating stream data in the process of executing a stream processing task;
the data calculator is used for calculating and processing the stream data according to an algorithm in the process of executing the stream processing task;
and the storage medium selector is used for selecting a certain storage medium to store according to the storage information parameters when the execution result is stored in the corresponding storage medium.
Further, the step 6 is followed by:
step 61, creating a task management table and an operation log table for the stream processing task in the execution engine, and inserting the progress information and the control information of the current stream processing task into the task management table in real time to check the progress of the current stream processing task and control the start, pause or close of the current stream processing task; and recording the progress information of the current stream processing task to an operation log table for log monitoring. .
Further, the step 7 is followed by:
and step 71, sending the stream processing result in the storage medium to a visual interface, and displaying through the visual interface, wherein the displaying mode comprises a table and a chart.
The invention has the advantages that:
1. the stream processing task is established in a visualization mode and can be completed without programming, so that the use threshold is reduced, and convenience is brought to users.
2. The matched visual operation function is convenient for starting and stopping the control flow processing task, and the flow processing result, the log and the like are convenient to observe.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
Fig. 1 is an execution flow chart of a visualization-based data stream processing method according to the present invention.
Detailed Description
In order that the invention may be more readily understood, a preferred embodiment thereof will now be described in detail with reference to the accompanying drawings.
As shown in fig. 1, a visualization-based data stream processing method of the present invention includes the following steps:
step 1, configuring stream data source information and structure information of the stream data source by a user (the data source is a device or original media providing certain required data, as the name implies, the source of the data), wherein the stream data source is a source for acquiring stream data and is used for specifying which data is accessed from, including accessing from a local real-time file, accessing from a service system server through an interface and the like, namely acquiring source data from the local real-time file or the service system server, and the structure of the stream data source is that the stream data fills in structure information such as separators, field names, codes and the like of the stream data according to a real data stream format; the configured stream data source information and the structure information of the stream data source are submitted to a database through a stream data source management page;
step 2, when the user creates the stream processing task on the visual interface, the front end of the stream processing task construction engine provides a corresponding stream processing component: the system comprises an SQL editor, a source selector, a data filter, a data manipulator (grouping, projecting, associating and the like), a data calculator (SUM, CPUNT and the like) and a storage medium selector, wherein a user selects a corresponding stream processing component to construct a stream processing task at the front end of a stream processing task construction engine through a visual interface, fills in input information of each stream processing component, configures corresponding execution parameters of the stream processing task, and submits the stream processing task, the input information of each stream processing component and the execution parameters to the stream processing task construction engine;
step 3, the rear end of the stream processing task construction engine analyzes the input information of each stream processing component, so that the stream processing task is converted into an SQL statement, and the stream processing task is stored in an SQL mode;
step 4, storing the execution parameters, the stream processing tasks converted into SQL statements and the storage information parameters in the input information into an execution plan table of a database to wait for the scheduling of an execution engine;
step 5, the execution engine starts a monitoring thread to scan the execution schedule at regular time, obtains the execution parameters of the stream processing task, and adds an execution plan for the stream processing task to be executed according to the execution parameters;
step 6, the flow processing task acquires flow data according to the flow data source information, formats the flow data according to the structure information of the flow data source, the execution engine creates a virtual table for the formatted flow data in the memory according to the defined structure of the flow data source, adds the formatted flow data into the virtual table, queries the virtual table through SQL statements, and executes the flow processing task on the flow data in the virtual table according to the execution plan timing to obtain a flow processing result; creating a task management table and an operation log table for the stream processing task in an execution engine, and inserting the progress information and the control information of the current stream processing task into the task management table in real time to check the progress of the current stream processing task and control the start, pause or close of the current stream processing task; recording the progress information of the current stream processing task to an operation log table for log monitoring;
and 7, storing the stream processing result into a corresponding storage medium according to the storage information parameters, sending the stream processing result in the storage medium to a visual interface, displaying the stream processing result through the visual interface in a mode of including a table and a chart, displaying the table information through a selection condition, and displaying the stream data in the data stream container in real time through the visual interface if the selected storage medium is the data stream container.
Specifically, the stream processing component comprises an SQL editor, a source selector, a data filter, a data manipulator, a data calculator and a storage medium selector;
the SQL editor is used for editing and converting the stream processing task into an SQL statement;
the source selector is used for selecting a certain stream data source according to the stream data source information to acquire the stream data when the stream data is acquired;
the data filter is used for filtering the streaming data in the process of executing the streaming processing task;
the data manipulator is used for grouping, projecting and associating stream data in the process of executing a stream processing task;
the data calculator is used for calculating and processing the stream data according to an algorithm in the process of executing the stream processing task;
and the storage medium selector is used for selecting a certain storage medium to store according to the storage information parameters when the execution result is stored in the corresponding storage medium.
The invention has the following advantages:
1. the stream processing task is established in a visualization mode and can be completed without programming, so that the use threshold is reduced, and convenience is brought to users.
2. The visual operation function is matched, the start and stop of a control flow processing task are facilitated, and the flow processing result, the log and the like are conveniently observed.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (3)
1. A visualization-based data stream processing method is characterized in that: the method comprises the following steps:
step 1, configuring stream data source information and structure information of a stream data source by a user, and submitting the configured stream data source information and the configured structure information of the stream data source to a database through a stream data source management page;
step 2, when a user creates a stream processing task on a visual interface, selecting a corresponding stream processing component through the visual interface to construct the stream processing task, filling input information of each stream processing component, configuring corresponding execution parameters of the stream processing task, and submitting the stream processing task, the input information of each stream processing component and the execution parameters to a stream processing task construction engine;
the stream processing component comprises an SQL editor, a source selector, a data filter, a data manipulator, a data calculator and a storage medium selector;
step 3, the stream processing task construction engine analyzes the input information of each stream processing component, so that the stream processing task is converted into an SQL statement;
step 4, storing the execution parameters, the stream processing tasks converted into SQL statements and the storage information parameters in the input information into an execution plan table of a database to wait for the scheduling of an execution engine;
step 5, the execution engine starts a monitoring thread to scan the execution schedule at regular time, obtains the execution parameters of the stream processing task, and adds an execution plan for the stream processing task to be executed according to the execution parameters;
step 6, the flow processing task acquires flow data according to the flow data source information, formats the flow data according to the structure information of the flow data source, the execution engine creates a virtual table for the formatted flow data in the memory according to the defined structure of the flow data source, adds the formatted flow data into the virtual table, queries the virtual table through SQL statements, and executes the flow processing task on the flow data in the virtual table according to the execution plan timing to obtain a flow processing result;
creating a task management table and an operation log table for the stream processing task in the execution engine, and inserting the progress information and the control information of the current stream processing task into the task management table in real time to check the progress of the current stream processing task and control the starting, pausing or closing of the current stream processing task; recording the progress information of the current stream processing task to an operation log table for log monitoring;
and 7, storing the stream processing result into a corresponding storage medium according to the storage information parameters.
2. The visualization-based data stream processing method of claim 1, wherein: the SQL editor is used for editing and converting the stream processing task into an SQL statement;
the source selector is used for selecting a certain stream data source to acquire the stream data according to the stream data source information when the stream data is acquired;
the data filter is used for filtering the streaming data in the process of executing the streaming processing task;
the data manipulator is used for grouping, projecting and associating stream data in the process of executing a stream processing task;
the data calculator is used for calculating and processing the stream data according to an algorithm in the process of executing the stream processing task;
and the storage medium selector is used for selecting a certain storage medium to store according to the storage information parameters when the execution result is stored in the corresponding storage medium.
3. A visualization-based data stream processing method as recited in claim 1, wherein: the step 7 further comprises the following steps:
and step 71, sending the stream processing result in the storage medium to a visual interface, and displaying through the visual interface, wherein the displaying mode comprises a table and a chart.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811017986.8A CN110020021B (en) | 2018-09-03 | 2018-09-03 | Visualization-based data stream processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811017986.8A CN110020021B (en) | 2018-09-03 | 2018-09-03 | Visualization-based data stream processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110020021A CN110020021A (en) | 2019-07-16 |
| CN110020021B true CN110020021B (en) | 2023-04-07 |
Family
ID=67188448
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811017986.8A Active CN110020021B (en) | 2018-09-03 | 2018-09-03 | Visualization-based data stream processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110020021B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457023A (en) * | 2019-07-23 | 2019-11-15 | 东软集团股份有限公司 | Task creation method, apparatus, storage medium and electronic equipment |
| CN111145038B (en) * | 2019-12-02 | 2023-08-01 | 积成电子股份有限公司 | Power grid regulation and control big data interactive analysis method based on visual data flow diagram |
| CN110908789A (en) * | 2019-12-04 | 2020-03-24 | 广东弓叶科技有限公司 | Visual data configuration method and system for multi-source data processing |
| CN111475564A (en) * | 2020-04-13 | 2020-07-31 | 恒安嘉新（北京）科技股份公司 | Streaming data processing method, system, computer equipment and storage medium |
| CN113742040B (en) * | 2021-08-09 | 2024-04-19 | 广州市易工品科技有限公司 | Method and device for quickly generating distributed batch processing task based on visual interface |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016029738A1 (en) * | 2014-08-26 | 2016-03-03 | 华为技术有限公司 | Method and device for processing stream data |
| CN106550045A (en) * | 2016-11-24 | 2017-03-29 | 北京中电普华信息技术有限公司 | A kind of distributed data processing method and component |
| CN107145532A (en) * | 2017-04-18 | 2017-09-08 | 北京思特奇信息技术股份有限公司 | The real-time analysis and processing method and system of a kind of flow data |
| CN107533453A (en) * | 2015-03-06 | 2018-01-02 | 思科技术公司 | System and method for generating data visualization application |
-
2018
- 2018-09-03 CN CN201811017986.8A patent/CN110020021B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016029738A1 (en) * | 2014-08-26 | 2016-03-03 | 华为技术有限公司 | Method and device for processing stream data |
| CN107533453A (en) * | 2015-03-06 | 2018-01-02 | 思科技术公司 | System and method for generating data visualization application |
| CN106550045A (en) * | 2016-11-24 | 2017-03-29 | 北京中电普华信息技术有限公司 | A kind of distributed data processing method and component |
| CN107145532A (en) * | 2017-04-18 | 2017-09-08 | 北京思特奇信息技术股份有限公司 | The real-time analysis and processing method and system of a kind of flow data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110020021A (en) | 2019-07-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110020021B (en) | Visualization-based data stream processing method | |
| CN110764753B (en) | Business logic code generation method, device, equipment and storage medium | |
| CN106682097B (en) | Method and device for processing log data | |
| CN106777101B (en) | Data processing engine | |
| CN109582647B (en) | Unstructured evidence file oriented analysis method and system | |
| CN108108297A (en) | The method and apparatus of automatic test | |
| CN110389898A (en) | Acquisition methods, device, terminal and the computer readable storage medium of Test Strategy | |
| CN108268529B (en) | Data summarization method and system based on business abstraction and multi-engine scheduling | |
| CN111881192A (en) | Method and system for generating visual configuration report, electronic equipment and storage medium | |
| US8417690B2 (en) | Automatically avoiding unconstrained cartesian product joins | |
| CN111125178B (en) | Data query method, device, terminal, presto query engine and storage medium | |
| CN111367524B (en) | Enumeration type design method and device | |
| WO2021068692A1 (en) | Method, apparatus and device for workflow migration, and computer-readable storage medium | |
| CN112306861A (en) | Unittest and Jenkins tool-based interface automatic testing system and method | |
| CN113258676B (en) | Power monitoring system interactively integrated with BIM (building information modeling) model and interactive integration method | |
| CN108959458B (en) | Data generation and use method, system, medium and computer device | |
| CN112163017B (en) | Knowledge mining system and method | |
| CN109614079A (en) | A kind of coding rule inspection system towards avionics field | |
| CN110909017B (en) | Data analysis method and system | |
| CN114610597A (en) | Pressure testing method, device, equipment and storage medium | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN106293862B (en) | A kind of analysis method and device of expandable mark language XML data | |
| CN111797111A (en) | Dynamic combined query method based on ES search engine | |
| CN103425490A (en) | Management method based on running object data in CRM system | |
| CN103793519A (en) | Automatic tool supporting exportation of mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 350000 21 / F, building 5, f District, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province Applicant after: FUJIAN SINOREGAL SOFTWARE CO.,LTD. Address before: Floor 20-21, building 5, area F, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province 350000 Applicant before: FUJIAN SINOREGAL SOFTWARE CO.,LTD. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |