CN109710215B - Visualization process processing engine for distributed flow calculation and use method thereof - Google Patents
Visualization process processing engine for distributed flow calculation and use method thereof Download PDFInfo
- Publication number
- CN109710215B CN109710215B CN201811589073.3A CN201811589073A CN109710215B CN 109710215 B CN109710215 B CN 109710215B CN 201811589073 A CN201811589073 A CN 201811589073A CN 109710215 B CN109710215 B CN 109710215B
- Authority
- CN
- China
- Prior art keywords
- processing
- engine
- stream
- flow
- visualization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a visualization process processing engine for distributed stream computing and a using method thereof, wherein the visualization process processing engine comprises the following steps: the system comprises a stream processing visualization arrangement engine, a stream processing process definition file generation engine and a stream processing release engine; the visualization arrangement engine comprises a foreground visualization interface and a background program code which are coupled; the foreground visual interface comprises a desktop and a plurality of labels; each label and a homonymy stream analysis framework operator in the background program code form a one-to-one mapping relation; the flow processing process definition file generation engine is used for converting a flow defined by a label for establishing connection on a desktop into a flow processing process definition file; the stream processing publishing engine includes a plurality of processing unit operator adapters for a streaming analytics platform. A set of flow conversion engine implementation method based on visual dragging, flow conversion and flow processing engine adaptation is realized; and realizing on-demand selection of a streaming analysis execution platform.
Description
Technical Field
The invention relates to the field of distributed stream computing, in particular to a visualization flow processing engine based on SPARK STREAMING/FLINK/STORM distributed stream computing and a using method thereof.
Background
Stream processing is an important big data processing means, and the main characteristic is that the processed data comes continuously and in real time. Distributed stream processing is a fine-grained processing mode oriented to dynamic data, and continuously generated dynamic data are processed based on a distributed memory. The method plays an increasingly important role in large data processing due to the characteristics of high speed, high efficiency, low delay and the like of the data processing.
Under the current explosive development of internet informatization, more and more data analysis and scene requirements for real-time processing of massive concurrent data are needed. Distributed stream computing, as an important technical means to address this need, has been proposed a variety of utility-based architectures, such as: SPARKTREAMING/FLINK/STORM etc.
However, for a specific application flow needing to adopt distributed stream computation, only a single flow can be selected currently
After the streaming processing platform, programming operation is performed, and then execution is performed, the operating process of the streaming processing platform has high requirements on the capability of technical personnel, the reusability of codes is poor, the execution cost is high, and when a large amount of specific application flows need to be processed, a large amount of workload can be generated.
Disclosure of Invention
In order to solve the problems of defects and shortcomings in the prior art, the invention specifically adopts the following technical scheme:
a visualization flow processing engine for distributed stream computing, comprising:
the system comprises a stream processing visualization arrangement engine, a stream processing process definition file generation engine and a stream processing release engine;
the visualization arrangement engine comprises a foreground visualization interface and a background program code which are coupled; the foreground visual interface comprises a desktop and a plurality of labels, and connection can be established between the labels on the desktop; each label and a homonymy stream analysis framework operator in the background program code form a one-to-one mapping relation;
the flow processing process definition file generation engine is used for converting a flow defined by a label for establishing connection on a desktop into a flow processing process definition file;
the stream processing publishing engine comprises a plurality of processing unit operator adapters for the stream analysis platform and is used for publishing the stream processing process definition file to one or more stream processing platforms through the steps of stream parsing, execution platform judgment, unit processor adaptation, application compiling, application packaging and application publishing.
Preferably, the streaming analysis platform and the streaming processing platform comprise STORM and/or SPARK and/or FLINK; the flow analysis framework operator is a STORM and/or SPARK and/or FLINK flow analysis framework operator.
Preferably, the label can also form a one-to-one mapping relation with the encapsulation of the same-name custom operator.
Preferably, the label comprises at least: data access (source), transformation (transition), concatenation (join), desensitization (masking), encoding (decode), projection (project), splitting (split), grouping (group by), counting (count), filtering (filter), encoding/decoding (encode/decode), output (sink).
The use method of the visualization flow processing engine for the optimized distributed flow calculation is characterized by comprising the following steps of:
step S1: decomposing the specific service requirement into a plurality of service processing processes, and expressing the service processing processes by taking an operator defined by a flow analysis framework as a minimum unit;
step S2: screening out labels which are in the same name as the streaming analysis frame operator in the visual layout engine, placing the labels on a desktop, and connecting the labels on the desktop according to the relation between business processing processes;
step S3: converting the flow defined in the step S2 into a stream processing procedure definition file according to the mapping relationship between the label and the package of the homonymy stream analysis framework operator, by the stream processing procedure definition file generation engine;
step S4: the stream processing issuing engine issues the stream processing process definition file to one or more stream processing platforms through the steps of stream parsing, execution platform judgment, unit processor adaptation, application compiling, application packaging and application issuing, and executes the running operation.
Preferably, in the step of performing platform judgment and element processor adaptation in step S4, a streaming processing platform capable of processing the streaming analysis framework operator is judged first through the type of the parsed streaming analysis framework operator; and performing unit processor adaptation according to the judgment result.
The invention and the preferred scheme thereof realize a set of stream conversion engine realization method based on visual dragging, flow conversion and stream processing engine adaptation; on-demand selection of the streaming analysis execution platform (STORM/SPARK/FLINK) is enabled. The stream analysis framework operator is used as a basic unit for packaging and is mapped to a label of a visual interface, stream processing editing is directly realized through operation on the label, and a visual process editing and highly generalized processing unit is adopted, so that the service processing process is simplified; and various flow analysis operations such as STORM, SPARK, FLINK and the like are supported, and can be selected according to needs. The scheme realizes the process of converting the flow definition into a program package which can run on various current mainstream flow analysis platforms (such as STORM, SPARK and FLINK), and can complete the arrangement of a series of analysis flows through visual operation without programming BY packaging a series of general processing units (such as SOURCE, JOIN, GROUP BY, COUNT, FILTER, ENCDER, MASKING, DECODER, SINK and the like).
Drawings
The invention is described in further detail below with reference to the following figures and detailed description:
FIG. 1 is a schematic diagram of an arrangement and a flow chart of an embodiment of the invention;
FIG. 2 is a schematic diagram of a logic for implementing a stream processing visualization orchestration engine according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a flow processing and issuing process according to an embodiment of the present invention.
Detailed Description
In order to make the features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail as follows:
as shown in fig. 1, the visualized flow processing engine for distributed stream computation provided by this embodiment includes:
the system comprises a stream processing visualization arrangement engine, a stream processing process definition file generation engine and a stream processing release engine.
The visualization arrangement engine comprises a foreground visualization interface and a background program code which are coupled; the foreground visual interface comprises a desktop and a plurality of labels, and connection can be established between the labels on the desktop; each label and the same-name streaming analysis frame operator in the background program code form a one-to-one mapping relation, so that the selection of the label in the foreground is equivalent to the calling of the corresponding streaming analysis frame operator, and the establishment of the connection relation between the labels is equivalent to the generation of the corresponding connection relation by the streaming analysis frame operator in the background. Wherein, the flow analysis frame operator is STORM and/or SPARK and/or FLINK flow analysis frame operator, mainly comprising: the method comprises the steps of data access (source), conversion (transition), connection (join), desensitization (masking), coding (decode), projection (project), splitting (split), grouping (group by), counting (count), filtering (filter), coding/decoding (encode/decode), output (sink) and other operators. Preferably, the tags may also form a one-to-one mapping relationship with the packages of the same-name custom operators, so as to extend the functions of the solution of this embodiment.
As shown in fig. 1 and fig. 2, a general complete flow definition includes an input unit, a processing unit, and an output unit; the business operation can be decomposed into a series of data processing procedures by analyzing the business requirements, each processing procedure corresponds to a processing unit, and the processing unit corresponds to a defined tag (including data access (source), conversion (transition), connection (join), desensitization (masking), coding (decode), projection (project), splitting (split), grouping (group by), counting (count), filtering (filter), coding/decoding (encode/decode), output (sink), and analysis model algorithm encapsulation, etc.), and the conversion from the business model to the calculation model can be realized by dragging the corresponding tag to the desktop, configuring the processing unit form information, and finally connecting each tag.
The stream processing process definition file generation engine is used for converting the flow defined by the label for establishing connection on the desktop into a stream processing process definition file;
as shown in fig. 3, the stream processing publishing engine includes multiple processing unit operator adapters for the streaming analysis platform, and is configured to publish the stream processing process definition file to one or more streaming processing platforms through steps of stream parsing, execution platform determination, unit processor adaptation, application compilation, application packaging, and application publishing. The streaming analysis platform and the streaming processing platform comprise STORM and/or SPARK and/or FLINK.
In order to more easily understand the overall scheme provided by the present embodiment, the following provides a method for using the visualization flow processing engine for distributed flow computation provided by the present embodiment, including the following steps:
step S1: decomposing the specific service requirement into a plurality of service processing processes, and expressing the service processing processes by taking an operator defined by a flow analysis framework as a minimum unit;
step S2: screening out labels with the same name as a streaming analysis frame operator in a visual arrangement engine, placing the labels on a desktop, and connecting the labels on the desktop according to the relation between business processing processes;
step S3: converting the process defined in the step S2 into a stream processing process definition file according to the one-to-one mapping relationship between the label and the package of the homonymous stream analysis framework operator by the stream processing process definition file generation engine; the file defines the dependency relationship of the flow processing unit, the attribute of the processing unit configured visually and the executable packaging implementation of the processing unit corresponding to the streaming engines such as STORM, SPARK, FLINK and the like;
step S4: as shown in fig. 3, the stream processing publishing engine publishes the stream processing process definition file to one or more stream processing platforms through the steps of stream parsing, execution platform judgment, unit processor adaptation, application compilation, application packaging, and application publishing, and executes the running job. And completing the operation conversion of the calculation model to different operation platforms.
In the step of determining the execution platform and adapting the unit processor in step S4, first, a streaming processing platform capable of processing the streaming analysis framework operator is determined according to the type of the parsed streaming analysis framework operator; and performing unit processor adaptation according to the judgment result.
The present invention is not limited to the above preferred embodiments, and all other various forms of visualization process engine for distributed stream computation and methods of using the same can be derived from the present invention.
Claims (4)
1. A visualization flow processing engine for distributed stream computing, comprising:
the system comprises a stream processing visualization arrangement engine, a stream processing process definition file generation engine and a stream processing release engine;
the visualization arrangement engine comprises a foreground visualization interface and a background program code which are coupled; the foreground visual interface comprises a desktop and a plurality of labels, and connection can be established between the labels on the desktop; each label and a homonymy stream analysis framework operator in the background program code form a one-to-one mapping relation;
the flow processing process definition file generation engine is used for converting a flow defined by a label for establishing connection on a desktop into a flow processing process definition file;
the stream processing publishing engine comprises a plurality of processing unit operator adapters for the stream analysis platform and is used for publishing the stream processing process definition file to one or more stream processing platforms through the steps of stream parsing, execution platform judgment, unit processor adaptation, application compiling, application packaging and application publishing;
the streaming analysis platform and the streaming processing platform comprise STORM and/or SPARK and/or FLINK; the flow analysis framework operator is a STORM and/or SPARK and/or FLINK flow analysis framework operator;
the label at least comprises: data access source, transformation transition, join, desensitization masking, coding decode, projection project, split, grouping by, count, filter, encoding/decoding encode/decode, and output sink.
2. The visualization flow processing engine of distributed flow computation of claim 1, characterized by: the label can also form a one-to-one mapping relation with the encapsulation of the same-name self-defined operator.
3. The method for using a visualization flow processing engine for distributed flow computation of claim 1, comprising the steps of:
step S1: decomposing the specific service requirement into a plurality of service processing processes, and expressing the service processing processes by taking an operator defined by a flow analysis framework as a minimum unit;
step S2: screening out labels which are in the same name as the streaming analysis frame operator in the visual layout engine, placing the labels on a desktop, and connecting the labels on the desktop according to the relation between business processing processes;
step S3: converting the process defined in the step S2 into a stream processing procedure definition file according to the one-to-one mapping relationship between the label and the package of the homonymous stream analysis framework operator by the stream processing procedure definition file generation engine;
step S4: the stream processing issuing engine issues the stream processing process definition file to one or more stream processing platforms through the steps of stream parsing, execution platform judgment, unit processor adaptation, application compiling, application packaging and application issuing, and executes the running operation.
4. The method for using a visualization flow processing engine for distributed flow computation of claim 3, wherein: in the step of determining the execution platform and adapting the unit processor in step S4, first, a streaming processing platform capable of processing the streaming analysis framework operator is determined according to the type of the parsed streaming analysis framework operator; and performing unit processor adaptation according to the judgment result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811589073.3A CN109710215B (en) | 2018-12-25 | 2018-12-25 | Visualization process processing engine for distributed flow calculation and use method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811589073.3A CN109710215B (en) | 2018-12-25 | 2018-12-25 | Visualization process processing engine for distributed flow calculation and use method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109710215A CN109710215A (en) | 2019-05-03 |
| CN109710215B true CN109710215B (en) | 2022-05-27 |
Family
ID=66257430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811589073.3A Active CN109710215B (en) | 2018-12-25 | 2018-12-25 | Visualization process processing engine for distributed flow calculation and use method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109710215B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| CN110647323A (en) * | 2019-09-17 | 2020-01-03 | 北京快威科创信息技术有限公司 | Code-free business application software development platform |
| CN112817573B (en) * | 2019-11-18 | 2024-03-01 | 北京沃东天骏信息技术有限公司 | Method, apparatus, computer system, and medium for building a streaming computing application |
| CN110868324A (en) * | 2019-11-22 | 2020-03-06 | 中国建设银行股份有限公司 | Service configuration method, device, equipment and storage medium |
| CN110908641B (en) * | 2019-11-27 | 2024-04-26 | 中国建设银行股份有限公司 | Visualization-based stream computing platform, method, device and storage medium |
| CN111414385A (en) * | 2020-02-28 | 2020-07-14 | 联想(北京)有限公司 | Stream calculation execution method, equipment and computer readable storage medium |
| CN111782371B (en) * | 2020-06-30 | 2024-05-14 | 北京百度网讯科技有限公司 | Stream computing method and device based on DAG interaction |
| CN112181511B (en) * | 2020-08-26 | 2021-09-14 | 北京大学 | Executable information analysis flow interaction configuration generation method |
| CN112162754A (en) * | 2020-10-19 | 2021-01-01 | 科技谷（厦门）信息技术有限公司 | Multi-source heterogeneous data processing system |
| CN112256258A (en) * | 2020-10-22 | 2021-01-22 | 北京神州数字科技有限公司 | Micro-service arrangement automatic code generation method and system |
| CN112364290B (en) * | 2020-11-18 | 2022-09-02 | 中睿信数字技术有限公司 | Method and system for constructing visual calculation model based on stream-oriented calculation |
| CN112506497B (en) * | 2020-11-30 | 2021-08-24 | 北京九章云极科技有限公司 | Data processing method and data processing system |
| CN112685004B (en) * | 2020-12-21 | 2022-08-05 | 福建新大陆软件工程有限公司 | Online component arrangement calculation method and system based on real-time stream calculation |
| CN116775041B (en) * | 2023-08-21 | 2023-12-05 | 四川兴川贷数字科技有限公司 | Real-time decision engine implementation method based on stream calculation and RETE algorithm |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101478431A (en) * | 2009-02-10 | 2009-07-08 | 浪潮通信信息系统有限公司 | Task scheduling system for management by visible process |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150170088A1 (en) * | 2013-12-17 | 2015-06-18 | The Cleveland Clinic Foundation | Logic flow generator system and method |
-
2018
- 2018-12-25 CN CN201811589073.3A patent/CN109710215B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101478431A (en) * | 2009-02-10 | 2009-07-08 | 浪潮通信信息系统有限公司 | Task scheduling system for management by visible process |
| CN108037919A (en) * | 2017-12-01 | 2018-05-15 | 北京博宇通达科技有限公司 | A kind of visualization big data workflow configuration method and system based on WEB |
| CN108628608A (en) * | 2018-04-16 | 2018-10-09 | 朗新科技股份有限公司 | A kind of method and device of data visualization |
Non-Patent Citations (1)
| Title |
|---|
| 流计算大数据技术在运营商试试信令处理中的应用;董斌等;《电信科学》;20151031(第10期);第2015212-1至2015212-7页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109710215A (en) | 2019-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109710215B (en) | Visualization process processing engine for distributed flow calculation and use method thereof | |
| CN110825488A (en) | Business processing method and device, electronic equipment and storage medium | |
| KR20040046463A (en) | System and method of generating EJB component from reusable business logic in servlet | |
| CN112416337B (en) | Software architecture development system for aerospace embedded system | |
| CN111352616A (en) | Real-time calculation visualization development system and application method thereof | |
| CN111290778A (en) | AI model packaging method, platform and electronic equipment | |
| CN104216766A (en) | Method and device for processing stream data | |
| CN113656503A (en) | Data synchronization method, device and system and computer readable storage medium | |
| CN104991810B (en) | Method and processing system in automatic addition APK to Android system | |
| CN114036183A (en) | Data ETL processing method, device, equipment and medium | |
| CN106681781B (en) | Method and system for realizing real-time computing service | |
| CN115794106A (en) | Method and system for analyzing configuration of binary protocol data of rail transit | |
| CN111176660A (en) | Distributed architecture-oriented micro-service contract management method and device, computer equipment and readable storage medium | |
| CN108874395B (en) | Hard compiling method and device in modular stream processing process | |
| CN105718429B (en) | The method and device of inspection report generation | |
| CN115392443B (en) | Pulse neural network application representation method and device of brain-like computer operating system | |
| CN104992379A (en) | Dynamic text exchange format management method of big data of power industry | |
| CN104731900A (en) | Hive scheduling method and device | |
| JP2010049439A (en) | System construction method using software model and modeling device | |
| CN106599244B (en) | General original log cleaning device and method | |
| Zhang et al. | GP-SWAT (v1. 0): A two-layer graph-based parallel simulation framework for the SWAT model | |
| US20210200177A1 (en) | Graph display device, graph display method, and recording medium | |
| CN110874508A (en) | Simulation scene construction method and device | |
| CN112527443A (en) | Prompt box display method and device, electronic equipment and computer readable storage medium | |
| CN112114817A (en) | COBOL language-based data dictionary field information acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |