CN111399826A - Online data exchange method and system for visual drag flow diagram ET L - Google Patents
Online data exchange method and system for visual drag flow diagram ET L Download PDFInfo
- Publication number
- CN111399826A CN111399826A CN202010197562.5A CN202010197562A CN111399826A CN 111399826 A CN111399826 A CN 111399826A CN 202010197562 A CN202010197562 A CN 202010197562A CN 111399826 A CN111399826 A CN 111399826A
- Authority
- CN
- China
- Prior art keywords
- task
- metadata
- submodule
- timing
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/34—Graphical or visual programming
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/0486—Drag-and-drop
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a visualization dragging flow diagram ET L online data exchange method and system, which are based on a highly-free B/S framework and combined with a corresponding user management module to meet the requirement of ET L task embedding diversification, and can perform corresponding model development and timing scheduling on an ET L task so as to execute corresponding ET L tasks in a distributed manner on different nodes and form corresponding task history logs, thereby effectively realizing visualization of ET L online data and improving conversion efficiency of the data.
Description
Technical Field
The invention relates to the technical field of ET L data exchange, in particular to a visualized drag flow diagram ET L online data exchange method and system.
Background
The application system in the prior art needs to be constructed with large manpower and material resources, and the data of the application system, particularly the application system is irreplaceable, while the newly-built business intelligence system aims to assist users in decision making through data analysis, and the difficulty of data integration is undoubtedly increased due to the fact that the sources and the formats of the data are different, ET L refers to the processes of data extraction, data cleaning, data conversion and data loading, ET L can solve the problems of data consistency and integration to a certain extent, and can integrate the existing traditional environment with the collected data in a platform and efficiently convert the data by using a single solution, however, the data exchange process of the application system in the prior art about the ET L task is still not complete, and the monitoring process of the data processing process related to the ET L task is seriously influenced.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention provides a visual dragging flow diagram ET L online data exchange method and a visual dragging flow diagram ET L online data exchange system, wherein the visual dragging flow diagram ET L online data exchange method and system realize model development of an ET L task on a B/S framework by reading metadata corresponding to an ET L task stored in a preset resource library and performing preset operation on the metadata, perform timed scheduling on the ET L task according to the preset operation on the metadata, acquire the node distribution characteristics of a plurality of deployed servers, perform corresponding ET L tasks on different nodes in a distributed manner by combining the result of the timed scheduling, generate corresponding task history logs and/or form corresponding task feedback operations according to the execution result of the ET L task, and accordingly, the visual dragging flow diagram ET L online data exchange method and system can meet the requirement of ET L task diversification on the highly-free B/S framework and combine with a corresponding user management module, can embed corresponding model and model L to perform corresponding task scheduling on the development and distributed tasks so as to effectively improve the ET data distribution efficiency and obtain the visual dragging flow diagram ET L online data.
The invention provides a visualized dragging flow diagram ET L online data exchange method, which is characterized by comprising the following steps of:
step S1, reading metadata corresponding to the ET L task stored in a predetermined resource library, and performing predetermined operation on the metadata to realize model development of the ET L task on a B/S architecture;
step S2, carrying out timing scheduling on the ET L task according to the predetermined operation carried out on the metadata;
step S3, acquiring the node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes by combining the result of the timing scheduling;
step S4, generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task;
further, in the step S1, reading metadata corresponding to the ET L task and stored in a predetermined resource library, and performing a predetermined operation on the metadata to realize the ET L task with respect to the model development of the B/S architecture specifically includes,
step S101, acquiring a storage organization structure of metadata in the preset resource library, and determining a positioning and reading mode of the metadata so as to read and obtain the metadata of the ET L task object;
step S102, after the metadata is analyzed, the ET L task is displayed to a system corresponding to the B/S framework in an MXgraph mode;
step S103, performing the model development on the ET L task displayed correspondingly in a system corresponding to the B/S framework so as to obtain a corresponding model development result;
further, in the step S2, the timing scheduling of the ET L task according to the predetermined operation performed on the metadata specifically includes,
step S201, monitoring the preset operation performed on the metadata so as to obtain the completion progress and/or the conversion progress of the ET L task corresponding to the metadata;
step S202, according to the completion progress and/or the conversion progress, mapping the ET L task to a Quartz timing frame, so as to determine the timing calculation information of the ET L task, wherein the ET L task comprises ET L extraction and ET L conversion, and the timing calculation information is specifically determined,
calculating the time length required for carrying out ET L extraction according to the following formula (1),
in the above formula (1), x1Extracting the corresponding required time length for the ET L, and extracting the corresponding completion progress, t, for the current ET L1In order to complete the time required by the current ET L extraction progress, n is the total extraction amount of ET L extraction;
calculating the time length required for carrying out the ET L conversion according to the following formula (2),
in the above formula (2), x2The time length required for the ET L conversion is the completion progress corresponding to the current ET L conversion, t2The time required to complete the current ET L conversion schedule;
calculating to obtain the expected timing information corresponding to the ET L task according to the following formula (3),
A＝{μ-t1,μ+x1,μ+x2+t2} (3)
in the above formula (3), a is the predicted timing information corresponding to the ET L task, μ is the current actual time, t1Time, x, required to complete the current ET L extraction schedule1Length of time, t, required for performing said ET L extraction2Time, x, required to complete the current ET L conversion schedule2When the ET L conversion is correspondingly neededA length of the interval;
calculating the timing calculation information of the ET L task according to the following formula (4),
f:A→B (4)
in the formula (4), f is a mapping function corresponding to mapping the ET L task to the Quartz timing frame, a is predicted timing information corresponding to the ET L task, and B is the timing calculation information of the ET L task;
step S203, carrying out timing scheduling on the ET L task according to the timing calculation information;
further, in the step S3, the acquiring node distribution characteristics of the deployed servers, and performing the corresponding ET L tasks in a distributed manner on different nodes in combination with the result of the timing scheduling specifically include,
step S301, according to a button execution engine, acquiring node topological structure information and/or node performance index information corresponding to the deployed servers, wherein the node topological structure information and/or the node performance index information are used as the node distribution characteristics;
step S302, sending HTTP requests to different remote nodes through a system corresponding to the B/S framework, so as to implement distributed execution of corresponding ET L tasks on different nodes;
further, in the step S4, generating a corresponding task history log and/or forming a corresponding task feedback operation according to the execution result of the ET L task specifically includes,
step S401, according to the execution result of the ET L task, mapping the management of the corresponding node to the management of the corresponding ET L task;
step S402, monitoring the execution process of the ET L task according to the management of the corresponding ET L task;
step S403, according to the monitoring result of the execution process, generating a corresponding task history log and forming a corresponding task feedback operation.
The invention also provides a visual dragging flow diagram ET L online data exchange system, which is characterized in that:
the visual dragging flow diagram ET L online data exchange system comprises a metadata reading module, a metadata operating module, an ET L task timing scheduling module, an ET L task executing module and an ET L task response module, wherein,
the metadata reading module is used for reading metadata which is stored in a preset resource library and corresponds to an ET L task;
the metadata operation module is used for carrying out predetermined operation on the metadata to realize model development of the ET L task on the B/S architecture;
the ET L task timing scheduling module is configured to perform timing scheduling on the ET L task according to the predetermined operation performed on the metadata;
the ET L task execution module is used for acquiring the node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes by combining the result of the timing scheduling;
the ET L task response module is used for generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task;
further, the metadata reading module comprises a storage organization structure determining submodule, a reading mode determining submodule and a reading execution submodule; wherein the content of the first and second substances,
the storage organization structure determining submodule is used for acquiring a storage organization structure of the metadata in the predetermined resource library;
the reading mode determining submodule is used for determining a positioning and reading mode of the metadata according to the storage organization structure;
the reading execution submodule is used for reading and obtaining the metadata of the ET L task object according to the positioning and reading mode;
alternatively, the first and second electrodes may be,
the metadata operation module comprises an analysis processing sub-module, a visual display sub-module and a model development sub-module; wherein the content of the first and second substances,
the analysis processing sub-module is used for analyzing the metadata;
the visualization display submodule is used for displaying the ET L task to a system corresponding to the B/S framework in an MXGraph mode according to the result of the analysis processing;
the model development submodule is used for carrying out model development on the ET L task which is correspondingly displayed in a system corresponding to the B/S framework so as to obtain a corresponding model development result;
further, the ET L task timing scheduling module comprises an ET L monitoring submodule, a timing calculation information determining submodule and a timing scheduling execution submodule;
the ET L monitoring submodule is used for monitoring the predetermined operation performed on the metadata so as to obtain the completion progress and/or the conversion progress of an ET L task corresponding to the metadata;
the timing calculation information determining submodule is used for mapping the ET L task to a Quartz timing frame according to the completion progress and/or the conversion progress so as to determine the timing calculation information of the ET L task;
the timing scheduling execution submodule is used for performing timing scheduling on the ET L task according to the timing calculation information;
further, the ET L task execution module comprises a node distribution characteristic determination submodule and a task execution submodule, wherein,
the node distribution characteristic determining submodule is used for acquiring node topological structure information and/or node performance index information corresponding to the deployed servers according to a button execution engine to serve as the node distribution characteristic;
the task execution submodule is used for sending HTTP requests to different remote nodes through a system corresponding to the B/S framework, so that corresponding ET L tasks are executed in a distributed mode on different nodes;
further, the ET L task response module comprises a node-task management interaction submodule, a task monitoring submodule and a task response execution submodule, wherein,
the node-task management interaction submodule is used for mapping the management of the corresponding node to the management of the corresponding ET L task according to the execution result of the ET L task;
the task monitoring submodule is used for monitoring the execution process of the ET L task according to the management of the corresponding ET L task;
and the task response execution submodule is used for generating a corresponding task history log and forming a corresponding task feedback operation according to the monitoring result of the execution process.
Compared with the prior art, the visual drag flow diagram ET L online data exchange method and system can perform corresponding model development and timing scheduling on L tasks based on a highly-free B/S framework and combined with a corresponding user management module to meet the ET L task embedded multi-diversification requirement, so that corresponding ET L online data can be performed in a distributed manner on different nodes, and the ET L task conversion efficiency can be effectively improved by reading metadata corresponding to an ET L task stored in a predetermined resource library, performing predetermined operation on the metadata to realize model development of the ET L task with respect to a B/S framework, performing timing scheduling on the ET L task according to the predetermined operation on the metadata, acquiring the node distribution characteristics of a plurality of deployed servers, performing corresponding ET L tasks in a distributed manner on different nodes according to the result of the timing scheduling, and generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flow diagram of a visualized drag flow diagram ET L online data exchange method according to the present invention.
Fig. 2 is a schematic structural diagram of a visualization drag flow diagram ET L online data exchange system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flow diagram of a visualized drag flow diagram ET L online data exchange method according to an embodiment of the present invention is shown, where the visualized drag flow diagram ET L online data exchange method includes the following steps:
step S1, reading metadata corresponding to the ET L task stored in a predetermined repository, and performing predetermined operations on the metadata to implement model development of the ET L task with respect to the B/S architecture, wherein the E L T task refers to corresponding data extraction (Extract), data transformation (Transform), and data loading (L oad) tasks, and the predetermined operations may be, but are not limited to, performing at least one of a recoverable editing operation, a reversible overlay operation, a pruning operation, and an adding operation on the metadata;
step S2, performing timing scheduling on the ET L task according to the predetermined operation performed on the metadata, wherein the performing of the timing scheduling on the ET L task may specifically be to invoke a corresponding ET L task from a task database and execute the ET L task according to a predetermined clock signal or a timing signal determined by a user;
step S3, obtaining node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes in combination with the result of the timing scheduling, where the node distribution characteristics may include, but are not limited to, data connection topology characteristics between any two servers of the plurality of servers and/or data stream input/output characteristics between any two servers;
and step S4, generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task.
Preferably, in the step S1, reading metadata corresponding to the ET L task and stored in a predetermined resource library, and performing a predetermined operation on the metadata to implement the ET L task, specifically includes regarding the model development of the B/S architecture,
step S101, obtaining a storage organization structure of the metadata in the predetermined resource library, determining a positioning and reading mode of the metadata, and reading the metadata of the ET L task object;
step S102, after the metadata is analyzed, the ET L task is displayed to a system corresponding to the B/S framework in an MXGraph mode;
step S103, in the system corresponding to the B/S framework, the model development is carried out on the ET L task which is correspondingly displayed, and therefore a corresponding model development result is obtained.
Preferably, in the step S2, the timing scheduling of the ET L task according to the predetermined operation performed on the metadata specifically includes,
step S201, monitoring the predetermined operation performed on the metadata, so as to obtain a completion progress and/or a conversion progress of an ET L task corresponding to the metadata;
step S202, according to the completion progress and/or the conversion progress, mapping the ET L task to a Quartz timing frame, so as to determine the timing calculation information of the ET L task, wherein the ET L task comprises ET L extraction and ET L conversion, and the timing calculation information is determined to be specific,
the length of time required for performing the ET L extraction is calculated according to the following formula (1),
in the above formula (1), x1Extracting the corresponding required time length for the ET L, extracting the corresponding completion progress for the current ET L, t1In order to complete the time required by the current ET L extraction progress, n is the total extraction amount of ET L extraction;
the length of time required for performing the ET L conversion is calculated according to the following equation (2),
in the above formula (2), x2The time length required for the ET L conversion is t, which is the completion progress corresponding to the current ET L conversion2The time required to complete the current ET L conversion schedule;
calculating to obtain the predicted timing information corresponding to the ET L task according to the following formula (3),
A＝{μ-t1,μ+x1,μ+x2+t2} (3)
in the above equation (3), a is the predicted timing information corresponding to the ET L task, μ is the current actual time, t1Time, x, required to complete the current ET L extraction schedule1The length of time, t, required for performing the ET L extraction2Time, x, required to complete the current ET L conversion schedule2The length of time required to perform the ET L transformation;
the timing calculation information of the ET L task is calculated according to the following formula (4),
f:A→B (4)
in the above formula (4), f is a mapping function corresponding to mapping the ET L task to the Quartz timing frame, a is predicted timing information corresponding to the ET L task, and B is the timing calculation information of the ET L task;
step S203, carrying out timing scheduling on the ET L task according to the timing calculation information;
in general, the process of determining the timing calculation information not only can determine the starting time corresponding to the process of extracting the ET L and converting the ET L into the subtask in the ET L task, but also can restore the execution process of the ET L task to the maximum extent according to the extraction progress of the ET L and the completion progress of the ET L, so that the determination result of the timing calculation information of the ET L task is closer to the actual situation.
Preferably, in the step S3, the obtaining of the node distribution characteristics of the deployed servers, and the performing of the corresponding ET L tasks in a distributed manner on different nodes in combination with the result of the timing scheduling specifically include,
step S301, according to a button execution engine, acquiring node topological structure information and/or node performance index information corresponding to the deployed servers, wherein the node topological structure information and/or the node performance index information are used as the node distribution characteristics;
step S302, an HTTP request is sent to different remote nodes through a system corresponding to the B/S framework, and therefore the corresponding ET L task is executed in a distributed mode on different nodes.
Preferably, in the step S4, the generating of the corresponding task history log and/or the forming of the corresponding task feedback operation according to the execution result of the ET L task specifically includes,
step S401, according to the execution result of the ET L task, mapping the management of the corresponding node to the management of the corresponding ET L task;
step S402, according to the management of the corresponding ET L task, monitoring the execution process of the ET L task;
step S403, according to the monitoring result of the execution process, generating a corresponding task history log and forming a corresponding task feedback operation.
Referring to fig. 2, a schematic structural diagram of a visualized drag-flow diagram ET L online data exchange system according to an embodiment of the present invention is shown, where the visualized drag-flow diagram ET L online data exchange system includes a metadata reading module, a metadata operating module, an ET L task timing scheduling module, an ET L task executing module, and an ET L task responding module,
the metadata reading module is used for reading metadata which is stored in a predetermined resource library and corresponds to an ET L task;
the metadata operation module is used for carrying out predetermined operation on the metadata to realize model development of the ET L task about the B/S architecture;
the ET L task timing scheduling module is configured to perform timing scheduling on the ET L task according to the predetermined operation performed on the metadata;
the ET L task execution module is used for acquiring node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes according to the result of the timing scheduling;
the ET L task response module is used for generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task.
Preferably, the metadata reading module comprises a storage organization structure determining submodule, a reading mode determining submodule and a reading executing submodule; wherein the content of the first and second substances,
the storage organization structure determining submodule is used for acquiring a storage organization structure of the metadata in the preset resource library;
the reading mode determining submodule is used for determining a positioning and reading mode of the metadata according to the storage organization structure;
the reading execution submodule is used for reading and obtaining the metadata of the ET L task object according to the positioning and reading mode.
Preferably, the metadata operation module comprises an analysis processing sub-module, a visual display sub-module and a model development sub-module; wherein the content of the first and second substances,
the analysis processing submodule is used for analyzing the metadata;
the visualization display submodule is used for displaying the ET L task to a system corresponding to the B/S framework in an MXGraph mode according to the analysis processing result;
the model development submodule is used for carrying out the model development on the ET L task which is correspondingly displayed in the system corresponding to the B/S framework, so that a corresponding model development result is obtained.
Preferably, the ET L task timing scheduling module includes an ET L monitoring submodule, a timing calculation information determining submodule and a timing scheduling execution submodule;
the ET L monitoring submodule is used for monitoring the predetermined operation performed on the metadata so as to obtain the completion progress and/or the conversion progress of the ET L task corresponding to the metadata;
the timing calculation information determining submodule is used for mapping the ET L task to a Quartz timing frame according to the completion progress and/or the conversion progress so as to determine the timing calculation information of the ET L task;
the timing scheduling execution submodule is used for performing timing scheduling on the ET L task according to the timing calculation information.
Preferably, the ET L task execution module comprises a node distribution characteristic determination sub-module and a task execution sub-module, wherein,
the node distribution characteristic determining submodule is used for acquiring node topological structure information and/or node performance index information corresponding to the deployed servers according to a button execution engine to serve as the node distribution characteristic;
the task execution submodule is used for sending an HTTP request to different remote nodes through a system corresponding to the B/S framework, so that the corresponding ET L task is executed in a distributed mode on different nodes.
Preferably, the ET L task response module comprises a node-task management interaction submodule, a task monitoring submodule and a task response execution submodule, wherein,
the node-task management interaction submodule is used for mapping the management of the corresponding node to the management of the corresponding ET L task according to the execution result of the ET L task;
the task monitoring submodule is used for monitoring the execution process of the ET L task according to the management of the corresponding ET L task;
the task response execution submodule is used for generating a corresponding task history log and forming a corresponding task feedback operation according to the monitoring result of the execution process.
From the content of the above embodiments, the visualization dragging flow diagram ET L online data exchange method and system meet the requirement of ET L task embedding diversification based on a highly-free B/S framework and in combination with a corresponding user management module, and can perform corresponding model development and timed scheduling on an ET L task, so that corresponding ET L tasks are executed in a distributed manner on different nodes and corresponding task history logs are formed, thereby effectively realizing visualization of ET L online data and improving conversion efficiency of data.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A visualized drag flow diagram ET L online data exchange method is characterized in that the visualized drag flow diagram ET L online data exchange method comprises the following steps:
step S1, reading metadata corresponding to the ET L task stored in a predetermined resource library, and performing predetermined operation on the metadata to realize model development of the ET L task on a B/S architecture;
step S2, carrying out timing scheduling on the ET L task according to the predetermined operation carried out on the metadata;
step S3, acquiring the node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes by combining the result of the timing scheduling;
and step S4, generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task.
2. The online data exchange method of the visual drag flow schema ET L as claimed in claim 1, wherein:
in the step S1, reading metadata corresponding to the ET L task and stored in a predetermined resource library, and performing a predetermined operation on the metadata to realize the model development of the ET L task with respect to the B/S architecture specifically comprises,
step S101, acquiring a storage organization structure of metadata in the preset resource library, and determining a positioning and reading mode of the metadata so as to read and obtain the metadata of the ET L task object;
step S102, after the metadata is analyzed, the ET L task is displayed to a system corresponding to the B/S framework in an MXgraph mode;
and step S103, carrying out model development on the ET L task correspondingly displayed in the system corresponding to the B/S framework so as to obtain a corresponding model development result.
3. The online data exchange method of the visual drag flow schema ET L as claimed in claim 1, wherein:
in the step S2, the timing scheduling of the ET L task according to the predetermined operation performed on the metadata specifically includes,
step S201, monitoring the preset operation performed on the metadata so as to obtain the completion progress and/or the conversion progress of the ET L task corresponding to the metadata;
step S202, according to the completion progress and/or the conversion progress, mapping the ET L task to a Quartz timing frame, so as to determine the timing calculation information of the ET L task, wherein the ET L task comprises ET L extraction and ET L conversion, and the timing calculation information is specifically determined,
calculating the time length required for carrying out ET L extraction according to the following formula (1),
in the above formula (1), x1Time required for performing the ET L extractionLength, extracting the corresponding completion schedule, t, for the current ET L1In order to complete the time required by the current ET L extraction progress, n is the total extraction amount of ET L extraction;
calculating the time length required for carrying out the ET L conversion according to the following formula (2),
in the above formula (2), x2The time length required for the ET L conversion is the completion progress corresponding to the current ET L conversion, t2The time required to complete the current ET L conversion schedule;
calculating to obtain the expected timing information corresponding to the ET L task according to the following formula (3),
A＝{μ-t1,μ+x1,μ+x2+t2} (3)
in the above formula (3), a is the predicted timing information corresponding to the ET L task, μ is the current actual time, t1Time, x, required to complete the current ET L extraction schedule1Length of time, t, required for performing said ET L extraction2Time, x, required to complete the current ET L conversion schedule2A time length corresponding to the ET L conversion;
calculating the timing calculation information of the ET L task according to the following formula (4),
f:A→B (4)
in the formula (4), f is a mapping function corresponding to mapping the ET L task to the Quartz timing frame, a is predicted timing information corresponding to the ET L task, and B is the timing calculation information of the ET L task;
and step S203, carrying out timing scheduling on the ET L task according to the timing calculation information.
4. The online data exchange method of the visual drag flow schema ET L as claimed in claim 1, wherein:
in the step S3, the acquiring node distribution characteristics of the deployed servers, and performing the corresponding ET L tasks in a distributed manner on different nodes according to the result of the timing scheduling specifically include,
step S301, according to a button execution engine, acquiring node topological structure information and/or node performance index information corresponding to the deployed servers, wherein the node topological structure information and/or the node performance index information are used as the node distribution characteristics;
step S302, an HTTP request is sent to different remote nodes through a system corresponding to the B/S framework, and therefore the corresponding ET L task is executed in a distributed mode on different nodes.
5. The online data exchange method of the visual drag flow schema ET L as claimed in claim 1, wherein:
in the step S4, the generating of the corresponding task history log and/or the forming of the corresponding task feedback operation according to the execution result of the ET L task specifically includes,
step S401, according to the execution result of the ET L task, mapping the management of the corresponding node to the management of the corresponding ET L task;
step S402, monitoring the execution process of the ET L task according to the management of the corresponding ET L task;
step S403, according to the monitoring result of the execution process, generating a corresponding task history log and forming a corresponding task feedback operation.
6. A visual drag flow diagram ET L online data exchange system is characterized in that:
the visual dragging flow diagram ET L online data exchange system comprises a metadata reading module, a metadata operating module, an ET L task timing scheduling module, an ET L task executing module and an ET L task response module, wherein,
the metadata reading module is used for reading metadata which is stored in a preset resource library and corresponds to an ET L task;
the metadata operation module is used for carrying out predetermined operation on the metadata to realize model development of the ET L task on the B/S architecture;
the ET L task timing scheduling module is configured to perform timing scheduling on the ET L task according to the predetermined operation performed on the metadata;
the ET L task execution module is used for acquiring the node distribution characteristics of a plurality of deployed servers, and executing corresponding ET L tasks in a distributed manner on different nodes by combining the result of the timing scheduling;
the ET L task response module is used for generating corresponding task history logs and/or forming corresponding task feedback operations according to the execution result of the ET L task.
7. The visual drag-flow schema ET L online data exchange system of claim 6, wherein:
the metadata reading module comprises a storage organization structure determining submodule, a reading mode determining submodule and a reading execution submodule; wherein the content of the first and second substances,
the storage organization structure determining submodule is used for acquiring a storage organization structure of the metadata in the predetermined resource library;
the reading mode determining submodule is used for determining a positioning and reading mode of the metadata according to the storage organization structure;
the reading execution submodule is used for reading and obtaining the metadata of the ET L task object according to the positioning and reading mode;
alternatively, the first and second electrodes may be,
the metadata operation module comprises an analysis processing sub-module, a visual display sub-module and a model development sub-module; wherein the content of the first and second substances,
the analysis processing sub-module is used for analyzing the metadata;
the visualization display submodule is used for displaying the ET L task to a system corresponding to the B/S framework in an MXGraph mode according to the result of the analysis processing;
the model development submodule is used for carrying out the model development on the ET L task which is correspondingly displayed in a system corresponding to the B/S framework, so that a corresponding model development result is obtained.
8. The visual drag-flow schema ET L online data exchange system of claim 6, wherein:
the ET L task timing scheduling module comprises an ET L monitoring submodule, a timing calculation information determining submodule and a timing scheduling execution submodule;
the ET L monitoring submodule is used for monitoring the predetermined operation performed on the metadata so as to obtain the completion progress and/or the conversion progress of an ET L task corresponding to the metadata;
the timing calculation information determining submodule is used for mapping the ET L task to a Quartz timing frame according to the completion progress and/or the conversion progress so as to determine the timing calculation information of the ET L task;
and the timing scheduling execution submodule is used for performing timing scheduling on the ET L task according to the timing calculation information.
9. The visual drag-flow schema ET L online data exchange system of claim 6, wherein:
the ET L task execution module comprises a node distribution characteristic determination submodule and a task execution submodule, wherein,
the node distribution characteristic determining submodule is used for acquiring node topological structure information and/or node performance index information corresponding to the deployed servers according to a button execution engine to serve as the node distribution characteristic;
the task execution submodule is used for sending an HTTP request to different remote nodes through a system corresponding to the B/S framework, so that the corresponding ET L task is executed in a distributed mode on different nodes.
10. The visual drag-flow schema ET L online data exchange system of claim 6, wherein:
the ET L task response module comprises a node-task management interaction submodule, a task monitoring submodule and a task response execution submodule, wherein,
the node-task management interaction submodule is used for mapping the management of the corresponding node to the management of the corresponding ET L task according to the execution result of the ET L task;
the task monitoring submodule is used for monitoring the execution process of the ET L task according to the management of the corresponding ET L task;
and the task response execution submodule is used for generating a corresponding task history log and forming a corresponding task feedback operation according to the monitoring result of the execution process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197562.5A CN111399826B (en) | 2020-03-19 | 2020-03-19 | Visual dragging flow diagram ETL online data exchange method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197562.5A CN111399826B (en) | 2020-03-19 | 2020-03-19 | Visual dragging flow diagram ETL online data exchange method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111399826A true CN111399826A (en) | 2020-07-10 |
| CN111399826B CN111399826B (en) | 2020-12-01 |
Family
ID=71436384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010197562.5A Active CN111399826B (en) | 2020-03-19 | 2020-03-19 | Visual dragging flow diagram ETL online data exchange method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111399826B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941397B2 (en) * | 2004-02-25 | 2011-05-10 | International Business Machines Corporation | Dynamically capturing data warehouse population activities for analysis, archival, and mining |
| CN104252381A (en) * | 2013-06-30 | 2014-12-31 | 北京百度网讯科技有限公司 | Method and equipment for scheduling ETL (Extraction-Transform-Loading) task |
| CN107256247A (en) * | 2017-06-07 | 2017-10-17 | 九次方大数据信息集团有限公司 | Big data data administering method and device |
| CN107992552A (en) * | 2017-11-28 | 2018-05-04 | 南京莱斯信息技术股份有限公司 | A kind of data interchange platform and method for interchanging data |
| US20180183864A1 (en) * | 2015-10-08 | 2018-06-28 | International Business Machines Corporation | Automated etl resource provisioner |
| CN109189846A (en) * | 2018-09-11 | 2019-01-11 | 北京易华录信息技术股份有限公司 | A kind of public security traffic control visual modeling system and method based on big data technology |
| CN109408495A (en) * | 2018-11-02 | 2019-03-01 | 北京亿友科技有限公司 | The automatically generating of communication engineering project account manages and display systems |
| CN109446274A (en) * | 2017-08-31 | 2019-03-08 | 北京京东尚科信息技术有限公司 | The method and apparatus of big data platform BI metadata management |
| WO2019060861A1 (en) * | 2017-09-24 | 2019-03-28 | Domo, Inc. | Systems and methods for data analysis and visualization spanning multiple datasets |
| CN109669976A (en) * | 2018-11-22 | 2019-04-23 | 武汉达梦数据库有限公司 | Data service method and equipment based on ETL |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
| CN110287245A (en) * | 2019-05-15 | 2019-09-27 | 北方工业大学 | Method and system for scheduling and executing distributed ETL (extract transform load) tasks |
| WO2020019038A1 (en) * | 2018-07-25 | 2020-01-30 | Make IT Work Pty Ltd | Data warehousing system and process |
-
2020
- 2020-03-19 CN CN202010197562.5A patent/CN111399826B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7941397B2 (en) * | 2004-02-25 | 2011-05-10 | International Business Machines Corporation | Dynamically capturing data warehouse population activities for analysis, archival, and mining |
| CN104252381A (en) * | 2013-06-30 | 2014-12-31 | 北京百度网讯科技有限公司 | Method and equipment for scheduling ETL (Extraction-Transform-Loading) task |
| US20180183864A1 (en) * | 2015-10-08 | 2018-06-28 | International Business Machines Corporation | Automated etl resource provisioner |
| CN107256247A (en) * | 2017-06-07 | 2017-10-17 | 九次方大数据信息集团有限公司 | Big data data administering method and device |
| CN109446274A (en) * | 2017-08-31 | 2019-03-08 | 北京京东尚科信息技术有限公司 | The method and apparatus of big data platform BI metadata management |
| WO2019060861A1 (en) * | 2017-09-24 | 2019-03-28 | Domo, Inc. | Systems and methods for data analysis and visualization spanning multiple datasets |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN107992552A (en) * | 2017-11-28 | 2018-05-04 | 南京莱斯信息技术股份有限公司 | A kind of data interchange platform and method for interchanging data |
| WO2020019038A1 (en) * | 2018-07-25 | 2020-01-30 | Make IT Work Pty Ltd | Data warehousing system and process |
| CN109189846A (en) * | 2018-09-11 | 2019-01-11 | 北京易华录信息技术股份有限公司 | A kind of public security traffic control visual modeling system and method based on big data technology |
| CN109408495A (en) * | 2018-11-02 | 2019-03-01 | 北京亿友科技有限公司 | The automatically generating of communication engineering project account manages and display systems |
| CN109669976A (en) * | 2018-11-22 | 2019-04-23 | 武汉达梦数据库有限公司 | Data service method and equipment based on ETL |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
| CN110287245A (en) * | 2019-05-15 | 2019-09-27 | 北方工业大学 | Method and system for scheduling and executing distributed ETL (extract transform load) tasks |
Non-Patent Citations (2)
| Title |
|---|
| 戴超凡等: ""数据仓库中元数据技术研究"", 《计算机工程与应用》 * |
| 杨弈: ""分布式环境下元数据驱动增量ETL的设计与实现"", 《东南大学硕士学位论文》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111399826B (en) | 2020-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10389602B2 (en) | Automated feature deployment for active analytics microservices | |
| Costa et al. | Capturing and querying workflow runtime provenance with PROV: a practical approach | |
| CN110908641B (en) | Visualization-based stream computing platform, method, device and storage medium | |
| US20090007146A1 (en) | Method and system to correlate and consolidate a plurality of events | |
| CN109062780A (en) | The development approach and terminal device of automatic test cases | |
| US20180165386A1 (en) | Chaining analytic models in tenant-specific space for a cloud-based architecture | |
| CN113282611B (en) | Method, device, computer equipment and storage medium for synchronizing stream data | |
| US20130346992A1 (en) | Computing system, method for controlling thereof, and computer-readable recording medium having computer program for controlling thereof | |
| CN110674083A (en) | Workflow migration method, device, equipment and computer readable storage medium | |
| US9164746B2 (en) | Automatic topology extraction and plotting with correlation to real time analytic data | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| Guo et al. | GEO-WMS: an improved approach to geoscientific workflow management system on HPC | |
| CN112559525A (en) | Data checking system, method, device and server | |
| CN111399826B (en) | Visual dragging flow diagram ETL online data exchange method and system | |
| CN115345600B (en) | RPA flow generation method and device | |
| CN111198916A (en) | Data transmission method and device, electronic equipment and storage medium | |
| CN116401025A (en) | Data processing system and data processing method | |
| EP3343372A1 (en) | Distributed cache cleanup for analytic instance runs processing operating data from industrial assets | |
| CN115309802A (en) | User distribution thermodynamic diagram acquisition method and device, electronic equipment and storage medium | |
| CN110955709B (en) | Data processing method and device and electronic equipment | |
| Plale et al. | Data provenance for preservation of digital geoscience data | |
| Di Martino et al. | A platform for mbdaaas based on patterns and skeletons: The python based algorithms compiler | |
| CN113806556A (en) | Method, device, equipment and medium for constructing knowledge graph based on power grid data | |
| CN108960433B (en) | Method and system for running machine learning modeling process | |
| Li et al. | Evaluating software architecture evolution using performance simulation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |