CN112637356A - Data synchronous transmission method, system, medium and terminal of remote data center - Google Patents
Data synchronous transmission method, system, medium and terminal of remote data center Download PDFInfo
- Publication number
- CN112637356A CN112637356A CN202011587500.1A CN202011587500A CN112637356A CN 112637356 A CN112637356 A CN 112637356A CN 202011587500 A CN202011587500 A CN 202011587500A CN 112637356 A CN112637356 A CN 112637356A
- Authority
- CN
- China
- Prior art keywords
- data
- transmission
- data center
- center
- centers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/25—Flow control; Congestion control with rate being modified by the source upon detecting a change of network conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to the technical field of data transmission management, and discloses a data synchronous transmission method, a system, a medium and a terminal of a remote data center, wherein the data synchronous transmission system comprises the following components: registering and managing data sources of a plurality of data centers of different regions or sites; configuring an Apache Nifi for each data center to respectively construct a data transmission network, and transmitting data among the data centers; displaying a classified list according to different data centers to which the data sources belong, and identifying the number of data transmission pipelines which exist and are running at present; and visually displaying the identified data transmission pipelines which exist and are running currently and the processing nodes and the final flow direction of the data transmitted by the data transmission pipelines. The invention realizes the high decoupling of the management and execution of the data stream, and is easy to use and maintain; the learning and using cost of the Apache Nifi of the user is shielded, and corresponding data are transmitted according to the user requirement.
Description
Technical Field
The invention belongs to the technical field of data transmission management, and particularly relates to a data synchronous transmission method, a system, a medium and a terminal of a remote data center.
Background
Currently, several problems are encountered when transferring data between different data centers (especially off-site data centers):
1. and the data sources are multiple. A data center often involves a plurality of data storage components, such as a big data storage component HDFS, a big data warehouse tool Hive, a big data query system Impala, a message queue Kafka, a file transfer system FTP, a local file system, and the like, so that it is impossible to interface multiple data sources through a single data transfer technology Sqoop or flash, and meanwhile, a plurality of data transfer scripts need to be developed and scheduled by using a special scheduling tool. Therefore, the data transmission system can involve a plurality of core technologies, the development difficulty is high, the maintenance cost is high, and once personnel loss occurs, a very large risk can be generated.
2. And (4) network security. Each storage data source has its own dedicated port (e.g., FTP 21, MySQL 3306, PostgreSQL 5432, etc.), especially when large data is involved, the number of ports used will be very large (e.g., when data is transferred from HDFS to HDFS, two clusters of dataode ports 50010 are needed to communicate). Such a situation, especially when a data center in a different place is involved, will open a large number of ports or protocols on the firewall, and adding a new data source will also add or modify the firewall policy, which will affect the network security and is not easy to manage. In addition, data is transmitted only by using the protocol of the data source, and if the data transmission is encrypted, the configuration of each data source needs to be modified.
3. The transmission rate. The bandwidth between data centers (< 100MB level) is often not as large as the bandwidth inside the data centers (10GB level), and is generally costly, carrying multiple services of data access. When data transmission is performed between two data centers, the data volume involved is often large, and some data have different priorities, so that a network department needs to be coordinated to perform bandwidth and QoS control.
4. And (5) management and maintenance. For the above reasons, a data transmission system needs to develop a large amount of scripts, involve multiple core technologies, and coordinate multiple departments to complete together. Therefore, the technical cost, the communication cost and the management cost cannot be effectively controlled, and various risks are faced in the subsequent maintenance process.
Through the above analysis, the problems and defects of the prior art are as follows: the existing data transmission management system has high management cost and maintenance cost, more data sources, low safety degree and low transmission rate.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a data synchronous transmission method, a system, a medium and a terminal of a remote data center.
The invention is realized in this way, a data synchronous transmission method of a data center at different places, the data synchronous transmission method of the data center at different places comprises:
registering and managing data sources of a plurality of data centers in different regions or sites; configuring an Apache Nifi for each data center to respectively construct a data transmission network, and transmitting data among the data centers;
step two, displaying a classification list according to different data centers to which the data sources belong, and identifying the number of data transmission pipelines which currently exist and are running;
and step three, visually displaying the identified data transmission pipelines which exist and are running currently, processing nodes of the data transmitted by the data transmission pipelines and the final flow direction of the data.
Further, each data center configures Apache Nifi to construct a data transmission network, and the data transmission between the data centers comprises:
and creating data flow, namely a data transmission pipeline, of corresponding data transmission according to the requirements of users, and synchronously transmitting data between data centers of different or same regions and sites.
Further, the performing data transmission among the data centers further includes:
the data transmission rate control may be performed according to the number of files or the size of data amount transmitted per unit time.
Further, the data synchronous transmission method of the allopatric data center further comprises the following steps:
extracting specific extracted characteristic attributes and variables of a certain data transmission pipeline, customizing recognizable names, creating a data flow template, and creating a similar data transmission pipeline based on the created data flow template.
Another objective of the present invention is to provide a data synchronous transmission system of a remote data center for implementing the data synchronous transmission method of the remote data center, wherein the data synchronous transmission system of the remote data center comprises:
the system comprises a system monitoring module, a data stream management module, a template management module and a site management module;
the system monitoring module is used for displaying the position of each data center in a national map mode and displaying the data transmission condition of each data center on the map in a connecting line mode; meanwhile, the method is used for displaying the load monitoring condition of the Apache Nifi host around the map and monitoring the running state of the system in real time;
the data stream management module is used for creating a data stream of corresponding data transmission according to the requirement of a user and modifying the data stream; meanwhile, the display device is used for displaying the list and the running state of the data stream and outputting a visual view of the data stream;
the template management module is used for creating and managing a data flow template;
and the site management module is used for registering the data center and managing the configuration file of the data center.
Further, the load monitoring condition of the Apache Nifi host computer comprises CPU, memory, disk, network, running time and other related monitoring information.
Further, the data synchronous transmission system of the remote data center further comprises:
the relational database MySQL is used for storing data;
and the memory database Redis is used for caching data.
Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to execute the data synchronous transmission method of the offsite data center.
Another object of the present invention is to provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the processor is enabled to execute the data synchronous transmission method of the remote data center.
Another objective of the present invention is to provide an information data processing terminal, where the information data processing terminal is configured to implement a data synchronous transmission method of the remote data center.
By combining all the technical schemes, the invention has the advantages and positive effects that: the invention can manage a plurality of data centers, display the information of the data flow in a list form, create the data flow in a pull-down menu form, and can transmit corresponding data between the two data centers only by filling in a data directory, thereby greatly reducing the learning and using threshold of the user.
The invention can construct data transmission pipelines crossing the data middle, each data stream transmission pipeline provides an independent view preview function, and the source and destination information and the middle processing state of the data stream transmission pipelines can be clearly and conveniently checked.
The invention provides a region/site monitoring schematic diagram based on a national map, and a data stream transmission pipeline of cross regions/sites is directly connected on the map by adopting a one-way line, so that the data stream transmission state of the cross regions/sites can be more intuitively displayed.
The invention provides real-time monitoring information of the self state of a data transmission server, which comprises system load, a CPU, a memory, a disk, network I/O and the like, and feeds back the current running state of the system in real time.
The invention realizes the high decoupling of the management and execution of the data stream. The management of the data flow is realized by a WEB interface which is easier to operate and manage, and the creation and classification management entries of the data flow are provided; the execution of the data stream is supported by Apache Nifi, the access of various data source components can be supported, the horizontal and vertical extension of the performance is supported at the same time, and the improvement of the transmission performance and the stability of the data stream can be rapidly realized.
The invention calls RESTAPI of Apache Nifi at the bottom layer to construct a data transmission pipeline, shields the construction and conversion of complex scripts in the traditional data transmission tool at the foreground and the background simultaneously, and is easy to use and maintain.
According to the method, the configuration file of the data center is added on the webpage, the data stream for data transmission across the data center is created, the operation frequency and the data transmission bandwidth of the data stream can be configured in the data stream, the method is simple and easy to use, the learning and use cost of Apache Nifi of a user is shielded, and corresponding data are transmitted according to the user requirements.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained from the drawings without creative efforts.
Fig. 1 is a flowchart of a data synchronous transmission method of a remote data center according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a data synchronous transmission system of a remote data center according to an embodiment of the present invention.
Fig. 3 is an architecture diagram of a data synchronous transmission system of a remote data center according to an embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a data synchronous transmission system of a remote data center according to an embodiment of the present invention;
in the figure: 1. a system monitoring module; 2. a data stream management module; 3. a template management module; 4. and a site management module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In view of the problems in the prior art, the present invention provides a method for synchronously transmitting data in a remote data center, which is described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for synchronously transmitting data in a remote data center provided in the embodiment of the present invention includes the following steps:
s101, registering and managing data sources of a plurality of data centers in different regions or sites; configuring an Apache Nifi for each data center to respectively construct a data transmission network, and transmitting data among the data centers;
s102, displaying a classified list according to different data centers to which data sources belong, and identifying the number of data transmission pipelines which exist and are running at present;
s103, visually displaying the identified data transmission pipeline which currently exists and is running, the processing node of the data transmitted by the data transmission pipeline and the final flow direction of the data.
The data transmission network is constructed by configuring Apache Nifi in each data center, and the data transmission among the data centers comprises the following steps:
and creating data flow, namely a data transmission pipeline, of corresponding data transmission according to the requirements of users, and synchronously transmitting data between data centers of different or same regions and sites.
The data transmission between the data centers provided by the embodiment of the invention further comprises the following steps:
the data transmission rate control may be performed according to the number of files or the size of data amount transmitted per unit time.
The data synchronous transmission method of the remote data center provided by the embodiment of the invention also comprises the following steps:
extracting specific extracted characteristic attributes and variables of a certain data transmission pipeline, customizing recognizable names, creating a data flow template, and creating a similar data transmission pipeline based on the created data flow template.
As shown in fig. 2 to fig. 4, the data synchronous transmission system of the remote data center provided in the embodiment of the present invention includes:
the system comprises a system monitoring module 1, a data stream management module 2, a template management module 3 and a site management module 4;
the system monitoring module 1 is used for displaying the position of each data center in a national map mode and displaying the data transmission condition of each data center on the map in a connecting line mode; meanwhile, the method is used for displaying the load monitoring condition of the Apache Nifi host around the map and monitoring the running state of the system in real time;
the data stream management module 2 is used for creating a data stream of corresponding data transmission according to the user requirement and modifying the data stream; meanwhile, the display device is used for displaying the list and the running state of the data stream and outputting a visual view of the data stream;
the template management module 3 is used for creating and managing a data flow template;
and the site management module 4 is used for registering the data center and managing the configuration file of the data center.
The load monitoring condition of the Apache Nifi host provided by the embodiment of the invention comprises CPU, memory, disk, network, running time and other related monitoring information.
The data synchronous transmission system of the remote data center provided by the embodiment of the invention also comprises:
the relational database MySQL is used for storing data;
and the memory database Redis is used for caching data.
The technical effects of the present invention will be further described with reference to specific embodiments.
Example 1:
the management system realizes a user interface based on a webpage, and the background calls REST API of Apache Nifi to realize the data transmission function. The user can add the configuration file of the data center on the webpage, create the data stream for data transmission across the data center, and can configure the running frequency and the data transmission bandwidth of the data stream in the data stream, so that the method is simple and easy to use, meanwhile, the learning and using cost of the user for Apache Nifi is shielded, and corresponding data is transmitted according to the user requirement.
The management system is divided into several core modules: system monitoring, data stream management, template management, site management, and the like.
And (3) system monitoring: the position of each data center is shown in the form of a national map, if data transmission in operation is carried out between two data centers, a moving connecting line is displayed, and the data transmission condition between all the data centers is clearly shown. Meanwhile, the load monitoring conditions of the Apache Nifi host are respectively displayed on the periphery of the map, the load monitoring conditions comprise monitoring information such as a CPU (central processing unit), a memory, a disk, a network and running time, and the running state of the system is monitored in real time.
Data stream management: and creating and modifying data streams, and creating data streams for corresponding data transmission according to user requirements. Viewing a list and running status of a data stream, and a visual view of the data stream (a complete data stream transmission channel from a data source end to a destination end)
Template management: and (3) creating a template, and only modifying the reserved data centers and the data directories in the template to directly create a data stream and transmit the selected data from one data center to another data center.
Site management: and registering a data center, and uploading a corresponding configuration file for data stream use.
The invention uses Apache Nifi to solve the problems of data source diversity, network security and transmission rate, and develops a management system to solve the problems of management and maintenance. Therefore, only one simple and easy-to-use Web page is provided for the user, and the requirement of data transmission between different places and data centers is met.
Example 2:
1. data processing and distribution system (Apache Nifi)
And configuring an Apache Nifi for each data center to construct a data transmission network, so as to realize data transmission pipelines between the data centers.
2. Management system
A management system based on a webpage is developed, a foreground provides a simple and clear selectable operation interface for a user, a background calls REST API of Apache Nifi, and a transmission pipeline of a data stream is constructed in real time.
The whole technical framework adopts a JAVA development framework, a JAVA security framework, a database and a cache (Springboot + Shiro + MySQL + Redis) framework, uses Log4j as a global Log system, records important Log information, and can find and solve problems on line in time when problems occur.
A JAVA security framework (Shiro) is used as a permission verification system, pages for user control are displayed, and different pages are displayed according to different permissions.
The relational database MySQL is used as a data storage system and is used for storing data.
The memory database Redis provides a high-performance data caching function for the system, and is mainly used for storing data information required by a home page data report.
The scheme realizes a visual website system presented for use, registers and manages data sources of a plurality of data centers of different regions or sites, can quickly establish a data transmission pipeline through simple text box filling or drop-down box selection, synchronously transmits data between the data centers of different or same regions and sites, and can control the transmission rate according to the number of files or data volume transmitted in unit time.
And performing classified list display according to different data centers to which the data sources belong, quickly identifying the number of data transmission pipelines existing and running in the current system, wherein each data transmission pipeline can display a visual state and clearly display each processing node and the final flow direction of the data.
A data flow template is created according to a specific certain data transmission pipeline, characteristic attributes and variables are extracted, an identifiable name is defined by a user, a similar data transmission pipeline can be rapidly created according to the template, and the access threshold of the system is reduced.
The above description is only for the purpose of illustrating the present invention and the appended claims are not to be construed as limiting the scope of the invention, which is intended to cover all modifications, equivalents and improvements that are within the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A data synchronous transmission method of a remote data center is characterized in that the data synchronous transmission method of the remote data center comprises the following steps:
registering and managing data sources of a plurality of data centers of different regions or sites; configuring an Apache Nifi for each data center to respectively construct a data transmission network, and transmitting data among the data centers;
displaying a classified list according to different data centers to which the data sources belong, and identifying the number of data transmission pipelines which exist and are running at present;
and visually displaying the identified data transmission pipelines which exist and are running currently and the processing nodes and the final flow direction of the data transmitted by the data transmission pipelines.
2. The method of claim 1, wherein the data centers are configured with Apache Nifi to construct a data transmission network, and the data transmission between the data centers comprises:
and creating data flow, namely a data transmission pipeline, of corresponding data transmission according to the requirements of users, and synchronously transmitting data between data centers of different or same regions and sites.
3. The method for synchronously transmitting data in a displaced data center according to claim 1, wherein the performing of data transmission between the data centers further comprises:
and controlling the data transmission rate according to the number of the transmitted files or the size of the data volume in unit time.
4. The method of synchronized data transfer in a data center off-site of claim 1, wherein the method of synchronized data transfer in a data center off-site further comprises:
extracting specific extracted characteristic attributes and variables of a certain data transmission pipeline, customizing recognizable names, creating a data flow template, and creating a similar data transmission pipeline based on the created data flow template.
5. A data synchronous transmission system of a data center at different places is characterized in that the data synchronous transmission system of the data center at different places comprises:
the system monitoring module is used for displaying the position of each data center in a national map mode and displaying the data transmission condition of each data center on the map in a connecting line mode; meanwhile, the method is used for displaying the load monitoring condition of the Apache Nifi host around the map and monitoring the running state of the system in real time;
the data stream management module is used for creating a data stream of corresponding data transmission according to the requirement of a user and modifying the data stream; meanwhile, the display device is used for displaying the list and the running state of the data stream and outputting a visual view of the data stream;
the template management module is used for creating and managing a data flow template;
and the site management module is used for registering the data center and managing the configuration file of the data center.
6. The data synchronous transmission system of the offsite data center according to claim 5, wherein the load monitoring condition of the Apache Nifi host includes CPU, memory, disk, network, running time and other related monitoring information.
7. The data synchronization transmission system of a offsite data center according to claim 5, wherein the data synchronization transmission system of the offsite data center further comprises:
the relational database MySQL is used for storing data;
and the memory database Redis is used for caching data.
8. A computer arrangement, characterized in that the computer arrangement comprises a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the method of data synchronized transfer of a displaced data center according to any one of claims 1 to 4.
9. A computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to execute the method for data synchronous transmission of a displaced data center according to any one of claims 1 to 4.
10. An information data processing terminal, characterized in that, the information data processing terminal is used to implement the data synchronous transmission method of the allopatric data center according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011587500.1A CN112637356A (en) | 2020-12-28 | 2020-12-28 | Data synchronous transmission method, system, medium and terminal of remote data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011587500.1A CN112637356A (en) | 2020-12-28 | 2020-12-28 | Data synchronous transmission method, system, medium and terminal of remote data center |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112637356A true CN112637356A (en) | 2021-04-09 |
Family
ID=75285778
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011587500.1A Pending CN112637356A (en) | 2020-12-28 | 2020-12-28 | Data synchronous transmission method, system, medium and terminal of remote data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112637356A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107302583A (en) * | 2017-07-10 | 2017-10-27 | 湖南智慧畅行交通科技有限公司 | A kind of automation operation management method that data integration is realized based on Kylo |
| CN108154341A (en) * | 2017-12-18 | 2018-06-12 | 千寻位置网络有限公司 | United Dispatching platform and method of work based on data flow and workflow |
| US20180254989A1 (en) * | 2015-08-28 | 2018-09-06 | Softnas Operating Inc. | Automated data flows using flow-based data processor blocks |
| CN109376153A (en) * | 2018-09-27 | 2019-02-22 | 浪潮软件集团有限公司 | System and method for writing data into graph database based on NiFi |
| CN109634652A (en) * | 2018-11-28 | 2019-04-16 | 郑州云海信息技术有限公司 | A kind of method, apparatus of data processing, computer storage medium and terminal |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
-
2020
- 2020-12-28 CN CN202011587500.1A patent/CN112637356A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180254989A1 (en) * | 2015-08-28 | 2018-09-06 | Softnas Operating Inc. | Automated data flows using flow-based data processor blocks |
| CN107302583A (en) * | 2017-07-10 | 2017-10-27 | 湖南智慧畅行交通科技有限公司 | A kind of automation operation management method that data integration is realized based on Kylo |
| CN108154341A (en) * | 2017-12-18 | 2018-06-12 | 千寻位置网络有限公司 | United Dispatching platform and method of work based on data flow and workflow |
| CN109376153A (en) * | 2018-09-27 | 2019-02-22 | 浪潮软件集团有限公司 | System and method for writing data into graph database based on NiFi |
| CN109634652A (en) * | 2018-11-28 | 2019-04-16 | 郑州云海信息技术有限公司 | A kind of method, apparatus of data processing, computer storage medium and terminal |
| CN110232085A (en) * | 2019-04-30 | 2019-09-13 | 中国科学院计算机网络信息中心 | A kind of method of combination and system of big data ETL task |
Non-Patent Citations (2)
| Title |
|---|
| VICTOR的博客: "Apache NiFi用户指南", 《HTTPS://WWW.CNBLOGS.COM/JOBBIBLE/P/12400784.HTML》 * |
| 引擎平台: "5分钟，用NiFi自动传输系统间的数据流", 《CSDN》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021017301A1 (en) | Management method and apparatus based on kubernetes cluster, and computer-readable storage medium | |
| Kristiani et al. | The implementation of a cloud-edge computing architecture using OpenStack and Kubernetes for air quality monitoring application | |
| CN112069265B (en) | Synchronization method of configuration data, business data system, computer system and medium | |
| US8634330B2 (en) | Inter-cluster communications technique for event and health status communications | |
| CN109819048B (en) | Data synchronization method, device, terminal and storage medium | |
| CN113709810B (en) | Method, equipment and medium for configuring network service quality | |
| CN107404417A (en) | A kind of processing method of monitoring data, processing unit and processing system | |
| CN111274223A (en) | One-key deployment big data and deep learning container cloud platform and construction method thereof | |
| CN110532322A (en) | O&M exchange method, system, computer readable storage medium and equipment | |
| CN102932431A (en) | Message-routing-based state monitoring data interface configuration method | |
| JP2021179994A (en) | Method for acquiring page resources, device, electronic apparatus, and readable storage medium | |
| WO2014153359A1 (en) | Systems and methods for scalable network monitoring | |
| CN110476154B (en) | Proxy server apparatus and method for data collection | |
| CN113542074A (en) | Method and system for visually managing east-west network traffic of kubernets cluster | |
| CN105530289A (en) | Platform and method for seamlessly integrating joint airport heterogeneous business systems | |
| EP4062624B1 (en) | Managed data export to a remote network from edge devices | |
| CN108063779A (en) | A kind of cloud storage method and system synchronous with realization is locally stored | |
| Calderon et al. | Monitoring Framework for the Performance Evaluation of an IoT Platform with Elasticsearch and Apache Kafka | |
| CN102769495B (en) | A kind of optical fiber access network equipment communication means, Apparatus and system | |
| KR20160110585A (en) | Method for Defining Quality of Things based on DDS in Internet of Things | |
| CN112637356A (en) | Data synchronous transmission method, system, medium and terminal of remote data center | |
| CN110020271A (en) | Method and system for cache management | |
| CN114756301B (en) | Log processing method, device and system | |
| JP2021513137A (en) | Data migration in a tiered storage management system | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210409 |
|
| RJ01 | Rejection of invention patent application after publication |