CN113132492A - Data exchange system - Google Patents
Data exchange system Download PDFInfo
- Publication number
- CN113132492A CN113132492A CN202110472182.2A CN202110472182A CN113132492A CN 113132492 A CN113132492 A CN 113132492A CN 202110472182 A CN202110472182 A CN 202110472182A CN 113132492 A CN113132492 A CN 113132492A
- Authority
- CN
- China
- Prior art keywords
- data
- data transmission
- transmitted
- module
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005540 biological transmission Effects 0.000 claims abstract description 153
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000012546 transfer Methods 0.000 claims description 18
- 238000013144 data compression Methods 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 4
- 230000006837 decompression Effects 0.000 claims description 4
- 238000012800 visualization Methods 0.000 claims description 4
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 19
- 230000006870 function Effects 0.000 description 10
- 238000007906 compression Methods 0.000 description 5
- 230000006835 compression Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 230000014509 gene expression Effects 0.000 description 4
- 230000000295 complement effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 101150039208 KCNK3 gene Proteins 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a data exchange system, comprising: a source system layer, a data transmission layer and a data receiving layer; the source system layer is used for generating a configuration file and data to be transmitted based on preset source data and sending the data to be transmitted and the configuration file to the data transmission layer; the data transmission layer is used for reading the configuration file, generating a data transmission task, and executing the data transmission task to send data to be transmitted to the data receiving layer; the data receiving layer is used for storing the data to be transmitted. The invention integrates the data exchange system into a complete system, improves the efficiency of the data exchange process, and ensures that the data exchange system can not influence the data source system and the data receiving system when the function is adjusted.
Description
Technical Field
The embodiment of the invention relates to the field of data exchange, in particular to a data exchange system.
Background
In the prior art, a data exchange process needs to integrate distributed application information systems, and information/data transmission and sharing among the application systems are realized through an information exchange system constructed by a computer network. The existing data exchange system has the functions of integrating protocol conversion, encryption, compression, exchange process monitoring and the like, and ensures effective exchange of data among the systems.
The existing data exchange process may involve a function adjustment by adjusting the applications connected by the switching system. The method has certain influence on a data source system and a data target system, so that the integration of the systems cannot be realized in the data exchange process. And meanwhile, data exchange among multiple systems cannot be supported.
Disclosure of Invention
The invention provides a data exchange system, which integrates the data exchange system into a complete system, improves the efficiency of the data exchange process, and ensures that the data exchange system can not influence a data source system and a data receiving system when the function is adjusted.
The present invention provides a data exchange system, comprising:
a source system layer, a data transmission layer and a data receiving layer;
the source system layer is used for generating a configuration file and data to be transmitted based on preset source data and sending the data to be transmitted and the configuration file to the data transmission layer;
the data transmission layer is used for reading the configuration file, generating a data transmission task, and executing the data transmission task to send data to be transmitted to the data receiving layer;
the data receiving layer is used for storing the data to be transmitted.
Further, the source system layer comprises a data acquisition module and a data configuration module, wherein the data acquisition module is used for acquiring source data, and the data configuration module is used for generating a configuration file and data to be transmitted based on the source data and sending the data to be transmitted and the configuration file to the data transmission layer.
Further, the data transmission layer includes a scanning module and a data transmission module, the scanning module is used for the data transmission module to read the configuration file and generate a data transmission task, and the data transmission module is used for executing the data transmission task to send the data to be transmitted to the data receiving layer.
Further, the data transmission layer further comprises a queue configuration module, wherein the queue configuration module is used for acquiring a setting period input by a user after reading the configuration file and generating a data transmission task, and taking the setting period as the data transmission period;
and/or
Acquiring data transmission priority input by a user;
sorting the data transmission tasks based on the priority to generate a sorting result;
and sequentially executing the data transmission tasks based on the sequencing result.
Further, the configuration file comprises source data information and data transmission information,
the source data information comprises a data source type, a data source address, a data source port and/or a data source user;
the data transmission information includes: data transmission paths, data storage periods and/or rules of data to be transmitted.
Further, the data transmission layer further comprises a data encryption module, which is used for encrypting the data to be transmitted based on the configuration file and sending the encrypted data to be transmitted to the data receiving layer;
the data receiving layer also comprises a data decryption module which is used for acquiring the encrypted data to be transmitted and decrypting the encrypted data.
Furthermore, the data transmission layer further comprises a data compression module, which is used for compressing the data to be transmitted based on the configuration file and sending the compressed data to be transmitted to the data receiving layer;
the data receiving layer further comprises a data decompression module used for obtaining the compressed data to be transmitted and decompressing the compressed data.
Further, still include: and the visualization module is used for visually displaying data transmission, a data transmission queue, a task execution state, an execution result and/or resource consumption information.
Further, the data transmission layer further comprises a queue configuration module, and the queue configuration module is used for preferentially allocating computing resources to the data transmission queue with the highest priority when one or more data transmission queues wait for executing data transmission tasks.
Further, the source system layer further comprises a data interface module for configuring the data interface to obtain the source data from different databases.
The invention integrates the data exchange system into a complete system, improves the efficiency of the data exchange process, and ensures that the data exchange system can not influence the data source system and the data receiving system when the function is adjusted.
Drawings
Fig. 1 is a block diagram of a data exchange system according to a first embodiment.
Fig. 2 is a block diagram of a data exchange system according to a second embodiment.
Fig. 3 shows the data transmission task recording representation intention of the second embodiment;
fig. 4 shows the data transmission task recording representation intention of the second embodiment.
Fig. 5 is a schematic diagram of the priority of the data transmission task in the second embodiment.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
Furthermore, the terms "first," "second," and the like may be used herein to describe various orientations, actions, steps, elements, or the like, but the orientations, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, the first feature information may be the second feature information or the third feature information, and similarly, the second feature information and the third feature information may be the first feature information without departing from the scope of the present application. The first characteristic information, the second characteristic information and the third characteristic information are characteristic information of the distributed file system, but are not the same characteristic information. The terms "first", "second", etc. are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "plurality", "batch" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Example one
The present embodiment provides a data transmission system, which is adapted to transmit data in a source database to a target database. The function of the embodiment is to realize the function of the data exchange system for exchanging data between different systems. Specifically, as shown in fig. 1, the data exchange system of the present invention includes:
a source system layer 100, a data transmission layer 200, a data reception layer 300;
the source system layer 100 is configured to generate a configuration file and data to be transmitted based on preset source data, and send the data to be transmitted and the configuration file to the data transmission layer 200.
The source of the source data includes but is not limited to various mainstream relational databases including Oracle, MySQL, SQL Server, and also includes various big data platforms including Spark, Hadoop, and ODPS. In the background of practical application, the data exchange tasks including data exchange requirements of systems among different departments in an enterprise, data exchange requirements of systems among three levels of departments of province, city and county, and data real-time synchronization between a main database and a standby database are supported.
The data transmission layer 200 is configured to read the configuration file, generate a data transmission task, and execute the data transmission task to send data to be transmitted to the data receiving layer 300; the data receiving layer 300 is configured to store the data to be transmitted.
The embodiment realizes a modularized data exchange system, ensures effective exchange of data between systems, and improves data exchange efficiency.
Example two
The present embodiment provides a data transmission system, which is adapted to transmit data in a source database to a target database. The function of the embodiment is to realize the function of the data exchange system for exchanging data between different systems.
Specifically, as shown in fig. 2, the data exchange system of the present invention includes:
a source system layer 100, a data transmission layer 200, a data reception layer 300;
the source system layer 100 is configured to generate a configuration file and data to be transmitted based on preset source data, and send the data to be transmitted and the configuration file to the data transmission layer 200;
the data transmission layer 200 is configured to read the configuration file, generate a data transmission task, and execute the data transmission task to send data to be transmitted to the data receiving layer 300;
the data receiving layer 300 is configured to store the data to be transmitted.
The source system layer 100 includes a data acquisition module 110 and a data configuration module 120, where the data acquisition module 110 is configured to acquire source data, and the data configuration module 120 is configured to generate a configuration file and data to be transmitted based on the source data, and send the data to be transmitted and the configuration file to the data transmission layer 200.
The data transmission layer 200 includes a scanning module 210 and a data transmission module 220, where the scanning module 210 is used by the data transmission module 220 to read the configuration file and generate a data transmission task, and the data transmission module 220 is used to execute the data transmission task to send data to be transmitted to the data reception layer 300.
In actual implementation, the source system layer 100 downloads offline data (i.e., preset source data) from an offline database while generating a configuration file, and the downloading process is performed in the form of a download task. The source data may be a raw data stream or a large data stream, and may also refer to data obtained from a system database before switching. The data to be transmitted is data that needs to be exchanged from the source system layer 100 to the data receiving layer 300, and may be source data or processed data generated after the source data is subjected to operations such as deduplication, screening, and data cleansing.
The configuration file is used for describing the downloading task so as to determine information such as data sources of the source data. The configuration file comprises source data information and data transmission information, wherein the source data information comprises a data source type, a data source address, a data source port and/or a data source user; the data transmission information includes: data transmission paths, data storage periods and/or rules of data to be transmitted.
For example, the configuration file is used to describe the type of data source (e.g., ftp, sftp, hdfs), the source address of the data (e.g., IP information, host information), the source port of the data (e.g., source transfer access port), the user of the data source (e.g., source transfer access user, and password). For example, one implementation of a code that describes port information, username and password, host information is as follows:
the configuration file is also used for describing data transmission information, for example, the configuration file is used for describing data transmission paths (such as data source paths, data transmission destination paths, paths which can be configured according to regular expressions and meet the requirements), data storage periods (data destination storage time periods, data transmission periods configured according to cron expressions and the like), and data rules to be transmitted (file names which can be configured according to regular expressions and the like). For example, the code for implementing the profile described above is as follows:
the configuration file may also describe other configuration information, such as file size filtering: by setting a threshold, the small filter file is not transmitted. File compression and transmission: and the compression type supports tar, gz and zip. File merging and transmission: txt, csv, etc. are supported.
It should be noted that, the scheduling mechanism of the scanning module 210 in the scanning process may perform task allocation according to the status and priority of the task, and the allocation mechanism is randomly and uniformly allocated to each driver node. Therefore, the module supports distributed scanning files, if a user adds or deletes a deliverer node, modification operation can be carried out on a page, and after the system receives the change of the number of the deliverer nodes, the system can automatically and self-adaptively carry out task allocation according to the latest deliverer configuration table. The principle of the data transmission process is similar to scanning, and the file to be transmitted is stored in the message queue to be transmitted and is randomly and uniformly distributed in each work node according to the priority. The transmission tasks can be adaptively distributed according to the nodes added and deleted by the user. The resources used by the Deliver and the work nodes are different, the deliverer mainly uses the cup computing power to scan the file, and the work mainly uses the memory and the IO port to process and transmit the file, so the two nodes can multiplex the same server. Thereby saving the number of servers used.
The data transmission layer 200 generates a data transmission task and a data transmission task by reading the configuration file describing the configuration information. And simultaneously stores the data exchange information of the data transmission task in the form of message records in the database of the data transmission layer 200. Optionally, the data transfer task in this step may set a timed commit.
Specifically, in an embodiment, the data transmission task is divided into two tables for recording, one table is hs _ drp _ access data (for example, fig. 3 is an example of a table of a certain transmission process) for describing data information to be exchanged in the data transmission task, and the other table is hs _ drp _ batchjobb (for example, fig. 4 is an example of a table of a certain transmission process) for mainly describing some information of the scheduling task itself in the batch timer exchange process.
In executing the data transfer task, for example, a data transfer cycle may be set to generate the data transfer task. In this process, optionally, the following steps are further included:
and acquiring a setting period input by a user, and taking the setting period as the data transmission period.
Optionally, the data transmission layer 200 further includes a queue configuration module 230, where the queue configuration module 230 is configured to: after the configuration file is read and a data transmission task is generated, acquiring a setting period input by a user, and taking the setting period as the data transmission period; and/or acquiring the data transmission priority input by the user; sorting the data transmission tasks based on the priority to generate a sorting result; and sequentially executing the data transmission tasks based on the sequencing result.
Illustratively, as shown in fig. 5, Task1 transmits every 60min, and generates 24 tasks each day after instantiation, and completes 24 transmissions. Similarly, task2 generates 48 data transfer tasks, task3 generates 96 data transfer tasks, task4 generates 144, and task5 generates 288 data transfer tasks.
At the point in time, the data transfer tasks of task1, task2, task3, task4 and task5 will all enter the ready state. Optionally, the system determines whether the task execution resources are sufficient, and if the task execution resources are sufficient, calls and executes one or more data transmission tasks in a ready state at the same time; and if the task execution resources are insufficient, preferentially calling and executing the task with the high priority based on the preset priority.
The queue configuration module 230 is configured to preferentially allocate computing resources to a data transmission queue with the highest priority when one or more data transmission queues wait to execute a data transmission task;
if the computing resources are left, executing data transmission tasks in other data transmission queues in sequence from high to low according to the priority; and if the computing resources are insufficient, after all the data transmission tasks in the data transmission queue with the highest priority are executed, executing the data transmission tasks in other data transmission queues in sequence from high to low according to the priority.
The process specifically comprises the following steps: the link source server downloads data to be transmitted in the data transmission tasks of the data transmission queue to the work memory in a stream form; encrypting and compressing the data to be transmitted based on configuration information (such as data encryption and data transmission) in the configuration file; based on the configuration information (such as the address of the destination server) in the configuration file, the processed data is pushed to the path of the specified destination server according to the configuration.
Optionally, the data transmission layer 200 further includes an exception handling module 240, configured to, when the task fails to be executed, determine whether the failed data transmission task has transmitted data, and if so, delete the transmitted data; and recalling and executing the data transmission task, and judging whether the task execution fails.
For example, when a data transfer task fails, if the data transfer task still fails after being retried n times (n is set by the user, for example, n is 3), the data transfer task is set to a failure state and is not called and executed.
Specifically, when a system which acquires a user provides a task re-run data complementary collection function, when the user initiates complementary collection, only the completed data transmission task needs to be re-started on an interface, the system scans the data which is specified in a destination address and is transmitted by the re-run task, compression backup processing is carried out on the old data, then file transmission is carried out from a source address again, and the backed-up files can be automatically recovered and deleted according to life cycle management. The rescheduled task can repeatedly scan and transmit the whole step, the priority of the re-run complementary acquisition task can inherit the priority of the original data transmission task by default, and the priority can be adjusted and reset.
In the process, because the user can set the task priority by himself, when data exchange among multiple data sources occurs, for example, data exchange is performed among different system databases among different departments in an enterprise, and data exchange of systems among three levels of departments of province, city and county, and the like.
Considering that when a plurality of data transmission tasks enter a ready state at the same time in the system, the priorities of the data transmission tasks are classified, and the task priorities are allocated according to the task importance, wherein the task in the first priority queue is allocated with a deliverer to scan files to be transmitted preferentially, and when residual resources exist, the task in the second priority queue enters an execution state and starts to be allocated with the deliverer to scan the files to be transmitted.
Specifically, when a data transmission task is allocated to each delivery device, the working state of the delivery device is preferentially acquired, and if the system has no idle delivery device, the data transmission task is stopped to be allocated until the idle delivery device exists.
When the file to be transmitted is scanned, the data transmission layer 200 determines information such as the data source attribute of the data transmission task by reading the configuration file, so as to determine the current file to be transmitted and place the current file to be transmitted in the message queue to be transmitted for transmission.
In an embodiment, optionally, during the data transmission process, the source data is acquired from the source system layer 100 based on the source data information in the configuration file; generating a configuration file based on source data, wherein the configuration file comprises source data information, and the source data information comprises information for describing data date, data time, file name, file size and file generation time;
generating a data transmission task based on a data source directory in the configuration file, filtering and screening the data transmission task based on source data information (illustratively, file basic attributes such as file name regular expressions, file names, file sizes, file generation time and the like, file types) and data transmission information (such as transmission task priority) in the configuration file, and storing the data transmission task meeting the transmission condition into a data transmission queue, or storing the data transmission task meeting the transmission condition into different types of data transmission queues based on the classification result.
In an alternative embodiment, the data transmission layer 200 further includes a data encryption module 250, configured to encrypt or compress the file to be transmitted based on the configuration file. Specifically, the data encryption module 250 is configured to encrypt the data to be transmitted based on the configuration file, and send the encrypted data to be transmitted to the data receiving layer 300; the data transmission layer 200 further includes a data compression module 260, configured to compress the data to be transmitted based on the configuration file, and send the compressed data to be transmitted to the data reception layer 300; by adding the data encryption module 250 and the data decryption module 310, the scheme can be applied to the encryption requirements when multi-party data exchange occurs, for example, when a door system of a government department performs data transmission, such as a civil department transmits household registration information data of a certain person to other departments, and since the household registration information data relates to the personal privacy of residents, in order to avoid data leakage, detail sensitive data needs to be hidden in the data transmission process.
The data receiving layer 300 further includes a data decryption module 310, configured to obtain the encrypted data to be transmitted and decrypt the encrypted data. The data receiving layer 300 further includes a data decompression module 320, configured to obtain and decompress the compressed data to be transmitted. In this step, exemplarily, data to be transmitted is compressed based on the compression format described by the configuration file, and when the data is transmitted to the data receiving layer 300, the data to be transmitted is written into a destination server path in the form of a file stream. Through the data compression module 260 and the data decompression module 320, the data transmission process can be more rapid.
Optionally, the data exchange system further includes a visualization module 400, where the visualization module 400 is configured to visually display and monitor information such as data transmission, data transmission queues, task execution states, execution results, and resource consumption. The user can monitor in real time in the data exchange process, and the user experience is improved.
The embodiment realizes a modularized data exchange system, ensures effective exchange of data between systems, and improves data exchange efficiency.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data exchange system, comprising:
a source system layer, a data transmission layer and a data receiving layer;
the source system layer is used for generating a configuration file and data to be transmitted based on preset source data and sending the data to be transmitted and the configuration file to the data transmission layer;
the data transmission layer is used for reading the configuration file, generating a data transmission task, and executing the data transmission task to send data to be transmitted to the data receiving layer;
the data receiving layer is used for storing the data to be transmitted.
2. The data exchange system according to claim 1, wherein the source system layer comprises a data acquisition module and a data configuration module, wherein the data acquisition module is configured to acquire source data, and the data configuration module is configured to generate a configuration file and data to be transmitted based on the source data, and send the data to be transmitted and the configuration file to the data transmission layer.
3. The data exchange system according to claim 1, wherein the data transmission layer comprises a scanning module and a data transmission module, the scanning module is used for the data transmission module to read the configuration file and generate a data transmission task, and the data transmission module is used for executing the data transmission task to send data to be transmitted to the data receiving layer.
4. The data exchange system according to claim 1, wherein the data transmission layer further comprises a queue configuration module, and the queue configuration module is configured to obtain a setting period input by a user after reading the configuration file and generating a data transmission task, and use the setting period as the data transmission period;
and/or
Acquiring data transmission priority input by a user;
sorting the data transmission tasks based on the priority to generate a sorting result;
and sequentially executing the data transmission tasks based on the sequencing result.
5. A data exchange system according to claim 1, characterized in that said configuration file comprises source data information and data transfer information,
the source data information comprises a data source type, a data source address, a data source port and/or a data source user;
the data transmission information includes: data transmission paths, data storage periods and/or rules of data to be transmitted.
6. The data exchange system according to claim 1, wherein the data transmission layer further comprises a data encryption module, configured to encrypt the data to be transmitted based on a configuration file, and send the encrypted data to be transmitted to the data reception layer;
the data receiving layer also comprises a data decryption module which is used for acquiring the encrypted data to be transmitted and decrypting the encrypted data.
7. The data exchange system according to claim 1, wherein the data transmission layer further comprises a data compression module, configured to compress the data to be transmitted based on a configuration file, and send the compressed data to be transmitted to the data reception layer;
the data receiving layer further comprises a data decompression module used for obtaining the compressed data to be transmitted and decompressing the compressed data.
8. The data exchange system of claim 1, further comprising: and the visualization module is used for visually displaying data transmission, a data transmission queue, a task execution state, an execution result and/or resource consumption information.
9. The method of claim 1, wherein the data transfer layer further comprises a queue configuration module configured to preferentially allocate computational resources to the highest priority data transfer queue when one or more data transfer queues are waiting to perform a data transfer task.
10. The method of claim 1, wherein the source system layer further comprises a data interface module for configuring the data interface to retrieve source data from different databases.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110472182.2A CN113132492A (en) | 2021-04-29 | 2021-04-29 | Data exchange system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110472182.2A CN113132492A (en) | 2021-04-29 | 2021-04-29 | Data exchange system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113132492A true CN113132492A (en) | 2021-07-16 |
Family
ID=76780876
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110472182.2A Pending CN113132492A (en) | 2021-04-29 | 2021-04-29 | Data exchange system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113132492A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109635016A (en) * | 2018-10-16 | 2019-04-16 | 深圳壹账通智能科技有限公司 | Data transfer device, device, equipment and computer readable storage medium |
| US20190289061A1 (en) * | 2012-03-31 | 2019-09-19 | Bmc Software, Inc. | Optimization of path selection for transfers of files |
| CN110311753A (en) * | 2019-06-21 | 2019-10-08 | 杭州飞步科技有限公司 | Communication means, device, node, storage medium and system |
| CN110830567A (en) * | 2019-10-31 | 2020-02-21 | 中国银行股份有限公司 | Data transmission method and device |
-
2021
- 2021-04-29 CN CN202110472182.2A patent/CN113132492A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190289061A1 (en) * | 2012-03-31 | 2019-09-19 | Bmc Software, Inc. | Optimization of path selection for transfers of files |
| CN109635016A (en) * | 2018-10-16 | 2019-04-16 | 深圳壹账通智能科技有限公司 | Data transfer device, device, equipment and computer readable storage medium |
| CN110311753A (en) * | 2019-06-21 | 2019-10-08 | 杭州飞步科技有限公司 | Communication means, device, node, storage medium and system |
| CN110830567A (en) * | 2019-10-31 | 2020-02-21 | 中国银行股份有限公司 | Data transmission method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11157324B2 (en) | Partitioning for delayed queues in a distributed network | |
| US20160275123A1 (en) | Pipeline execution of multiple map-reduce jobs | |
| CN109144785B (en) | Method and apparatus for backing up data | |
| US10908834B2 (en) | Load balancing for scalable storage system | |
| CN109194711B (en) | Synchronization method, client, server and medium for organization architecture | |
| US11429566B2 (en) | Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo | |
| WO2019197918A1 (en) | Fault-tolerant federated distributed database | |
| US20190179565A1 (en) | Global occupancy aggregator for global garbage collection scheduling | |
| George et al. | Hadoop MapReduce for mobile clouds | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| JP2004038516A (en) | Work processing system, operation management method and program for performing operation management | |
| US20140304237A1 (en) | Apparatus and Method for Handling Partially Inconsistent States Among Members of a Cluster in an Erratic Storage Network | |
| CN113885780A (en) | Data synchronization method, device, electronic equipment, system and storage medium | |
| CN113434312A (en) | Data blood relationship processing method and device | |
| CN114237892A (en) | Key value data processing method and device, electronic equipment and storage medium | |
| CN113132492A (en) | Data exchange system | |
| US20160359953A1 (en) | Migration enhanced by presence, instant messaging, and last log-on time | |
| US11042454B1 (en) | Restoration of a data source | |
| CN113312316B (en) | Data processing method and device | |
| CN115757270A (en) | File version management system and method and computer equipment | |
| CN113672665A (en) | Data processing method, data acquisition system, electronic device and storage medium | |
| CN112905386A (en) | Table data backup cleaning method and device based on life cycle | |
| CN116954870B (en) | Migration method, recovery method and device of cross-cluster application and cluster system | |
| Yasmin et al. | A constraint programming-based resource allocation and scheduling of map reduce jobs with service level agreement | |
| US20220291998A1 (en) | Backing-up application data from cloud-native applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210716 |
|
| RJ01 | Rejection of invention patent application after publication |