CN113259424A - Cross-regional data transmission method and system - Google Patents
Cross-regional data transmission method and system Download PDFInfo
- Publication number
- CN113259424A CN113259424A CN202110475989.1A CN202110475989A CN113259424A CN 113259424 A CN113259424 A CN 113259424A CN 202110475989 A CN202110475989 A CN 202110475989A CN 113259424 A CN113259424 A CN 113259424A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- file
- compressed
- data transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Abstract
The invention provides a trans-regional data transmission method and a trans-regional data transmission system, wherein the method is applied to a data sending end and comprises the following steps: responding to a message data sending request, sending message data, and compressing the message data to obtain a compressed message file; sending the compressed message file; the compressed message file is used for being uploaded to a cloud service object for storage and being downloaded by a second data sending server for data transmission. The invention aims to provide a cross-region data transmission method and a cross-region data transmission system. The invention provides a method which is based on an object storage function in cloud service, is simple to deploy and can effectively solve the problem of trans-regional data transmission.
Description
Technical Field
The invention relates to the technical field of advertisement putting, in particular to a trans-regional data transmission method and a trans-regional data transmission system.
Background
In the internet field, data is becoming the core of all services and applications, and with the rapid development of the internet, more and more service-level applications are serving users worldwide. This means that data for services at these application levels may need to be transmitted to the data center for the service through globally deployed servers. In this service scenario, a huge amount of data needs to be transferred across regions, for example from asian regions to north america and from south america to europe. Such data transmission across a long physical territory, the delay of data transmission is high, the network bandwidth is limited and the connection is relatively unstable.
For such a large amount of trans-regional data transmission, existing solutions generally employ message queue components for data transmission, such as Kafka or rocktmq. The message system is used for trans-regional data transmission, and as shown in fig. 1 to 3, the trans-regional data transmission can be performed mainly in three ways:
1) the message system client in the initial area of data transmission directly sends the message to the message system server in the target area, and the target area server directly completes data transmission by consuming the message on the local message system server.
2) The message system client in the initial area of data transmission directly sends the message to the local message system server, and the target area server completes the data transmission by consuming the message on the initial area message system server.
3) The message system client side in the initial area of data transmission directly sends the message to the local message system server, the initial area message system server and the target area message system server carry out data synchronization, and the target area server directly completes the data transmission by consuming the synchronized message on the local message system server.
In the above mode 1, the messaging system client needs to write data across the region, and the sending pressure of the messaging system client server is too high due to network delay and unstable connection in the cross-region scene.
In the above mode 2, the message system client needs to consume data across regions, and the consumption client server is under excessive pressure due to network delay and unstable connection in a cross-region scene. At the same time, the overall message consumption speed may be slowed.
In the method 3, the message system server needs to be deployed in both the real area and the target area of the data transmission to complete data synchronization, and although the problem of cross-area delay of the sending end and the consuming end is solved, the service cost is greatly increased.
In addition, although the message system data supports data compression, the compression is mainly performed on a single message, and the overall data compression is low, which brings high data transmission cost.
Disclosure of Invention
The invention aims to provide a cross-region data transmission method and a cross-region data transmission system. The invention provides a method which is based on an object storage function in cloud service, is simple to deploy and can effectively solve the problem of trans-regional data transmission.
In order to achieve the purpose, the invention adopts the following technical scheme:
a cross-region data transmission method is applied to a data sending end and comprises the following steps:
in response to the message data transmission request, issuing message data,
compressing the message data to obtain a compressed message file;
sending the compressed message file; the compressed message file is used for being uploaded to a cloud service object for storage and being downloaded by a second data sending server for data transmission.
As a further refinement of the invention, the message data is a kafka message.
As a further improvement of the invention, the compressed message file is uploaded by adopting a segmented uploading method.
As a further improvement of the present invention, after the compressed message file is uploaded, the cloud service object storage triggering message notifies the second data sending server.
As a further improvement of the present invention, the second data sending server obtains a position of the compressed message file in the cloud service object storage by consuming the message queue message of the cloud service, and then downloads the compressed data file.
As a further improvement of the present invention, after the downloading of the compressed data file is completed, the second data transmission server deletes the corresponding message in the message queue.
A cross-region data transmission method is applied to a data receiving end and comprises the following steps:
uploading a compressed message file, wherein the compressed message file is a file which is sent in response to a message data sending request and is obtained by compression;
responding to a downloading request, and downloading the compressed message file;
and receiving the compressed message file and performing data transmission.
A cross-regional data transmission system comprising:
a data transmitting end, comprising:
a first data transmission server for issuing message data in response to a message data transmission request,
a message system server for receiving the message data and sending the message data to a connection system,
the connection system is used for compressing and sending message data;
and a data receiving end comprising:
the cloud service object storage is used for uploading the compressed message file and receiving a downloading request to download the compressed message file;
and the second data sending server is used for receiving the compressed message file and carrying out data transmission.
As a further improvement of the present invention,
the cloud service object storage is also used for triggering a message to inform the second data sending server after the compressed message file is uploaded.
As a further improvement of the present invention,
the message system server is a kafka message system server;
the connecting system is a kafka connect system.
The invention has the beneficial effects that:
the invention uploads the compressed message file by the cloud service object storage arranged at the data receiving end after compressing the message information, and receives the downloading request to download the compressed message file so as to realize cross-region data transmission.
The system of the invention utilizes the kafka message system and the object storage component in the cloud service to realize a trans-regional data transmission method. It has the following advantages:
firstly, the method comprises the following steps: both the data sending end and the data receiving end send/receive data in the area, the data sending delay is low, and the pressure on the server is low.
Secondly, the method comprises the following steps: by means of the characteristic of high throughput of kafka, mass data quasi-real-time transmission can be achieved.
Thirdly, the method comprises the following steps: the kafka message system only needs to be deployed in a data sending area, and the cost is low.
Fourthly: the kafka connect can be used for compressing large batches of data and then uploading the data to an object for storage, the compression ratio is higher, multiple compression algorithms are informed to be supported, the compression ratio can be further improved, and the flow cost of data cross-regional transmission is reduced.
Fifth, the method comprises the following steps: by means of the characteristics of low cost and high availability of cloud service object storage, the transmission data can be stored for a long time so as to deal with data backtracking query or disaster recovery data recovery.
Furthermore, when the data receiving end generates a scene that the data consumption is not timely due to abnormality, the data is mainly accumulated in the object storage, and extra pressure cannot be caused on the message system.
Drawings
Fig. 1 illustrates a first conventional method for transferring data across regions by means of a messaging system.
Fig. 2 shows a second conventional method for transferring data across regions by means of a messaging system.
Fig. 3 shows a third conventional method for transferring data across regions by means of a messaging system.
Fig. 4 is a flow chart of a method for transferring data across regions through kafka and object storage in the invention.
Fig. 5 is a flow chart of a cross-region data transmission method according to the present invention.
Fig. 6 is a flow chart of a cross-region data transmission method according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For the convenience of clearly describing the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", and the like are used to distinguish the same items or similar items with basically the same functions or actions, and those skilled in the art can understand that the words "first", "second", and the like do not limit the quantity and execution order.
The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
The term "comprises/comprising" when used herein refers to the presence of a feature, element or component, but does not preclude the presence or addition of one or more other features, elements or components.
The present invention will be described in further detail with reference to the accompanying drawings and examples. The examples are given solely for the purpose of illustration and are not intended to limit the scope of the invention.
The invention relates to a trans-regional data transmission method based on kafka and object storage, which comprises the following steps:
step 1: and a kafka message system is deployed at a data sender, and the sender sends data to a local kafka message server through a kafka client.
Step 2: uploading kafka data to a cloud service object storage (e.g., S3 of AWS or OSS of aliyun) in the form of a compressed file in the data transmission area, the object storage being deployed in the target area;
and step 3: after the compressed file is uploaded, the object storage triggering message informs a target area data receiving server;
and 4, step 4: and the target area data receiving server downloads the compressed data file from the cloud service object storage to complete data transmission.
As shown in fig. 4, a cross-region data transmission system includes:
a data transmitting end, comprising:
a first data transmission server for sending message data in response to a message data transmission request, a message system server for receiving the message data and transmitting the message data to a connection system,
the connection system is used for compressing and sending message data;
and a data receiving end comprising:
the cloud service object storage is used for uploading the compressed message file and receiving a downloading request to download the compressed message file;
and the second data sending server is used for receiving the compressed message file and transmitting data.
The cloud service object storage is also used for triggering a message to inform the second data sending server after the compressed message file is uploaded.
As a preferred embodiment, the message system server is a kafka message system server; the connecting system is a kafka connect system.
Specifically, Kafka connect is a scalable and reliable means for data transfer between Apache Kafka and other systems. The kafka connection in the invention should include a kafka connection cluster service deployed in the target sending area and a connector component for realizing data transmission between kafka and object storage.
The cloud service components required by the invention include object storage and message queues, which are S3 and SQS in AWS, and OSS and MNS in Alice cloud. And configuring a trigger in a data bucket stored by the object, and after the file transmission is finished, triggering a message queue to inform a data receiving server of a data downloading position so as to finish the data transmission.
The invention realizes a trans-regional data transmission method by utilizing a kafka message system and an object storage component in cloud service. As shown in fig. 4, the following parts are mainly divided:
kafka message system and data transmission
Apache Kafka is an open source messaging system project developed by the Apache software foundation. The Kafka message system is deployed in a data transmission starting area, and the data transmission server transmits data to the Kafka server deployed in the same area through the Kafka client. Because the client and the server are deployed in the same area, low delay and high throughput of data transmission are ensured.
Kafka connect System
Kafka connect is a scalable and reliable means for data transfer between Apache Kafka and other systems. The Kafka connect service is deployed in a real area of data transmission (the same area as the Kafka system service), and a connect component for realizing compressed uploading of Kafka message data to an object storage is required. The component can compress data by adopting different compression algorithms according to the needs of a service scene, and uploads the data to an object storage data bucket in a file compression mode through the api stored in the cloud service object. Because the object is stored in the message receiving area, file uploading is trans-regional transmission, and data can be uploaded by a segmented uploading method provided by the object storage service, so that the data uploading speed and the data integrity are ensured.
3. Object storage and data reception
The cloud object storage service is a massive, safe, low-cost and highly reliable cloud storage service, and is suitable for storing any type of files. The capacity and the processing capacity are flexibly expanded, a plurality of storage types are selected, and the storage cost is comprehensively optimized. The cloud object storage in the present invention is not limited to the object storage service provided by a certain cloud service provider, and may be applied to the present invention, for example, the object storage service provided by Amazon (AWS) and aristoloc. The object storage service used by the present invention should be deployed in the same area as the data receiving service. When the compressed data file is uploaded in a designated data bucket through the kafka connect service, the object storage triggers a message queue notification service of the cloud service through a pre-configured trigger, the SQS service is performed in the AWS, and the MNS service is performed in the Alice cloud. The data receiving server obtains the position of the data file in the cloud object storage through the message queue message of the consumption cloud service, and then the cross-region transmission of the data is completed through downloading the data. After completing data reception, the data reception server needs to delete the corresponding message in the message queue to prevent message duplicate processing.
The invention has application in a variety of product services involving cross-regional transmission of data. By the trans-regional data transmission method provided by the invention, the on-line service can realize efficient low-delay transmission of the data of the daily TB level. In addition, after the method provided by the invention is applied, the cost of the whole data transmission is reduced by more than 50%.
In order to explain the method of the present invention in detail, the following description is made with respect to a receiving side and a transmitting side, respectively.
As shown in fig. 6, a cross-region data transmission method is applied to a data transmitting end, and includes the following steps:
in response to the message data transmission request, issuing message data,
compressing the message data to obtain a compressed message file;
sending the compressed message file; the compressed message file is used for being uploaded to a cloud service object for storage and being downloaded by a second data sending server for data transmission.
As shown in fig. 6, a cross-region data transmission method is applied to a data receiving end, and includes the following steps:
uploading a compressed message file, wherein the compressed message file is a file which is sent in response to a message data sending request and is obtained by compression;
responding to a downloading request, and downloading the compressed message file;
and receiving the compressed message file and performing data transmission.
The message data of the present invention is a kafka message. And uploading the compressed message file by adopting a segmented uploading method. And after the compressed message file is uploaded, the cloud service object stores a trigger message to inform the second data sending server.
And the second data sending server obtains the position of the compressed message file in the cloud service object storage through the message queue message of the consumption cloud service, and then downloads the compressed data file. And after the compressed data file is downloaded, the second data sending server deletes the corresponding message in the message queue.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. A cross-region data transmission method is applied to a data sending end and comprises the following steps:
in response to the message data transmission request, issuing message data,
compressing the message data to obtain a compressed message file;
sending the compressed message file; the compressed message file is used for being uploaded to a cloud service object for storage and being downloaded by a second data sending server for data transmission.
2. The method of claim 1,
the message data is a kafka message.
3. The method of claim 1,
and uploading the compressed message file by adopting a segmented uploading method.
4. The method of claim 1,
and after the compressed message file is uploaded, the cloud service object stores a trigger message to inform the second data sending server.
5. The method of claim 1,
and the second data sending server obtains the position of the compressed message file in the cloud service object storage through the message queue message of the consumption cloud service, and then downloads the compressed data file.
6. The method of claim 1,
and after the compressed data file is downloaded, the second data sending server deletes the corresponding message in the message queue.
7. A cross-region data transmission method is applied to a data receiving end and comprises the following steps:
uploading a compressed message file, wherein the compressed message file is a file which is sent in response to a message data sending request and is obtained by compression;
responding to a downloading request, and downloading the compressed message file;
and receiving the compressed message file and performing data transmission.
8. A cross-regional data transmission system, comprising:
a data transmitting end, comprising:
a first data transmission server for issuing message data in response to a message data transmission request,
a message system server for receiving the message data and sending the message data to a connection system,
the connection system is used for compressing and sending message data;
and a data receiving end comprising:
the cloud service object storage is used for uploading the compressed message file and receiving a downloading request to download the compressed message file;
and the second data sending server is used for receiving the compressed message file and carrying out data transmission.
9. The system of claim 1,
the cloud service object storage is also used for triggering a message to inform the second data sending server after the compressed message file is uploaded.
10. The system of claim 1,
the message system server is a kafka message system server;
the connecting system is a kafka connect system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475989.1A CN113259424A (en) | 2021-04-29 | 2021-04-29 | Cross-regional data transmission method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110475989.1A CN113259424A (en) | 2021-04-29 | 2021-04-29 | Cross-regional data transmission method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113259424A true CN113259424A (en) | 2021-08-13 |
Family
ID=77223649
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110475989.1A Pending CN113259424A (en) | 2021-04-29 | 2021-04-29 | Cross-regional data transmission method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113259424A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991134A (en) * | 2017-03-13 | 2017-07-28 | 人和未来生物科技（长沙）有限公司 | A kind of large data cloud storage method stored based on object |
| CN107071073A (en) * | 2017-06-27 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of document transmission method and device applied to object storage system |
| CN108718345A (en) * | 2018-09-05 | 2018-10-30 | 电子科技大学 | A kind of digitlization workshop industrial data Network Transmitting system |
| CN109325006A (en) * | 2018-08-23 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of method and apparatus for compressing the method and apparatus stored, decompression downloading |
| US20190199832A1 (en) * | 2017-12-27 | 2019-06-27 | Fujitsu Limited | Data transmitting program, data transmitting device, and data transmitting method |
| CN111858099A (en) * | 2020-07-24 | 2020-10-30 | 海尔优家智能科技（北京）有限公司 | Message subscription method and device |
| CN112084154A (en) * | 2020-09-01 | 2020-12-15 | 浙江工业大学 | Cross-platform multi-host joint log compression method |
-
2021
- 2021-04-29 CN CN202110475989.1A patent/CN113259424A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991134A (en) * | 2017-03-13 | 2017-07-28 | 人和未来生物科技（长沙）有限公司 | A kind of large data cloud storage method stored based on object |
| CN107071073A (en) * | 2017-06-27 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of document transmission method and device applied to object storage system |
| US20190199832A1 (en) * | 2017-12-27 | 2019-06-27 | Fujitsu Limited | Data transmitting program, data transmitting device, and data transmitting method |
| CN109325006A (en) * | 2018-08-23 | 2019-02-12 | 郑州云海信息技术有限公司 | A kind of method and apparatus for compressing the method and apparatus stored, decompression downloading |
| CN108718345A (en) * | 2018-09-05 | 2018-10-30 | 电子科技大学 | A kind of digitlization workshop industrial data Network Transmitting system |
| CN111858099A (en) * | 2020-07-24 | 2020-10-30 | 海尔优家智能科技（北京）有限公司 | Message subscription method and device |
| CN112084154A (en) * | 2020-09-01 | 2020-12-15 | 浙江工业大学 | Cross-platform multi-host joint log compression method |
Non-Patent Citations (1)
| Title |
|---|
| 王晓薇等: "基于SpringMVC的图片处理系统的研究与设计", 《电子设计工程》 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108958922B (en) | Method and device for executing task | |
| CN110703980B (en) | File transmission method and device | |
| CN104918072A (en) | Low latency live video streaming | |
| CN105095364A (en) | Data synchronizing system and method | |
| CN105025391A (en) | Video playing method and apparatus | |
| CN102111437A (en) | Simple object access protocol (SOAP) extension-based WEB service transmission compression method | |
| CN111431730B (en) | Service processing method, system, computer equipment and readable medium | |
| CN113391979A (en) | Processing method, equipment and system for monitoring data display and storage medium | |
| CN110572422A (en) | Data downloading method and device | |
| CN111541555A (en) | Group chat optimization method and related product | |
| CN103547339A (en) | File uploading method, client side and server side | |
| US11647103B1 (en) | Compression-as-a-service for data transmissions | |
| CN111491037A (en) | Communication method with object storage server through SFTP data stream | |
| CN110582998A (en) | Group communication method and device, computer storage medium and computer equipment | |
| CN108614820B (en) | Method and device for realizing streaming source data analysis | |
| CN104092754A (en) | File storage system and method | |
| CN110233791B (en) | Data deduplication method and device | |
| CN113259424A (en) | Cross-regional data transmission method and system | |
| CN101958934B (en) | Electronic program guide incremental content synchronization method, device and system | |
| US20130227162A1 (en) | Management of Data Upload Speed | |
| CN104602047B (en) | Sharing method, server, terminal and the system of EPG data | |
| CN103731511A (en) | Data acquiring method and data acquiring device in P2P (Peer-to-Peer) system | |
| CN107302551B (en) | Service data publishing method and device | |
| CN107071038A (en) | A kind of method and device of transmitting multimedia data | |
| CN113114662A (en) | Method and device for processing concurrent requests by single TCP (Transmission control protocol) connection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |