WO2017167119A1 - 数据上传方法、装置及系统 - Google Patents

数据上传方法、装置及系统 Download PDF

Info

Publication number
WO2017167119A1
WO2017167119A1 PCT/CN2017/077912 CN2017077912W WO2017167119A1 WO 2017167119 A1 WO2017167119 A1 WO 2017167119A1 CN 2017077912 W CN2017077912 W CN 2017077912W WO 2017167119 A1 WO2017167119 A1 WO 2017167119A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
idc
uploading
proxy module
client
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/077912
Other languages
English (en)
French (fr)
Inventor
郑振宇
卿银波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Publication of WO2017167119A1 publication Critical patent/WO2017167119A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/289—Intermediate processing functionally located close to the data consumer application, e.g. in same machine, in same home or in same sub-network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements
    • H04L67/2895—Intermediate processing functionally located close to the data provider application, e.g. reverse proxies

Definitions

  • the invention belongs to the field of data processing, and in particular relates to a data uploading method, device and system.
  • the client directly uploads to the IDC: a connection is established between the client and the IDC, and the data is directly uploaded to the IDC through the established connection.
  • FIG. 1 it is a schematic diagram of an existing client directly uploading data to an IDC.
  • the physical distance between the client and the IDC may be relatively long, resulting in a long Round Trip Time (RTT) between the client and the IDC.
  • RTT Round Trip Time
  • the upload throughput is relatively low when the TCP transmission window does not reach the bandwidth delay product (BDP).
  • BDP bandwidth delay product
  • CDN Content Delivery Network
  • the CDN is set between the client and the IDC.
  • the client first uploads the data to be uploaded to the CDN, and the CDN returns a successful upload to the client.
  • FIG. 2 it is a schematic diagram of an existing client uploading data to an IDC through a CDN.
  • the CDN In the CDN transfer mode, the CDN needs to store the uploaded data in the middle. When the server storing the uploaded data is down, data loss may occur. CDN will be connected After the received upload data is stored, the client can receive the message that the CDN returns the upload successfully. At this time, the data remains on the CDN node and does not reach the Internet data center. Moreover, when the server that stores the uploaded data is down, the data is lost. At this time, although the client upload is successful, the uploaded data does not exist.
  • the CDN Since the CDN needs to store data on the ground, there may be data hotspot issues. In practical applications, most of the disk write efficiency is not high, which will increase the total delay of data upload. In order to speed up the forwarding of the uploaded data, the CDN needs to compress or convert the uploaded data. The actual stored data is not the same as the uploaded data, and the data upload takes a long time.
  • the present invention provides a data uploading method, device and system for solving the problem that an existing client has a long time to upload data to an Internet data center IDC.
  • the present invention provides a data uploading method, including:
  • the proxy module receives data uploaded by the client based on the streaming upload technology
  • the agent module uploads the data to an internet data center IDC based on a streaming uploading technology
  • the location where the proxy module is deployed meets a preset condition, where the preset condition is that the minimum time that the client spends uploading data to the IDC based on the proxy module is less than or equal to the client directly to the The minimum amount of time that IDC spends uploading data.
  • the present invention provides a data uploading apparatus, including:
  • a receiving unit configured to receive data uploaded by the client based on the streaming uploading technology
  • Uploading unit configured to upload the data to an Internet data center IDC based on a streaming uploading technology
  • the deployment location of the data uploading device meets a preset condition, where the preset condition is that the minimum time that the client spends uploading data to the IDC based on the data uploading device is less than or equal to the direct directing of the client.
  • the minimum length of time that IDC spends uploading data is less than or equal to the direct directing of the client.
  • the present invention provides a data uploading system, including:
  • the agent module is configured to receive the data uploaded by the client based on a streaming uploading technology, and upload the data to the IDC based on a streaming uploading technology;
  • the IDC is configured to receive the data uploaded by the client forwarded by the proxy module;
  • the location of the proxy module is configured to meet a preset condition, where the preset condition is that the minimum time that the client spends uploading data to the IDC based on the proxy module is less than or equal to the client directly The minimum length of time that the IDC spends uploading data.
  • the data uploading method, device and system provided by the present invention deploys a proxy module by using a proxy module between the client and the Internet data center IDC, and the proxy module respectively transmits data based on the streaming uploading technology with the client and the Internet data center IDC.
  • the proxy module by increasing the proxy module to shorten the round-trip transmission time RTT between the user and the IDC, after the RTT becomes shorter, the data transmission speed is correspondingly increased, and the proxy module uploads the data based on the streaming, and no longer needs to Data is stored and converted to save resources and time, further increasing the speed of uploading.
  • FIG. 1 is a schematic diagram of an existing client directly uploading data to an IDC
  • FIG. 2 is a schematic diagram of an existing client uploading data to an IDC through a CDN
  • FIG. 3 is a schematic flowchart of a data uploading method according to Embodiment 1 of the present invention.
  • FIG. 5 is a second schematic diagram of application of the data uploading method according to Embodiment 1;
  • FIG. 6 is a third schematic diagram of application of the data uploading method according to Embodiment 1;
  • FIG. 7 is a fourth schematic diagram of application of the data uploading method according to Embodiment 1;
  • FIG. 8 is a fifth schematic diagram of application of the data uploading method according to Embodiment 1;
  • FIG. 8 is a fifth schematic diagram of application of the data uploading method according to Embodiment 1;
  • FIG. 9 is a schematic flowchart of a data uploading method according to Embodiment 2 of the present invention.
  • FIG. 10 is a schematic structural diagram of a data uploading apparatus according to Embodiment 3 of the present invention.
  • FIG. 11 is a schematic structural diagram of a data uploading system according to Embodiment 4 of the present invention.
  • FIG. 1 is a schematic flowchart diagram of a data uploading method according to Embodiment 1 of the present invention.
  • the data uploading method includes the following steps:
  • the proxy module receives data uploaded by the client based on the streaming uploading technology.
  • a proxy module is deployed between the client (Client) and the Internet data center IDC, and the client module is used to upload data to the IDC.
  • the proxy module in this embodiment may be a proxy server or a program module (Proxy) that performs proxy functions on other servers.
  • the deployment agent module needs to meet the following preset conditions: the minimum time that the client uploads data to the Internet data center IDC based on the deployed proxy module is less than or equal to the minimum time that the client spends directly uploading data to the Internet data center IDC.
  • the location of the deployment of the proxy module is the location of the proxy module in the transmission link.
  • the RTT between the proxy module and the client is used to represent the transmission distance between the proxy module and the client, and the proxy module and the Internet are used.
  • the RTT between the data center IDCs represents the transmission distance between the proxy module and the Internet data center IDC.
  • the data transmission time required by the client and the Internet data center IDC is the first round-trip transmission time RTT1, and the connection establishment time of the client and the Internet data center IDC is at least one.
  • RTT1 the minimum time taken by the client to upload data directly to the Internet data center IDC is 2RTT1.
  • RTT2 the data transmission time required by the client and the proxy module is the second round-trip transmission time
  • RTT3 the data transmission time required by the proxy module and the Internet data center IDC is The third round trip transmission time is RTT3.
  • connection establishment time between the proxy modules is at least one RTT2, and the minimum time taken by the client to upload data to the Internet data center IDC based on the proxy module is 2RTT1+RTT3.
  • the preset condition is: 2RTT2+RTT3 ⁇ 2RTT1.
  • the client and the proxy module are connected by using a Transmission Control Protocol (TCP), and the proxy module and the Internet data center IDC are connected by a long connection.
  • TCP Transmission Control Protocol
  • the long connection between the proxy module and the Internet data center IDC uses a dedicated line or connection guarantee technology to reduce the packet loss rate between the proxy module and the data network center.
  • the proxy module will reduce the transmission speed. If you want to restore the last transmission rate, you need an RTT time.
  • the shorter the RTT required by the proxy module and the client and the Internet data center IDC the faster the rate recovery between the proxy module and the Internet data center IDC and the client, thereby facilitating the accelerated transmission of data.
  • the client When the client attempts to upload data to the Internet Data Center IDC, the client establishes a TCP connection with the proxy module. When the connection is established, the client sends data to the proxy module based on the streaming upload technology.
  • the proxy module uploads the data to an Internet data center IDC based on a streaming uploading technology.
  • a long connection between the proxy module and the Internet data center IDC After the long connection between the proxy module and the Internet data center IDC is completed, after the proxy module and the Internet data center IDC complete a data read and write, the long connection between them Will continue to maintain, and will not actively close, subsequent read and write operations can continue to use this long connection in the hold state.
  • each time the client requests to upload data to the proxy module it is no longer necessary to re-establish the connection between the proxy module and the Internet data center IDC, thereby saving an RTT required to establish a connection.
  • the proxy module After receiving the data, the proxy module does not perform any processing on the data, and directly uploads the received data to the IDC based on the streaming uploading technology.
  • the uploading data is performed based on the streaming uploading technology, and the proxy module does not need to perform intermediate storage, so that the server that stores the data in the CDN may not be in the event of data loss, and the data is not required to be stored. Avoid data hotspot issues.
  • the proxy module does not need to convert the data to speed down the data and accelerate the transmission speed. Accordingly, the Internet data center IDC can directly store the source data of the client.
  • the Internet data center receives the data accordingly, and the two are synchronized in real time.
  • the data size of the data uploaded by the client to the Internet data center in this embodiment needs to be controlled within a range of 14 KB to 5 MB, and the data in the range is faster based on the uploading by the proxy module.
  • the uploading time is set to rt
  • the initial sending window value of the user system is initwnd
  • the user bandwidth is bandwidth
  • the uploaded data size is filesize
  • the round-trip transmission time of the client to the server where the Internet data center is located is rttOfClientToIDC
  • the client The round-trip transmission time of the server to the proxy module is rttOfClientToProxy
  • the round-trip transmission time of the server where the proxy module is located to the server of the Internet data center is rttOfProxyToIDC
  • the number of rttOfClients that need to be consumed by the send window to full bandwidth is N
  • the bandwidth delay product bdp A logarithmic function log with a base of 2.
  • the sending window can be uploaded at the maximum bandwidth allowed.
  • the uploading time is:
  • the uploading time is:
  • the size of the data that can be transmitted by the window is as follows:
  • the sending window can be uploaded at the maximum bandwidth allowed.
  • the uploading time is:
  • the uploading time is:
  • the uploading time is:
  • the uploading time of different data sizes under common bandwidth can be calculated.
  • 4 to 8 are schematic diagrams showing the uploading time of different data sizes under different bandwidths.
  • the experimental results in the figure are based on the RTT between the proxy module and the client being 5 ms, the RTT of the proxy module to the IDC is 30 ms, and the RTT of the client to the IDC is 30 ms.
  • FIG. 4 is a schematic diagram of the time taken for uploading in different bandwidths when the data size is 10 KB.
  • FIG. 5 is a schematic diagram of the time taken for uploading at different bandwidths when the data size is 50 KB.
  • FIG. 6 is a schematic diagram of the time taken for uploading under different bandwidths when the data size is 100K.
  • FIG. 7 is a schematic diagram of the time taken for uploading in different bandwidths when the data size is 1 MB.
  • FIG. 8 is a schematic diagram of the time taken for uploading in different bandwidths when the data size is 5 MB.
  • the proxy module-based data uploading method provided in this embodiment has no acceleration effect, and has a certain delay compared with the existing direct upload, and the added delay is a round trip from a client to the proxy module. Transmission time.
  • the acceleration effect of the proxy module-based data uploading method provided by this embodiment is more obvious as the bandwidth increases.
  • the acceleration effect will be different under different bandwidths of users and different congestion conditions of the network.
  • packet loss occurs in the network, because the rtt of the proxy module is relatively small, the recovery speed of the transmission window will be faster, so there will be an advantage in the case of unstable network.
  • the R&D staff used the Linux tc tool to simulate the 10% to 30% transmission environment of the packet loss in the common bandwidth. The conclusion is as follows: When the packet loss rate exceeds 30%, the data cannot be successfully uploaded. In the case of packet loss, the test result is random. The performance is too large, but the proxy module-based data uploading method provided by this embodiment has a performance improvement of 30% to 3 times compared with direct uploading.
  • a proxy module is added between the client and the IDC, and the proxy module and the client use a TCP connection to maintain a long connection with the IDC.
  • the client needs to upload a 20KB file and set the transfer data to be 10KB in one RTT.
  • R0 30ms
  • R1 5ms
  • R2 25ms.
  • the RTT of the client to the IDC is R0
  • the RTT of the client to the proxy module is R1
  • the RTT of the proxy module to the IDC is R2.
  • a target node is obtained according to R0, R1, and R2 and a preset condition, and the proxy is set on the target node.
  • the client sends data to the proxy module based on the streaming connection technology through the TCP connection.
  • the sending window size is 10
  • the client needs to send the data through two R1s, and the client needs to establish a TCP connection with the proxy.
  • the client shares 3 R1 from establishing the connection to the data to the proxy.
  • the proxy also needs to send data to the IDC, which consumes an R2.
  • the proxy module adopts the streaming uploading technology between the client and the IDC, respectively, and does not require intermediate storage, and does not require compression conversion processing, which not only saves storage space but also reduces time consumption compared to forwarding through CDN. .
  • Uploading data based on the method provided in this embodiment can save half of the time compared to the direct uploading method, and improve the timeliness of uploading.
  • the agent module is deployed near the client and the Internet data center, and the agent module respectively transmits data based on the streaming uploading technology with the client and the Internet data center IDC.
  • the proxy module by increasing the proxy module to shorten the round-trip transmission time RTT between the user and the IDC, after the RTT becomes shorter, the data transmission speed is increased accordingly, and the proxy module completes data uploading based on the streaming uploading technology, and the proxy The module no longer needs to store and convert the received data, saving resources and time, and further increasing the upload speed.
  • FIG. 9 is a schematic flowchart diagram of a data uploading method according to Embodiment 2 of the present invention.
  • the data uploading method includes the following steps:
  • the node deploying the proxy module can reuse the nodes in the CDN, and the nodes of the CDN are multiplexed. Since the coverage of the CDN is wide, the node deploying the proxy module can simultaneously consider more customers. Machines, because of the large number of CDN nodes, can deploy agent modules in a wide range.
  • the preset conditions refer to the description of related content in the foregoing Embodiment 1, and details are not described herein again.
  • the client and the agent module establish a connection through a TCP connection, and the agent module connects to the Internet data center IDC through a long connection.
  • the number of long connections before the proxy module and the Internet data center IDC is changed, and the proxy module can dynamically adjust the number of long connections according to the bandwidth occupation. When the bandwidth is sufficient, set up multiple long connections to upload data faster. When the bandwidth is insufficient, the number of long connections is reduced accordingly.
  • the proxy module receives data uploaded by the client based on the streaming uploading technology.
  • the proxy module caches the received data.
  • the proxy module may cache the received data, wherein the proxy module caches the data in the memory, and does not need to store the data.
  • the proxy module maintains the sending window by a long connection with the Internet data center IDC to match the available bandwidth, so that the cached data is sent through the sending window at one time.
  • the user can set the initial size of the sending window based on the client.
  • the initial size of the sending window is the default setting, which can be set by the user through the corresponding application.
  • the client As the data continues to be sent continuously, the operating system in the client can continually attempt to increase the send window to resize the send window to match the available bandwidth.
  • the proxy module maintains a transmission window between the proxy module and the Internet data center IDC through a long connection with the Internet data center IDC.
  • the appropriate size the appropriate size can ensure that the proxy module can cache the data through the send window at one time.
  • the transmission window between the proxy module and the network number center IDC can be maintained at a stable size, without the proxy module and the client.
  • the adjustment between the sending windows is constantly changing.
  • the proxy module After the data volume of the cached data reaches a threshold, the proxy module simultaneously uploads the cached data to the IDC.
  • a threshold is preset for the data amount of the buffered data, and the data amount of the buffered data is within the threshold.
  • the cached data is simultaneously uploaded to the Internet data center IDC. in.
  • the amount of data after buffering is less than or equal to the bandwidth delay product (BDP) of the proxy module and the IDC.
  • the proxy module clears the cached data.
  • the proxy module caches the data in the memory, and when the data is sent out, the cached data can be cleared to release the memory space.
  • the proxy module does not need to store the cached data to save resources.
  • the proxy module receives an upload completion indication returned by the IDC.
  • the proxy module forwards the upload completion indication to the client.
  • the upload completion indication is sent by the IDC after receiving all the data uploaded by the proxy module.
  • the client when the client uploads the data to the CDN, it can receive an indication that the upload is completed, but at this time, the Internet data center IDC does not receive the data.
  • the Internet data center after receiving all the data, the Internet data center will generate an upload completion indication and send it to the proxy module. After receiving the upload completion indication, the proxy module returns the upload completion indication to the client.
  • the agent module is deployed in the vicinity of the client and the Internet data center, and the agent module transmits data to the client and the Internet data center IDC based on the streaming uploading technology respectively.
  • the proxy module is used to shorten the round-trip transmission time RTT between the user and the IDC. After the RTT becomes shorter, the data transmission speed is increased accordingly, and the proxy module completes the data upload based on the streaming uploading technology, and the proxy module no longer needs to receive the data. Data conversion saves time and further increases upload speed. Further, the proxy module caches the data in the memory, and simultaneously uploads the cached data to the IDC of the Internet data center, and no longer stores the data, thereby saving resources.
  • FIG. 10 is a schematic structural diagram of a data uploading apparatus according to Embodiment 3 of the present invention.
  • the data uploading device includes: a receiving unit 11 and an uploading unit 12.
  • the receiving unit 11 is configured to receive data uploaded by the client based on the streaming uploading technology.
  • the uploading unit 12 is configured to upload the data to the Internet data center IDC based on a streaming uploading technology.
  • the location where the data uploading device is deployed meets a preset condition, where the preset condition is that the minimum time that the client spends uploading data to the IDC based on the data uploading device is less than or equal to the client directly. The minimum amount of time it takes to upload data to the IDC.
  • the receiving unit 11 and the client are connected by using a TCP method.
  • the uploading unit 12 and the IDC are connected by a long connection.
  • the uploading unit 12 is further configured to dynamically adjust the number of long connections with the IDC.
  • the uploading unit 12 is configured to cache the received data, and after the amount of data of the cached data reaches a threshold, upload the cached data to the IDC at the same time, and clear the data. The data that is cached.
  • the uploading unit 12 is further configured to maintain a size of the sending window by a long connection with the IDC to a size matching the available bandwidth, so that the buffered data is completely sent through the sending window.
  • the amount of data of the cached data is less than or equal to the bandwidth delay product BDP between the data uploading device and the IDC.
  • the receiving unit 11 is further configured to receive an upload completion indication returned by the IDC, where the upload completion indication is sent by the IDC after receiving all data uploaded by the data uploading device.
  • the uploading unit 12 is further configured to forward the upload completion indication to the client.
  • the data uploading device is deployed on a node that is a node in the IDC.
  • the amount of data of the data uploaded by the client to the IDC ranges from 14 KB to 5 MB.
  • a data uploading device is deployed in the vicinity of the client and the Internet data center, and the data uploading device transmits data to the client and the Internet data center IDC based on the streaming uploading technology, respectively.
  • the data uploading device by increasing the data uploading device to shorten the round-trip transmission time RTT between the user and the IDC, after the RTT becomes shorter, the data transmission speed is correspondingly increased, and the data uploading device completes the data uploading based on the streaming uploading technology. , no longer need to receive data conversion, saving time.
  • the data uploading device caches the data in the memory, and simultaneously uploads the cached data to the Internet data center IDC, and no longer stores the data, which not only saves resources, but also improves the uploading speed.
  • FIG. 11 is a schematic structural diagram of a data uploading system according to Embodiment 4 of the present invention.
  • the data uploading system includes a client 1 and a proxy module 2 and an internet data center IDC 3.
  • the client 1 is used to upload data to the Internet data center IDC 3 through the proxy module 2.
  • the proxy module 2 is configured to receive the data uploaded by the client 1 based on the streaming uploading technology, and upload the data to the Internet data center IDC 3 based on the streaming uploading technology.
  • the Internet data center IDC3 is configured to receive the data uploaded by the client 1 forwarded by the proxy module 2.
  • the location of the proxy module 2 is configured to meet a preset condition, where the preset condition is that the minimum time that the client 1 uploads data to the Internet data center IDC 3 based on the proxy module 2 is less than or equal to the direct data of the client 1 to the Internet.
  • the minimum length of time that IDC 3 uploads data is configured to meet a preset condition, where the preset condition is that the minimum time that the client 1 uploads data to the Internet data center IDC 3 based on the proxy module 2 is less than or equal to the direct data of the client 1 to the Internet. The minimum length of time that IDC 3 uploads data.
  • the proxy module 2 is the data uploading device in the third embodiment.
  • the structure of the proxy module 2 refer to the structure of the data uploading device in the foregoing embodiment.
  • the proxy module 2 and the client 1 are connected by TCP, and the Internet data center IDC 3 is connected by a long connection.
  • the proxy module 2 is also used to dynamically adjust the number of long connections between the Internet data center IDC 3.
  • the proxy module 2 is configured to cache the received data, and after the buffered data volume reaches a threshold, upload the cached data to the Internet data center IDC 3, and clear the cached device. Data.
  • the proxy module 2 is specifically configured to maintain the transmission window at a size matching the available bandwidth through a long connection with the Internet data center IDC 3, so that the buffered data is completely transmitted through the sending window.
  • the amount of data of the cached data is less than or equal to the bandwidth delay product BDP between the proxy module 2 and the Internet data center IDC 3.
  • the Internet data center IDC 3 is also used to return an upload completion indication to the proxy module 2; wherein the upload completion indication is sent by the Internet data center IDC 3 after receiving all the data uploaded by the proxy module 2.
  • the proxy module 2 is further configured to receive the upload completion indication returned by the Internet data center IDC 3, and forward the upload completion indication to the client 1.
  • the client 1 is further configured to receive the upload completion indication.
  • the proxy module 2 is deployed on a node that is a node in the Internet data center IDC 3.
  • the data amount of the data uploaded by the client 1 to the Internet data center IDC3 ranges from 14 KB to 5 MB.
  • a proxy module is deployed near the client and the Internet data center, and the proxy module respectively transmits data based on the streaming upload technology with the client and the Internet data center IDC, and completes data upload based on the streaming upload technology.
  • the proxy module by increasing the proxy module to shorten the round-trip transmission time RTT between the user and the IDC, after the RTT becomes shorter, the data transmission speed is correspondingly increased, and the proxy module no longer needs to convert the received data, saving time.
  • the proxy module caches the data in the memory, and simultaneously uploads the cached data to the Internet data center IDC, and no longer stores the data, saves resources, and improves the upload speed.
  • the aforementioned program can be stored in a computer readable storage medium.
  • the program when executed, performs the steps including the foregoing method embodiments; and the foregoing storage medium includes various media that can store program codes, such as a ROM, a RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

本发明提供了一种数据上传方法、装置及系统，通过在客户机与互联网数据中心IDC之间就近部署代理模块，代理模块分别与客户机和互联网数据中心IDC基于流式上传技术传输数据。本发明中，通过增加代理模块来缩短用户到IDC之间的往返传输时间RTT，相应地提高数据的传输速度，而且代理模块基于流式上传技术完成数据的上传，代理模块不再需要对接收到数据进行落地存储和转换，节省资源和时间，进一步提高上传的速度。

Description

数据上传方法、装置及系统 技术领域
本发明属于数据处理领域，尤其涉及一种数据上传方法、装置及系统。
背景技术
目前，用户通过客户机向互联网数据中心(Internet Data Center，简称IDC)上传数据主要有以下两种方式：
(1)客户机向IDC直接上传：客户机与IDC之间建立连接，通过建立的连接向IDC直接上传数据。如图1所示，其为现有的客户机直接向IDC上传数据的示意图。
实际应用中，客户机与IDC之间的物理距离可能比较远，导致客户机与IDC之间的往返传输时间(Round Trip Time，简称RTT)比较长。当RTT比较长时，往往导致客户机与IDC之间建立连接的时候耗时较长，而且处于无线场景下建连的难度更大。而且由于RTT比较长，在TCP发送窗口没有达到带宽时延乘积(Bandwidth Delay Product，简称BDP)情况下，上传吞吐量比较低。进一步地，由于RTT比较长，客户机和IDC之间的路由跳数就会比较多，丢包率较高。
(2)通过内容分发网络(Content Delivery Network，简称CDN)进行中转：CDN设置在客户机与IDC之间，客户机首先将待上传的数据上传到CDN上，CDN向客户机返回一个上传成功的消息，然后CDN将接收到的数据进行压缩或者转换后转发给IDC。如图2所示，其为现有的客户机通过CDN向IDC上传数据的示意图。
在CDN中转方式下，CDN需要对上传的数据进行中间落地存储，当存储上传数据的服务器出现宕机时，可能出现数据丢失的情况。CDN将接 收到的上传数据存储结束后客户机就能接收到CDN返回上传成功的消息，此时，数据还停留在CDN节点上，并未到达互联网数据中心。而且当出现存储上传数据的服务器宕机导致丢失数据时，此时虽然客户机上传成功，但是上传的数据却不存在。
由于CDN需要落地存储数据，可能出现数据热点问题。实际应用中，大部分磁盘写入效率不高，会增加数据上传的总延时。为了加速转发上传的数据，时CDN需要压缩或者转换处理上传的数据，实际存储的数据和上传的数据并不一样，而且导致数据上传的耗时较长。
发明内容
本发明提供一种数据上传方法、装置及系统，用于解决现有客户机向互联网数据中心IDC上传数据时存在耗时较长的问题。
为了实现上述目的，本发明提供了一种数据上传方法，包括：
代理模块接收客户机基于流式上传技术上传的数据；
所述代理模块基于流式上传技术将所述数据上传到互联网数据中心IDC；
其中，所述代理模块部署的位置满足预设条件，其中所述预设条件为：客户机基于所述代理模块向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
为了实现上述目的，本发明提供了一种数据上传装置，包括：
接收单元，用于接收客户机基于流式上传技术上传的数据；
上传单元，用于基于流式上传技术将所述数据上传到互联网数据中心IDC；
其中所述数据上传装置的部署位置满足预设条件，其中所述预设条件为：客户机基于所述数据上传装置向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
为了实现上述目的，本发明提供了一种数据上传系统，包括：
客户机，用于向通过代理模块向互联网数据中心IDC上传数据；
所述代理模块，用于接收所述客户机基于流式上传技术上传的所述数据，以及基于流式上传技术将所述数据上传到所述IDC上；
所述IDC，用于接收所述代理模块转发的所述客户机上传的所述数据；
其中，所述代理模块部署的位置满足预设条件，其中所述预设条件为：所述客户机基于所述代理模块向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
本发明提供的数据上传方法、装置及系统，通过在客户机与互联网数据中心IDC之间就近部署代理模块，代理模块分别与客户机和互联网数据中心IDC基于流式上传技术传输数据。本实施例中，通过增加代理模块来缩短用户到IDC之间的往返传输时间RTT，RTT变短后，则相应地提高数据的传输速度，而且代理模块基于流式传输上传数据，不再需要对数据进行落地存储和转换，节省资源和时间，进一步地提高上传的速度。
附图说明
图1为现有的客户机直接向IDC上传数据的示意图；
图2为现有的客户机通过CDN向IDC上传数据的示意图；
图3为本发明实施例一的数据上传方法的流程示意图；
图4为本实施例一的数据上传方法的应用示意图之一；
图5为本实施例一的数据上传方法的应用示意图之二；
图6为本实施例一的数据上传方法的应用示意图之三；
图7为本实施例一的数据上传方法的应用示意图之四；
图8为本实施例一的数据上传方法的应用示意图之五；
图9为本发明实施例二的数据上传方法的流程示意图；
图10为本发明实施例三的数据上传装置的结构示意图；
图11为本发明实施例四的数据上传系统的结构示意图。
具体实施方式
下面结合附图对本发明实施例提供的数据上传方法、装置及系统进行详细描述。
实施例一
如图1所示，其为本发明实施例一的数据上传方法的流程示意图。该数据上传方法包括以下步骤：
S101、代理模块接收客户机基于流式上传技术上传的数据。
本实施例中，客户机(Client)与互联网数据中心IDC之间部署有一个代理模块，通过该代理模块实现客户机向IDC上传数据。本实施例中的代理模块可以是代理服务器也可以是设置在其他服务器上的执行代理功能的程序模块(Proxy)。
部署代理模块需要满足以下预设条件：客户机基于部署的代理模块向互联网数据中心IDC上传数据的最小时长小于等于客户机直接向互联网数据中心IDC上传数据所耗费的最小时长。代理模块的部署的位置就是代理模块在传输链路中的位置，本实施例中，用代理模块与客户机之间的RTT来表示代理模块与客户机之间的传输距离，用代理模块与互联网数据中心IDC之间的RTT，来表示代理模块与互联网数据中心IDC之间的传输距离。
例如，客户机直接向互联网数据中心IDC上传数据时，客户机与互联网数据中心IDC所需的数据传输时间为第一往返传输时间RTT1，而客户机与互联网数据中心IDC的连接建立时间至少为一个RTT1，则客户机直接向互联网数据中心IDC上传数据所耗费的最小时长为2RTT1。客户机通过部署的代理模块向互联网数据中心IDC上传数据时，客户机与代理模块所需的数据传输时间为第二往返传输时间为RTT2，代理模块与互联网数据中心IDC所需的数据传输时间为第三往返传输时间为RTT3。客户机与 代理模块之间的连接建立时间至少一个RTT2，则客户机基于代理模块向互联网数据中心IDC上传数据所耗费的最小时长为2RTT1+RTT3。预设条件为：2RTT2+RTT3≤2RTT1。
本实施例中，客户机与代理模块之间采用传输控制协议(Transmission Control Protocol，简称TCP)方式进行连接，代理模块与互联网数据中心IDC之间采用长连接方式进行连接。其中，代理模块与互联网数据中心IDC之间的长连接使用专线或者连接保障技术，以减少代理模块与数据网络中心之间的丢包率。
一般情况下，当代理模块与互联网数据中心IDC或者客户机之间因为故障或者异常情况出现丢包时，代理模块会降低传输速度，如果要想恢复上次的传输速率，则需要一个RTT的时间，这样代理模块与客户机以及互联网数据中心IDC所需的RTT越短，代理模块与互联网数据中心IDC以及客户机之间的速率恢复越快，进而更加有利于数据的加速传输。
当客户机试图向互联网数据中心IDC上传数据时，客户机就会与代理模块建立TCP连接，当连接建立完成后，客户机基于流式上传技术向代理模块发送数据。
S102、代理模块基于流式上传技术将所述数据上传到互联网数据中心IDC。
代理模块与互联网数据中心IDC之间采用长连接，当代理模块与互联网数据中心IDC之间长连接建立完成之后，在代理模块与互联网数据中心IDC完成一次数据读写之后，它们之间的长连接会继续保持，并不会主动关闭，后续的读写操作可以继续使用这个处于保持状态的长连接。
本实施例中，每当客户机请求向代理模块上传数据后，代理模块与互联网数据中心IDC之间不再需要耗费时间重新建立连接，从而节省了建立连接所需的一个RTT。
实际应用中，设备之间的连接从开始建立到该连接达到可使用的最大带宽，需要耗费一定的时间。而长连接的方式可以让发送窗口保持在BDP附近，这样就无需不断重复从开始建立到该连接达到可使用的最大带宽的过程，从而节省了上述从开始建立到该连接达到可使用的最大带宽过程所需耗费的时间。
在接收到数据后，代理模块并不对数据进行任何处理，直接基于流式上传技术直接将接收到的数据上传到IDC上。本实施例中，基于流式上传技术上传数据，代理模块不需要进行中间存储，这样不会出现在CDN中存储数据的服务器出现宕机而导致数据丢失的情况，而且不需要对数据落地存储，避免出现数据热点问题。进一步地，代理模块不需要对数据进行转换，以将数据缩小加速传输速度，相应地，互联网数据中心IDC就可以直接存储客户机的源数据。而且基于流式上传技术传输数据，当客户机上传完所有的数据后，相应地互联网数据中心也就接收到了该数据，两者是实时同步的。
此处需要说明，本实施例中的客户机向互联网数据中心上传的数据的数据量大小需要控制在14KB～5MB范围内，在该范围内的数据基于代理模块上传的速度较快。
本实施例中，设定上传耗时为rt，用户系统初始发送窗口值为initwnd，用户带宽为bandwidth，上传的数据大小为filesize，客户机到互联网数据中心所在服务器的往返传输时间为rttOfClientToIDC，客户机到代理模块所在服务器的往返传输时间为rttOfClientToProxy，代理模块所在服务器到互联网数据中心所在服务器的往返传输时间为rttOfProxyToIDC，发送窗口上升到满带宽需要消耗的rttOfClient数量为N，带宽延时积bdp，以2为底数的对数函数log。
设定在不丢失包的传输条件，分别对客户机直接将数据上传到互联网 数据中心IDC的方式以及本实施例中提供的数据上传方法的上传耗时进行测试。
客户机直接将数据上传到互联网数据中心IDC方式的上传耗时：
当bdp小于initwnd时，无需提升发送窗口的带宽，发送窗口可以在允许使用的最大带宽进行上传，此时上传耗时为：
rt＝filesize/bandwidth+rttOfClientToIDC
当数据大小小于满带宽时发送窗口可传输的数据大小情况下，此时上传耗时为：
rt＝(log(filesize/initwnd)+2)*rttOfClientToIDC
当数据大小大小满带宽时发送窗口可传输的数据大小情况下，此时上传耗时为：
N＝log(bdp/initwnd)
rt＝((filesize-2^(n-1)*initwnd)/bandwidth)+(n+1)*rttOfClientToIDC
本实施例中提供的数据上传方法的上传耗时：
当bdp小于initwnd时，无需提升发送窗口的带宽，发送窗口可以在允许使用的最大带宽进行上传，此时上传耗时为：
rt＝filesize/bandwidth+rttOfClientToProxy+rttOfProxyToIDC
当数据大小小于满带宽时发送窗口可传输的数据大小情况下，此时上传耗时为：
rt＝(log(filesize/initwnd)+2)*rttOfClientToProxy+rttOfProxyToIDC
当数据大小大于满带宽时tcp窗口可传输的数据大小情况下，此时上传耗时为：
N＝log(bdp/initwnd)
rt＝((filesize-2^(n-1)*initwnd)/bandwidth)+(n+1)*rttOfClientToProxy+rttOfProxyToIDC
由上述公式可以计算出常见带宽下，不同数据大小的上传耗时。
图4～图8为不同数据大小在不同带宽下的上传耗时的示意图。图中的实验结果基于代理模块到客户机之间的RTT为5ms，代理模块到到IDC的RTT为30ms，客户机到IDC的RTT为30ms得到的。
图4其为数据大小为10KB时不同带宽下上传耗时的示意图。
图5其为数据大小为50KB时不同带宽下上传耗时的示意图。
图6其为数据大小为100K时不同带宽下上传耗时的示意图。
图7其为数据大小为1MB时不同带宽下上传耗时的示意图。
图8其为数据大小为5MB时不同带宽下上传耗时的示意图。
由上述公式和图4～图8可以得出以下结论：
当带宽小于500KB时候，本实施例提供的基于代理模块的数据上传方法无加速效果，反而与现有的直接上传相比存在一定的时延，增加的时延为一个客户机到代理模块的往返传输时间。
本实施例提供的基于代理模块的数据上传方法加速效果随着带宽增大越明显。
在常用的带宽(4Mb～100Mb)范围内，待传输的数据越大，本实施例提供的基于代理模块的数据上传方法加速效果越不明显，
而且在实际应用中，加速效果会在用户不同带宽以及网络的不同拥塞情况会下有不同。当网络中出现丢包情况，由于代理模块的rtt比较小，发送窗口的恢复速度会比较快，所以在网络不稳定的情况下会有优势。
研发人员使用Linux tc工具模拟了常用带宽下丢包10％～30％传输环境，得出如下结论：当超过30％的丢包率时基本无法成功上传数据，在丢包情况下，测试结果随机性太大，但本实施例提供的基于代理模块的数据上传方法比直接上传有30％到3倍的性能提升。
在实验测试后，可以得出代理模块对14KB以下的加速效果不明显，对14KB～2MB大小内加速效果会越来越明显，当文件超过5M以上，加速 效果又会减小。
为了更好地理解本实施例提供的数据上传方法，举例说明：
在客户机与IDC之间增加一个代理模块，代理模块与客户机之间采用TCP连接，与IDC之间保持长连接。
客户机需要上传20KB的文件，设置传输数据为一个RTT内传输10KB。R0＝30ms，R1＝5ms，R2＝25ms。其中，客户机到IDC的RTT为R0，客户机到代理模块的RTT为R1，代理模块到IDC的RTT为R2。
首先根据R0、R1和R2以及预设条件得到一个目标节点，将代理设置在该目标节点上。客户机通过TCP连接基于流式上传技术向代理模块发送数据，当发送窗口大小为10时，客户机需要通过两个R1将数据发完，而客户机与代理建立TCP连接还需要1个R1，则客户机从建立连接到将数据到代理上共用3个R1，代理还需要将数据发送到IDC，需要消耗一个R2。则总耗时为：3*R1+R2＝40ms。而且在该示例中，代理模块分别与客户机和IDC之间采用流式上传技术，则不需要中间落地存储，而且不需要压缩转换处理，相对于通过CDN转发不仅节省存储空间，而且减少耗时。
而通过直接上传的方式，客户机需要1个R0时间建立连接，用两个R0时间发送数据，则总耗时3*R0＝90ms。基于本实施例提供的方法上传数据，能够比直接上传方式节省一半的时间，提高上传的时效性。
本实施例提供的数据上传方法，通过在客户机与互联网数据中心之间就近部署代理模块，代理模块分别与客户机和互联网数据中心IDC基于流式上传技术传输数据。本实施例中，通过增加代理模块来缩短用户到IDC之间的往返传输时间RTT，RTT变短后，则相应地提高数据的传输速度，而且代理模块基于流式上传技术完成数据的上传，代理模块不再需要对接收到数据进行落地存储和转换，节省资源和时间，进一步提高上传的速度。
实施例二
如图9所示，其为本发明实施例二的数据上传方法的流程示意图。该数据上传方法包括以下步骤：
S201、在满足预设条件的节点上部署代理模块。
在客户机与互联网数据中心，设置一个满足预设条件的节点，在该节点上部署代理模块。可选地，为了节省部署的成本，部署代理模块的节点可以复用CDN中的节点，而且复用CDN的节点，由于CDN覆盖范围较广，部署了代理模块的节点可以同时兼顾到更多客户机，由于CDN的节点较多，能够在大范围内部署代理模块。其中预设条件可参见上述实施例一中相关内容的记载，此处不再赘述。
在部署完代理模块后，在部署完代理模块后，客户机与代理模块通过TCP连接建立连接，代理模块通过长连接与互联网数据中心IDC连接。本实施例中，代理模块与互联网数据中心IDC之前的长连接个数是变化的，代理模块可以根据带宽占用情况，动态地调整长连接的个数。当带宽充足时，设置多个长连接，可以更快地上传数据。当带宽不足时，则相应地减少长连接的个数。
S202、代理模块接收客户机基于流式上传技术上传的数据。
S203、代理模块对接收到的数据进行缓存。
本实施例中，代理模块可以对接收的数据进行缓存，其中，代理模块将数据缓存在内存中，并不需要对数据进行落地存储。
S204、代理模块通过与互联网数据中心IDC之间的长连接将发送窗口维持在与可利用带宽匹配的大小，以使缓存后的数据通过发送窗口一次性发送完成
用户可以基于客户机上设置发送窗口的初始大小，一般情况下，在客户机与代理模块建立连接之初，发送窗口的初始大小为默认设置，该设置可以由用户通过相应的应用程序进行设置。但是在TCP连接建立后，客户 机随着数据不断地发送成功后，客户机中的操作系统可以不断地尝试将发送窗口增大，以使发送窗口的大小调整到与可利用带宽匹配的大小。当客户机与代理模块之间的发送窗口调整到合适大小后，相应的，代理模块通过与互联网数据中心IDC之间的长连接，将代理模块与互联网数据中心IDC之间的发送窗口维持到一个合适的大小，该合适大小可以保证代理模块将缓存的数据能够一次性通过发送窗口。本实施例中，由于代理模块和互联网数据中心IDC之间进行长连接，从而使得代理模块与网络数中心IDC之间的发送窗口可以维持在一个稳定的大小，不会随着代理模块与客户机之间的发送窗口的调整而不断地变化。
S205、代理模块在缓存的数据的数据量达到阈值后，将缓存的数据同时上传到IDC。
本实施例中，为缓存的数据的数据量预设一个阈值，缓存后的数据的数据量要在该阈值内，当缓存的数据量达到阈值后，将缓存的数据同时上传到互联网数据中心IDC中。例如，缓存后的数据量小于等于代理模块与IDC之前的带宽时延乘积(Bandwidth Delay Product，简称BDP)。
S206、代理模块清除缓存的数据。
本实施例中，代理模块将数据缓存在内存中，当数据发送出去后，则可以对缓存的数据进行清除，释放内存空间。代理模块并不需要将缓存的数据进行落地存储，节省资源。
S207、代理模块接收IDC返回的上传完成指示。
S208、代理模块将上传完成指示转发给客户机。
其中，上传完成指示是由IDC在接收到由代理模块上传完所有数据后发送的。
在通过CDN中转的方案中，当客户机将数据上传到CDN后，就可以接收到上传完成的指示，但是此时互联网数据中心IDC并未接收到该数据。 而本实施例中互联网数据中心在接收完所有数据后，将会生成一个上传完成指示并发送给代理模块，代理模块接收到该上传完成指示后，将该上传完成指示返回给客户机。
本实施例提供的数据上传方法，通过在客户机与互联网数据中心之间就近部署代理模块，代理模块分别与客户机和互联网数据中心IDC基于流式上传技术传输数据，本实施例中，通过增加代理模块来缩短用户到IDC之间的往返传输时间RTT，RTT变短后，则相应地提高数据的传输速度，而且代理模块基于流式上传技术完成数据的上传，代理模块不再需要对接收到数据转换，节省时间，进一步地提高上传速度。进一步地，代理模块在内存中对数据进行缓存，并同时将缓存的数据一次性上传到互联网数据中心IDC中，不再对数据进行落地存储，节省资源。
实施例三
如图10所示，其为本发明实施例三的数据上传装置的结构示意图。该数据上传装置包括：接收单元11和上传单元12。
其中，接收单元11，用于接收客户机基于流式上传技术上传的数据。
上传单元12，用于基于流式上传技术将所述数据上传到互联网数据中心IDC。
本实施例中，数据上传装置部署的位置满足预设条件，其中所述预设条件为：客户机基于所述数据上传装置向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
接收单元11与所述客户机之间采用TCP方式进行连接。
上传单元12与所述IDC之间采用长连接方式进行连接。
上传单元12，还用于动态调整与所述IDC之间长连接的个数。
上传单元12，具体用于对接收到的所述数据进行缓存，在缓存的所述数据的数据量达到阈值后，将缓存的所述数据同时上传到所述IDC，清除 缓存的所述数据。
上传单元12，还用于通过与所述IDC之间的长连接将发送窗口的大小维持在与可利用带宽匹配的大小，以使缓存后的所述数据通过所述发送窗口一次性发送完成。
缓存的所述数据的数据量小于等于所述数据上传装置与所述IDC之间的带宽时延乘积BDP。
接收单元11，还用于接收所述IDC返回的上传完成指示；其中，所述上传完成指示是由所述IDC在接收到由所述数据上传装置上传完所有数据后发送的。
上传单元12，还用于将所述上传完成指示转发给所述客户机。
数据上传装置部署在节点上，所述节点为所述IDC中的节点。
所述客户机向所述IDC上传的所述数据的数据量范围为14KB～5MB。
本实施例中，在客户机与互联网数据中心之间就近部署数据上传装置，该数据上传装置分别与客户机和互联网数据中心IDC基于流式上传技术传输数据。本实施例中，通过增加数据上传装置来缩短用户到IDC之间的往返传输时间RTT，RTT变短后，则相应地提高数据的传输速度，而且数据上传装置基于流式上传技术完成数据的上传，不再需要对接收到数据转换，节省时间。进一步地，该数据上传装置在内存中对数据进行缓存，并同时将缓存的数据一次性上传到互联网数据中心IDC中，不再对数据进行落地存储，不仅节省资源，而且提高上传的速度。
实施例四
如图11所示，其为本发明实施例四的数据上传系统的结构示意图。该数据上传系统包括：客户机1和代理模块2和互联网数据中心IDC 3。
其中，客户机1，用于向通过代理模块2向互联网数据中心IDC 3上传数据。
代理模块2，用于接收客户机1基于流式上传技术上传的所述数据，以及基于流式上传技术将所述数据上传到互联网数据中心IDC 3上。
互联网数据中心IDC3，用于接收代理模块2转发的客户机1上传的所述数据。
其中，代理模块2部署的位置满足预设条件，其中所述预设条件为：客户机1基于代理模块2向互联网数据中心IDC 3上传数据所耗费的最小时长小于等于客户机1直接向互联网数据中心IDC 3上传数据耗费的最小时长。
本实施例中，代理模块2为上述实施例三中的数据上传装置，关于代理模块2的结构，可参见上述实施例中数据上传装置的结构。
代理模块2与客户机1之间采用TCP方式进行连接，与互联网数据中心IDC 3之间采用长连接方式进行连接。
代理模块2，还用于动态调整与互联网数据中心IDC 3之间长连接的个数。
代理模块2，具体用于对接收到的所述数据进行缓存，在缓存的所述数据的数据量达到阈值后，将缓存的所述数据同时上传到互联网数据中心IDC 3，以及清除缓存的所述数据。
代理模块2，具体用于通过与互联网数据中心IDC 3之间的长连接将发送窗口维持在与可利用带宽匹配的大小，以使缓存后的所述数据通过所述发送窗口一次性发送完成。
其中，缓存的所述数据的数据量小于等于代理模块2与互联网数据中心IDC 3之间的带宽时延乘积BDP。
互联网数据中心IDC 3，还用于向代理模块2返回上传完成指示；其中，上传完成指示是由互联网数据中心IDC 3在接收到由代理模块2上传完所有数据后发送的。
代理模块2，还用于接收互联网数据中心IDC 3返回的所述上传完成指示，以及将所述上传完成指示转发给客户机1。
客户机1，还用于接收所述上传完成指示。
代理模块2部署在节点上，所述节点为互联网数据中心IDC 3中的节点。
客户机1向互联网数据中心IDC3上传的所述数据的数据量范围为14KB～5MB。
关于代理模块2与客户机1和互联网数据中心3之间的交互过程，可参见上述实施例中相关内容的记载，此处不再赘述。
本实施例中，在客户机与互联网数据中心之间就近部署代理模块，该代理模块分别与客户机和互联网数据中心IDC基于流式上传技术传输数据，基于流式上传技术完成数据的上传。本实施例中，通过增加代理模块来缩短用户到IDC之间的往返传输时间RTT，RTT变短后，则相应地提高数据的传输速度，而且代理模块不再需要对接收到数据转换，节省时间。进一步地，该代理模块在内存中对数据进行缓存，并同时将缓存的数据一次性上传到互联网数据中心IDC中，不再对数据进行落地存储，节省资源，而且提高上传的速度。
本领域普通技术人员可以理解：实现上述各方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成。前述的程序可以存储于一计算机可读取存储介质中。该程序在执行时，执行包括上述各方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上各实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述各实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进 行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (27)

  1. 一种数据上传方法，其特征在于，包括：
    代理模块接收客户机基于流式上传技术上传的数据；
    所述代理模块基于流式上传技术将所述数据上传到互联网数据中心IDC；
    其中，所述代理模块部署的位置满足预设条件，其中所述预设条件为：所述客户机基于所述代理模块向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
  2. 根据权利要求1所述的方法，其特征在于，所述代理模块接收客户机基于流式上传技术上传的数据之前，包括：
    所述代理模块与所述客户机之间采用TCP方式进行连接；
    所述代理模块与所述IDC之间采用长连接方式进行连接。
  3. 根据权利要求2所述的方法，其特征在于，所述代理模块动态调整与所述IDC之间长连接的个数。
  4. 根据权利要求1所述的方法，其特征在于，所述代理模块基于所述流式上传技术将接收到的所述数据上传到互联网数据中心IDC，包括：
    所述代理模块对接收到的所述数据进行缓存；
    所述代理模块在缓存的所述数据的数据量达到阈值后，将缓存的所述数据同时上传到所述IDC；
    所述代理模块清除缓存的所述数据。
  5. 根据权利要求4所述的方法，其特征在于，所述代理模块通过与所述IDC之间的长连接将发送窗口维持在与可利用带宽匹配的大小，以使缓存后的所述数据通过所述发送窗口一次性发送完成。
  6. 根据权利要求5所述的方法，其特征在于，缓存的所述数据的数据量小于等于所述代理模块与所述IDC之间的带宽时延乘积BDP。
  7. 根据权利要求6所述的方法，其特征在于，所述代理模块基于流式上传技术将所述数据上传到互联网数据中心IDC之后，还包括：
    所述代理模块接收所述IDC返回的上传完成指示；其中，所述上传完成指示是由所述IDC在接收到由所述代理模块上传完所有数据后发送的；
    所述代理模块将所述上传完成指示转发给所述客户机。
  8. 根据权利要求1-7任一项所述的方法，其特征在于，所述代理模块部署在节点上，所述节点为所述IDC中的节点。
  9. 根据权利要求1-7任一项所述的方法，其特征在于，所述客户机向所述IDC上传的所述数据的数据量范围为14KB～5MB。
  10. 一种数据上传装置，其特征在于，包括：
    接收单元，用于接收客户机基于流式上传技术上传的数据；
    上传单元，用于基于流式上传技术将所述数据上传到互联网数据中心IDC；
    其中，所述数据上传装置部署的位置满足预设条件，其中所述预设条件为：所述客户机基于所述数据上传装置向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长。
  11. 根据权利要求10所述的装置，其特征在于，所述接收单元与所述客户机之间采用TCP方式进行连接；
    所述上传单元与所述IDC之间采用长连接方式进行连接。
  12. 根据权利要求11所述的装置，其特征在于，所述上传单元，还用于动态调整与所述IDC之间长连接的个数。
  13. 根据权利要求12所述的装置，其特征在于，所述上传单元，具体用于对接收到的所述数据进行缓存，在缓存的所述数据的数据量达到阈值后，将缓存的所述数据同时上传到所述IDC，清除缓存的所述数据。
  14. 根据权利要求13所述的装置，其特征在于，所述上传单元，还 用于通过与所述IDC之间的长连接将发送窗口的大小维持在与可利用带宽匹配的大小，以使缓存后的所述数据通过所述发送窗口一次性发送完成。
  15. 根据权利要求14所述的装置，其特征在于，缓存的所述数据的数据量小于等于所述数据上传装置与所述IDC之间的带宽时延乘积BDP。
  16. 根据权利要求15所述的装置，其特征在于，所述接收单元，还用于接收所述IDC返回的上传完成指示；其中，所述上传完成指示是由所述IDC在接收到由所述数据上传装置上传完所有数据后发送的；
    所述上传单元，还用于将所述上传完成指示转发给所述客户机。
  17. 根据权利要求10-16任一项所述的装置，其特征在于，所述数据上传装置部署在节点上，所述节点为所述IDC中的节点。
  18. 根据权利要求10-16任一项所述的装置，其特征在于，所述客户机向所述IDC上传的所述数据的数据量范围为14KB～5MB。
  19. 一种数据上传系统，其特征在于，包括：
    客户机，用于向通过代理模块向互联网数据中心IDC上传数据；
    所述代理模块，用于接收所述客户机基于流式上传技术上传的所述数据，以及基于流式上传技术将所述数据上传到所述IDC上；
    所述IDC，用于接收所述代理模块转发的所述客户机上传的所述数据；
    其中，所述代理模块部署的位置满足预设条件，其中所述预设条件为：所述客户机基于所述代理模块向所述IDC上传数据所耗费的最小时长小于等于所述客户机直接向所述IDC上传数据耗费的最小时长；
    所述代理模块为上述权利要求10-18任一项所述的数据上传装置。
  20. 根据权利要求19所述的系统，其特征在于，所述代理模块与所述客户机之间采用TCP方式进行连接，与所述IDC之间采用长连接方式进行连接。
  21. 根据权利要求20所述的系统，其特征在于，所述代理模块，还 用于动态调整与所述IDC之间长连接的个数。
  22. 根据权利要求19所述的系统，其特征在于，所述代理模块，具体用于对接收到的所述数据进行缓存，在缓存的所述数据的数据量达到阈值后，将缓存的所述数据同时上传到所述IDC，以及清除缓存的所述数据。
  23. 根据权利要求22所述的系统，其特征在于，所述代理模块，具体用于通过与所述IDC之间的长连接将发送窗口维持在与可利用带宽匹配的大小，以使缓存后的所述数据通过所述发送窗口一次性发送完成。
  24. 根据权利要求23所述的系统，其特征在于，缓存的所述数据的数据量小于等于所述代理模块与所述IDC之间的带宽时延乘积BDP。
  25. 根据权利要求24所述的系统，其特征在于，所述IDC，还用于向所述代理模块返回上传完成指示；其中，所述上传完成指示是由所述IDC在接收到由所述代理模块上传完所有数据后发送的；
    所述代理模块，还用于接收所述IDC返回的所述上传完成指示，以及将所述上传完成指示转发给所述客户机；
    所述客户机，还用于接收所述上传完成指示。
  26. 根据权利要求19-25任一项所述的系统，其特征在于，所述代理模块部署在节点上，所述节点为所述IDC中的节点。
  27. 根据权利要求19-25任一项所述的系统，其特征在于，所述客户机向所述IDC上传的所述数据的数据量范围为14KB～5MB。
PCT/CN2017/077912 2016-03-31 2017-03-23 数据上传方法、装置及系统 Ceased WO2017167119A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610201099.0A CN107295040A (zh) 2016-03-31 2016-03-31 数据上传方法、装置及系统
CN201610201099.0 2016-03-31

Publications (1)

Publication Number Publication Date
WO2017167119A1 true WO2017167119A1 (zh) 2017-10-05

Family

ID=59962601

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/077912 Ceased WO2017167119A1 (zh) 2016-03-31 2017-03-23 数据上传方法、装置及系统

Country Status (3)

Country Link
CN (1) CN107295040A (zh)
TW (1) TW201737109A (zh)
WO (1) WO2017167119A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113439416A (zh) * 2018-12-13 2021-09-24 亚马逊技术有限公司 连续校准的网络系统

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108121507B (zh) * 2017-12-06 2021-04-02 北京奇艺世纪科技有限公司 一种数据处理方法、装置以及电子设备
CN111193789B (zh) * 2019-12-25 2023-07-11 微民保险代理有限公司 订阅信息推送方法、装置、计算机设备和可读存储介质
CN113992662A (zh) * 2021-12-29 2022-01-28 深圳竹云科技有限公司 文件传输方法、装置及存储介质
CN114710548B (zh) * 2022-03-22 2024-04-05 阿里巴巴（中国）有限公司 报文转发方法及装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977234A (zh) * 2010-11-02 2011-02-16 中南大学 基于并行tcp技术的广域网通信加速方法
CN103731398A (zh) * 2012-10-11 2014-04-16 北京百度网讯科技有限公司 基于cdn网络的数据访问方法、系统及装置
CN103731933A (zh) * 2014-01-10 2014-04-16 北京百度网讯科技有限公司 建立无线连接的方法、系统以及移动终端
CN104010001A (zh) * 2014-06-17 2014-08-27 合一网络技术(北京)有限公司 移动终端中同类联网请求进行连接通信的方法和系统
CN104009938A (zh) * 2014-06-17 2014-08-27 合一网络技术(北京)有限公司 基于路由层面的长连接的方法和系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101977234A (zh) * 2010-11-02 2011-02-16 中南大学 基于并行tcp技术的广域网通信加速方法
CN103731398A (zh) * 2012-10-11 2014-04-16 北京百度网讯科技有限公司 基于cdn网络的数据访问方法、系统及装置
CN103731933A (zh) * 2014-01-10 2014-04-16 北京百度网讯科技有限公司 建立无线连接的方法、系统以及移动终端
CN104010001A (zh) * 2014-06-17 2014-08-27 合一网络技术(北京)有限公司 移动终端中同类联网请求进行连接通信的方法和系统
CN104009938A (zh) * 2014-06-17 2014-08-27 合一网络技术(北京)有限公司 基于路由层面的长连接的方法和系统

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113439416A (zh) * 2018-12-13 2021-09-24 亚马逊技术有限公司 连续校准的网络系统
CN113439416B (zh) * 2018-12-13 2023-12-26 亚马逊技术有限公司 连续校准的网络系统

Also Published As

Publication number Publication date
CN107295040A (zh) 2017-10-24
TW201737109A (zh) 2017-10-16

Similar Documents

Publication Publication Date Title
EP3780542B1 (en) Data transmission method and device
EP2586164B1 (en) System and method for content and application acceleration in a wireless communications system
WO2017167119A1 (zh) 数据上传方法、装置及系统
JP2018508151A (ja) 伝送制御プロトコルｔｃｐデータパケットを送信する方法及び装置、並びにシステム
CN104243090B (zh) 一种基于无线信道反馈的发送速率调整方法和设备
US11411892B2 (en) Packet fragment processing method and apparatus and system
US10931585B2 (en) Data flow control for dual ended transmission control protocol performance enhancement proxies
US12184550B2 (en) Flow rate control method and apparatus
WO2012142844A1 (zh) 一种基于p2p的文件分发系统及分片方法
CN116131923B (zh) 一种基于卫星通信的数据传输方法、装置及存储介质
US20200076718A1 (en) High bandwidth using multiple physical ports
WO2017114231A1 (zh) 一种报文发送方法、tcp代理以及tcp客户端
CN102014058B (zh) 一种上行流量的调度方法及装置
CN104202268A (zh) 无线传感器网络IPv6路由协议报文分片重组的方法
CN105577557B (zh) 一种卫星网络中并发tcp连接复用方法
CN105027515B (zh) 用于在内容传输系统中处理数据的设备和方法
CN107026716A (zh) 一种数据中心网络中基于并发度感知的传输控制方法
KR102390522B1 (ko) 적응형 비디오 스트리밍 서비스를 위한 블록체인 기반의 안정적인 체감품질 가속기
CN106101005A (zh) 一种数据中心网络中基于分组长度的拥塞控制方法
WO2019004013A1 (ja) データ送信装置、方法および記録媒体
US20150133120A1 (en) Method, system and computer-readable recording medium for transmitting contents with supporting hand-over
CN106713432A (zh) 数据缓存方法及网络代理设备
HK1246042A1 (zh) 数据上传方法、装置及系统
CN115695311B (zh) 一种RoCE v2网络环境流量控制方法、系统和存储介质
CN115174690B (zh) 一种弱网或断网情况下大流量业务加速系统及方法

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17773149

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17773149

Country of ref document: EP

Kind code of ref document: A1