CN103036967B

CN103036967B - A kind of download management equipment, method and data downloading system

Info

Publication number: CN103036967B
Application number: CN201210528632.6A
Authority: CN
Inventors: 徐铁城; 陈超
Original assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Priority date: 2012-12-10
Filing date: 2012-12-10
Publication date: 2017-03-15
Anticipated expiration: 2032-12-10
Also published as: CN106850817A; CN103036967A

Abstract

The invention discloses a kind of download management equipment, method and data downloading system, wherein, the download management equipment includes：Buffer, is configured to the synopsiss for caching each file and each file obtained from data source nodes, and the synopsiss of this document are that the content to file changes the data for generating using particular data；Resolver, is configured to obtain the file download request from client device, and parses the synopsiss of asked file according to the download address of file download request；Finger, is configured to be searched according to the synopsiss of file in both the buffers, if found, the file that is asked is transmitted to client device；And source device is returned, when being configured as not finding in both the buffers, from the asked file of related data source nodes acquisition, and transmit to client device, and the file that is asked be supplied to buffer being cached.

Description

Download management equipment and method and data download system

Technical Field

The invention relates to the technical field of data downloading, in particular to a downloading management device used in an edge node, a data downloading system and a downloading management method used in the data downloading system.

Background

An existing CDN (Content Delivery Network) improves data access speed by delivering data to each edge node and reducing a distance to a client. The basic idea is to avoid bottlenecks and links possibly influencing data transmission speed and stability on the Internet as far as possible, so that content transmission is faster and more stable. The CDN system can redirect a request of a user to a service node closest to the user in real time according to the network traffic and comprehensive information such as connection of each node, a load condition, a distance to the user, and response time.

However, each existing CDN node caches all files in the data source as much as possible, which causes a large storage cost. Moreover, the content of some files is not changed, only the download address, such as the URL of the file, is changed, and according to the existing CDN caching method, multiple files with the same content and different URLs are cached in one node, that is, the duplicate data is cached, thereby further causing higher storage cost.

Disclosure of Invention

In view of the above problems, the present invention has been made to provide a download management apparatus for use in an edge node, a data download system, and a download management method for use in a data download system that overcome or at least partially solve the above problems.

According to an aspect of the present invention, there is provided a download management apparatus for use in an edge node, comprising: the cache is configured to cache each file obtained from the data source node and a content abstract of each file, wherein the content abstract of each file is data generated by converting the content of each file by adopting specific data; the analyzer is configured to obtain a file downloading request from the client equipment and analyze the content abstract of the requested file according to the downloading address of the file downloading request; the searching device is configured to search the requested file in the buffer according to the content abstract of the requested file analyzed by the analyzer, and if the requested file is searched, the requested file is transmitted to the client equipment; and the source returning device is configured to acquire the requested file from the data source node related to the requested file and transmit the requested file to the client device when the requested file is not found in the cache by the finder, and provide the requested file to the cache for caching.

Optionally, the back-source is adapted to retrieve portions of the requested file from the data source node portion by portion, and to simultaneously transmit the retrieved portions of the requested file to the client device until the requested file is fully retrieved.

Optionally, the source returning device includes: the data source query module is configured to query a data source node related to the requested file according to a known source return table and the download address of the requested file; a direct source returning module configured to obtain the requested file directly from the data source node of the first operator network when the edge node is the edge node of the first operator network and the queried data source node related to the requested file includes the data source node of the first operator network; the proxy source returning module is configured to acquire the requested file from the data source node related to the second operator network through a proxy cluster from the first operator network to the second operator network when the edge node is the edge node of the first operator network and the queried data source node related to the requested file is the data source node of the second operator network; and the cache notification module is configured to notify the cache to cache the requested file after the requested file is acquired by the direct source returning module or the proxy source returning module.

Optionally, the content summary of the file includes: one of sha series data of secure hash algorithm, or one of MD series data of information summary algorithm.

Optionally, the download management device is a varniash cache server.

According to another aspect of the present invention, there is provided a data downloading system, which includes an edge node, an agent cluster providing information interaction service between different operator networks, and a data source node, wherein the edge node includes the above-mentioned download management device.

Optionally, the method further includes: in the management device for providing a file download address described above, the download address of the file download request received by the edge node is provided by the management device for providing a file download address.

Optionally, the agent cluster providing information interaction service between different operator networks includes an agent cluster from a first operator network to a second operator network, and an agent cluster from the second operator network to the first operator network, where the agent cluster from the first operator network to the second operator network includes: an edge side first operator network agent cluster configured to receive a file download request from an edge node of a first operator network and return the requested file to the edge node of the first operator network; the transmission channel is configured to transmit information from the edge side first operator network agent cluster to the source side second operator network agent cluster, and transmit information from the source node side second operator network agent cluster to the edge side first operator network agent cluster; the source node side second operator network agent cluster is configured to send a file downloading request to a data source node related to a second operator network according to a file downloading request received from the edge side first operator network agent cluster through a transmission channel, receive a requested file returned by the data source node related to the second operator network, and transmit the requested file to the edge side first operator network agent cluster through the transmission channel;

the second operator network to first operator network proxy cluster comprises: an edge side second operator network agent cluster configured to receive a file download request from an edge node of the second operator network and return the requested file to the edge node of the second operator network; the transmission channel is configured to transmit information from the edge side second operator network agent cluster to the source node side first operator network agent cluster, and transmit information from the source node side first operator network agent cluster to the edge side second operator network agent cluster; and the source node side first operator network agent cluster is configured to send a file downloading request to a data source node related to the first operator network according to a file downloading request received from the edge side second operator network agent cluster through a transmission channel, receive a requested file returned by the data source node related to the first operator network, and transmit the requested file to the edge side second operator network agent cluster through the transmission channel.

According to another aspect of the present invention, there is provided a download management method for use in a data download system, the data download system including at least an edge node and a data source node, the download management method comprising: the edge node acquires a file downloading request from the client equipment and analyzes the content abstract of the requested file according to the downloading address of the file downloading request; the edge node searches whether the requested file exists in the cache according to the content abstract of the requested file, and if so, transmits the requested file to the client equipment; if the requested file is not found in the cache, the edge node acquires the requested file from a data source node related to the requested file and transmits the requested file to the client equipment; the edge node caches the file obtained from the data source node and the content abstract of the file, wherein the content abstract of the file is data generated by adopting specific data conversion on the content of the file.

According to the embodiment of the invention, the file content abstract analyzed from the download address of the requested file can be used as a cache file query basis, whether the requested file is stored or not is queried in the cache instead of being used as an index query according to the whole URL of the file, and similarly, whether the files are the same file or not is judged according to whether the content abstract of the file is the same when the files are stored in the cache. Therefore, if the URLs of two files are different, but the content data of the files are substantially the same, the content digests of the files are the same, and further, if the files have been cached in the edge node before, even if the client device sends a different file download address URL again, as long as the content digests of the files are consistent with those in the cache, the edge node does not go to the data source node to download the files repeatedly, but directly finds the files from the cache according to the content digests of the files and provides the files to the client device. Therefore, on one hand, the repeated data in the cache is reduced, and on the other hand, the efficiency of downloading the file for the client equipment is improved.

Further, the edge node acquires the file from the data source node through the proxy cluster, when the file is large, each part of the requested file can be acquired from the data source node one by one, and the acquired part of the requested file, namely the source return in the non-blocking mode, is transmitted to the client device at the same time, so that the client device can download the file while caching the file. The problem that the downloading can be carried out only by waiting for the complete caching of the whole file when the CDN is used is avoided, and the client equipment does not need to wait in the whole process.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 shows a schematic diagram of a data download system according to one embodiment of the invention;

FIG. 2 shows a schematic diagram of a management device for providing file download addresses according to one embodiment of the invention;

FIG. 3 illustrates a schematic diagram of buffering logic in a buffer according to one embodiment of the invention;

FIG. 4 illustrates a proxy cluster diagram for a telecom-to-netcom according to one embodiment of the invention;

FIG. 5 shows a schematic diagram of a data download system according to one embodiment of the invention;

FIG. 6 illustrates a flow diagram of a management method for providing file download addresses according to one embodiment of the present invention; and

fig. 7 shows a flowchart of a download management method for use in a data download system, according to one embodiment of the invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 is a schematic diagram of a data downloading system according to an embodiment of the present invention. The download system comprises an edge node 100 of a first operator network, an edge node 200 of a second operator network, a proxy cluster 300 of the first operator network to the second operator network, a proxy cluster 400 of the second operator network to the first operator network, a data source node 500 of the first operator network and a data source node 600 of the second operator network. Further, a client device 700 of a first carrier network and a client device 800 of a second carrier network having data interaction with the data download system are shown. It should be noted that the edge nodes, data source nodes, proxy clusters, etc. included in the system may all be one or more.

The edge node 100 of the first operator network and the edge node 200 of the second operator network each include a download management device 110, and the download management device 110 specifically includes a buffer 102, a parser 104, a finder 106, and a source return device 108. The agent cluster 300 from the first operator network to the second operator network specifically includes an edge side first operator network agent cluster 302, a first transmission channel 304, and a source node side second operator network agent cluster 306. The second-operator-network-to-first-operator-network proxy cluster 400 specifically includes an edge-side second-operator-network proxy cluster 402, a second transmission channel 404, and a source-node-side first-operator-network proxy cluster 406. The data source nodes 500 of the first operator network include a plurality of data source nodes in the first operator network, such as a data source node 1, a data source node 2, and a data source node n. The data source node 600 of the second operator network includes a plurality of data source nodes in the second operator network, such as a data source node 1, a data source node 2, and a data source node n.

Since the structures of the download management devices 110 included in the edge node 100 of the first operator network and the edge node 200 of the second operator network are the same, and the data processing flows are also the same, only the operator networks where the two edge nodes are located are different and are responsible for connecting client devices of different operator networks, the following description mainly takes the working principle of the edge node 100 of the first operator network as an example, and the internal structure and the working principle of the edge node 200 of the second operator network only need to refer to the content of the relevant embodiment of the edge node 100 of the first operator network, and are not repeated.

In an embodiment of the present invention, a client device 700 connects to the edge node 100 to request to download a file, specifically, the client device 700 sends a file download request to the edge node 100, where the download request includes a download address of the requested file, and the parser 104 parses the content digest of the requested file according to the download address of the file download request after obtaining the file download request. The download request includes the download address of the requested file, which is most commonly the URL (Uniform Resource Locator) of the requested file. In one embodiment of the invention, the URL of a requested file sent by the client device 700 to the edge node 100 of the first carrier network is different from the URL in the prior art, which typically includes the protocol type, the host domain name where the file is stored, the path and the file name, and the general format of the URL is (optional with brackets [): protocol:// hostname [: port ]/path/[;, parameters ] [; in addition to the above conventional URL content, the URL of the requested file in the embodiment of the present invention also includes a content digest of the requested file.

For example, traditionally, the existing conventional URL of a request file is:

http. Wherein XXXX represents specific information of a file path and a file name.

In the embodiment of the present invention, the URL of the request file is:

http://wsdl11.yunpan.cn/share.php?method=Share.download&fhash=4c9a055de0a290341abf7fff6a4d8c0f2af7f155&xqid=XXXX&nid=XXXX&cqid=XXXX&fname=1.txt&e=XXXX&st=XXXXX。

by comparing the two URLs, the second URL has a part of more content than the first URL, that is:

"fhash =4c9a055de0a290341abf7fff6a4d8c0f2af7f 155", which is the content digest of the requested file. Content summarization of a file mainly refers to generating data capable of identifying file content by adopting a specific data conversion method according to the content of the file, and a content summarization calculation algorithm is, for example, one of sha series algorithms of secure hash algorithm (sha 1, sha224, sha256, and the like), one of MD series algorithms of information summarization algorithm (such as MD3, MD4, MD5, and the like), and other algorithms known in the art. As long as the content digest can be used as a unique identifier of the file content, the content digests corresponding to the files with the same content are the same, and the content digests corresponding to the files with different contents are different, the method for generating the content digest is within the protection scope of the present invention. Generally, the unique identification of the content summary generated by performing specific data conversion on the whole content of the file is stronger, but it is not excluded that in some cases, the unique identification of the content summary generated only from part of the key content of the file can also meet the actual requirement, and this is also possible in such cases.

The URL of a file is typically generated when the file is uploaded to the data source node. In connection with an example, how the URL of the requested file is obtained in a download request issued by the client device 100 to the edge node 100 of the first operator network is described below. Referring also to fig. 2, a schematic diagram of a management device for providing a file download address according to an embodiment of the present invention is shown, wherein the management device 210 includes a resource locator 212, a summary generator 214, and a download address generator 216. The management device 210 may be disposed in a corresponding data source node, such as the data source node 500 of the first operator network or the data source node 600 of the second operator network, or may be disposed independently from the data source node. When the management device 210 is provided independently of the data source nodes, it is equivalent to that the data source nodes basically only serve as file storage servers, and the management device 210 serves as a management device for performing operations related to file management, maintenance, and the like, other than file storage, but the data source nodes of the management device 210 can communicate with each other and can obtain various information required by each other.

Specifically, on the one hand, after a certain file is uploaded to a certain data source node, the resource locator 212 in the management device 210 may obtain a storage path of the file on the data source node, and then generate resource locating information of the file according to the storage path, that is, content equivalent to a part of a path and a file name in a URL of the file, for example, the part of the content in the foregoing URL instance: wsdl11. yunnan. cn/share. php.

On the other hand, the digest generator 214 in the management device 210 may know the specific content of the file through the data source node, and then calculate the content of the file by using various content digest calculation methods to generate the content digest of the file. For example, generate the portion of the content in the previous URL instance:

fhash=4c9a055de0a290341abf7fff6a4d8c0f2af7f155。

thereafter, download address generator 216 in management device 210 generates a complete download address, such as a URL, for the file based on the resource locator information for the file provided by resource locator 212 and the content digest for the file provided by digest generator 214, including at least the resource locator information for the file and the content digest for the file in the URL. E.g., to the full URL in the previous URL instance:

it can be seen that the management device 210 can generate corresponding download addresses, such as URLs, for various files that have been uploaded to a certain data source node. Furthermore, when the client device 100 needs to download a file, the client device can obtain the download address of the file by clicking the file on the website or looking up the URL of the file, and then send a download request including the download address to the edge node. So that an edge node, such as edge node 100 of the first operator network, obtains the URL of the file requested by the client device 100, and includes therein not only the conventional URL parameters but also the content digest of the requested file. It can be seen that, in essence, the URL of the file requested by the client device 700 obtained by the edge node 100 is provided by the management device 210.

In an embodiment of the present invention, in order to better ensure the security of data transmission, the management device 210 may generate a signature according to a preset signature generation manner and all or part of the content of the file URL while generating a URL for the file, for example, generate a signature according to key information such as a file content digest and a file storage path in the URL, when the client device 700 needs to download the file, provide the file URL and the corresponding signature to the client device 700, and when the client device 700 sends a download request to the edge node 100, the file URL and the corresponding signature are included. The edge node 100 maintains the same signature generation mode as the management device 210, after the edge node 100 receives the file download request of the client device 700, the parser 104 generates a signature corresponding to the file URL by itself according to the requested file URL and a preset signature generation mode, compares the signature generated by itself with the signature carried in the file download request sent by the client device 700, and if the signature is consistent with the signature carried in the file download request sent by the client device 700, determines that the URL is not modified, and performs related processing on a subsequent downloaded file; if the URL content of the file sent by the client device 700 is inconsistent with the URL content of the file sent by the client device 700, the URL content of the file sent by the client device 700 is modified, for example, the URL content may be maliciously tampered with, and an error or illegal prompt may be returned to the client device 700, so that subsequent related file downloading processing is not performed.

As mentioned above, the parser 104 of the edge node 100, after receiving the download request from the client device 700, is further configured to parse the content digest of the requested file according to the download address of the file download request. Furthermore, the finder 106 in the edge node 100 finds the requested file in the buffer 102 according to the content digest of the requested file parsed by the parser 104, and if the requested file is found, the requested file is directly transmitted to the client device 700 without going to a data source node to obtain the file.

The cache 102 caches each file and a content digest of each file obtained from each data source node (e.g., the data source node 500 of the first operator network and the data source node 600 of the second operator network). In one embodiment of the invention, if a file requested by the client device 700 is present in the cache 102, the file in the cache is provided directly to the client device 700; if not, the file is downloaded to the relevant data source node, and the requested file is obtained from the data source node, cached in the cache 102 for subsequent use on the one hand, and provided to the client device 700 on the other hand.

As can be seen from the above process, each file stored in the cache 102 is obtained from the data source node. Moreover, since the parser 104 can parse the content digest of the file, if there is no file corresponding to the content digest in the buffer 102, the file is obtained from the relevant data source node according to the file download request, and the download address of the file download request contains the content digest of the file, so that after the file is obtained from the data source node, the buffer 102 already knows what the content digest corresponding to the file is, and the file and the content digest of the file are cached in the buffer 102. As mentioned above, there are various specific implementations of the content summary of the file, such as sha1 of the file and MD 5. Further, sha1 and MD5 may be used in combination, since sha1 is generally longer but more uniquely identifiable and MD5 is shorter and more storage-friendly. For example, the digest of the file content recorded in the URL of the requested file is sha1, but the cache 102 may perform data conversion on sha1 again to generate a shorter hash value, that is, MD5, and then store the file and the MD5 corresponding to the file in the cache, and use this MD5 as an index key for searching the file data in the cache 102, as shown in fig. 3, which is a schematic diagram of the cache logic in the cache according to an embodiment of the present invention.

Optionally, the buffer 102 may also set a buffer valid time for each file, because the buffer capacity is limited, and by setting the buffer valid time, occupation of the buffer by the file that is not used for a long time may be removed in time. In addition, it can also be determined whether the file needs to be cached, if some files are not suitable for caching, even if the needed file is obtained from the data source node, the cache 108 will not cache the file, and subsequently, if the client device 700 requests the file again, the edge node 100 will obtain the file from the data source node again and provide the file to the client device 700. In addition, the cache 102 may also maintain a list of prohibited downloads, and if the content summary of the file requested by the client device 700 is in the list of prohibited downloads, the edge node 100 neither looks up the file from the cache 102 to send to the client device 700, nor informs the client device 700 that the file is prohibited from downloading by obtaining the file from the data source node to send to the client device 700.

Since the cache 102 has the above-mentioned caching logic, the finder 106 may find in the cache 102 according to the content digest of the requested file, determine whether a file corresponding to the content digest of the file exists, and if so, directly send the file in the cache to the client device 700.

It should be noted that, generally, each operator network may have a plurality of edge nodes, and there may also be a plurality of operator networks, the caching logic of the cache 102 may be that all edge nodes of each operator network maintain one cache, for example, all telecommunications edge nodes maintain one cache logic, that is, files obtained by each telecommunications edge node all exist in the same logical cache 102, and further, files cached in the cache 102 may be used by all telecommunications edge nodes, that is, the finder 106 may find required files in the cache 102 maintained by all telecommunications edge nodes. In addition, the method can also be implemented by taking a region as a unit, that is, a plurality of edge nodes in a certain region of the same operator maintain a buffer 102 together, for example, a plurality of telecommunication edge nodes in beijing maintain a buffer together, a plurality of telecommunication edge nodes in shanghai maintain another buffer together, the telecommunication edge nodes in beijing can search whether a required file exists in the buffer in beijing, and if not, the required file is considered to be absent, and only the data source node can be obtained; the telecommunication edge node in Shanghai can search whether a needed file exists in the cache in Shanghai, and if not, the telecommunication edge node can only go to the data source node to obtain the file. Or, each edge node may also maintain a cache logic separately, that is, each edge node maintains its own cache 102, each edge node can only search its own cache, and if not found, it needs to obtain a file from the data source node. In summary, the buffering logic of the buffer 102 may be configured differently according to actual needs, and these are all within the scope of the present invention.

If the finder 106 does not find the corresponding file in the buffer 102 according to the content summary of the file, it notifies the back-to-source device 108, and the back-to-source device 108 obtains the requested file from the relevant data source node according to the back-to-source table and the download address of the requested file, and transmits the client device 700, and provides the requested file to the buffer 102 for buffering. Specifically, the source returning device 108 maintains a source returning table, in which the content summary and/or the data source node number of the file and the corresponding relationship between the data source node addresses are recorded.

The data source node number is understood to be the number of the data source node server or server cluster, and still the above URL example is taken as an example for explanation:

http. The "11" following "wsdl" in the URL is the number of the data source node, by which the host address corresponding to the number, such as the specific I P address of the data source node, can be found in the feed back table. If the label corresponds to a cluster of data source nodes, the host addresses of the data source nodes can be found in the back source table by the number, and the back source 108 can obtain the required file from any host.

As mentioned above, since the correspondence between the content digest of the file and the address of the data source node storing the file can also be stored in the source return table, the address of the data source node can also be searched by using the content digest of the requested file. In the feed back table, the content digest of the same file may also correspond to a plurality of data source node addresses, and the feed back device 108 selects one address from the plurality of addresses to the corresponding data source node to obtain the required file.

Whether the data source node storing the file is searched according to the data source node number in the URL of the requested file or the data source node storing the file is searched according to the content abstract of the requested file, if only one data source node storing the requested file is found, the file data is directly obtained according to the data source node address provided by the source returning table. If a plurality of data source nodes are found, a plurality of selection modes can be provided according to actual needs. For example, if the client device 700 is of the first carrier network and the edge node 100 is of the first carrier network, the back-source 108 preferably selects to obtain the required file from the data source node of the first carrier network if the found data source nodes storing the required file have both the data source node of the first carrier network and the data source node of the second carrier network. For another example, the load conditions of a plurality of data source nodes storing the required file may be considered, that is, one data source node with the relatively lightest load is selected according to the principle of load balancing to acquire the required file.

The contents of the source-back table maintained by the source-back unit 108 may be generated based on information obtained from the various data source nodes. For example, each time a file is uploaded to a data source node, a record is updated to the source return table, the record records a content summary of the file and/or a data source node number storing the file, and a specific address of the corresponding data source node, and further, address information of each file and the data source node storing the file can be recorded in the source return table. The source return table is not limited to be stored in the edge node 100, and may also be stored in other special management devices, as long as the data source node can communicate with the edge node 100 or the special management device, so that the information in the source return table can be updated according to the update of the data source node storage file, and the source return device 108 in the edge node 100 can acquire the information in the source return table in time. As mentioned above, the source returning device 108 may query the relevant data source node address only according to the data source node number in the URL of the requested file, or may query the relevant data source node address only according to the content summary of the requested file, and therefore, correspondingly, the source returning table may also record only the association relationship between the data source node number and the data source node address, or only record the association relationship between the content summary of the file and the data source node address. Certainly, the data source node number may also be preferentially used to query the address of the relevant data source node, and if the data source node is not found properly, for example, in the same operator network, the content abstract of the file is used to continue to query whether there are data source nodes storing the file in the same operator network.

The back-source 108 may include a data source query module, a direct back-source module, a proxy back-source module, and a cache notification module. Specifically, if the data source query module finds that the data source node storing the file and the edge node 100 are in the same operator network, such as the data source node 500 of the first operator network, according to the known data source table and the download address of the file download request, specifically, according to the data source node number obtained from the data source table and the download address or the content abstract of the file, the direct data source query module forwards the file download request to the found data source node 500, and obtains the required file from the data source node 500. Since the edge node 100 and the data source node 500 are both nodes of the first operator network, they can communicate directly with each other quickly without going through a proxy cluster, so that they can be connected directly for downloading. Meanwhile, the cache notification module notifies the cache 102 to cache the requested file acquired by the direct source returning module.

If the data source query module in the edge node 100 of the first operator network does not find in the source return table that the data source node belonging to the same operator network as the edge node 100 stores the requested file, and only finds that the data source node 600 of the second operator network stores the requested file, the proxy source return module obtains the requested file from the data source node 600 related to the second operator network through the proxy cluster 300 from the first operator network to the second operator network. At the same time, the cache notification module notifies the cache 102 to cache the requested file obtained by the proxy back-to-source module.

To better describe how the proxy back-source module obtains the requested file from the data source node 600 of the second operator network through the proxy cluster 300 from the first operator network to the second operator network, the following describes a specific structure of the proxy cluster 300. The proxy cluster 300 includes an edge side first operator network proxy cluster 302, a source node side second operator network proxy cluster 306, and a first transport channel 304. For example, fig. 4 is a schematic diagram of a proxy cluster for telecommunication to internet according to an embodiment of the present invention. The agent cluster comprises a telecommunication agent cluster, a network communication agent cluster and a fiber transmission channel between the two agent clusters. The telecommunications proxy cluster 412 corresponds to the edge-side first operator network proxy cluster 302, and the internet access proxy cluster 416 corresponds to the source-node-side second operator network proxy cluster 306. The telecommunications proxy cluster 412 communicates with telecommunications download nodes (i.e., telecommunications edge nodes) through a telecommunications egress port, and the network proxy cluster 416 communicates with a network-wide data source node through a network egress port.

First, the source returning device 108 of the edge node 100 of the first operator network sends a request for downloading a file from the data source node 600 to the edge-side first operator network proxy cluster 302 in the proxy cluster 300 according to the file downloading request of the client device 700 and the found address of the data source node 600, and then the edge-side first operator network proxy cluster 302 transmits the file downloading request to the source-node-side second operator network proxy cluster 306 through the first transmission channel 304, which is generally an optical fiber. Further, the source node side second operator network proxy cluster 306 sends a file download request to the data source node 600 of the relevant second operator network, and obtains the requested file from the data source node 600. After the source node side second operator network proxy cluster 306 obtains the requested file from the data source node 600, the requested file is transmitted to the edge side first operator network proxy cluster 302 through the first transmission channel 304, and then the edge side first operator network proxy cluster 302 transmits the file to the proxy source returning module of the edge node 100, and finally the proxy source returning module 108 of the edge node 100 returns the file to the client device 100. To this end, the client device 700 of the first operator network successfully downloads the requested file from the data source node of the second operator network through the edge node 100 of the first operator network, the proxy cluster 300 of the first operator network to the second operator network.

The structure of the proxy cluster 400 from the second operator network to the first operator network is the same as that of the proxy cluster from the first operator network to the second operator network, and therefore, the description is omitted and only a brief introduction is made. The proxy cluster 400 comprises an edge side second operator network proxy cluster 402, receives a file download request from the edge node 200 of the second operator network, and returns the requested file to the edge node 200 of the second operator network. The second transmission channel 404 transmits information from the edge side second operator network agent cluster 402 to the source node side first operator network agent cluster 406, and transmits information from the source node side first operator network agent cluster 406 to the edge side second operator network agent cluster 402; the source node side first operator network agent cluster 406 sends a file download request to the data source node 500 related to the first operator network according to a file download request from the edge side second operator network agent cluster 402 received through the second transmission channel 404, receives a requested file returned by the data source node 500 related to the first operator network, and transmits the requested file to the edge side second operator network agent cluster 200 through the second transmission channel 404.

As can be seen from the above description, the source feedback device 108 in the edge node 100 of the first carrier network may acquire a file required by the client device 700 from a data source node (e.g., the data source node 500 of the first carrier network or the data source node 600 of the second carrier network), and transmit the file to the client device 700. In one embodiment of the invention, the back source 108 may retrieve portions of the requested file from the data source node on a portion-by-portion basis and simultaneously transmit the retrieved portions of the file to the client device 700 until the file is completely retrieved. Specifically, after the data source node returns file header information to the source return 108 in the edge node 100, the edge node 100 immediately returns the header information to the client device 700, and the client device 100 knows the size of the requested file after obtaining the header information, and then keeps the link with the edge node 100, waits for and accepts the data returned by the edge node until the entire file is downloaded. After connecting to the data source node, the edge node 100 continuously obtains the content data of the file from the data source node, and continuously sends the obtained content data directly to the client device 700 until the file is completely transmitted. The edge node 100 will put the header information and the file content in the cache after obtaining the entire file from the data source node.

It can be seen that the edge node 100 can obtain the requested file from the data source node and transmit the partial content of the obtained file to the client device 700, and does not need to wait for the edge node 100 to completely download the entire file content from the data source node and then start transmitting the file to the client device 700. In this case, the back-source 108 in the edge node 100 can be considered a non-blocking back-source.

In an embodiment of the present invention, the download management device 110 may be implemented by a varnia sh cache server, or all or part of the edge node may be implemented by the varnia sh cache server. The varniash cache server can realize synchronous file transmission to the client device in the process of downloading the file from the data source node, namely, the file is provided with a non-blocking source returning device. Referring specifically to fig. 5, a schematic diagram of a data downloading system according to an embodiment of the present invention is shown, in which a telecommunications downloading node 512 is an embodiment of the edge node 100 of the first operator network in fig. 1, and the telecommunications downloading node 512 includes a plurality of varniash nodes; the netcom download node 514 is a specific example of the edge node 200 of the second operator network in fig. 1, and the netcom download node 514 also includes a plurality of varniash; telecom- > netcom agent cluster 516 is a specific example of the agent cluster 300 from the first operator network to the second operator network in fig. 1, and may include a plurality of agent clusters such as agent cluster 1 and agent cluster 2, which may be backup of each other; the netcom-telecommunications agent cluster 518 is a specific example of the agent cluster 400 from the second operator network to the first operator network in fig. 1, and also includes a plurality of agent clusters such as agent cluster 1 and agent cluster 2, which may be backup to each other; the telecommunication data source 520 is a specific example of the data source node 500 of the first operator network in fig. 1, which may include a plurality of data source nodes; the network data source 522 is a specific example of the data source node 600 of the second operator network in fig. 1, which may include a plurality of data source nodes. Each varniash node corresponds to a group of agent clusters, so that by using a failover (failure backup) mechanism of the Varnish node, when a certain agent cluster fails, other agent clusters can be automatically switched to, thereby reducing the risk of service unavailability.

Referring to fig. 6, a flowchart of a management method for providing a file download address according to an embodiment of the present invention is shown.

The management method starts in step S610, and in step S610, resource location information of a file is first generated according to a storage path of the file in a data source node. After a certain file is uploaded to a certain data source node, the data source node or other management devices capable of communicating with the data source node can obtain a storage path of the file on the data source node, and then resource location information of the file can be generated according to the storage path, namely, the resource location information is equivalent to the content of a path and a file name in a file URL. This step can be performed by the resource locator 212 in fig. 2, and the related art can refer to the description of the resource locator 212 in the foregoing embodiments, which is not described herein again.

On one hand, the resource location information of the file is generated through step S610, and on the other hand, in step S620, the content summary of the file is generated by applying specific data transformation to the content of the file. The content abstract of the file is essentially identification information of the file content, the file content abstract corresponds to file data one by one, and the content abstract of the file with different content is different. This step can be implemented by the digest generator 214 in fig. 2, and the related art may refer to the description of the digest generator 214 in the foregoing embodiments, which is not described herein again.

It should be noted that the order between step S610 and step S620 may be adjustable, for example, step S610 may be performed before, after, or simultaneously with step S620.

After the resource location information of the file is obtained through step S610 and the content digest of the file is obtained through step S620, a download address, such as a URL, of the file is generated at step S630 according to at least the resource location information of the file and the content digest of the file, where the download address includes at least the resource location information of the file and the content digest of the file. The download address of the file is generated and can be provided for the client device which needs to download the file for use. For example, when a client device clicks a certain file or a file list on a website and needs to download the file, the URL of the file can be obtained, and then, a subsequent client device can send a file download request to an edge node according to the download address, so that the edge node obtains the URL of the file, where the URL includes not only resource location information such as a storage path of the file, but also a content summary of the file. This step can be implemented by the download address generator 216 in fig. 2, and the related art may refer to the description of the download address generator 216 in the foregoing embodiments, which is not described herein again.

Turning next to fig. 7, a flowchart of a download management method for use in a data download system is shown, in accordance with one embodiment of the present invention. The data downloading system may include the edge node and the data source node described in the foregoing embodiments, and further may include the proxy cluster described in the foregoing embodiments.

The download management method starts in step S710, and in step S710, the edge node obtains a file download request from the client device and parses a content summary of the requested file according to a download address of the file download request. As described in the flow of the embodiment shown in fig. 6, the client device may obtain the URL of the file to be downloaded, and the URL also contains the content digest of the file, so that when the client device sends a download request to the edge node, the URL of the download address in the download request contains the content digest of the requested file.

After the edge node obtains the download request of the client device through step S710 and parses out the content digest of the requested file according to the download request, the edge node searches whether the requested file exists in the cache according to the content digest of the requested file in step S720. If yes, go to step S730: and acquiring the requested file from the cache and transmitting the file to the client device. If not, i.e. there is no file in the cache, step S740 is entered: the edge node obtains the requested file from the data source node related to the requested file and transmits the requested file to the client device.

In step S740 of one embodiment, the edge node may obtain portions of the requested file from the data source node portion by portion and simultaneously transmit the obtained portions of the requested file to the client device until the requested file is completely obtained. By the non-blocking source returning mode, the edge node does not need to wait for obtaining the whole file from the data source node and then transmit the file to the client equipment, and can simultaneously transmit the file to the client equipment in the process of obtaining the requested file from the data source node, so that the downloading speed of the file is further improved.

In step S740 of an embodiment, in order to further improve the downloading efficiency, if the edge node and the data source node are not nodes of the same operator network, the edge node may further obtain the requested file from the data source node through a proxy cluster, for example, the data downloading system further includes a proxy cluster from the first operator network to the second operator network. Step S740 specifically includes: firstly, the edge node inquires a data source node related to a requested file according to a known source return table and a download address of the requested file; when the edge node is an edge node of a first operator network and the data source node related to the requested file comprises the data source node of the first operator network, directly acquiring the requested file from the data source node of the first operator network; and when the edge node is the edge node of the first operator network and the data source node related to the requested file is the data source node of the second operator network, acquiring the requested file from the data source node related to the second operator network through the proxy cluster from the first operator network to the second operator network.

The process of the edge node directly obtaining the requested file from the data source node of the same operator network is simple and will not be described in detail, and the following mainly introduces the process of obtaining the requested file from the data source node of the non-same operator network through the proxy cluster. Specifically, the agent cluster from the first operator network to the second operator network comprises an edge side first operator network agent cluster, a transmission channel and a source node side second operator network agent cluster, wherein firstly, the edge side first operator network agent cluster receives a file downloading request from an edge node of the first operator network and transmits the file downloading request to the source node side second operator network agent cluster through the transmission channel; after receiving the file downloading request, the source node side second operator network agent cluster sends a file downloading request to a data source node related to a second operator network, receives a requested file returned by the data source node related to the second operator network, and transmits the file to the edge side first operator network agent cluster through a transmission channel; and then, the edge side first operator network agent cluster transmits the file to the edge node of the first operator network.

When or after the edge node obtains the file requested by the client in step S740, in step S750, the content summary of the file and the file content obtained from the data source node are cached, so that when there is a subsequent download request for downloading the file, the edge node does not need to go to the data source node to obtain the file, and the file is directly transmitted to the client device from its cache in step S730. Through the continuous accumulation in step S750, more and more files are stored in the edge node cache, and the requested file can be directly provided to the client device to a great extent without going to the data source node for downloading, so that the file downloading efficiency is improved, the number of times of communication with the data source node is reduced, and the load burden of the data source node is reduced.

In the above data processing procedure, step S710 may be performed by the parser 104 in fig. 1, steps S720 and S730 may be performed by the finder 106 in fig. 1, step S740 may be performed by the back source 108 in fig. 1, step S750 may be performed by the buffer 102 in fig. 1, and the steps performed by the above edge node may be performed by the varniash buffer server. For related technology implementation, reference may be made to the description of corresponding components in the foregoing embodiments, which is not described herein again.

It can be seen from the above embodiments of the present invention that, the edge node queries whether the requested file is already stored in the cache by using the file content digest parsed from the requested file download address as an index key, instead of using the entire URL of the file as an index query, and similarly, when the file is stored in the cache, it is determined whether the file is the same file according to whether the file content digests are the same, that is, the file content digest is an identifier of the file content data, and the file data itself is different as long as the file content digests are different, whereas the file data itself is the same as long as the file content digests are the same. Therefore, if the URLs of two files are different, but the content data of the files are substantially the same, the content digests of the files are the same, and further, if the files have been cached in the edge node before, even if the client device sends a different file download address URL again, as long as the content digests of the files in the URLs are consistent with those in the cache, the edge node does not go to the data source node to download the files repeatedly and cache the files, but directly finds the files from the cache according to the content digests of the files and provides the files to the client device. Therefore, on one hand, the repeated data in the cache is reduced, on the other hand, the efficiency of downloading the file for the client device is improved, meanwhile, the communication times from the edge node to the data source node are reduced, and further, the load of the data source node and the bandwidth consumption of the proxy cluster are reduced.

Further, there are cases where the file URL is not changed but the content data of the essence file has been updated. If the URL which does not carry the summary information of the file content in the prior art is used as the index key word for inquiring the cache file, the file with the changed substantial content data and unchanged URL is probably sent to the client device, and the latest and most correct file cannot be downloaded by the client device. By adopting the scheme of the embodiment of the invention, whether the file exists in the cache is judged according to the file content abstract in the file URL, if the URL is not changed and the actual data content is changed, the corresponding file can not be found when the cache is inquired through the file content abstract, and then the latest file can be automatically acquired from the data source node and provided for the client equipment, namely, the file in the cache is updated in time. In other words, the updating of the cache in the embodiment of the present invention is initiated by the edge node actively, rather than by the data source node in the existing CDN network, so that the problem of low updating efficiency when using the existing CDN data cache is avoided.

Furthermore, the edge node obtains the file from the data source node through the proxy cluster, and when the file is large, the file can be cached and downloaded by the client device by using a Stream non-blocking mode of an edge node server (such as Varnish). The problem that the downloading can be carried out only by waiting for the complete caching of the whole file when the CDN is used is avoided, and the client equipment does not need to wait in the whole process.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components of the download management device, the management device and the data download system according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Claims

1. A data download system comprises an edge node, an agent cluster for providing information interaction service between different operator networks, and a data source node, wherein the edge node comprises a download management device,

the agent cluster for providing information interaction service between different operator networks comprises an agent cluster from a first operator network to a second operator network and an agent cluster from the second operator network to the first operator network, wherein,

the cluster of proxies from the first operator network to the second operator network comprises:

an edge side first operator network agent cluster configured to receive a file download request from an edge node of a first operator network and return the requested file to the edge node of the first operator network;

a transmission channel configured to transmit information from the edge side first operator network proxy cluster to a source node side second operator network proxy cluster, and to transmit information from the source node side second operator network proxy cluster to the edge side first operator network proxy cluster; and the number of the first and second groups,

the source node side second operator network proxy cluster is configured to send a file downloading request to a data source node related to the second operator network according to a file downloading request received from the edge side first operator network proxy cluster through the transmission channel, receive a requested file returned by the data source node related to the second operator network, and transmit the requested file to the edge side first operator network proxy cluster through the transmission channel;

the second operator network to first operator network proxy cluster comprises:

an edge side second operator network agent cluster configured to receive a file download request from an edge node of a second operator network and return the requested file to the edge node of the second operator network;

a transmission channel configured to transmit information from the edge side second operator network proxy cluster to a source node side first operator network proxy cluster, and to transmit information from the source node side first operator network proxy cluster to the edge side second operator network proxy cluster; and the number of the first and second groups,

a source node side first operator network agent cluster configured to send a file download request to a data source node related to the first operator network according to a file download request from the edge side second operator network agent cluster received through the transmission channel, and receive a requested file returned by the data source node related to the first operator network and transmit the requested file to the edge side second operator network agent cluster through the transmission channel;

the download management apparatus includes:

the cache is configured to cache each file obtained from a data source node and a content abstract of each file, wherein the content abstract of each file is data generated by adopting specific data conversion on the content of each file;

the analyzer is configured to obtain a file downloading request from the client equipment and analyze the content abstract of the requested file according to the downloading address of the file downloading request;

the finder is configured to find the requested file in the cache according to the content abstract of the requested file analyzed by the analyzer, and if the requested file is found, the requested file is transmitted to the client device; and the number of the first and second groups,

and the source returning device is configured to acquire the requested file from a data source node related to the requested file and transmit the requested file to the client device when the requested file is not found in the cache by the finder, and provide the requested file to the cache for caching.

2. The system of claim 1, the back source comprising:

the data source query module is configured to query a data source node related to the requested file according to a known source return table and a download address of the requested file;

a direct source-returning module configured to obtain the requested file directly from the data source node of the first operator network when the edge node is an edge node of the first operator network and the queried data source node related to the requested file includes the data source node of the first operator network;

the proxy source returning module is configured to acquire the requested file from a data source node related to a second operator network through a proxy cluster from the first operator network to the second operator network when the edge node is an edge node of the first operator network and the queried data source node related to the requested file is a data source node of the second operator network; and

and the cache notification module is configured to notify the cache to cache the requested file after the requested file is acquired by the direct source returning module or the proxy source returning module.

3. The system of claim 1, the back source adapted to retrieve portions of the requested file from the data source node on a portion-by-portion basis and simultaneously transmit the retrieved portions of the requested file to the client device until the requested file is fully retrieved.

4. The system of claim 1, 2 or 3, the content digest of the file comprising: one of sha series data of secure hash algorithm, or one of MD series data of information summary algorithm.

5. A system as claimed in claim 1, 2 or 3, the download management device being a varniash cache server.

6. The system of claim 1, further comprising: the management equipment is used for providing a file downloading address, and the downloading address of the file downloading request received by the edge node is provided by the management equipment used for providing the file downloading address; wherein,

the management apparatus for providing a file download address includes:

the resource locator is configured to generate resource locating information of the file according to a storage path of the file in a data source node;

a summary generator configured to generate a summary of the content of the file by applying a specific data transformation to the content of the file; and

a download address generator configured to generate a download address of the file according to at least the resource location information of the file provided by the resource locator and the content digest of the file provided by the digest generator, wherein the download address at least includes the resource location information of the file and the content digest of the file.

7. The system of claim 6, the download address for the file generated by the download address generator being a uniform resource locator, URL, of the file.

8. A download management method for use in a data download system, the data download system including at least an edge node and a data source node, the download management method comprising:

the method comprises the steps that an edge node obtains a file downloading request from client equipment, and analyzes the content abstract of a requested file according to the downloading address of the file downloading request;

the edge node searches whether the requested file exists in the cache according to the content abstract of the requested file, and if so, transmits the requested file to the client equipment; and

if the requested file is not found in the cache, the edge node acquires the requested file from a data source node related to the requested file and transmits the requested file to the client device;

caching a file obtained from a data source node and a content abstract of the file by an edge node, wherein the content abstract of the file is data generated by converting the content of the file by adopting specific data;

the data download system further comprises a proxy cluster from the first carrier network to the second carrier network, the step of obtaining the requested file from the data source node associated with the requested file comprising:

inquiring a data source node related to the requested file according to the known source returning table and the download address of the requested file;

when the edge node is an edge node of a first operator network and the inquired data source node related to the requested file comprises the data source node of the first operator network, directly acquiring the requested file from the data source node of the first operator network;

when the edge node is an edge node of a first operator network and the inquired data source node related to the requested file is a data source node of a second operator network, acquiring the requested file from the data source node related to the second operator network through a proxy cluster from the first operator network to the second operator network;

the proxy cluster from the first operator network to the second operator network comprises an edge side first operator network proxy cluster, a transmission channel and a source node side second operator network proxy cluster, and the step that the edge node acquires the requested file from a data source node related to the second operator network through the proxy cluster from the first operator network to the second operator network comprises the following steps:

the method comprises the steps that a first operator network agent cluster at an edge side receives a file downloading request from an edge node of a first operator network and transmits the file downloading request to a second operator network agent cluster at a source node side through a transmission channel;

a second operator network agent cluster at a source node side sends a file downloading request to a data source node related to a second operator network according to a file downloading request from a first operator network agent cluster at an edge side, and receives a requested file returned by the data source node related to the second operator network;

the second operator network agent cluster at the source node side transmits the requested file to the first operator network agent cluster at the edge side through the transmission channel;

the edge side first operator network agent cluster transmits the requested file to an edge node of the first operator network.

9. The download management method as claimed in claim 8, said step of obtaining the requested file from the data source node to which the requested file relates comprising:

portions of the requested file are retrieved from the data source node on a portion-by-portion basis, and the retrieved portions of the requested file are simultaneously transmitted to the client device until the requested file is fully retrieved.

10. A download management method as claimed in claim 8 or 9, wherein said steps are performed by a varniash cache server.