CN103281394A - File acquiring method, node servers and system - Google Patents
File acquiring method, node servers and system Download PDFInfo
- Publication number
- CN103281394A CN103281394A CN2013102265360A CN201310226536A CN103281394A CN 103281394 A CN103281394 A CN 103281394A CN 2013102265360 A CN2013102265360 A CN 2013102265360A CN 201310226536 A CN201310226536 A CN 201310226536A CN 103281394 A CN103281394 A CN 103281394A
- Authority
- CN
- China
- Prior art keywords
- file
- popular
- node server
- information
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a file acquiring method, node servers and a system. The file acquiring method of the node servers in the content distribution network system includes the steps of acquiring information of one or more hot files from a hot file server, acquiring file location information of at least one hot file of the one or more hot files from a file location server, wherein the file location information indicates information of at least one node server where the hot files are stored, and downloading the hot files from the at least one node server where the hot files are stored and storing the hot files in the node servers according to the file location information of the hot files, wherein the hot files are files with file visiting times larger than pre-determined times.
Description
Technical field
The present invention relates to technical field of the computer network, particularly the file acquisition method of the node server in a kind of content distribution network system, the node server in the content distribution network system and content distribution network system.
Background technology
CDN (Content Delivery Network, content distributing network) is a kind of by place one deck intelligent virtual network on existing basis, the Internet that node server constitutes everywhere at network.The basic ideas of CDN are to avoid as far as possible might influencing on the Internet bottleneck and the link of data transmission bauds and stability, make the faster, more stable of content delivery.
The CDN system can be in real time leads user's request on the nearest node server of user again according to being connected of network traffics and each node, load state and with integrated informations such as user's distance and response times, its objective is and make the user can obtain required content nearby, solve the situation of Internet (internet) network congestion, improve the response speed of user's access websites.
When the scale of CDN system need enlarge, the new node server of need in the CDN system, reaching the standard grade.Prior art is when newly reaching the standard grade a node server, specify a source server, by the Clone instrument, rysnc (remote synchronize for example, remote synchronization) instrument, with the All Files on the source server according to identical clone as a result to the new node server.After Clone finishes, carry out the verification of data full dose after, the new node server is reached the standard grade.There is following defective at least in this existing scheme:
1, length consuming time.The size of full dose data is usually in the T magnitude on the source server, and is even with other transmission speed of per second M level, consuming time also one day or a couple of days.
2, the occupied bandwidth resource is many.Because the data volume of clone is bigger, need bigger data bandwidth, particularly source server and new node server not when an IDC (Internet Data Center, Internet data center), can take the bandwidth resources of a large amount of preciousnesses for a long time.
3, transfer of data complexity and verification complexity are higher.Owing to need the data file of transmission in the T magnitude, have complicated bibliographic structure, can cause the verification complexity very high, and millions of files, when bust this, retransmitting complexity also can be higher.
4, real-time is poor.Since need be in the data file of T magnitude the data that need of search request end, cause the real-time satisfaction lower.
5, take the memory space height.Data file has taken the memory space of number T in the new node server, yet a lot of file all is the file that can requestedly not have access to.
Perhaps, when prior art is disposed the new node server, be empty during the cache initialization of new node server, without any content.When the user asks to arrive this new node server, check that file whether in cache, if file exists, then reads the back returned content and gives the user from cache; If file does not exist, then go after the source station is grasped, to be stored in the cache or local disk by returning the source side formula.There is following defective at least in this existing scheme:
1, real-time is relatively poor.Put into local cache because want to reach back data from the source station earlier, just can send to the request end.Time and network transmission cost all can be double, and real-time is poor.As running into the situation of the big file of acquisition request, real-time can worsen more, can consume a large amount of internal memories simultaneously, and hit rate can be because file is lower greatly.
2, less stable.When source station and local network were relatively poor, the file of request can't be fetched, and can't send to the request end.If cache just in internal memory, then can't recover the data among the cache after the power down.
3, source station pressure is bigger.Because a large amount of requests all need to obtain by returning the source side formula, the data processing pressure of source station is big.As the sudden generation of popular file, source station pressure can be very big.
Summary of the invention
In view of the above problems, the present invention has been proposed in order to file acquisition method, the node server in the content distribution network system and the content distribution network system of the node server in a kind of a kind of content distribution network system that overcomes the problems referred to above or address the above problem at least in part are provided.
According to one aspect of the present invention, the embodiment of the invention provides the file acquisition method of the node server in a kind of content distribution network system.This content distribution network system comprise the information that is suitable for providing popular file popular file server, be suitable for providing document location server and one or more node server of file location information, this document acquisition methods comprises: the information of obtaining one or more popular file from popular file server; Obtain the file location information of at least one the popular file one or more popular file from the document location server, the indication of this document positional information stores the information of at least one node server of this hot topic file; And according to the file location information of popular file, download popular file from least one node server that stores this hot topic file and be stored in the node server; Wherein, the above-mentioned popular file access times that are file surpass the file of pre-determined number.
Alternatively, the information of above-mentioned popular file comprises Message Digest Algorithm 5 MD5 value, file name and/or the bibliographic structure of file.
Alternatively, above-mentioned file location information according to popular file, downloading popular file from least one node server that stores this hot topic file comprises: when knowing that popular file is stored in a plurality of node servers, judge whether the size of this hot topic file surpasses file download threshold value; When the size of popular file surpasses file and downloads threshold value, according to the quantity of the node server at this hot topic file place popular file is divided into a plurality of parts; From each node server at this hot topic file place, download the part of this hot topic file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.
Whether alternatively, surpass before file downloads threshold value in the above-mentioned size of judging this hot topic file, said method: know the size of popular file according to the information of popular file, also comprise the size of popular file in the information of this hot topic file if also comprising; Perhaps, to arbitrary node server transmission query messages at popular file place, know the size of popular file according to the Query Result that returns.
Alternatively, above-mentioned download popular file from least one node server that stores this hot topic file after, said method also comprises: MD5 value that calculate to download the popular file that obtains; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
Alternatively, said method also comprises: when receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, the file destination that inquires is back to the request end, if do not exist, from the document location server, obtains the file location information of file destination, after download obtains file destination according to this document positional information, file destination is back to the request end.
According to another aspect of the present invention, the embodiment of the invention provides the node server in a kind of content distribution network system.This node server comprises: the information getter is suitable for obtaining the information of one or more popular file from content distributing network; File retainer is suitable for obtaining the file location information of at least one the popular file in one or more popular file from content distributing network, the indication of this document positional information stores the information of at least one node server of this hot topic file; And file downloader is suitable for the file location information according to popular file, downloads popular file from least one node server that stores this hot topic file; File memory, the popular file that is suitable for downloading to is stored in the node server; Wherein, the above-mentioned popular file access times that are file surpass the file of pre-determined number.
Alternatively, the information of the popular file that gets access to of information getter comprises Message Digest Algorithm 5 MD5 value, file name and/or the bibliographic structure of file.
Alternatively, file downloader is suitable for when knowing that popular file is stored in a plurality of node servers, judges whether the size of this hot topic file surpasses file download threshold value; When the size of popular file surpasses file and downloads threshold value, according to the quantity of the node server at this hot topic file place popular file is divided into a plurality of parts; From each node server at this hot topic file place, download the part of popular file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.
Alternatively, file downloader is suitable for knowing according to the information of popular file the size of popular file, also comprises the size of popular file in the information of this hot topic file; Perhaps, file downloader is suitable for sending query messages to arbitrary node server at popular file place; Know the size of popular file according to the Query Result that returns.
Alternatively, file downloader is suitable for calculating the MD5 value of downloading the popular file that obtains; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
Alternatively, node server also comprises the access request processor, when being suitable for receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, the file destination that inquires is back to the request end, if do not exist, the access request processor triggers file retainer is obtained file destination from content distributing network file location information, and trigger file downloader according to this document positional information download obtain file destination after, file destination is back to the request end.
According to another aspect of the present invention, the embodiment of the invention provides a kind of content distribution network system.This system comprise one or more as the node server in the above-mentioned content distribution network system, be suitable for providing popular file information popular file server and be suitable for providing the document location server of file location information.This node server is suitable for obtaining the information of one or more popular file from popular file server; And this node server is suitable for obtaining from the document location server file location information of at least one the popular file one or more popular file.
Therefore, the embodiment of the invention has adopted the popular file in the definite CDN system, information and file location information by the popular file that obtains are downloaded to the technological means of new node server to realize that the new node server is reached the standard grade with popular file, because the size of popular file is usually in the G magnitude, be far smaller than the T magnitude data of transmitting under the Clone scheme, thereby compare the scheme with existing C lone, shortened data transmission period greatly, the bandwidth resources that transfer of data takies have been reduced, reduced the transfer of data complexity, the verification complexity, and data search speed is fast, and the real-time of system is better, and the memory space that data file takies is less.
And, because the embodiment of the invention has adopted the popular file that will download to be stored in this locality, the technological means of popular document source information is provided by file location information, than existing cache scheme, response speed is fast, real-time and better, the stability of data file is better, and the data processing pressure of source station is little.
Above-mentioned explanation only is the general introduction of technical solution of the present invention, for can clearer understanding technological means of the present invention, and can be implemented according to the content of specification, and for above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Description of drawings
By reading hereinafter detailed description of the preferred embodiment, various other advantage and benefits will become cheer and bright for those of ordinary skills.Accompanying drawing only is used for the purpose of preferred implementation is shown, and does not think limitation of the present invention.And in whole accompanying drawing, represent identical parts with identical reference symbol.In the accompanying drawings:
Fig. 1 shows the structural representation of the node server in the content distribution network system according to an embodiment of the invention;
Fig. 2 shows the configuration diagram of content distribution network system in accordance with another embodiment of the present invention;
Fig. 3 shows the file acquisition method schematic flow sheet of the node server in the content distribution network system of another embodiment according to the present invention.
Embodiment
Exemplary embodiment of the present disclosure is described below with reference to accompanying drawings in more detail.Though shown exemplary embodiment of the present disclosure in the accompanying drawing, yet should be appreciated that and to realize the disclosure and the embodiment that should do not set forth limits here with various forms.On the contrary, it is in order to understand the disclosure more thoroughly that these embodiment are provided, and can with the scope of the present disclosure complete convey to those skilled in the art.
The technical conceive of this programme mainly is to solve existing CDN system handles new node server when reaching the standard grade, linear velocity is slow on the Clone mode, transfer of data is complicated, bandwidth waste is serious, cache mode stability and real-time problem such as high inadequately again, arrange by rational at aspects such as on-line time, data stability, real-times, reach node server and reach the standard grade fast, have the purpose of high real-time and reliability simultaneously.
One embodiment of the invention provides the node server in a kind of content distribution network system.Referring to Fig. 1, this node server comprises information getter 110, file retainer 112, file downloader 114, file memory 116 and access request processor 118.Respectively these devices are described below.
The information of the popular file that information getter 110 gets access to comprises one or more in MD5 (Message Digest Algorithm 5) value, file name and the bibliographic structure of file, but be not limited to this, as comprising the information of the size of indicating popular file in the information of popular file.The MD5 value is the information that uniqueness identifies popular file in content distributing network, and bibliographic structure is the information of indication file store path structure in content distributing network.
Be appreciated that the information that can comprise other uniqueness identification document in content distributing network in the information of popular file, to be alternatively to above-mentioned MD5 value; And comprise that other and bibliographic structure have the information of identical function, to be alternatively to above-mentioned bibliographic structure.The information of above-mentioned popular file can be recorded in the popular listed files of setting.As shown in fig. 1, the information of file of the every line item of popular listed files (being popular file), in the example of Fig. 1, popular listed files itemize has recorded the fileinfo of three files (file 1, file 2, file 3).
The example of the information of the popular file of storing in the popular listed files can be as follows:
19a4b94ba6603c0c0537eda5df4c1196, "/v3/libleak.dat/libleak.dat.1.0.1.2415-1.0.1.2416.cab ", wherein, 19a4b94ba6603c0c0537eda5df4c1196 represents the MD5 value of file, part in the quotation marks has been represented the version (v3) of file, file name and bibliographic structure.
3012a03fe2f0fca74f794cd6b4da5424, "/cse/switcher/switcher_4.0.1.748.cab ", wherein, 3012a03fe2f0fca74f794cd6b4da5424 represents the MD5 value of file, and the part in the quotation marks has been represented file name and bibliographic structure.
7c97a331614dccb98c8dc1be5c5df49f, "/v3/urllib.dat/urllib.dat.1.0.0.6469-1.0.0.6471.cab ", wherein, 7c97a331614dccb98c8dc1be5c5df49f represents the MD5 value of file, part in the quotation marks has been represented the version (v3) of file, file name and bibliographic structure.
14c198e95e66635b77c625c8cfacaba2, "/v3/urllib.dat/urllib.dat.1.0.0.6473-1.0.0.6475.cab ", wherein, 14c198e95e66635b77c625c8cfacaba2 represents the MD5 value of file, part in the quotation marks has been represented the version (v3) of file, file name and bibliographic structure.
71dfcb4e3e407be657a032eaf33d3ad5, "/v3/urllib.dat/urllib.dat.1.0.0.6467-1.0.0.6471.cab ", wherein, 71dfcb4e3e407be657a032eaf33d3ad5 represents the MD5 value of file, part in the quotation marks has been represented the version (v3) of file, file name and bibliographic structure.
Need to prove, present embodiment can screen the file in the content distributing network in advance, distinguish popular file and non-popular file (being the unexpected winner file), the access times that popular file is file surpass the file of pre-determined number, for example total access times surpass the file of pre-determined number, and perhaps the file access number of times in certain time interval surpasses the file of pre-determined number.For example, content distributing network can utilize the Map Reduce algorithm on the distributed system architecture Hadoop to carry out computing, obtains having ageing focus listed files.Information getter 110 obtains the operation of the information of popular file from content distributing network, specifically can be as follows:
1), moving the statistics program of file access number of times on each node server, this statistics program periodically is submitted to popular file server after can the number of times that file is accessed adding up;
2), after popular file server gathers these data, data are stored in the Hadoop cluster;
3), every the scheduled time (as 5 minutes), popular file server calculates these data by Map Reduce algorithm, obtains the focus listed files and preserves according to result of calculation;
4), information getter 110 is downloaded from popular file server and is obtained popular listed files.
An example of the file location information that present embodiment provides is as follows:
19a4b94ba6603c0c0537eda5df4c1196，″/v3/libleak.dat/libleak.dat.1.0.1.2415-1.0.1.2416.cab″，“machine1；machine2；machine3；machine4”
In above-mentioned example, file location information has comprised the information of file and the information of indication file place node server, i.e. machine1; Machine2; Machine3; Machine4, the node server at indication file place comprises machine1, machine2, machine3 and machine4 in the above-mentioned example.
Need to prove, above-mentioned popular file server and document location server can be realized by the node server among the CDN, such node server can also be carried out the function of popular file server and/or document location server except the function with general node server; Perhaps, above-mentioned popular file server and document location server are realized by the proprietary server that is arranged among the CDN, this proprietary server is only realized the function of popular file server and/or document location server, and need not to have the function of general node server.
Can store popular file in one or more node server.Be stored in the scene of a plurality of node servers for popular file, it is that the multithreading download is carried out in the source with a plurality of node servers that present embodiment also provides a kind of, the processing mode that multifile is downloaded simultaneously, at this moment, file downloader 114 is suitable for when knowing that popular file is stored in a plurality of node servers, judges whether the size of this hot topic file surpasses file download threshold value (as 5MB); When the size of popular file surpasses file download threshold value, quantity according to the node server at this hot topic file place is divided into a plurality of parts with popular file, for example can be divided into a plurality of parts, also can be according to the weight of respective nodes server (can connection speed Network Based, server performance waits to calculate weight) and divide; From each node server at this hot topic file place, download the part of popular file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.Adopt this downloading mode, file synchronization speed promotes obviously, and the required bandwidth of file synchronization disperses more, can not cause local bandwidth congestion, and, the unsteadiness of having avoided the source single-point to download, reliability is stronger.
Mode one, file downloader 114 are known according to the information of popular file the size of popular file under this mode, need comprise the information of the size of indicating popular file in the information of this hot topic file.
Mode two, file downloader 114 are suitable for sending query messages to arbitrary node server at popular file place; Know the size of popular file according to the Query Result that returns.As arbitrary node server transmission HTTP (the Hypertext Transfer Protocol of file downloader 114 to popular file place, HTML (Hypertext Markup Language)) query messages, the parameter of acquisition request file size is set in the head (packet header) of this HTTP query messages, file downloader 114 receiving node servers are replied this HTTP query messages, know the size of popular file.
Alternatively, file downloader 114 can verify that after downloading and making up the popular file that finishes at this moment, file downloader 114 is calculated the MD5 value of downloading the popular file that obtains to the correctness of this hot topic file; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
The popular file that file memory 116 is suitable for downloading to is stored in the node server.Popular file and alternative document that 116 pairs of node servers of newly reaching the standard grade of file memory get access to are stored, and in the scene shown in Figure 1, the file itemize that 116 pairs of file memories get access to is stored, as file 1, file 2 and file 3 etc.
Node server also comprises access request processor 118, in order to after node server is reached the standard grade access request is handled.When access request processor 118 is suitable for receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, show that file destination is focus file or the unexpected winner file of before having asked, the file destination that inquires is back to the request end, if do not exist, show that file destination is the unexpected winner file of before not asking, the access request processor triggers file retainer 112 is obtained file destination from content distributing network file location information, and trigger file downloader 114 according to this document positional information download obtain file destination after, file destination is back to the request end.
When the unexpected winner file of downloading as file destination, can adopt equally with a plurality of node servers is that the multithreading download is carried out in the source, the processing mode that multifile is downloaded simultaneously, concrete mode is identical with the mode that above-mentioned file downloader 114 multithreadings are downloaded popular file, does not repeat them here.
The framework of the content distribution network system 200 that provides below in conjunction with the another embodiment of the present invention of Fig. 2 describes.Content distribution network system 200 comprises node server 210 in one or more content distribution network system, be suitable for providing popular file information popular file server 212 and be suitable for providing the document location server 214 of file location information.
Node server comprises reached the standard grade in the content distribution network system node server that uses and the new node server that is about to reach the standard grade and use.The new node server that uses that is about to reach the standard grade establishes and is connected with popular file server 212, document location server 214, and the new node server that uses that is about to reach the standard grade also needs to connect with the node server of having reached the standard grade that needs the downloaded files place.
The new node server 210 that uses of being about to reach the standard grade is suitable for obtaining the information of one or more popular file from popular file server 212; And this node server 210 is suitable for obtaining from document location server 214 file location information of at least one the popular file one or more popular file.After new node server 210 is reached the standard grade, reception is during from the access request of the request access destination file of request end, according to access request at the local search file destination, if find file destination, this file destination is back to the request end, if do not find file destination, from document location server 214, obtains the file location information of file destination, and according to this document positional information download obtain file destination after, file destination is back to the request end.
The concrete mode that the new node server that uses of being about to reach the standard grade obtains file can not repeat them here referring to other embodiment of the present invention.
Further embodiment of this invention provides the file acquisition method of the node server in a kind of content distribution network system.This content distribution network system comprise the information that is suitable for providing popular file popular file server, be suitable for providing document location server and one or more node server of file location information, referring to Fig. 3, this document acquisition methods starts from step S300, comprises the steps:
S300: the information of from popular file server, obtaining one or more popular file.
The access times that above-mentioned popular file is file surpass the file of pre-determined number, and for example total access times surpass the file of pre-determined number, and perhaps the file access number of times in certain time interval surpasses the file of pre-determined number.The information of above-mentioned popular file is including, but not limited to MD5 value, file name and/or the bibliographic structure of file.The MD5 value is the information that uniqueness identifies popular file in content distributing network, and bibliographic structure is the information of indication file store path structure in content distributing network.
After knowing the synchronous popular file of needs according to the information of popular file, enter step S302.
S302: obtain the file location information of at least one the popular file one or more popular file from the document location server, the indication of this document positional information stores the information of at least one node server of this hot topic file, enters step S304 then.
S304: according to the file location information of popular file, download popular file from least one node server that stores this hot topic file and be stored in the node server.
When knowing that popular file is stored in a plurality of node servers, judge whether the size of this hot topic file surpasses file download threshold value (as 5MB) in this step; When the size of popular file surpasses file download threshold value, quantity according to the node server at this hot topic file place is divided into a plurality of parts with popular file, for example can be divided into a plurality of parts, also can be according to the weight of respective nodes server (can connection speed Network Based, server performance waits to calculate weight) and divide; From each node server at this hot topic file place, download the part of popular file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.Adopt this downloading mode, file synchronization speed promotes obviously, and the required bandwidth of file synchronization disperses more, can not cause local bandwidth congestion, and, the unsteadiness of having avoided the source single-point to download, reliability is stronger.
In addition, whether surpass before file downloads threshold value in the above-mentioned size of judging this hot topic file, can know the size of popular file at least by following dual mode: know the size of popular file according to the information of popular file, also comprise the size of popular file in the information of this hot topic file; Perhaps, to arbitrary node server transmission query messages at popular file place, know the size of popular file according to the Query Result that returns.
After execution of step S304, when needed, can also carry out verification to the file that download obtains, guarantee integrality and the correctness of file, at this moment, enter step S306.After not needing when downloading file and carry out verification, in step S304, downloading to file, file is stored in the new node server that is about to reach the standard grade end operation.
S306: judge whether the file download is successful, if download successfully, in node server, preserve the popular file that downloads to, finish this operation; If failed download, the file that deletion downloads to finishes this operation.Can initiate the down operation to the file of this failed download again.
When judge downloading whether success, calculate the MD5 value of downloading the popular file that obtains; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
After the new node server of execution of step S300 to S306 is reached the standard grade, said method also comprises: when receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, show that this file destination is popular file or the unexpected winner file of before having asked, the file destination that inquires is back to the request end, if do not exist, show the unexpected winner file of this file destination for before not asking, from the document location server, obtain the file location information of file destination, after download obtains file destination according to this document positional information, file destination is back to the request end.
The concrete executive mode of each step can not repeat them here referring to other embodiment of the present invention among the inventive method embodiment.
From the above mentioned, the embodiment of the invention has following advantage at least:
1, by the popular file of Map Reduce algorithm picks and safeguard the technological means of the information of popular file, removed the transmission of unexpected winner file.Because popular file shared ratio in conceptual data is less, need synchronous quantity of documents thereby greatly reduce;
2, adopt multi-thread concurrent to download the technological means of file respectively from a plurality of node servers based on file location information, file synchronization speed is obviously promoted, and the required bandwidth of file synchronization is disperseed more, can not cause local bandwidth congestion, and, the unsteadiness of having avoided the source single-point to download, reliability is stronger.
3, for user's access request, search in the popular file of this locality earlier, the technological means of obtaining from other node servers when this locality does not have corresponding file reaches under the situation of uninterrupted service, replenishes the effect of disappearance file fast;
4, this programme can significantly reduce invalid bandwidth consumption, obviously shorten the node on-line time, and reliability and stability are higher.
Therefore, the embodiment of the invention has adopted the popular file in the definite CDN system, information and file location information by the popular file that obtains are downloaded to the technological means of new node server to realize that the new node server is reached the standard grade with popular file, because the size of popular file is usually in the G magnitude, be far smaller than the T magnitude data of transmitting under the Clone scheme, thereby compare the scheme with existing C lone, shortened data transmission period greatly, the bandwidth resources that transfer of data takies have been reduced, reduced the transfer of data complexity, the verification complexity, and data search speed is fast, and the real-time of system is better, and the memory space that data file takies is less.
And, because the embodiment of the invention has adopted the popular file that will download to be stored in this locality, the technological means of popular document source information is provided by file location information, than existing cache scheme, response speed is fast, real-time and better, the stability of data file is better, and the data processing pressure of source station is little.
A5, according to any described method among the A1-4, wherein, described download described popular file from least one node server that stores this hot topic file after, described method also comprises: MD5 value that calculate to download the popular file that obtains; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
A6, according to any described method among the A1-5, wherein, described method also comprises: when receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, the file destination that inquires is back to the request end, if do not exist, from the document location server, obtain the file location information of described file destination, after download obtains file destination according to this document positional information, file destination is back to the request end.
B12, according to any described node server among the B7-11, wherein, described node server also comprises the access request processor, when being suitable for receiving the access request of request access destination file, according to whether having file destination in this access request query node server, if exist, the file destination that inquires is back to the request end, if do not exist, described access request processor triggers file retainer is obtained described file destination from content distributing network file location information, and trigger file downloader according to this document positional information download obtain file destination after, file destination is back to the request end.
Intrinsic not relevant with any certain computer, virtual system or miscellaneous equipment with demonstration at this algorithm that provides.Various general-purpose systems also can be with using based on the teaching at this.According to top description, it is apparent constructing the desired structure of this type systematic.In addition, the present invention is not also at any certain programmed language.Should be understood that and to utilize various programming languages to realize content of the present invention described here, and the top description that language-specific is done is in order to disclose preferred forms of the present invention.
In the specification that provides herein, a large amount of details have been described.Yet, can understand, embodiments of the invention can be put into practice under the situation of these details not having.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand one or more in each inventive aspect, in the description to exemplary embodiment of the present invention, each feature of the present invention is grouped together in single embodiment, figure or the description to it sometimes in the above.Yet the method for the disclosure should be construed to the following intention of reflection: namely the present invention for required protection requires the more feature of feature clearly put down in writing than institute in each claim.Or rather, as following claims reflected, inventive aspect was to be less than all features of the disclosed single embodiment in front.Therefore, follow claims of embodiment and incorporate this embodiment thus clearly into, wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are appreciated that and can adaptively change and they are arranged in one or more equipment different with this embodiment the module in the equipment among the embodiment.Can become a module or unit or assembly to the module among the embodiment or unit or combination of components, and can be divided into a plurality of submodules or subelement or sub-component to them in addition.In such feature and/or process or unit at least some are mutually repelling, and can adopt any combination to disclosed all features in this specification (comprising claim, summary and the accompanying drawing followed) and so all processes or the unit of disclosed any method or equipment make up.Unless clearly statement in addition, disclosed each feature can be by providing identical, being equal to or the alternative features of similar purpose replaces in this specification (comprising claim, summary and the accompanying drawing followed).
In addition, those skilled in the art can understand, although embodiment more described herein comprise some feature rather than further feature included among other embodiment, the combination of features of different embodiment means and is within the scope of the present invention and forms different embodiment.For example, in the following claims, the one of any of embodiment required for protection can be used with compound mode arbitrarily.
Each parts embodiment of the present invention can realize with hardware, perhaps realizes with the software module of moving at one or more processor, and perhaps the combination with them realizes.It will be understood by those of skill in the art that and to use microprocessor or digital signal processor (DSP) to realize according to some or all some or repertoire of parts in the node server in the content distribution network system of the embodiment of the invention in practice.The present invention can also be embodied as for part or all equipment or the device program (for example, computer program and computer program) of carrying out method as described herein.Such realization program of the present invention can be stored on the computer-readable medium, perhaps can have the form of one or more signal.Such signal can be downloaded from internet website and obtain, and perhaps provides at carrier signal, perhaps provides with any other form.
It should be noted above-described embodiment the present invention will be described rather than limit the invention, and those skilled in the art can design alternative embodiment under the situation of the scope that does not break away from claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and is not listed in element or step in the claim.Being positioned at word " " before the element or " one " does not get rid of and has a plurality of such elements.The present invention can realize by means of the hardware that includes some different elements and by means of the computer of suitably programming.In having enumerated the unit claim of some devices, several in these devices can be to come imbody by same hardware branch.Any order is not represented in the use of word first, second and C grade.Can be title with these word explanations.
Claims (10)
1. the file acquisition method of the node server in the content distribution network system, this content distribution network system comprise the information that is suitable for providing popular file popular file server, be suitable for providing document location server and one or more node server of file location information, this document acquisition methods comprises:
From described popular file server, obtain the information of one or more popular file;
Obtain the file location information of at least one the popular file described one or more popular file from described document location server, described file location information indication stores the information of at least one node server of this hot topic file; And
According to the file location information of described popular file, download described popular file from least one node server that stores this hot topic file and be stored in the node server;
Wherein, the described popular file access times that are file surpass the file of pre-determined number.
2. method according to claim 1, wherein, the information of described popular file comprises Message Digest Algorithm 5 MD5 value, file name and/or the bibliographic structure of file.
3. according to claim 1 or 2 described methods, wherein, described file location information according to described popular file, download described popular file from least one node server that stores this hot topic file and comprise:
When knowing that popular file is stored in a plurality of node servers, judge whether the size of this hot topic file surpasses file download threshold value;
When the size of popular file surpasses file and downloads threshold value, according to the quantity of the node server at this hot topic file place popular file is divided into a plurality of parts;
From each node server at this hot topic file place, download the part of this hot topic file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.
4. whether method according to claim 3 wherein, surpasses before file downloads threshold value in the described size of judging this hot topic file, and described method also comprises:
Know according to the information of popular file and the size of popular file wherein, also to comprise the size of popular file in the information of described popular file; Perhaps,
Arbitrary node server to popular file place sends query messages, knows the size of popular file according to the Query Result that returns.
5. the node server in the content distribution network system, described node server comprises:
The information getter is suitable for obtaining the information of one or more popular file from content distributing network;
File retainer is suitable for obtaining the file location information of at least one the popular file in described one or more popular file from content distributing network, described file location information indication stores the information of at least one node server of this hot topic file; And,
File downloader is suitable for the file location information according to described popular file, downloads described popular file from least one node server that stores this hot topic file;
File memory, the popular file that is suitable for downloading to is stored in the node server;
Wherein, the described popular file access times that are file surpass the file of pre-determined number.
6. node server according to claim 5, wherein, the information of the popular file that described information getter gets access to comprises Message Digest Algorithm 5 MD5 value, file name and/or the bibliographic structure of file.
7. according to claim 5 or 6 described node servers, wherein, described file downloader is suitable for when knowing that popular file is stored in a plurality of node servers, judges whether the size of this hot topic file surpasses file download threshold value; When the size of popular file surpasses file and downloads threshold value, according to the quantity of the node server at this hot topic file place popular file is divided into a plurality of parts; From each node server at this hot topic file place, download the part of popular file respectively, and the each several part that downloads to is carried out combination of files obtain this hot topic file.
8. node server according to claim 7, wherein, described file downloader is suitable for knowing according to the information of popular file the size of popular file, wherein, also comprises the size of popular file in the information of described popular file; Perhaps,
Described file downloader is suitable for sending query messages to arbitrary node server at popular file place; Know the size of popular file according to the Query Result that returns.
9. according to any described node server among the claim 5-8, wherein, described file downloader is suitable for calculating the MD5 value of downloading the popular file that obtains; Whether corresponding MD5 value is identical in the MD5 value that relatively calculates and the information of popular file, if identical, confirms to download successfully, keeps this hot topic file, and as if inequality, the affirmation failed download is deleted this hot topic file.
10. content distribution network system, described system comprise one or more as the node server in each described content distribution network system of above-mentioned claim 5 to 9, be suitable for providing popular file information popular file server and be suitable for providing the document location server of file location information
Described node server is suitable for obtaining the information of one or more popular file from described popular file server;
Described node server is suitable for obtaining from described document location server the file location information of at least one the popular file described one or more popular file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102265360A CN103281394A (en) | 2013-06-07 | 2013-06-07 | File acquiring method, node servers and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102265360A CN103281394A (en) | 2013-06-07 | 2013-06-07 | File acquiring method, node servers and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103281394A true CN103281394A (en) | 2013-09-04 |
Family
ID=49063839
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013102265360A Pending CN103281394A (en) | 2013-06-07 | 2013-06-07 | File acquiring method, node servers and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103281394A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103747049A (en) * | 2013-12-24 | 2014-04-23 | 乐视网信息技术（北京）股份有限公司 | CDN file distribution method, control center and system |
| CN103986977A (en) * | 2014-04-15 | 2014-08-13 | 上海聚力传媒技术有限公司 | Method and device for preloading video in content distribution network |
| CN104392008A (en) * | 2014-12-19 | 2015-03-04 | 北京奇虎科技有限公司 | Webpage data acquisition method, browser client end and CDN (content distribution network) server |
| CN104935648A (en) * | 2015-06-03 | 2015-09-23 | 北京快网科技有限公司 | High-cost-performance CDN system, and file pre-push and fragment buffer memory methods |
| CN105827541A (en) * | 2016-04-06 | 2016-08-03 | 中国建设银行股份有限公司 | Data message processing method and system for online trade |
| CN106603627A (en) * | 2016-11-09 | 2017-04-26 | 北京奇艺世纪科技有限公司 | Data center online video file synchronization method and synchronous scheduler |
| CN108073350A (en) * | 2016-11-10 | 2018-05-25 | 成都赫尔墨斯科技股份有限公司 | A kind of object storage system rendered for cloud and method |
| CN108074210A (en) * | 2016-11-10 | 2018-05-25 | 成都赫尔墨斯科技股份有限公司 | A kind of object acquisition system and method rendered for cloud |
| CN108234632A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | A kind of data distributing method and device of content distributing network CDN |
| CN108737475A (en) * | 2017-04-21 | 2018-11-02 | 贵州白山云科技有限公司 | A kind of file transmits in cloud distribution network fault-tolerance approach, device and system |
| CN109981751A (en) * | 2019-03-06 | 2019-07-05 | 珠海金山网络游戏科技有限公司 | A kind of document transmission method and system, computer equipment and storage medium |
| WO2020029380A1 (en) * | 2018-08-10 | 2020-02-13 | 网宿科技股份有限公司 | Method for processing superhot file, load balancing device, and download server |
| WO2021068740A1 (en) * | 2019-10-10 | 2021-04-15 | 深圳前海微众银行股份有限公司 | File management method and device |
| CN107562926B (en) * | 2017-09-14 | 2023-09-26 | 丙申南京网络技术有限公司 | Multi-hadoop distributed file system for big data analysis |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040193702A1 (en) * | 2003-03-31 | 2004-09-30 | Microsoft Corp. | System and method of network content location for roaming clients |
| CN101420452A (en) * | 2008-12-05 | 2009-04-29 | 深圳市迅雷网络技术有限公司 | Video file publishing method and device |
| CN101710901A (en) * | 2009-10-22 | 2010-05-19 | 乐视网信息技术（北京）股份有限公司 | Distributed type storage system having p2p function and method thereof |
| CN101741730A (en) * | 2009-12-02 | 2010-06-16 | 成都市华为赛门铁克科技有限公司 | Method and equipment for downloading file and method and system for providing file downloading service |
-
2013
- 2013-06-07 CN CN2013102265360A patent/CN103281394A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040193702A1 (en) * | 2003-03-31 | 2004-09-30 | Microsoft Corp. | System and method of network content location for roaming clients |
| CN101420452A (en) * | 2008-12-05 | 2009-04-29 | 深圳市迅雷网络技术有限公司 | Video file publishing method and device |
| CN101710901A (en) * | 2009-10-22 | 2010-05-19 | 乐视网信息技术（北京）股份有限公司 | Distributed type storage system having p2p function and method thereof |
| CN101741730A (en) * | 2009-12-02 | 2010-06-16 | 成都市华为赛门铁克科技有限公司 | Method and equipment for downloading file and method and system for providing file downloading service |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103747049A (en) * | 2013-12-24 | 2014-04-23 | 乐视网信息技术（北京）股份有限公司 | CDN file distribution method, control center and system |
| CN103986977A (en) * | 2014-04-15 | 2014-08-13 | 上海聚力传媒技术有限公司 | Method and device for preloading video in content distribution network |
| CN104392008A (en) * | 2014-12-19 | 2015-03-04 | 北京奇虎科技有限公司 | Webpage data acquisition method, browser client end and CDN (content distribution network) server |
| CN104392008B (en) * | 2014-12-19 | 2017-12-05 | 北京奇虎科技有限公司 | Web data acquisition methods, browser client and CDN server |
| CN104935648B (en) * | 2015-06-03 | 2018-07-17 | 北京快网科技有限公司 | The CDN system and file of a kind of high performance-price ratio push away in advance, the method for fragment cache memory |
| CN104935648A (en) * | 2015-06-03 | 2015-09-23 | 北京快网科技有限公司 | High-cost-performance CDN system, and file pre-push and fragment buffer memory methods |
| CN105827541A (en) * | 2016-04-06 | 2016-08-03 | 中国建设银行股份有限公司 | Data message processing method and system for online trade |
| CN106603627A (en) * | 2016-11-09 | 2017-04-26 | 北京奇艺世纪科技有限公司 | Data center online video file synchronization method and synchronous scheduler |
| CN106603627B (en) * | 2016-11-09 | 2019-11-08 | 北京奇艺世纪科技有限公司 | The synchronous method of video file and isochronous schedules device when data center online |
| CN108074210A (en) * | 2016-11-10 | 2018-05-25 | 成都赫尔墨斯科技股份有限公司 | A kind of object acquisition system and method rendered for cloud |
| CN108073350A (en) * | 2016-11-10 | 2018-05-25 | 成都赫尔墨斯科技股份有限公司 | A kind of object storage system rendered for cloud and method |
| CN108737475A (en) * | 2017-04-21 | 2018-11-02 | 贵州白山云科技有限公司 | A kind of file transmits in cloud distribution network fault-tolerance approach, device and system |
| CN107562926B (en) * | 2017-09-14 | 2023-09-26 | 丙申南京网络技术有限公司 | Multi-hadoop distributed file system for big data analysis |
| CN108234632A (en) * | 2017-12-29 | 2018-06-29 | 北京奇虎科技有限公司 | A kind of data distributing method and device of content distributing network CDN |
| WO2020029380A1 (en) * | 2018-08-10 | 2020-02-13 | 网宿科技股份有限公司 | Method for processing superhot file, load balancing device, and download server |
| US11201914B2 (en) | 2018-08-10 | 2021-12-14 | Wangsu Science & Technology Co., Ltd. | Method for processing a super-hot file, load balancing device and download server |
| CN109981751A (en) * | 2019-03-06 | 2019-07-05 | 珠海金山网络游戏科技有限公司 | A kind of document transmission method and system, computer equipment and storage medium |
| CN109981751B (en) * | 2019-03-06 | 2022-06-17 | 珠海金山网络游戏科技有限公司 | File transmission method and system, computer equipment and storage medium |
| WO2021068740A1 (en) * | 2019-10-10 | 2021-04-15 | 深圳前海微众银行股份有限公司 | File management method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103281394A (en) | File acquiring method, node servers and system | |
| CN103856569B (en) | A kind of method and apparatus of synchronous domain name system asset information | |
| CN103002010B (en) | A kind of data-updating method based on incremental data, device and system | |
| CN102394880B (en) | Method and device for processing jump response in content delivery network | |
| CN103327415A (en) | Method and device for accelerating network video downloading | |
| CN104468807A (en) | Processing method, cloud end device, local devices and system for webpage cache | |
| CN101710902B (en) | Unstructured P2P network, data searching method thereof and index updating method thereof | |
| CN103036967A (en) | Data download system and device and method for download management | |
| RU2012118601A (en) | SYSTEM AND METHOD FOR PROVIDING MORE FAST AND MORE EFFECTIVE DATA TRANSFER | |
| CN103607424B (en) | Server connection method and server system | |
| CN107302582B (en) | Data acquisition and weak push method for million-level Internet of things scene | |
| CN107079011A (en) | Long-tail content in process content transmission network | |
| CN103036969A (en) | Management device and method for providing file download addresses | |
| CN104284201A (en) | Video content processing method and device | |
| CN107888666A (en) | A kind of cross-region data-storage system and method for data synchronization and device | |
| CN103297528A (en) | Ticket information acquisition method and device | |
| CN106453460B (en) | File distribution method, device and system | |
| CN103731507A (en) | Data processing method and device of distributed data storage device | |
| CN105306585A (en) | Data synchronization method for plurality of data centers | |
| CN103227826A (en) | Method and device for transferring file | |
| US9503541B2 (en) | Fast mobile web applications using cloud caching | |
| CN102111449A (en) | Method, device and system for updating data | |
| CN103024082A (en) | Point-to-point communication method used for digital media distribution | |
| CN103379115A (en) | Data synchronism method and equipment for local storage and network storage | |
| CN108347459A (en) | A kind of high in the clouds data quick storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information |
Inventor after: Wang Pengcheng Inventor after: Liu Hao Inventor after: Feng Gu Inventor after: Hu Zhenyong Inventor after: Cao Shu Inventor before: Wang Pengcheng Inventor before: Liu Hao Inventor before: Feng Gu Inventor before: Hu Zhenyong |
|
| COR | Change of bibliographic data | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130904 |
|
| RJ01 | Rejection of invention patent application after publication |