CN102571959A - System and method for downloading data - Google Patents
System and method for downloading data Download PDFInfo
- Publication number
- CN102571959A CN102571959A CN2012100076730A CN201210007673A CN102571959A CN 102571959 A CN102571959 A CN 102571959A CN 2012100076730 A CN2012100076730 A CN 2012100076730A CN 201210007673 A CN201210007673 A CN 201210007673A CN 102571959 A CN102571959 A CN 102571959A
- Authority
- CN
- China
- Prior art keywords
- nginx
- lvs
- hdfs
- download
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000008569 process Effects 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 7
- 238000012546 transfer Methods 0.000 claims description 5
- 239000000872 buffer Substances 0.000 claims description 4
- 239000012467 final product Substances 0.000 description 3
- 241000238366 Cephalopoda Species 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 208000004350 Strabismus Diseases 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
Images
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a system and method for downloading data. The system comprises a plurality of downloading clusters. Each downloading cluster comprises a Linux virtual server (LVS), at least two downloading node Nginx servers and a Hadoop distributed file system (HDFS), wherein an operating system of each downloading node Nginx server is used for mounting the storage service of the HDFS through a filesystem in userspace (FUSE); each LVS is used for receiving a downloading request of a user, performing scheduling selection on all Nginx servers and forwarding the downloading request to the selected Nginx server; the Nginx servers are used for accessing the data stored in the HDFS through the FUSE after receiving the downloading request forwarded by each LVS and responding to the downloading request of the user; and each HDFS is used for storing the data. According to the system and the method for downloading the data, the requirements of high stored data volume and great file number can be met, and moreover, the downloading service requirements when the access hotspots of part of service users are not concentrated can also be met.
Description
Technical field
The present invention relates to field of computer technology, particularly relate to a kind of data downloading system and method.
Background technology
Along with the development of information technology, people get used to obtaining various data through network day by day.For example, a kind of common mode is exactly to download required content through data downloading system.
Use more download framework to mainly contain two kinds at present, a kind of is traditional CDN (Content Delivery Network, content distributing network) multilayer cache structure, and another kind is that general video website is used more download framework.
The sketch map of CDN multilayer cache structure is as shown in Figure 1; In this framework; Global data only is stored in the equipment of top layer source station, and each edge service node is done reverse proxy by Nginx, request is given to the Squid of rear end; Squid receives the user asks back time source station extracting data that service is provided, and front end uses the DNS mode to carry out load balancing.This framework is applicable to that usually the overall data amount is little, and the concentrated downloading service of focus.
Video website uses more download framework as shown in Figure 2; In this framework; The overall situation is set up some distributed storage clusters; Data do not have redundancy or redundancy lower between each cluster, use based on the layer 7 load-balancing device of HTTP (HyperText Transfer Protocol, HTTP) the Nginx server of the overall situation is carried out load balancing.This framework is applicable to the video downloading service usually, and whole concurrency is lower, is mostly to read continuously.
But the download service in some occasions (for example the download of installation procedure such as large software, recreation etc.) has following characteristics usually now: the storage data volume is big, quantity of documents is big, and certain partial service user capture focus is not concentrated or the like.Therefore, the technical problem that presses for those skilled in the art's solution just is how a kind of new download framework is provided, can satisfies above-mentioned requirements.
Summary of the invention
The invention provides a kind of data downloading system and method, can satisfy that the storage data volume is big, quantity of documents is big, and the download service demand of certain partial service user capture focus when not concentrating.
The invention provides following scheme:
A kind of data downloading system; Comprise a plurality of download clusters; Each is downloaded and comprises Linux virtual server LVS, at least two download node Nginx and a distributed file system HDFS in the cluster, and the operating system of said Nginx is through the stores service of user's space file system FUSE carry HDFS; Wherein:
Said LVS is used to receive user's download request, and each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses;
Said Nginx is used for after the download request that receives the LVS forwarding, through the data of storing among the FUSE visit HDFS, response user's download request;
Said HDFS is used to store data.
Wherein, Said LVS specifically is used for: after receiving user's download request; Performance and/or present load situation according to each Nginx are dispatched selection to each Nginx, and said download request is sent to the Nginx that performance and/or present load state meet prerequisite.
Wherein, said LVS is two.
Wherein, two LVS are active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, have an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.
Wherein, said HDFS comprises namenode and at least two data nodes, the multiplexing station server of said namenode and LVS, each back end and the multiplexing station server of Nginx.
Wherein, the number of Nginx is identical with the number of back end.
A kind of data download method; Be applied in the data downloading system; Said data downloading system comprises a plurality of download clusters; Each is downloaded and comprises Linux virtual server LVS, at least two download node Nginx and a distributed file system HDFS in the cluster, and said HDFS is used to store data, and the operating system of said Nginx is through the stores service of user's space file system FUSE carry HDFS; Said method comprises:
Receive user's download request through said LVS, and each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses by said LVS;
Nginx is after the download request that receives the LVS forwarding, through the data of storing among the FUSE visit HDFS, response user's download request.
Wherein, said each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses comprises:
Performance and/or present load situation according to each Nginx are dispatched selection to each Nginx, and said download request is sent to the Nginx that performance and/or present load state meet prerequisite.
Wherein, said LVS is two.
Wherein, two LVS are active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, have an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.
Wherein, said HDFS comprises namenode and at least two data nodes, the multiplexing station server of said namenode and LVS, each back end and the multiplexing station server of Nginx.
Wherein, the number of Nginx is identical with the number of back end.
Wherein, also comprise:
Parameter to Nginx is adjusted, and the parameter of being adjusted comprises one or more among Sendfile option, worker process number, one process maximum number of connections, backlog parameter, the output-buffers.
Wherein, also comprise:
The HDFS parameter is adjusted, and the parameter of being adjusted comprises one or more in read function parameters number, IPCserver listen queue length, IPC Server active line number of passes, the transfer of data maximum thread.
According to specific embodiment provided by the invention, the invention discloses following technique effect:
The first, owing in download system, used HDFS, therefore, make single download cluster that the storage of hundred TB ranks can be provided, can increase storage server through simple pluggable mode, improve storage capacity, satisfy the demand of big storage data volume; Simultaneously, owing to do not use hardware RAID card, read but each disk is walked abreast through the HDFS service; Therefore; Can realize that each disk of unit independently reads, can satisfy user capture well and do not have focus, file reads the high I/O demand of dispersion, and improve data throughout.Experimental data shows, is using under the situation of changeing the SATA dish for 12 7200, and total IOPS of unit can reach more than 1000; The highlyest realize the concurrent connection of unit more than 15000, the above bandwidth of 1Gb is handled up.
The second, because each cluster only is exposed to the user with LVS, therefore; Make each cluster have the characteristics of high cohesion, also promptly, if certain or some Nginx can't outwards provide service owing to break down etc.; Can (for example solve in cluster inside through LVS; Can carry out availability monitor to rear end Nginx, unusual Nginx carried out that second level is extractd etc.), can not have influence on other cluster.
The 3rd, can directly carry out load balancing in cluster inside by LVS.
The 4th, because each cluster content is all stored data, therefore, can realize the fully redundance of data between different clusters, can guarantee to carry out apace flow scheduling and switching.
The 5th, adopt said structure, can realize expansion easily to system.For example, become bottleneck, can increase independently the LVS server or on existing server, increase network interface card, the expansion of Rapid Realization performance if find the network performance of front end LVS; If data space is not enough or download service can not satisfy the demands, can simply carry out memory space and the expansion of download service performance through the back end that increases HDFS, corresponding, HDFS can accomplish the data sync of new node automatically.
Description of drawings
In order to be illustrated more clearly in the embodiment of the invention or technical scheme of the prior art; To do to introduce simply to the accompanying drawing of required use among the embodiment below; Obviously, the accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills; Under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is a download configuration diagram of the prior art;
Fig. 2 is that of the prior art another downloaded configuration diagram;
Fig. 3 is the sketch map of the system that provides of the embodiment of the invention;
Fig. 4 is that physics of each service in the system that provides of the embodiment of the invention is disposed sketch map;
Fig. 5 is the flow chart of the method that provides of the embodiment of the invention.
Embodiment
To combine the accompanying drawing in the embodiment of the invention below, the technical scheme in the embodiment of the invention is carried out clear, intactly description, obviously, described embodiment only is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, the every other embodiment that those of ordinary skills obtained belongs to the scope that the present invention protects.
The embodiment of the invention at first provides a kind of data downloading system, and its sketch map is as shown in Figure 3.In this system; Comprise a plurality of download clusters (only illustrate among the figure and download cluster), each is downloaded and all comprises LVS (Linux Virtual Server, Linux virtual server), at least two download node Nginx and a HDFS (Hadoop Distributed File System in the cluster; Distributed file system); Wherein, the operating system that Nginx moved is through the stores service of FUSE (Filesystem in Userspace, user's space file system) carry HDFS.
For the ease of understanding the embodiment of the invention, at first HDFS, LVS, Nginx and FUSE are introduced simply below.
HDFS has the characteristics of high fault tolerance, and design is used for being deployed on the cheap hardware, and it provides high-throughput to visit application's data, and being fit to those has the application program of super large data set.Have These characteristics just because of HDFS, the embodiment of the invention attempts HDFS is applied in the online download service, and corresponding solution is provided.Certainly, in the prior art, HDFS generally is used for the storage under the line, and Shang Weiyou is applied to online data with HDFS and downloads the application in the framework.
LVS is a software of increasing income, and can realize the simple load under the LINUX platform.LVS adopts IP load-balancing technique and content-based request distribution technology; Its scheduler has good throughput; Carry out asking balancedly to transfer on the different servers; And scheduler can mask the fault of server automatically, thereby one group of server is constituted high performance, a high available virtual server.The structure of whole server cluster is transparent to the client, and need not to revise the program of client and server end.
Nginx is a high performance HTTP and Reverse Proxy, connects under the concurrent situation at height, and Nginx is a good selection.
FUSE is the notion in the (SuSE) Linux OS, refers to the complete file system that realizes in user's attitude, is used for some cyberspace of carry, to the module of local file system.Operating system provides support to file system on the kernel aspect traditionally, and the code of kernel state is difficult to debugging usually, and productivity ratio is lower; But after having utilized FUSE, can significantly boost productivity, be reduced to the workload that operating system provides new file system, be specially adapted to various Virtual File Systems and NFS.
Based on These characteristics, the embodiment of the invention provides a kind of novel data downloading system.Wherein, inner at each cluster, data are stored through HDFS, and the cluster outermost layer exposes the empty IP of LVS provides service to the user, also promptly, is received user's download request by LVS; And LVS can dispatch each Nginx after receiving user's download request, selects suitable Nginx, and user's download request is transmitted to the Nginx that this is chosen.Nginx just can visit the data of storing among the HDFS through FUSE after receiving user's download request, respond user's download request with this.Wherein, LVS is specifically when dispatching Nginx; Can carry out according to performance (comprising hardware of server performance index etc.) and/or the present load situation of each Nginx; Select the Nginx that performance and/or present load state meet prerequisite, and user's download request is sent to this Nginx get final product.
Through the said system framework, can be so that the data downloading system that the embodiment of the invention provides has following characteristics:
The first, owing in download system, used HDFS, therefore, make single download cluster that the storage of hundred TB ranks can be provided, can increase storage server through simple pluggable mode, improve storage capacity, satisfy the demand of big storage data volume; Simultaneously, owing to do not use hardware RAID card, read but each disk is walked abreast through the HDFS service; Therefore; Can realize that each disk of unit independently reads, can satisfy user capture well and do not have focus, file reads the high I/O demand of dispersion, and improve data throughout.Experimental data shows, is using under the situation of changeing the SATA dish for 12 7200, and total IOPS of unit can reach more than 1000; The highlyest realize the concurrent connection of unit more than 15000, the above bandwidth of 1Gb is handled up.
The second, because each cluster only is exposed to the user with LVS, therefore; Make each cluster have the characteristics of high cohesion, also promptly, if certain or some Nginx can't outwards provide service owing to break down etc.; Can (for example solve in cluster inside through LVS; Can carry out availability monitor to rear end Nginx, unusual Nginx carried out that second level is extractd etc.), can not have influence on other cluster.
The 3rd, can directly carry out load balancing in cluster inside by LVS.
The 4th, because each cluster content is all stored data, therefore, can realize the fully redundance of data between different clusters, can guarantee to carry out apace flow scheduling and switching.
The 5th, adopt said structure, can realize expansion easily to system.For example, become bottleneck, can increase independently the LVS server or on existing server, increase network interface card, the expansion of Rapid Realization performance if find the network performance of front end LVS; If data space is not enough or download service can not satisfy the demands, can simply carry out memory space and the expansion of download service performance through the back end that increases HDFS, corresponding, HDFS can accomplish the data sync of new node automatically.
Owing to can expand system, therefore, in practical application, LVS can be a plurality of.For example, in sketch map shown in Figure 3, LVS can provide download service to the user simultaneously for two.At this moment, these two LVS can be active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, has an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.Through this mechanism, further guaranteed the high reliability that single cluster is inner, avoid because LVS breaks down and causes the failure of download service.
In addition; It is a hypotactic system that HDFS also has following characteristics: HDFS; HDFS cluster be by namenode (NameNode, NN) and back end (DataNode DN) forms; Namenode is the NameSpace and the master server of regulating the client-access file of a management document, and back end is used for managed storage.In other words, the file or the directory operation of namenode operation file NameSpace, as open, close, rename, or the like, the mapping of its while specified data and back end.Back end is responsible for the read-write requests from the file system user, also will carry out the establishment of data simultaneously, deletion and duplicate indication from the data of namenode.Wherein, namenode and back end all are that running software is on common machine.
In practical application, in order to realize making full use of of hardware resource, reduce system scale, improve the maintainability of system, each service physically can disposed in the following manner:
Referring to Fig. 4, the namenode of HDFS (NN) service is carried out the multiplexing of server with the LVS service, for example, among Fig. 4, when having two LVS, also can have two namenodes simultaneously in the system.Also promptly,, and dispose FUSE carry HDFS, Update Information to HDFS through FUSE at namenode deploy data sync client end.In the download service process, namenode only responds the inner a spot of RPC request of cluster, and load is very low; LVS service belongs to CPU and the service of network-intensive type, but efficient is very high, therefore, carries out multiplexingly like this, can make full use of system resource and can not serve HDFS and exert an influence.
In addition, as shown in Figure 4, each back end can with a multiplexing station server of Nginx, also promptly, download service is provided at each back end (DN) deploy FUSE and Nginx.At this moment; The data of storing in each back end can be whole data, and the number of Nginx can be identical with the number of back end, also are; Between each inner different downloading node of a cluster, also can realize the fully redundance of data.In a word, this mechanism can further improve the reliability of system, simultaneously; Have benefited from the characteristic of HDFS, the HDFS that overwhelming majority's request of HDFS data is all directly visited this machine serves, and can not stride the machine visit; Therefore, can improve response speed, evade the Intranet transmission bottleneck.Certainly, in practical application, between different Nginx, can realize that also section data is redundant.
Need to prove that LVS has two kinds of route patterns, a kind of is the NAT pattern, and another kind is DR (Direct Routing) pattern.In embodiments of the present invention, LVS can work in the DR pattern, and under this pattern, inbound traffics (user's request) are through LVS, but outflow (response data) can be directly from behind back end return, LVS no longer flows through.Can avoid like this going out inbound traffics and all walking LVS in the NAT pattern, make LVS become the problem of network bandwidth bottleneck.
When concrete the realization,, can also adjust Nginx parameter and HDFS service in order to adapt to the requirement of concurrent, the low delay of download service height, high data throughput better.
Wherein, when adjustment Nginx parameter, can comprise following aspect: (a) close the Sendfile option; (b) worker process number is increased to 256 from 1 of acquiescence; (c) the one process maximum number of connections is adjusted into 512; (d) be 204800 with the backlog parameter adjustment; (e) output-buffers is set to " 1128k ".When adjustment HDFS service; Can comprise aspect down: (a) former read function publicsynchronized int read (byte buf []; Int off, int len) throws IOException comprises three parameter: Buf (buffer memory), Off (data squint at buf), Len (reading length), and the embodiment of the invention can increase a parameter p osition (document misregistration) on this basis; Reduce data pre-head with this, reduce read request pressure; (b) IPC server listen queue length is increased to 1024 by 128; (c) increase of IPC Server active line number of passes is 1024 by 3; (d) the transfer of data maximum thread increases to 4096 by 256.Certainly, in practical application, also can be according to actual needs, adjustment a part of parameter wherein, perhaps the concrete number adjustment with these parameters is other value, perhaps adjusts other parameter, or the like, do not limit here.
Corresponding with the data downloading system that the embodiment of the invention provides, the embodiment of the invention also provides a kind of data download method, and this method is applied in the data downloading system shown in Figure 3, and referring to Fig. 5, this method may further comprise the steps:
S501: receive user's download request through said LVS, and each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses by said LVS;
S502:Nginx is after the download request that receives the LVS forwarding, through the data of storing among the FUSE visit HDFS, response user's download request.
Wherein, When concrete the realization; LVS specifically can dispatch selection according to performance and/or the present load situation of each Nginx to each Nginx after receiving user's download request, said download request is sent to the Nginx that performance and/or present load state meet prerequisite.In addition, the LVS in the system is two, and two LVS can be active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, has an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.
In order to realize making full use of of hardware resource, reduce system scale, improve the maintainability of system, the namenode of HDFS can with the multiplexing station server of LVS, each back end of HDFS and the multiplexing station server of Nginx.And the number of Nginx can be identical with the number of back end.
In addition, in practical application,, can also adjust Nginx parameter and HDFS service in order to adapt to the requirement of concurrent, the low delay of download service height, high data throughput better.
Description through above execution mode can know, those skilled in the art can be well understood to the present invention and can realize by the mode that software adds essential general hardware platform.Based on such understanding; The part that technical scheme of the present invention contributes to prior art in essence in other words can be come out with the embodied of software product; This computer software product can be stored in the storage medium, like ROM/RAM, magnetic disc, CD etc., comprises that some instructions are with so that a computer equipment (can be a personal computer; Server, the perhaps network equipment etc.) carry out the described method of some part of each embodiment of the present invention or embodiment.
Each embodiment in this specification all adopts the mode of going forward one by one to describe, and identical similar part is mutually referring to getting final product between each embodiment, and each embodiment stresses all is the difference with other embodiment.Especially, for device or system embodiment, because it is basically similar in appearance to method embodiment, so describe fairly simplely, relevant part gets final product referring to the part explanation of method embodiment.Apparatus and system embodiment described above only is schematic; Wherein said unit as the separating component explanation can or can not be physically to separate also; The parts that show as the unit can be or can not be physical locations also; Promptly can be positioned at a place, perhaps also can be distributed on a plurality of NEs.Can realize the purpose of present embodiment scheme according to the needs selection some or all of module wherein of reality.Those of ordinary skills promptly can understand and implement under the situation of not paying creative work.
More than to a kind of data downloading system provided by the present invention and method; Carried out detailed introduction; Used concrete example among this paper principle of the present invention and execution mode are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof; Simultaneously, for one of ordinary skill in the art, according to thought of the present invention, part all can change on embodiment and range of application.In sum, this description should not be construed as limitation of the present invention.
Claims (14)
1. data downloading system; It is characterized in that; Comprise a plurality of download clusters; Each is downloaded and comprises Linux virtual server LVS, at least two download node Nginx and a distributed file system HDFS in the cluster, and the operating system of said Nginx is through the stores service of user's space file system FUSE carry HDFS; Wherein:
Said LVS is used to receive user's download request, and each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses;
Said Nginx is used for after the download request that receives the LVS forwarding, through the data of storing among the FUSE visit HDFS, response user's download request;
Said HDFS is used to store data.
2. system according to claim 1; It is characterized in that; Said LVS specifically is used for: after receiving user's download request; Performance and/or present load situation according to each Nginx are dispatched selection to each Nginx, and said download request is sent to the Nginx that performance and/or present load state meet prerequisite.
3. system according to claim 1 is characterized in that, said LVS is two.
4. system according to claim 3 is characterized in that, two LVS are active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, has an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.
5. system according to claim 1 is characterized in that, said HDFS comprises namenode and at least two data nodes, the multiplexing station server of said namenode and LVS, each back end and the multiplexing station server of Nginx.
6. system according to claim 5 is characterized in that, the number of Nginx is identical with the number of back end.
7. data download method; Be applied in the data downloading system; It is characterized in that said data downloading system comprises a plurality of download clusters, each is downloaded and comprises Linux virtual server LVS, at least two download node Nginx and a distributed file system HDFS in the cluster; Said HDFS is used to store data, and the operating system of said Nginx is through the stores service of user's space file system FUSE carry HDFS; Said method comprises:
Receive user's download request through said LVS, and each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses by said LVS;
Nginx is after the download request that receives the LVS forwarding, through the data of storing among the FUSE visit HDFS, response user's download request.
8. method according to claim 7 is characterized in that, said each Nginx is dispatched selection, said download request is transmitted to the Nginx that chooses comprises:
Performance and/or present load situation according to each Nginx are dispatched selection to each Nginx, and said download request is sent to the Nginx that performance and/or present load state meet prerequisite.
9. method according to claim 7 is characterized in that, said LVS is two.
10. method according to claim 9 is characterized in that, two LVS are active and standby each other, and each LVS provides download service through an empty IP who is in holotype to the user, has an empty IP who is in standby mode simultaneously; When a LVS can't provide service, another LVS was through starting the download service that the said empty IP that is in standby mode takes over this LVS.
11. method according to claim 7 is characterized in that, said HDFS comprises namenode and at least two data nodes, the multiplexing station server of said namenode and LVS, each back end and the multiplexing station server of Nginx.
12. method according to claim 11 is characterized in that, the number of Nginx is identical with the number of back end.
13. method according to claim 7 is characterized in that, also comprises:
Parameter to Nginx is adjusted, and the parameter of being adjusted comprises one or more among Sendfile option, worker process number, one process maximum number of connections, backlog parameter, the output-buffers.
14. method according to claim 7 is characterized in that, also comprises:
The HDFS parameter is adjusted, and the parameter of being adjusted comprises one or more in read function parameters number, IPCserver listen queue length, IPC Server active line number of passes, the transfer of data maximum thread.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510145970.5A CN104702702B (en) | 2012-01-11 | 2012-01-11 | A kind of data downloading system and method |
| CN201210007673.0A CN102571959B (en) | 2012-01-11 | 2012-01-11 | System and method for downloading data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210007673.0A CN102571959B (en) | 2012-01-11 | 2012-01-11 | System and method for downloading data |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510145970.5A Division CN104702702B (en) | 2012-01-11 | 2012-01-11 | A kind of data downloading system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102571959A true CN102571959A (en) | 2012-07-11 |
| CN102571959B CN102571959B (en) | 2015-05-06 |
Family
ID=46416419
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210007673.0A Expired - Fee Related CN102571959B (en) | 2012-01-11 | 2012-01-11 | System and method for downloading data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102571959B (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024018A (en) * | 2012-12-04 | 2013-04-03 | 北京蓝汛通信技术有限责任公司 | Method and device for operating multiple content distribution network (CDN) service processes in single device |
| CN103207785A (en) * | 2013-04-23 | 2013-07-17 | 北京奇虎科技有限公司 | Method, device, and system for processing data download request |
| CN103685331A (en) * | 2012-08-31 | 2014-03-26 | 腾讯科技（深圳）有限公司 | Information processing method and server |
| CN103685348A (en) * | 2012-09-03 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | Cloud enforcing system and method |
| CN104301442A (en) * | 2014-11-17 | 2015-01-21 | 浪潮电子信息产业股份有限公司 | Method for achieving client of access object storage cluster based on fuse |
| CN104394204A (en) * | 2014-11-12 | 2015-03-04 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing information interaction in load balancing |
| CN104407946A (en) * | 2014-12-31 | 2015-03-11 | 浪潮电子信息产业股份有限公司 | Method for conveniently backing up mail to HDFS |
| CN104580304A (en) * | 2013-10-18 | 2015-04-29 | 北大方正集团有限公司 | Data downloading method, device and system for distributed platform |
| CN105049244A (en) * | 2015-07-02 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Load balancing method and system |
| CN105162878A (en) * | 2015-09-24 | 2015-12-16 | 网宿科技股份有限公司 | Distributed storage based file distribution system and method |
| CN105187489A (en) * | 2015-08-05 | 2015-12-23 | 深圳联友科技有限公司 | File transfer method and system capable of clustering and supporting multiple users to upload simultaneously |
| US9348707B2 (en) | 2013-12-18 | 2016-05-24 | International Business Machines Corporation | Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system |
| CN105824707A (en) * | 2016-04-21 | 2016-08-03 | 乐视控股（北京）有限公司 | Merging back-source method and device for multiple processes of streaming media service |
| CN105978939A (en) * | 2016-04-25 | 2016-09-28 | 乐视控股（北京）有限公司 | Data downloading method and data downloading device |
| CN106254544A (en) * | 2016-09-28 | 2016-12-21 | 乐视控股（北京）有限公司 | Data access method and device |
| WO2017084393A1 (en) * | 2015-11-20 | 2017-05-26 | 华为技术有限公司 | Content distribution method, virtual server management method, cloud platform and system |
| CN106844562A (en) * | 2016-12-30 | 2017-06-13 | 北京航天泰坦科技股份有限公司 | Geographical image transparent caching mechanism based on FUSE |
| CN106850836A (en) * | 2017-03-06 | 2017-06-13 | 武汉虹旭信息技术有限责任公司 | Front end and backstage multiclass pattern synthesis connection method for WiFi acquisition systems |
| WO2017107483A1 (en) * | 2015-12-23 | 2017-06-29 | 中兴通讯股份有限公司 | Load balancing method for virtualized network management file downloading, and network management server |
| CN107547654A (en) * | 2017-09-12 | 2018-01-05 | 郑州云海信息技术有限公司 | A kind of distributed objects storage cluster, deployment, method of servicing and system |
| CN107566466A (en) * | 2017-08-24 | 2018-01-09 | 新华三大数据技术有限公司 | Load-balancing method and device |
| CN108055157A (en) * | 2017-12-19 | 2018-05-18 | 北京奇艺世纪科技有限公司 | The acquisition methods and device of a kind of service node |
| CN108337328A (en) * | 2018-05-17 | 2018-07-27 | 广东铭鸿数据有限公司 | A kind of data exchange system, data uploading method and data download method |
| CN108595616A (en) * | 2018-04-23 | 2018-09-28 | 南京大学 | A kind of method of the naming space management of Based on Distributed file system |
| CN109144529A (en) * | 2018-08-02 | 2019-01-04 | 郑州市景安网络科技股份有限公司 | Brush method, apparatus, equipment and readable storage medium storing program for executing under operating system template |
| CN109753242A (en) * | 2018-12-10 | 2019-05-14 | 航天信息股份有限公司 | Based on the method and apparatus that SaltStack realizes HDFS dilatation, storage medium and electronic equipment |
| CN110933145A (en) * | 2019-11-14 | 2020-03-27 | 光通天下网络科技股份有限公司 | Remote scheduling method, device, equipment and medium |
| CN111432039A (en) * | 2020-06-15 | 2020-07-17 | 北京金山云网络技术有限公司 | Data request method, device and equipment in CDN and CDN node |
| CN112929414A (en) * | 2021-01-22 | 2021-06-08 | 北京北信源软件股份有限公司 | Upgrade package downloading method and device, electronic equipment and storage medium |
| CN114627571A (en) * | 2022-03-18 | 2022-06-14 | 广州文远知行科技有限公司 | Data uploading method and device, vehicle and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101996250A (en) * | 2010-11-15 | 2011-03-30 | 中国科学院计算技术研究所 | Hadoop-based mass stream data storage and query method and system |
| US20110078666A1 (en) * | 2009-05-26 | 2011-03-31 | University Of California | System and Method for Reproducing Device Program Execution |
| CN102012933A (en) * | 2010-12-02 | 2011-04-13 | 清华大学 | Distributed file system and method for storing data and providing services by utilizing same |
| CN102073741A (en) * | 2011-01-30 | 2011-05-25 | 宇龙计算机通信科技(深圳)有限公司 | Method for realizing file reading and/or writing and data server |
-
2012
- 2012-01-11 CN CN201210007673.0A patent/CN102571959B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110078666A1 (en) * | 2009-05-26 | 2011-03-31 | University Of California | System and Method for Reproducing Device Program Execution |
| CN101996250A (en) * | 2010-11-15 | 2011-03-30 | 中国科学院计算技术研究所 | Hadoop-based mass stream data storage and query method and system |
| CN102012933A (en) * | 2010-12-02 | 2011-04-13 | 清华大学 | Distributed file system and method for storing data and providing services by utilizing same |
| CN102073741A (en) * | 2011-01-30 | 2011-05-25 | 宇龙计算机通信科技(深圳)有限公司 | Method for realizing file reading and/or writing and data server |
Non-Patent Citations (4)
| Title |
|---|
| 奇虎360: "360HDFS下载平台介绍", 《HADOOP中国2011云计算大会》, 3 December 2011 (2011-12-03), pages 3 - 15 * |
| 赵攀，等: "LVS集群技术在校园网中的研究与设计", 《四川理工学院学报（自然科学版）》, vol. 22, no. 2, 30 April 2009 (2009-04-30) * |
| 赵殿奎: "基于LVS负载调度器的双机热备份研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》, 28 February 2007 (2007-02-28), pages 6 * |
| 黄晓云: "基于HDFS的云存储服务器系统研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, 30 September 2010 (2010-09-30) * |
Cited By (42)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103685331A (en) * | 2012-08-31 | 2014-03-26 | 腾讯科技（深圳）有限公司 | Information processing method and server |
| CN103685348A (en) * | 2012-09-03 | 2014-03-26 | 百度在线网络技术（北京）有限公司 | Cloud enforcing system and method |
| CN103685348B (en) * | 2012-09-03 | 2016-12-21 | 百度在线网络技术（北京）有限公司 | The system and method that cloud performs |
| CN103024018A (en) * | 2012-12-04 | 2013-04-03 | 北京蓝汛通信技术有限责任公司 | Method and device for operating multiple content distribution network (CDN) service processes in single device |
| CN103207785A (en) * | 2013-04-23 | 2013-07-17 | 北京奇虎科技有限公司 | Method, device, and system for processing data download request |
| CN104580304A (en) * | 2013-10-18 | 2015-04-29 | 北大方正集团有限公司 | Data downloading method, device and system for distributed platform |
| US9348707B2 (en) | 2013-12-18 | 2016-05-24 | International Business Machines Corporation | Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system |
| US10025676B2 (en) | 2013-12-18 | 2018-07-17 | International Business Machines Corporation | Dynamically adjusting the number of replicas of a file according the probability that the file will be accessed within a distributed file system |
| US10353784B2 (en) | 2013-12-18 | 2019-07-16 | International Business Machines Corporation | Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system |
| US11176005B2 (en) | 2013-12-18 | 2021-11-16 | International Business Machines Corporation | Dynamically adjusting the number of replicas of a file according to the probability that the file will be accessed within a distributed file system |
| CN104394204A (en) * | 2014-11-12 | 2015-03-04 | 浪潮(北京)电子信息产业有限公司 | Method and system for realizing information interaction in load balancing |
| CN104394204B (en) * | 2014-11-12 | 2018-06-12 | 浪潮(北京)电子信息产业有限公司 | Information interacting method and system in a kind of realization load balancing |
| CN104301442A (en) * | 2014-11-17 | 2015-01-21 | 浪潮电子信息产业股份有限公司 | Method for achieving client of access object storage cluster based on fuse |
| CN104407946A (en) * | 2014-12-31 | 2015-03-11 | 浪潮电子信息产业股份有限公司 | Method for conveniently backing up mail to HDFS |
| CN105049244A (en) * | 2015-07-02 | 2015-11-11 | 浪潮(北京)电子信息产业有限公司 | Load balancing method and system |
| CN105187489A (en) * | 2015-08-05 | 2015-12-23 | 深圳联友科技有限公司 | File transfer method and system capable of clustering and supporting multiple users to upload simultaneously |
| CN105162878A (en) * | 2015-09-24 | 2015-12-16 | 网宿科技股份有限公司 | Distributed storage based file distribution system and method |
| CN105162878B (en) * | 2015-09-24 | 2018-08-31 | 网宿科技股份有限公司 | Document distribution system based on distributed storage and method |
| US11431791B2 (en) | 2015-11-20 | 2022-08-30 | Huawei Technologies Co., Ltd. | Content delivery method, virtual server management method, cloud platform, and system |
| WO2017084393A1 (en) * | 2015-11-20 | 2017-05-26 | 华为技术有限公司 | Content distribution method, virtual server management method, cloud platform and system |
| US10911530B2 (en) | 2015-11-20 | 2021-02-02 | Huawei Technologies Co., Ltd. | Content delivery method, virtual server management method, cloud platform, and system |
| WO2017107483A1 (en) * | 2015-12-23 | 2017-06-29 | 中兴通讯股份有限公司 | Load balancing method for virtualized network management file downloading, and network management server |
| CN105824707A (en) * | 2016-04-21 | 2016-08-03 | 乐视控股（北京）有限公司 | Merging back-source method and device for multiple processes of streaming media service |
| WO2017185631A1 (en) * | 2016-04-25 | 2017-11-02 | 乐视控股（北京）有限公司 | Data downloading method and device |
| CN105978939A (en) * | 2016-04-25 | 2016-09-28 | 乐视控股（北京）有限公司 | Data downloading method and data downloading device |
| CN106254544A (en) * | 2016-09-28 | 2016-12-21 | 乐视控股（北京）有限公司 | Data access method and device |
| CN106844562A (en) * | 2016-12-30 | 2017-06-13 | 北京航天泰坦科技股份有限公司 | Geographical image transparent caching mechanism based on FUSE |
| CN106850836A (en) * | 2017-03-06 | 2017-06-13 | 武汉虹旭信息技术有限责任公司 | Front end and backstage multiclass pattern synthesis connection method for WiFi acquisition systems |
| CN107566466A (en) * | 2017-08-24 | 2018-01-09 | 新华三大数据技术有限公司 | Load-balancing method and device |
| CN107547654A (en) * | 2017-09-12 | 2018-01-05 | 郑州云海信息技术有限公司 | A kind of distributed objects storage cluster, deployment, method of servicing and system |
| CN108055157A (en) * | 2017-12-19 | 2018-05-18 | 北京奇艺世纪科技有限公司 | The acquisition methods and device of a kind of service node |
| CN108055157B (en) * | 2017-12-19 | 2021-02-23 | 北京奇艺世纪科技有限公司 | Service node acquisition method and device |
| CN108595616A (en) * | 2018-04-23 | 2018-09-28 | 南京大学 | A kind of method of the naming space management of Based on Distributed file system |
| CN108595616B (en) * | 2018-04-23 | 2022-04-26 | 南京大学 | Unified namespace management method for distributed file system |
| CN108337328A (en) * | 2018-05-17 | 2018-07-27 | 广东铭鸿数据有限公司 | A kind of data exchange system, data uploading method and data download method |
| CN109144529A (en) * | 2018-08-02 | 2019-01-04 | 郑州市景安网络科技股份有限公司 | Brush method, apparatus, equipment and readable storage medium storing program for executing under operating system template |
| CN109753242A (en) * | 2018-12-10 | 2019-05-14 | 航天信息股份有限公司 | Based on the method and apparatus that SaltStack realizes HDFS dilatation, storage medium and electronic equipment |
| CN110933145A (en) * | 2019-11-14 | 2020-03-27 | 光通天下网络科技股份有限公司 | Remote scheduling method, device, equipment and medium |
| CN111432039A (en) * | 2020-06-15 | 2020-07-17 | 北京金山云网络技术有限公司 | Data request method, device and equipment in CDN and CDN node |
| CN112929414A (en) * | 2021-01-22 | 2021-06-08 | 北京北信源软件股份有限公司 | Upgrade package downloading method and device, electronic equipment and storage medium |
| CN114627571A (en) * | 2022-03-18 | 2022-06-14 | 广州文远知行科技有限公司 | Data uploading method and device, vehicle and storage medium |
| CN114627571B (en) * | 2022-03-18 | 2024-05-24 | 广州文远知行科技有限公司 | Data uploading method and device, vehicle and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102571959B (en) | 2015-05-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102571959B (en) | System and method for downloading data | |
| CN104702702A (en) | System and method for downloading data | |
| EP1330907B1 (en) | Method and apparatus for real-time parallel delivery of segments of a large payload file | |
| US7877511B1 (en) | Method and apparatus for adaptive services networking | |
| US7734778B2 (en) | Distributed intelligent virtual server | |
| JP2023532947A (en) | Data transfer method, proxy server, storage medium and electronic device | |
| EP1892921A2 (en) | Method and sytem for managing distributed content and related metadata | |
| CN101631143B (en) | Multi-server system in load-balancing environment and file transmission method thereof | |
| US20020133491A1 (en) | Method and system for managing distributed content and related metadata | |
| US8701113B2 (en) | Switch-aware parallel file system | |
| US9037618B2 (en) | Distributed, unified file system operations | |
| WO2011044829A1 (en) | Resource buffering method, resource acquisition method, apparatus and system thereof | |
| CN105005611B (en) | A kind of file management system and file management method | |
| CN102594852A (en) | Data access method, node and system | |
| US8489709B2 (en) | Method of managing a file access in a distributed file storage system | |
| CN109756573B (en) | File system based on block chain | |
| CN104168323A (en) | Cloud service system and method | |
| CN105468296B (en) | Nothing based on virtual platform shares memory management method | |
| US20030154246A1 (en) | Server for storing files | |
| KR101236477B1 (en) | Method of processing data in asymetric cluster filesystem | |
| CN104539982A (en) | Method and system for dispatching of video resource point-to-point sharing and node terminal | |
| CN107818111A (en) | A kind of method, server and the terminal of cache file data | |
| CN103442034A (en) | Streaming media service method and system based on cloud computing technology | |
| KR101901266B1 (en) | System and method for parallel file transfer between file storage clusters | |
| KR100751075B1 (en) | The Remote Mirroring System using the Parallel Transfer Technique on NAS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20150506 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |