CN106254833B - Method and device for downloading distributed files - Google Patents
Method and device for downloading distributed files Download PDFInfo
- Publication number
- CN106254833B CN106254833B CN201610770461.6A CN201610770461A CN106254833B CN 106254833 B CN106254833 B CN 106254833B CN 201610770461 A CN201610770461 A CN 201610770461A CN 106254833 B CN106254833 B CN 106254833B
- Authority
- CN
- China
- Prior art keywords
- bandwidth
- downloading
- file
- download
- data node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
- H04N21/2385—Channel allocation; Bandwidth allocation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
Abstract
The invention discloses a method and a device for downloading a distributed file, wherein a plurality of concurrent threads are started according to acquired information of data nodes, each concurrent thread is connected with one of the data nodes, a file downloading request containing a downloading file name and an initial bandwidth is sent to the corresponding data node, the file downloading is carried out according to the downloading bandwidth distributed by the data node according to the file downloading request, and then a downloading bandwidth updating request is periodically sent to redistribute the bandwidth according to the actual downloading bandwidth. The device comprises a node request module, a file query module, a download module and an update module. The method and the device of the invention ensure that the downloading speed of the client is maximized while the downloading bandwidth of the data node is reasonably utilized, thereby improving the speed of simultaneous downloading of multiple clients.
Description
Technical Field
The invention belongs to the field of data transmission, and particularly relates to a method and a device for downloading a distributed file.
Background
With the development of video monitoring technology, video monitoring is more and more widely used in various industries, and besides being used for real-time video monitoring of various monitoring places, video monitoring systems in subways, prisons, banks, safe cities and the like also often encounter the situation that video monitoring videos need to be downloaded to the local for viewing. And along with the development of science and technology, video monitoring is more and more intelligent, wherein when carrying out intelligent analysis to the video surveillance record, in order to carry out intelligent analysis more conveniently fast, also need download the video surveillance record to local.
In the prior art, as shown in fig. 1, video surveillance videos are stored in a distributed storage manner on a plurality of data nodes DN, and each video surveillance client UI downloads a video surveillance video from a data node DN through a video on demand server VOD. Specifically, when the video monitoring client needs to download the video monitoring videos, a downloading request is sent to the video on demand server VOD, the video on demand server VOD downloads the corresponding video monitoring videos from the data nodes DN at first, and then pushes the downloaded video monitoring videos to the corresponding video monitoring client, so that the downloading of the video monitoring videos is completed.
Although the prior art can download the video surveillance videos, the downloading of all the surveillance videos needs to be performed by the VOD server, so the VOD server becomes a bottleneck for downloading the surveillance videos, which is especially obvious in the case that there are many video surveillance clients and the downloading requirement of the surveillance videos is high. In addition, if the bandwidth of the video monitoring client and the bandwidth of the data node DN are both high, but the VOD bandwidth of the video-on-demand server is not high, the downloading speed of the video monitoring video can be greatly reduced.
Disclosure of Invention
The invention aims to provide a method and a device for downloading a distributed file, which aim to solve the problem that the downloading speed of a video monitoring record is seriously influenced because a video on demand server VOD becomes a bottleneck of downloading the monitoring video record because the video on demand server VOD is required to be carried out by the video on demand server VOD when the video monitoring record is downloaded in the prior art.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a method for downloading a distributed file is applied to a file downloading client, and comprises the following steps:
sending a data node request message to a metadata management server, and acquiring information of a data node storing a downloaded file;
sending a query message to a data node for storing the download file, and acquiring the file name of the download file stored in the data node;
starting a plurality of concurrent threads according to the acquired information of the data nodes, wherein each concurrent thread corresponds to one data node, sends a file downloading request containing a downloading file name and an initial bandwidth to the corresponding data node, and downloads the file according to the downloading bandwidth distributed by the data node according to the file downloading request;
and counting the actual download bandwidth, periodically sending a download bandwidth updating request containing the actual download bandwidth to the data node, and downloading the file according to the download bandwidth allocated to the data node according to the download bandwidth updating request.
The initial bandwidth in the file downloading request is a set basic bandwidth, and the data node allocates the downloading bandwidth for the data node according to the file downloading request and is a preset proportion of the residual downloading bandwidth of the data node.
The download bandwidth updating request also comprises a preset bandwidth step length, the predicted bandwidth requested in the download bandwidth updating request is the sum of the actual download bandwidth and the preset bandwidth step length, and the download bandwidth allocated to the data node by the data node according to the download bandwidth updating request is the smaller value of the predicted bandwidth and the preset proportion of the residual download bandwidth of the data node.
The invention also provides a method for downloading the distributed file, which applies the data node and comprises the following steps:
receiving a query message of a file downloading client, querying a file name of a downloading file stored in the device, and sending the file name to the file downloading client;
receiving a file downloading request which is sent by a file downloading client and contains a downloading file name and initial bandwidth, and distributing downloading bandwidth for the file downloading request according to the initial bandwidth in the file downloading request;
and receiving a downloading bandwidth updating request which is periodically sent by the file downloading client and contains the actual downloading bandwidth obtained by statistics, and distributing the downloading bandwidth for the file downloading client according to the actual downloading bandwidth in the downloading bandwidth updating request.
The allocating the download bandwidth for the file download request according to the initial bandwidth in the file download request comprises:
and calculating the residual downloading bandwidth of the data node, and taking the preset proportion of the residual downloading bandwidth of the data node as the downloading bandwidth distributed for the file downloading client.
The downloading bandwidth updating request further includes a preset bandwidth step length, the predicted bandwidth requested in the downloading bandwidth updating request is the sum of the actual downloading bandwidth and the preset bandwidth step length, and the allocating of the downloading bandwidth according to the actual downloading bandwidth in the downloading bandwidth updating request includes:
and calculating the residual downloading bandwidth of the data node, and taking the smaller value of the preset proportion of the residual downloading bandwidth of the data node and the predicted bandwidth as the downloading bandwidth allocated to the file downloading client.
The invention also provides a download device of the distributed file, which is applied to a file download client, and the download device of the distributed file comprises:
the node request module is used for sending a data node request message to the metadata management server and acquiring information of a data node for storing the downloaded file;
the file query module is used for sending a query message to a data node for storing the downloaded file and acquiring the file name of the downloaded file stored in the data node;
the download module is used for starting a plurality of concurrent threads according to the acquired information of the data nodes, sending a file download request containing a download file name and an initial bandwidth to the corresponding data nodes when each concurrent thread corresponds to one of the data nodes, and downloading the file according to the download bandwidth allocated to the data nodes according to the file download request;
and the updating module is used for counting the actual downloading bandwidth, periodically sending a downloading bandwidth updating request containing the actual downloading bandwidth to the data node, and downloading the file according to the downloading bandwidth distributed by the data node according to the downloading bandwidth updating request.
The invention also provides a download device of the distributed file, which is applied to the data node and comprises:
the query module is used for receiving a query message of the file downloading client, querying the file name of the downloaded file stored in the equipment, and sending the file name to the file downloading client;
the system comprises an initial bandwidth allocation module, a file downloading module and a bandwidth allocation module, wherein the initial bandwidth allocation module is used for receiving a file downloading request which is sent by a file downloading client and contains a downloading file name and initial bandwidth, and allocating downloading bandwidth for the file downloading request according to the initial bandwidth in the file downloading request;
and the bandwidth updating module is used for receiving a downloading bandwidth updating request which is periodically sent by the file downloading client and contains the actual downloading bandwidth obtained through statistics, and allocating the downloading bandwidth for the file downloading client according to the actual downloading bandwidth in the downloading bandwidth updating request.
When the initial bandwidth allocation module allocates the download bandwidth for the file download request according to the initial bandwidth in the file download request, the following operations are executed:
and calculating the residual downloading bandwidth of the data node, and taking the preset proportion of the residual downloading bandwidth of the data node as the downloading bandwidth distributed for the file downloading client.
The download bandwidth updating request also comprises a preset bandwidth step length, the predicted bandwidth requested in the download bandwidth updating request is the sum of the actual download bandwidth and the preset bandwidth step length, and the bandwidth updating module executes the following operations when allocating the download bandwidth according to the actual download bandwidth in the download bandwidth updating request:
and calculating the residual downloading bandwidth of the data node, and taking the smaller value of the preset proportion of the residual downloading bandwidth of the data node and the predicted bandwidth as the downloading bandwidth allocated to the file downloading client.
The invention provides a method and a device for downloading a distributed file.A client side starts a plurality of concurrent threads, each concurrent thread is connected with a data node in a distributed storage system and downloads the distributed file, so that the downloading speed of the distributed file is greatly accelerated, and the downloading bandwidth is periodically updated according to the downloading capacity of the client side and the residual downloading bandwidth of the data nodes in the file downloading process, so that the downloading bandwidth of the data nodes is reasonably utilized, the downloading speed of the client side is maximized, and the simultaneous downloading speed of a plurality of client sides is improved. The problem of among the prior art client carry out the video monitoring video download, all need go on through video on demand server VOD, cause video on demand server VOD to become the bottleneck that the video of control downloaded, seriously influence video monitoring video download speed is solved. The invention also reasonably distributes the total storage bandwidth and the total download bandwidth in the data nodes by counting the file storage speed in the data nodes and according to the generated file upload speed curve, so that the data nodes can not influence the storage of the files when downloading the files.
Drawings
FIG. 1 is a schematic diagram of a prior art downloading video surveillance video;
FIG. 2 is a flowchart of a distributed file downloading method according to the present invention;
fig. 3 is a schematic diagram of a network connection structure according to the embodiment.
Detailed Description
The technical solutions of the present invention are further described in detail below with reference to the drawings and examples, which should not be construed as limiting the present invention.
A method for downloading a distributed file, applied to a file downloading client, as shown in fig. 2, includes:
step S1, sending a data node request message to the metadata management server, and acquiring information of the data node storing the download file.
The present embodiment is described by taking downloading of video surveillance videos stored in a distributed manner in a video surveillance system as an example. In the schematic view of downloading a video surveillance record of this embodiment, as shown in fig. 3, an MD is a metadata management server for managing data nodes; the Client is a storage Client, is deployed on the camera or the streaming server, and is used for storing the video monitoring videos generated by the camera on the data nodes distributed by the metadata management server to realize distributed storage of the video monitoring data; DN is a data node and is used for storing video monitoring data; the UI is a video monitoring client used for downloading video monitoring videos, namely a file downloading client.
The storage process of the video surveillance video of the embodiment is as follows: before storing video monitoring data, the storage client sends a request message to the metadata management server to request to acquire information of the data nodes, the metadata management server allocates corresponding data nodes according to a distributed storage principle and feeds back the allocated information of the data nodes to the storage client, and the storage client writes the video monitoring data into the data nodes according to the received information of the data nodes, such as IP addresses. The specific steps of writing the video monitoring data into the data nodes by the storage client side are as follows: the storage client creates a directory with the camera code as the directory name on the data node, then creates a subdirectory with the current date as the directory name under the directory with the camera code as the directory name, then creates a file with the file name containing the start time and the end time of the written video monitoring video in the subdirectory with the current date as the directory name, and writes the video monitoring video in the file. For example, the storage client creates a file in DN 1: cam1/20160819/S0830000E103000B. h3crd, which shows that the camera code of the video surveillance video is cam1, the date of the video surveillance video is 2016, 08 and 19 days, and the time of the video surveillance video is 08:30:00 to 10:30: 00.
It should be noted that, in order to implement distributed storage of video surveillance videos, the storage client in this embodiment stores video surveillance data of a camera on different data nodes according to a preset time period. For example, the preset time period is two hours, after the camera is turned on (the turn-on time is 8: 30), the storage client requests the metadata management server to allocate a data node, the data node allocated by the metadata management server is DN1, and the storage client allocates the data node of the camera 8: 30-10: video surveillance data for 30 is written to data node DN1, in the morning 10:30 newly requests the metadata management server to allocate a data node, the metadata management server allocates a data node DN2, and the storage client stores the video camera 10: 30-12: the video monitoring data of 30 is written into the data node DN2, and the operation is repeated in a circulating way, so that the distributed storage of the video monitoring videos of the camera is realized.
In this embodiment, when the video monitoring client needs to download the video monitoring record, first, a data node acquisition request message is sent to the metadata management server, where the request message carries a camera code of the video monitoring record that needs to be downloaded. The metadata management server inquires all data nodes storing the video monitoring videos of the camera according to the camera codes in the request message, and sends the information (such as IP addresses of the data nodes) of the inquired data nodes to the video monitoring client.
By the method, the information of the data node storing the download file is acquired.
Step S2, sending a query message to the data node storing the downloaded file, and obtaining the file name of the downloaded file stored in the data node.
In this embodiment, after information of a data node storing a downloaded file is acquired, a camera code and a time period of a video surveillance record to be downloaded are sent to each data node, the data node searches whether a video surveillance record of the camera in the time period is stored in a file system of the device, if the video surveillance record of the camera in the time period exists, a file name of the video surveillance record is sent to a video surveillance client, and if the video surveillance record of the camera in the time period does not exist, a message that the file does not exist is sent to the video surveillance client.
For example, the video monitoring client in this embodiment needs to download the video monitoring video with the camera code cam1 and the time of 2016, 08, 19, 9: 00-12: 30; the DN1 stores video monitoring videos with camera codes of cam1 and the time of 2016, 08, month, 19, 8: 30-10: 30, and the stored directory and file names are as follows: (iii) cam1/20160819/S0830000E103000B. h3crd; the DN2 stores video monitoring videos with camera codes of cam1 and the time of 2016, 08 and 19 months, 10: 30-12: 30, and the stored directory and file names are as follows: (ii) cam1/20160819/S1030000E123000B. h3crd; the DN3 stores video monitoring videos with camera codes of cam1 and the time of 2016, 08 and 19 months, 12: 30-14: 30, and the stored directory and file names are as follows: /cam 1/20160819/S1230000E143000B.h3crd. The video client sends a camera code cam1 of a video monitoring video to be downloaded and 10: 30-12: 30 of 2016, 08, 19, months and 19 days to DN1, DN2 and DN3 respectively; DN1 inquires the video monitoring video in the device, finds that there is a video monitoring video meeting the above condition, and sends the file name of the video monitoring video S0830000E103000B. h3crd to the video monitoring client; DN2 inquires the video monitoring video in the device, finds the video monitoring video meeting the above condition, and sends the file name of the video monitoring video S1030000E123000B. h3crd to the video monitoring client; the DN3 inquires the video monitoring record in the device, finds that there is no video monitoring record meeting the above conditions, and sends a message that the file does not exist to the video monitoring client.
By the method, the file name of the download file stored in the data node is obtained.
Step S3, according to the acquired information of the data nodes, starting a plurality of concurrent threads, wherein each concurrent thread corresponds to one of the data nodes, sending a file downloading request containing a downloading file name and an initial bandwidth to the corresponding data node, and downloading the file according to the downloading bandwidth allocated to the data node according to the file downloading request.
In this embodiment, after data node information for storing the video surveillance videos to be downloaded and file names of the video surveillance videos stored on the data nodes are acquired, a plurality of concurrent threads are started, the number of the concurrent threads is the same as the number of the acquired data nodes, each concurrent thread establishes a connection with one data node, then a corresponding video surveillance video file is downloaded through the established connection, and each concurrent thread downloads one video surveillance video file each time.
The process of downloading the video monitoring video file by each concurrent thread is as follows:
firstly, a file downloading request is sent to a data node, wherein the file downloading request comprises a file name of a downloaded video monitoring record and an initial bandwidth, and the initial bandwidth is a set basic bandwidth and is generally set to be 0. The data node is configured with a maximum concurrent download file number and a minimum download bandwidth allocation threshold, after receiving a download request, the data node queries the number of files currently being downloaded and the remaining download bandwidth of the data node, and if the number of the files currently being downloaded is equal to the maximum concurrent download file number or the remaining download bandwidth of the data node is less than the minimum download bandwidth allocation threshold, a download refusal message is returned, at this moment, a download failure occurs, and the concurrent thread can re-initiate the download request according to a preset time period.
And if the number of the files currently being downloaded is less than the maximum concurrent download file number and the residual download bandwidth of the data node is greater than the minimum download bandwidth allocation threshold value, allocating the download bandwidth of the thread according to the residual download bandwidth of the data node and the predicted bandwidth in the download request.
The residual download bandwidth of the data node is calculated by the following method:
the total download bandwidth of the data node is first determined, which may be determined in two ways.
1. And calculating the total download bandwidth of the data nodes according to the total bandwidth of the data nodes and the preset proportion of the storage bandwidth to the download bandwidth. For example, the total bandwidth of the data nodes is 1000Mbit/s, the preset ratio of the storage bandwidth to the download bandwidth is 8: and 2, obtaining the total download bandwidth of the data node to be 200 Mbit/s.
2. The flow of the video storage in the video monitoring system is relatively constant, so that the flow curve of the video storage can be obtained by counting the real-time storage flow of the data nodes, and the total download bandwidth of the data nodes is determined as the total bandwidth minus the storage bandwidth according to the flow curve of the video storage.
And then subtracting the allocated download bandwidth from the total download bandwidth of the data node to obtain the residual download bandwidth of the data node.
Since the initial bandwidth in the download request is 0, that is, the thread sending the download request does not know the download bandwidth value supported by its own network environment, in order to maximize the download speed, in this embodiment, 80% of the remaining download bandwidth of the data node is allocated to the thread, and the thread downloads the video monitoring video file according to the download bandwidth allocated by the data node.
It is easy to understand that, when the data node allocates the bandwidth, it may adopt a plurality of methods to allocate, and allocate within the allowable range of the remaining download bandwidth of the data node, for example, directly allocate a fixed bandwidth, and the like, which is not described herein again.
Step S4, counting the actual download bandwidth, periodically sending a download bandwidth update request containing the actual download bandwidth to the data node, and downloading the file according to the download bandwidth allocated by the data node according to the download bandwidth update request.
In order to make the download bandwidth of the data node reasonably utilized and maximize the download speed of the thread, the thread of this embodiment periodically sends a download bandwidth update request to the data node according to a preset period threshold value in the download process of the video surveillance video file. The download bandwidth update request carries the actual download bandwidth, and the actual download is obtained according to the flow statistics.
After receiving the download bandwidth updating request, the data node allocates bandwidth to the video monitoring client according to the actual download bandwidth carried in the download bandwidth updating request:
for example, when the counted actual download bandwidth is 30Mbit/s, a preset bandwidth step size (assuming that the preset download bandwidth step size is 10Mbit/s) may be directly added on the basis of 30Mbit/s, so as to obtain that the bandwidth actually allocated to the video monitoring client is 40 Mbit/s.
The preset bandwidth step length can be set at the video monitoring client, and can also be set at the data node. When the preset bandwidth step length is set by the video monitoring client, the preset bandwidth step length needs to be included in the download bandwidth update request. When the data node is set, the download bandwidth update request only contains the actual download bandwidth.
Because the download bandwidth of the data node is limited, in this embodiment, first, whether the proportion of the download bandwidth allocated to the data node to the total download bandwidth is greater than a preset proportion threshold (for example, 80%), and if the proportion is smaller than the preset proportion threshold, it indicates that the remaining download bandwidth is still sufficient, the smaller value between 80% of the remaining download bandwidth and the predicted bandwidth is used as the download bandwidth allocated to the thread; and if the ratio is larger than or equal to the preset ratio threshold, the residual download bandwidth is tense, and the smaller value of 50% of the current residual download bandwidth and the predicted bandwidth is taken as the download bandwidth allocated to the thread.
And the thread continues to download the video monitoring video file according to the download bandwidth redistributed by the data node, and repeats the process until the video monitoring video file is downloaded.
It should be noted that, in this embodiment, the data node updates the allocated download bandwidth after allocating the download bandwidth each time, for example, the allocated bandwidth of the data node before allocating the download bandwidth is 60Mbit/s, and the download bandwidth allocated this time is 40Mbit/s, and then the allocated download bandwidth is updated to 100Mbit/s after allocating the download bandwidth this time. In addition, before the data node allocates the download bandwidth each time, if the recoverable download bandwidth exists, the recoverable download bandwidth is recovered first, and then the download bandwidth allocation is performed. For example, when a certain thread sends a download request, the download bandwidth allocated to the thread by a data node is 100Mbit/s, when the thread sends a download bandwidth update request for the first time, the residual download bandwidth of the data node is 20Mbit/s, the 100Mbit/s download bandwidth allocated to the thread is recycled first, the residual download bandwidth is 120Mbit/s, and then the download bandwidth allocation is performed.
The following description will be given of downloading a video surveillance recording file from a data node by a thread of a video surveillance client by way of example:
suppose that the maximum download bandwidth of a certain data node DN1 is 500Mbit/s, the allocated download bandwidth is 100Mbit/s, the minimum download bandwidth allocation threshold is 1Mbit/s, the download bandwidth step length preset by the thread on the video monitoring client is 10Mbit/s, and the cycle threshold of the preset download bandwidth updating request is 5 seconds.
A certain concurrent thread of the video monitoring client firstly sends a downloading request message to DN1, wherein the predicted bandwidth is 0Mbit/s, DN1 allocates 80% of the residual downloading bandwidth after receiving the downloading request message, namely 320Mbit/s, to the concurrent thread, and downloads the video monitoring video file.
After a period threshold value of a download bandwidth updating request is 5s, the actual download bandwidth counted by the concurrency thread is 30Mbit/s, the download bandwidth step length is 10Mbit/s, the predicted bandwidth is 40Mbit/s, the download bandwidth updating request message is sent to DN1, DN1 receives the message, the 320Mbit/s distributed before is recycled, the download bandwidth distributed to the concurrency thread 40MMbit/s is redistributed, and the downloading of the video monitoring video file is continued.
After a period threshold value of 5s of a download bandwidth updating request, the actual download bandwidth counted by the concurrent thread is 38Mbit/s, the download bandwidth step length is 10Mbit/s, the expected bandwidth is 48Mbit/s, then the download bandwidth updating request message is sent to DN1, DN1 receives the message, firstly recovers the previously distributed 40Mbit/s, redistributes the download bandwidth to the concurrent thread 48Mbit/s, and continues downloading the video monitoring video file.
And repeating the process until the video monitoring video file is downloaded.
By the method, the multiple concurrent threads on each video monitoring client simultaneously download the video monitoring video files on each data node, so that the download bandwidth of the data nodes and the download capability of the video monitoring client are fully utilized, and the download of the video monitoring video files is greatly accelerated.
Corresponding to the method, as an embodiment of the present invention, a corresponding device for downloading a distributed file is also provided, which is applied to a file downloading client. The download device of the distributed file comprises:
the node request module is used for sending a data node request message to the metadata management server and acquiring information of a data node for storing the downloaded file;
the file query module is used for sending a query message to a data node for storing the downloaded file and acquiring the file name of the downloaded file stored in the data node;
the download module is used for starting a plurality of concurrent threads according to the acquired information of the data nodes, sending a file download request containing a download file name and an initial bandwidth to the corresponding data nodes when each concurrent thread corresponds to one of the data nodes, and downloading the file according to the download bandwidth allocated to the data nodes according to the file download request;
and the updating module is used for counting the actual downloading bandwidth, periodically sending a downloading bandwidth updating request containing the actual downloading bandwidth to the data node, and downloading the file according to the downloading bandwidth distributed by the data node according to the downloading bandwidth updating request.
Corresponding to the method, the invention further provides a corresponding device for downloading the distributed file and an application data node as an embodiment of the invention. The download device of the distributed file comprises:
the query module is used for receiving a query message of the file downloading client, querying the file name of the downloaded file stored in the equipment, and sending the file name to the file downloading client;
the system comprises an initial bandwidth allocation module, a file downloading module and a bandwidth allocation module, wherein the initial bandwidth allocation module is used for receiving a file downloading request which is sent by a file downloading client and contains a downloading file name and initial bandwidth, and allocating downloading bandwidth for the file downloading request according to the initial bandwidth in the file downloading request;
and the bandwidth updating module is used for receiving a downloading bandwidth updating request which is periodically sent by the file downloading client and contains the actual downloading bandwidth obtained through statistics, and allocating the downloading bandwidth for the file downloading client according to the actual downloading bandwidth in the downloading bandwidth updating request.
Wherein, the initial bandwidth allocation module executes the following operations when allocating the download bandwidth for the file download request according to the initial bandwidth in the file download request:
and calculating the residual downloading bandwidth of the data node, and taking the preset proportion of the residual downloading bandwidth of the data node as the downloading bandwidth distributed for the file downloading client.
When the download bandwidth update request further includes a preset bandwidth step, the predicted bandwidth requested in the download bandwidth update request is the sum of the actual download bandwidth and the preset bandwidth step, and the bandwidth update module allocates the download bandwidth for the actual download bandwidth in the download bandwidth update request according to the actual download bandwidth, the following operations are executed:
and calculating the residual downloading bandwidth of the data node, and taking the smaller value of the preset proportion of the residual downloading bandwidth of the data node and the predicted bandwidth as the downloading bandwidth allocated to the file downloading client.
The above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and those skilled in the art can make various corresponding changes and modifications according to the present invention without departing from the spirit and the essence of the present invention, but these corresponding changes and modifications should fall within the protection scope of the appended claims.
Claims (10)
1. A method for downloading a distributed file is applied to a file downloading client, and is characterized in that the method for downloading the distributed file comprises the following steps:
sending a data node request message to a metadata management server, and acquiring information of a data node storing a downloaded file;
sending a query message to a data node for storing the download file, and acquiring the file name of the download file stored in the data node;
starting a plurality of concurrent threads according to the acquired information of the data nodes, wherein each concurrent thread corresponds to one data node, sends a file downloading request containing a downloading file name and an initial bandwidth to the corresponding data node, and downloads the file according to the downloading bandwidth distributed by the data node according to the file downloading request;
counting actual download bandwidth, periodically sending a download bandwidth updating request containing the actual download bandwidth to a data node, and downloading a file for the download bandwidth allocated to the data node according to the download bandwidth updating request by the data node;
the data node allocates the download bandwidth to the data node according to the download bandwidth updating request, wherein the download bandwidth is a smaller value of a preset proportion of the residual download bandwidth of the data node and a predicted bandwidth corresponding to the download bandwidth updating request, and the predicted bandwidth corresponding to the download bandwidth updating request is the sum of the actual download bandwidth and a preset bandwidth step length.
2. The method for downloading a distributed file according to claim 1, wherein the download bandwidth update request further includes a preset bandwidth step size.
3. A method for downloading a distributed file, which is applied to a data node, is characterized in that the method for downloading the distributed file comprises the following steps:
receiving a query message of a file downloading client, querying a file name of a downloading file stored in the device, and sending the file name to the file downloading client;
receiving a file downloading request which is sent by a file downloading client and contains a downloading file name and initial bandwidth, and distributing downloading bandwidth for the file downloading request according to the initial bandwidth in the file downloading request;
receiving a downloading bandwidth updating request which is periodically sent by a file downloading client and contains the actual downloading bandwidth obtained through statistics, and distributing the downloading bandwidth for the file downloading client according to the actual downloading bandwidth in the downloading bandwidth updating request;
wherein, the allocating the download bandwidth for the actual download bandwidth in the update request of the download bandwidth comprises:
and calculating the residual download bandwidth of the data node, and taking the smaller value of the preset proportion of the residual download bandwidth of the data node and the predicted bandwidth corresponding to the download bandwidth updating request as the download bandwidth allocated to the file download client, wherein the predicted bandwidth corresponding to the download bandwidth updating request is the sum of the actual download bandwidth and the preset bandwidth step length.
4. The method for downloading the distributed file according to claim 3, wherein the allocating the download bandwidth for the file download request according to the initial bandwidth thereof comprises:
and calculating the residual downloading bandwidth of the data node, and taking the preset proportion of the residual downloading bandwidth of the data node as the downloading bandwidth distributed for the file downloading client.
5. The method for downloading the distributed file according to claim 3, wherein the request for updating the download bandwidth further comprises a preset bandwidth step size.
6. The utility model provides a distributed file's download device, is applied to file download client, its characterized in that, distributed file's download device includes:
the node request module is used for sending a data node request message to the metadata management server and acquiring information of a data node for storing the downloaded file;
the file query module is used for sending a query message to a data node for storing the downloaded file and acquiring the file name of the downloaded file stored in the data node;
the download module is used for starting a plurality of concurrent threads according to the acquired information of the data nodes, sending a file download request containing a download file name and an initial bandwidth to the corresponding data nodes when each concurrent thread corresponds to one of the data nodes, and downloading the file according to the download bandwidth allocated to the data nodes according to the file download request;
the updating module is used for counting the actual downloading bandwidth, periodically sending a downloading bandwidth updating request containing the actual downloading bandwidth to the data node, and downloading the file according to the downloading bandwidth allocated to the data node according to the downloading bandwidth updating request;
the data node allocates the download bandwidth to the data node according to the download bandwidth updating request, wherein the download bandwidth is a smaller value of a preset proportion of the residual download bandwidth of the data node and a predicted bandwidth corresponding to the download bandwidth updating request, and the predicted bandwidth corresponding to the download bandwidth updating request is the sum of the actual download bandwidth and a preset bandwidth step length.
7. The apparatus for downloading distributed files according to claim 6, wherein the request for updating download bandwidth further comprises a preset bandwidth step size.
8. The utility model provides a download device of distributed file, uses data node, characterized in that, download device of distributed file includes:
the query module is used for receiving a query message of the file downloading client, querying the file name of the downloaded file stored in the equipment, and sending the file name to the file downloading client;
the system comprises an initial bandwidth allocation module, a file downloading module and a bandwidth allocation module, wherein the initial bandwidth allocation module is used for receiving a file downloading request which is sent by a file downloading client and contains a downloading file name and initial bandwidth, and allocating downloading bandwidth for the file downloading request according to the initial bandwidth in the file downloading request;
the bandwidth updating module is used for receiving a downloading bandwidth updating request which is periodically sent by a file downloading client and contains the actual downloading bandwidth obtained through statistics, and allocating the downloading bandwidth for the downloading bandwidth according to the actual downloading bandwidth in the downloading bandwidth updating request;
when the bandwidth updating module allocates the download bandwidth to the actual download bandwidth in the download bandwidth updating request according to the actual download bandwidth, the following operations are executed:
and calculating the residual download bandwidth of the data node, and taking the smaller value of the preset proportion of the residual download bandwidth of the data node and the predicted bandwidth corresponding to the download bandwidth updating request as the download bandwidth allocated to the file download client, wherein the predicted bandwidth corresponding to the download bandwidth updating request is the sum of the actual download bandwidth and the preset bandwidth step length.
9. The apparatus for downloading distributed files according to claim 8, wherein the initial bandwidth allocation module, when allocating the download bandwidth to the file according to the initial bandwidth in the file download request, performs the following operations:
and calculating the residual downloading bandwidth of the data node, and taking the preset proportion of the residual downloading bandwidth of the data node as the downloading bandwidth distributed for the file downloading client.
10. The apparatus for downloading distributed files according to claim 8, wherein the request for updating download bandwidth further comprises a preset bandwidth step size.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610770461.6A CN106254833B (en) | 2016-08-30 | 2016-08-30 | Method and device for downloading distributed files |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610770461.6A CN106254833B (en) | 2016-08-30 | 2016-08-30 | Method and device for downloading distributed files |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106254833A CN106254833A (en) | 2016-12-21 |
| CN106254833B true CN106254833B (en) | 2020-01-14 |
Family
ID=58079564
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610770461.6A Active CN106254833B (en) | 2016-08-30 | 2016-08-30 | Method and device for downloading distributed files |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106254833B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109753849B (en) * | 2017-11-03 | 2021-08-17 | 杭州海康威视数字技术股份有限公司 | Method and device for processing video analysis task |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002035359A3 (en) * | 2000-10-26 | 2003-09-18 | Prismedia Networks Inc | Method and system for managing distributed content and related metadata |
| CN1941681A (en) * | 2005-09-30 | 2007-04-04 | 财团法人工业技术研究院 | Internet wide band distribution |
| CN101141627A (en) * | 2007-10-23 | 2008-03-12 | 深圳市迅雷网络技术有限公司 | Storage system and method of stream media file |
| CN102368833A (en) * | 2011-11-29 | 2012-03-07 | 乐视网信息技术（北京）股份有限公司 | Method for carrying out segment download equalization on video information |
| CN102761530A (en) * | 2011-04-29 | 2012-10-31 | 华为终端有限公司 | Method, device and system for media code stream transmission |
| CN102811258A (en) * | 2012-07-27 | 2012-12-05 | 北京星网锐捷网络技术有限公司 | Data parallel-downloading method, apparatus and network device |
| CN103716719A (en) * | 2013-12-02 | 2014-04-09 | 乐视网信息技术（北京）股份有限公司 | Multimedia file downloading method and user terminal |
-
2016
- 2016-08-30 CN CN201610770461.6A patent/CN106254833B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002035359A3 (en) * | 2000-10-26 | 2003-09-18 | Prismedia Networks Inc | Method and system for managing distributed content and related metadata |
| CN1941681A (en) * | 2005-09-30 | 2007-04-04 | 财团法人工业技术研究院 | Internet wide band distribution |
| CN101141627A (en) * | 2007-10-23 | 2008-03-12 | 深圳市迅雷网络技术有限公司 | Storage system and method of stream media file |
| CN102761530A (en) * | 2011-04-29 | 2012-10-31 | 华为终端有限公司 | Method, device and system for media code stream transmission |
| CN102368833A (en) * | 2011-11-29 | 2012-03-07 | 乐视网信息技术（北京）股份有限公司 | Method for carrying out segment download equalization on video information |
| CN102811258A (en) * | 2012-07-27 | 2012-12-05 | 北京星网锐捷网络技术有限公司 | Data parallel-downloading method, apparatus and network device |
| CN103716719A (en) * | 2013-12-02 | 2014-04-09 | 乐视网信息技术（北京）股份有限公司 | Multimedia file downloading method and user terminal |
Non-Patent Citations (2)
| Title |
|---|
| 分布式结构下视频流的缓存优化分配;王江涛，彭伟，关冬亮;《哈尔滨理工大学学报》;20111231;第16卷(第6期);全文 * |
| 基于流媒体的小型分布式视频点播系统研究;朱晓辉，陈苏蓉;《微电子学与计算机》;20101031;第27卷(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106254833A (en) | 2016-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US6128656A (en) | System for updating selected part of configuration information stored in a memory of a network element depending on status of received state variable | |
| US8037200B2 (en) | Media organization for distributed sending of media data | |
| CN101136911B (en) | Method to download files using P2P technique and P2P download system | |
| CN102333126B (en) | Streaming media on demand method based on Hadoop and virtual streaming media server cluster | |
| US7614071B2 (en) | Architecture for distributed sending of media data | |
| US6721850B2 (en) | Method of cache replacement for streaming media | |
| DE602004004200T2 (en) | System and method for managing buffered objects using messaging connections | |
| CN111200657B (en) | Method for managing resource state information and resource downloading system | |
| US9445162B2 (en) | Interactive personal/internet protocol television reservation system, reservation plan management method and device | |
| US10191909B2 (en) | File system creating and deleting apparatus and driving method thereof | |
| US11032394B1 (en) | Caching techniques | |
| US7545812B2 (en) | Scheduling scheme for distributed sending of media data | |
| CN108509297A (en) | A kind of data back up method and system | |
| CN114125489B (en) | Data stream distribution method and system | |
| CN101094389A (en) | Method for downloading video, video server, and monitoring center system | |
| CN102957972B (en) | Video on demand method and system thereof | |
| CN106254833B (en) | Method and device for downloading distributed files | |
| CN106534336B (en) | A kind of video subscribing dynamically realizes system and method | |
| WO2017215415A1 (en) | Resource control method and apparatus, and iptv server | |
| CN108174147A (en) | Video source and the Automatic Optimal distribution method of Scout service device in video monitoring system | |
| JP2011191907A (en) | Data distribution system, load balancing method and storage server | |
| CN104469539A (en) | A cooperation buffering method, streaming media managing subsystem and server | |
| CN109981484B (en) | Monitoring system, monitoring method and monitoring center | |
| CN114860483B (en) | Method and system for quickly collecting equipment data and realizing high-efficiency persistence | |
| KR101986850B1 (en) | A method for managing information of M2M system and apparatus therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |