CN102333126B - Streaming media on demand method based on Hadoop and virtual streaming media server cluster - Google Patents
Streaming media on demand method based on Hadoop and virtual streaming media server cluster Download PDFInfo
- Publication number
- CN102333126B CN102333126B CN201110312612.0A CN201110312612A CN102333126B CN 102333126 B CN102333126 B CN 102333126B CN 201110312612 A CN201110312612 A CN 201110312612A CN 102333126 B CN102333126 B CN 102333126B
- Authority
- CN
- China
- Prior art keywords
- stream media
- media server
- virtual
- virtual stream
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The invention discloses a streaming media on demand method based on Hadoop and a virtual streaming media server cluster. The virtual streaming media server cluster is constructed on a virtual machine server cluster, and directly acquires stored streaming media file data from a Hadoop storage cluster to realize streaming media on demand. Particularly, in the method, a virtual machine server management node monitors the service conditions of memory and network bandwidth resources of the virtual streaming media server cluster in real time to realize the dynamic scheduling of virtual streaming media servers; hotspot file prefixes on the Hadoop storage cluster are deployed on the virtual streaming media servers according to a virtual streaming media server cluster balanced-deployment principle; requests of users are cached at intervals on the virtual streaming media servers according to a time sequence; and cache sizes on each virtual streaming media server are controlled and a virtual machine scheduling strategy is triggered according to the file reading bandwidth of the virtual streaming media server cluster for the Hadoop storage cluster.
Description
Technical field
The present invention relates to a kind ofly, use the order method of virtual machine scheduling policy and file cache strategy based on Hadoop (a kind of high fault tolerance, high transmission rates, the distributed storage cluster that conducts interviews in the mode of stream) storage resources and at the virtual stream media server of rolling reamer machine deploy.
Background technology
Along with Internet technology use extensive day by day, video-on-demand service has become the important component part of people's daily life, the quality of video-on-demand service quality largely depends on the support of backstage vod server.The method that makes up the backstage vod server has a lot, and the applicant is new through looking into, and retrieves the patent of 3 pieces of relevant stream media server systems or stream media ordering method, and they are respectively:
1. stream media service dynamic load method (application number: 200610106932.X)
2. the cluster streaming media server system of a suitable large-scale consumer program request (application number: 201010117647.4)
3. video grid method for self-adapted load balance scheduling (application number: 200610144274.3)
The common problem that exists of above-mentioned existing patent is: streaming media server substantially all is a real server and do not relate to virtual stream media server cluster; The loading condition that dynamic load method all is based on server carries out balanced distribution with user's request, and does not consider user's focus degree; Do not consider that back-end data reads bandwidth thereby can't dynamically control virtual machine cache size at interval.
Summary of the invention
The deficiency that the objective of the invention is to solve in the background technology relevant stream media server system and stream media ordering method patent, providing a kind of leaves files in stream media on the Hadoop storage cluster in, the clustered deploy(ment) of virtual stream media server is on blade server, reception is obtained the files in stream media data from Hadoop storage cluster, and adopt virtual stream media server colony dispatching and metadata cache strategy, realize the method for streaming media on demand function.
For reaching above purpose, the present invention takes following technical scheme to be achieved:
A kind of stream media ordering method based on Hadoop and virtual stream media server cluster, it is characterized in that, comprise the steps: that (1) files in stream media leaves on the Hadoop storage cluster, (2) streaming media server is structured on the Virtual Server Cluster, (3) virtual stream media server cluster directly obtains the files in stream media data from Hadoop storage cluster, and employing virtual stream media server colony dispatching and data cache method, realize the streaming media on demand function, wherein, the described virtual stream media server of step (3) colony dispatching method is, the virtual machine server management node is used for controlling the unlatching of virtual stream media server and closing, the virtual machine server management node with virtual stream media server cluster network bandwidth usage surpass 80% or memory usage to surpass 90% be foundation, trigger virtual stream media server set-up mechanism; With virtual stream media server cluster overall bandwidth occupancy less than 20% and memory usage less than 30% and to have user's request amount be that 0 virtual stream media server is a foundation, trigger virtual stream media server reclaim mechanism;
The described virtual stream media server of step (3) cluster data cache method is, adopted the balanced deployment strategy of metadata cache of virtual stream media server cluster, carries out the prefix of file for the focus files in stream media and disposes; According to the real-time focus degree of files in stream media, real-time focus Documents Department is deployed on the virtual stream media server of new unlatching; And when carrying out the streaming media on demand service, consider the time order and function order and the relevance of request same stream media file, to the The data corresponding intervals buffer memory that obtains from Hadoop storage cluster; Read bandwidth according to virtual stream media server cluster to the file of Hadoop storage cluster, dynamically control the cache size on each virtual stream media server, trigger virtual machine scheduling policy.
In the such scheme, the concrete steps of described triggering virtual stream media server set-up mechanism are:
Step1: virtual machine server management node initialization;
Step2: the virtual machine server management node obtains the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
iObtain current unlatching virtual stream media server number N;
Step3: virtual machine server management node real-time update detects the network bandwidth α that each virtual stream media server takies
iAnd the β of committed memory
iIf virtual stream media server cluster network bandwidth usage
Surpass 80% or memory usage
Surpass 90%, then jump to
Step4, otherwise continue to detect;
Step4: the virtual stream media server is reserved internal memory θ for the virtual stream media server that will open is provided with from assignable resource
N+1, network bandwidth μ
N+1, open the virtual stream media server and serve for user's program request.
The concrete steps of described triggering virtual stream media server reclaim mechanism are:
Step1: virtual machine server management node initialization;
Step2: the virtual machine server management node obtains the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
iObtain current unlatching virtual stream media server number N;
Step3: the virtual machine server management node detects current each virtual stream media server and takies network bandwidth α
iAnd the β of committed memory
iIf the bandwidth usage of current virtual stream media server cluster
Less than 20% and memory usage
Less than 30% and exist the virtual stream media server to satisfy network bandwidth occupation rate β simultaneously
i/ μ
iBe 0%, then jump to Step4, otherwise continue to detect;
Step4: the virtual server management node, manage the internal memory α of this virtual stream media server to be recycled
L, network bandwidth β
LResource adds resource in the middle of the allowable resource, closes this virtual stream media server, reclaims virtual machine.
The balanced deployment strategy of the metadata cache of described virtual stream media server cluster is deployed in each file prefix on the virtual stream media server cluster for the mode of disposing with prefix, and concrete steps are as follows:
Step1:, obtain the focus files in stream media θ of server in time T according to the record of Hadoop storage cluster convection current media file access
i, focus files in stream media θ
iDefinition be in time T, its access times N
iGreater than 5% of the total access times N of Hadoop storage cluster file;
Step2: according to focus files in stream media θ
iThe times N of visit
i, can obtain θ
iRelative temperature
Wherein, n is the number of whole focus files, N
jBe focus files in stream media θ wherein
jAccess times;
Step3: the virtual stream media server number of establishing current startup is M, according to focus files in stream media θ
iRelative temperature
Obtain θ
iNeed carry out the number ρ of the virtual server of prefix deployment
iFor
Step4: according to each focus files in stream media θ
iPrefix is disposed number ρ
i, from M virtual stream media server, choose ρ
iIndividual is file θ
iCarrying out the prefix of load balancing disposes.
Described concrete steps at the new real-time focus file of opening of virtual stream media server deploy are:
Step1: obtain the visit situation of current Hadoop storage cluster convection current media file, establish file θ this moment
iThere is N
iIndividual request visit, the total request of Hadoop storage cluster response number is N;
Step2: for R
iSatisfy R
iThe file θ of＞5%M
i, it is added real-time focus file set, the focus file set is { θ during document
i, θ
j..., θ
x, θ
y, and the focus file is k altogether;
Step3: establishing the virtual stream media server number that virtual stream media services this moment cluster opened is M, from focus file set { θ
i, θ
j..., θ
x, θ
yIn select at random
Individual, these focus files are carried out prefix dispose on the virtual stream media server of newly opening;
Described interval caching method is,
Step1: when the t moment, the user asks q to arrive the virtual machine server management node, checks whether the files in stream media θ of current request is present in the middle of the prefix deployment of virtual stream media server cluster.If exist, jump to Step2; Otherwise jump to Step3;
Step2: for the virtual stream media server set { S that exists files in stream media θ prefix to dispose
1, S
2..., S
n, the response request q of memory usage minimum serves the user in the selection virtual stream media server; When this moment this virtual stream media server had request q ' to Hadoop storage cluster files in stream media θ, the data that q ' beginning buffer memory reads, and the data after the Hadoop storage cluster demand file θ prefix, the data of getting off up to q ' buffer memory can be request q service; If there is not the request to Hadoop storage cluster convection current media file θ in current virtual stream media server, then uses the prefix buffer memory to serve, and from Hadoop storage cluster, obtain the further part of θ prefix file;
Step3: check (whether t-Δ t t) exists identical file request to the virtual stream media server in time in the time to file θ.If have identical file request, jump to Step4; Otherwise jump to Step5;
Step4: will ask q to distribute to exist the virtual stream media server m of internal memory occupation rate minimum in the virtual stream media server of file θ request, m begins the data of cache file θ.This virtual stream media server m begins the file θ of request q request is served, the virtual stream media server uses the prefix buffer memory to serve, and beginning is to the further part of Hadoop storage cluster demand file θ prefix file, up to asking to begin data in buffer, utilize the interval data in buffer to serve at time t;
Step5: if there is no identical file request, then distribute the virtual stream media server of a memory usage minimum, begin to respond the service of this time asking q;
Step6: in the time of new request, be repeated to Step1, carry out Cyclic Service.
Cache size concrete steps on each virtual stream media server of described dynamic control are as follows:
Step1: the virtual machine server management node is obtained current Hadoop storage cluster service bandwidth occupation rate φ.If φ greater than 90%, jumps to Step2; If φ less than 10%, jumps to Step3;
Step2: the virtual machine server management node sends to each virtual stream media server increases the message of cache size at interval; Each virtual stream media server is received increases buffered message at interval, increases the interval buffer memory of each virtual stream media server, and size is half of each virtual stream media server free memory;
Step3: the virtual machine server management node sends the message of initialization interval cache size to each virtual stream media server; Each virtual stream media server is received the initialization interval buffered message, and interval cache size separately is set to initial value.
Compared with prior art, advantage of the present invention is:
1, virtual stream media services clustered deploy(ment) can be passed through the virtual machine server management node on blade server, according to virtual machine scheduling policy virtual machine is dynamically opened and is closed.
2, consider the focus degree of Hadoop storage cluster files in stream media, according to the focus degree file has been carried out corresponding prefix and dispose, can realize the load balancing of virtual bulk flow media server cluster.
3, consider the bandwidth that reads of virtual stream media server cluster and Hadoop storage cluster, can dynamically control virtual machine cache size at interval.Like this, the virtual machine server management node can trigger virtual machine creating and reclaim mechanism according to the network and the internal memory operating position of virtual stream media server cluster.
Description of drawings
Below in conjunction with the drawings and the specific embodiments content of the present invention is described in further detail.
Fig. 1 is the system architecture diagram of the stream media ordering method based on Hadoop and virtual stream media server cluster of the present invention.
Fig. 2 is that virtual stream media server of the present invention is created the block diagram on opportunity.
Fig. 3 is that virtual stream media server of the present invention reclaims the block diagram on opportunity.
Fig. 4 is the virtual stream media server visioning procedure among Fig. 1.
Fig. 5 is the virtual stream media server recovery process among Fig. 1.
Fig. 6 is that the present invention obtains Streaming Media focus file step.
Fig. 7 is the deployment umber step that the present invention generates each focus file.
Fig. 8 is the balanced step of disposing of file of the present invention.
Fig. 9 is that the present invention newly opens the flow process that virtual stream media server prefix is disposed.
Figure 10 is known interval cache policy schematic diagram.
Figure 11 is that buffer memory of the present invention is dynamically adjusted flow process.
Embodiment
System architecture
With reference to shown in Figure 1, described stream media ordering method framework based on Hadoop and virtual stream media server cluster is become by virtual machine server management node, virtual stream media server and Hadoop saveset group, and is specific as follows:
Virtual machine server management node: be used to monitor the state of virtual stream media server, the unlatching and the shutoff operation of control virtual stream media server;
Virtual stream media server cluster: the server that specifically carries out order program service to the user; Formed virtual stream media services cluster by a plurality of virtual stream media servers;
Hadoop stores cluster: be in charge of files in stream media, the files in stream media request of response virtual stream media server;
Be example explanation specific embodiment with virtual resource monitor component IBM Tivoli Monitoring and Virtual Machine Manager serviced component Virtual Computing Lab below: the virtual machine server management node is the server end of virtual resource monitoring software IBM Tivoli Monitoring, hereinafter referred to as ITM, be used for monitoring the resource such as internal memory, the network bandwidth of virtual stream media server, and adopt Virtual Machine Manager serviced component virtual Computing Lab, hereinafter referred to as VCL, finish the management of virtual machine; The virtual stream media server is the monitoring client of ITM, when VCL carries out virtual machine distribution startup, this virtual machine ITM Agent is registered among the ITM Sever; Hadoop stores cluster, is in charge of files in stream media, the files in stream media request of response virtual stream media server.
The set-up mechanism of virtual stream media server
According to current virtual stream media server network bandwidth occupancy and memory usage is index, triggers the creation operation of virtual stream media server.
1) virtual machine server management node initialization
The virtual machine server management node is collected the state information of virtual machine streaming media server, thereby realizes the monitoring to virtual machine streaming media server real-time status.The initialization of virtual machine server management node is responsible for safeguarding the streaming media server network bandwidth that all and virtual stream media apparatus interaction are collected and the tables of data of internal storage state information, handles the state parameter of its internal memory and network interface respectively.
2) the virtual machine server management node obtains the reservation internal memory θ that the virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i
Virtual machine server management node and virtual stream media server are undertaken alternately by the running state parameter information of XML form, collect the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i, opened virtual stream media server number N.
3) virtual machine server management node real-time update detects the network bandwidth α that the virtual stream media server takies
iAnd the β of committed memory
i
Virtual machine server management node real-time servicing upgrades the internal memory of each virtual stream media server and the state parameter of network interface, detects to obtain its network bandwidth α that takies
iAnd the β of committed memory
iOpening new virtual stream media server when satisfying following condition serves for user's program request:
(1) virtual stream media server cluster network bandwidth usage
(2) virtual stream media server cluster memory usage
Otherwise continue to detect each state parameter, repeat this step until satisfying above-mentioned condition.
4) open new virtual stream media server
The virtual server management node is safeguarded the mirror image of virtual stream media server by the virtual machine image manager, when needs are opened new virtual machine, selects corresponding virtual machine mirror image to operate from the mirror image manager.
The virtual stream media server is reserved internal memory θ for the virtual stream media server that is about to open is provided with from assignable resource
N+1, reservation of network bandwidth μ
N+1By the control of the virtual machine image manager in virtual server management node virtual machine image storehouse, be that identical streaming media server copy is copied on the physical server by internal network afterwards with the virtual server mirror image.After finishing the virtual server mirror-image copies, system carries out the startup of N+1 platform virtual server and the setting of streaming media server software environment, for user's program request is served.
With reference to shown in Figure 2, described virtual machine creating mechanism.The virtual machine server management node is realized the system running state data acquisition by middleware ITM, and the calling interface of Web mode is provided to the upper strata application service.The virtual machine server management node is set up the real-time running state parameter information that data are connected and obtain virtual machine by soap protocol with ITM condition monitoring server.
The virtual machine server management node encapsulates all state informations and collection method thereof, every kind of state parameter all has corresponding class to manage, comprise: MemoryParameters class, NetworkParameters class, be responsible for safeguarding the state parameter of internal memory and network interface respectively.
State information between virtual machine server management node and the ITM middleware is to transmit according to the XML form, so management node need call the parsing that the XML data content is carried out in the Dom4j storehouse.ITM Parse class is resolved the XML file that system obtains according to the setting form of ITM, extract various system running state parameters.The main flow process of system running state parameter collection is:
To intended target virtual machine and required running state parameter, generate the system running state parameter collection request of XML form.
1. the object of initialization HttpURLConnection class is set up HTTP with the virtual machine server management node and is linked and use setRequestProperty () method to carry out corresponding configuration to linking.
2. call getOutputStream () method and form output stream, and the object of initialization OutputStream class, the request of Write () method transmit status parameter collection called.
3. call the return results that getInputStream () obtains server, with its with and the object of XML form initialization Document class.
4. obtain the root element of this XML content by calling getRootElement () method.Parse required running state parameter by CalculatePara () method.
Collect the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i, current unlatching virtual stream media server number is N; Then detect and obtain its network bandwidth θ that takies
iAnd the μ of committed memory
iOpening new virtual stream media server when satisfying following condition serves for user's program request:
(1) virtual stream media server cluster network bandwidth usage
(2) virtual stream media server cluster memory usage
Otherwise continue to detect each state parameter, after satisfying above-mentioned condition, the virtual stream media server is from assignable resource, and the virtual stream media server that for this reason is about to open is provided with reserves internal memory θ
N+1, network bandwidth μ
N+1
The virtual machine server management node uses Virtual Machine Manager assembly VCL to manage a plurality of virtual machine operations tasks, VCL is by vcld assembly poll inquiry virtual machine server leader information, if find the virtual machine creating request then call its esx module and carry out corresponding operating.Add the support of the action need mirror site of virtual machine.As shown in Figure 4, this operation implementation procedure is as follows:
Step1:VCL assembly poll inquiry virtual machine server leader information, its esx module is then called in the request of discovery virtual machine creating;
Step2: judge whether the physical server disk space is enough,, otherwise go to Step8 if enough then continue Step3;
Step3: check and add virtual machine title legitimacy, then continue Step4 as if legal, otherwise go to Step8;
Step4: duplicate virtual machine image;
Step5: start virtual machine;
Step6: registration virtual machine IP;
Step7: add virtual machine Control Node management information;
Step8: finish the virtual machine creating operation.
Finally finish the startup of N+1 platform virtual server, for user's program request is served.
The reclaim mechanism of virtual stream media server
Judged whether that according to current virtual stream media server network bandwidth occupancy and virutal machine memory occupancy the virtual stream media server is in unloaded idle state, when current virtual stream media server does not have user capture, triggered the reclaimer operation of virtual stream media server.
1) virtual machine server management node initialization
The virtual machine server management node is collected the state information of virtual machine streaming media server, thereby realizes the real-time status of virtual machine streaming media server is monitored.The initialization of virtual machine server management node is responsible for safeguarding the streaming media server network bandwidth that all and virtual stream media apparatus interaction are collected and the tables of data of internal storage state information.
2) the virtual machine server management node obtains the reservation internal memory θ that the virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i
Virtual server management node and virtual stream media server are undertaken alternately by the running state parameter information of XML form, collect the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i, having opened virtual stream media server number is N.
3) judge whether current virtual stream media server is in unloaded idle state and need be recovered
Unloaded idle state promptly exists the virtual stream media server to satisfy unloaded idle condition β
i/ μ
i=0% is that network bandwidth occupation rate is 0%.When carrying out virtual machine recovery decision-making, need the special circumstances under the consideration front end request distribution mechanisms.Supposing the system finds have a virtual stream media server M not have user capture in the Virtual Server Cluster according to state information this moment, and promptly M is in unloaded idle state, then has two kinds of possible situations to distinguish.
First kind of situation is that M is the virtual machine that leaves unused, and also will leave unused in following a period of time always.Virtual machine under this state should be recovered, the system resource of release busy; Second kind of situation is that M is the new virtual stream media server that is created recently, just do not have user's program request this moment, satisfies unloaded idle condition β
i/ μ
i=0%, but follow-up request will be positioned on this virtual server very soon.The virtual machine of this moment should continue to keep, and handles the follow-up user who is about to arrive and asks.
Judge the virtual stream media server M that is in unloaded idle state is in above-mentioned which kind of situation:
The bandwidth usage of current virtual stream media server cluster
The memory usage of current virtual stream media server cluster
Satisfy above-mentioned two conditions, then M satisfies second kind of situation, need be recovered.Otherwise satisfy first kind of situation, be the new virtual machine server of creating, continue to detect each state parameter, repeat this step until satisfying above-mentioned condition.
4) close the virtual stream media server, reclaim virtual machine
The virtual server management node is managed the internal memory α of this virtual stream media server to be recycled
L, network bandwidth β
LThe physical resource that distributes before the resource etc., current unlatching virtual stream media server number is N; To reclaim resource and add in the middle of the allowable resource, close this virtual stream media server, and finish and reclaim this virtual stream media server.
With reference to shown in Figure 3, described virtual machine reclaim mechanism.To intended target virtual machine and required running state parameter, generate the system running state parameter collection request of XML form.
1. the object of initialization HttpURLConnection class is set up HTTP with the virtual machine server management node and is linked and use setRequestProperty () method to carry out corresponding configuration to linking.
2. call getOutputStream () method and form output stream, and the object of initialization OutputStream class, the request of Write () method transmit status parameter collection called.
3. call the return results that getInputStream () obtains server, with its with and the object of XML form initialization Document class.
4. obtain the root element of this XML content by calling getRootElement () method.Parse required running state parameter by CalculatePara () method.
Collect the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
i, current unlatching virtual stream media server number is N; Then detect and obtain its network bandwidth α that takies
iAnd the β of committed memory
iWhen satisfying following condition, reclaim the virtual stream media server:
1) network bandwidth occupancy of current virtual stream media server cluster
2) memory usage of current virtual stream media server cluster
3) exist virtual stream media server M to satisfy unloaded idle condition β
i/ μ
i=0%,
Otherwise continue to detect each state parameter, the virtual server management node is managed the internal memory θ of this virtual stream media server to be recycled after satisfying above-mentioned condition
i, network bandwidth μ
iThe physical resource that distributes before the resource etc., current unlatching virtual stream media server number is N;
The virtual machine server management node uses Virtual Machine Manager assembly VCL to manage a plurality of virtual machine operations tasks, VCL is by vcld assembly poll inquiry virtual machine server leader information, if find the virtual machine creating request then call its esx module and carry out corresponding operating.As shown in Figure 5, this operation implementation procedure is as follows:
Step1:VCL assembly poll inquiry virtual machine server leader information, discovery virtual machine removal request is then called its esx module;
Step2: nullify virtual machine IP;
Step3: stop virtual machine service;
Step4: nullify the virtual machine server leader information;
Step5: finish the virtual machine deletion action.
Reclaim resource the most at last and add in the middle of the allowable resource, close the virtual stream media server and finish the recovery of this virtual stream media server M.
The balanced prefix deployment strategy of files in stream media
This strategy is used for balanced deployment of prefix of carrying out the focus file on the virtual stream media server, mainly comprises and obtaining focus file, generation deployment umber, three steps of balanced deployment strategy, and is specific as follows:
With reference to shown in Figure 6, the described step of obtaining Streaming Media focus file is:
In the Step1:Hadoop storage cluster acquisition time T,, obtain files in stream media access times N according to Hadoop cluster Visitor Logs;
If Hadoop storage cluster file access record format is:＜file mark θ
i, the file access time T
i; File access times N like this is: satisfy T
iIn time interval T, the bar number of file access record;
Step2:, obtain files in stream media θ according to Hadoop cluster Visitor Logs
iAccess times N
iFor files in stream media θ
iAccess times N
iFor: satisfy T
iIn time interval T, file mark is θ
iThe bar number of file access record;
Step3: for each files in stream media θ
i, calculate visit ratio N
iIf/N is N
i/ N jumps to Step4 greater than 5%; Otherwise check whether also have files in stream media not calculate,, to repeat step3, otherwise jump to Step5 if having;
Step4: with this files in stream media θ
iAdd focus files in stream media data sequence;
Step5: the focus files in stream media sequence of generation is exactly the focus file sequence in time T.
With reference to shown in Figure 7, the deployment umber of each focus file of described generation, concrete steps are as follows:
Step1: virtual stream media server number M from the Virtual Service of service to the virtual machine server management node that inquire, obtain; The current virtual machine of virtual machine server management node inquiry VCL is opened tabulation, the virtual stream media server number that statistics is opened;
Step2: the relative temperature of calculating each focus files in stream media
Be designated as ρ
i
Step3: the deployment umber M * ρ that calculates each focus files in stream media
i, and condition below satisfying:
If M * ρ
iLess than 1, n then
iBe designated as 1;
If M * ρ
iGreater than 1, n then
iBe designated as M * ρ
iTake off integer.
Step4: generated the focus files in stream media and disposed number sequence n
1, n
2..., n
n
With reference to shown in Figure 8, described is the balanced step of disposing of file, specific as follows:
Step1: if files in stream media θ
iCorresponding deployment umber is n
i, then in deployment file, increase file n
i-1 part; Each files in stream media that need dispose is carried out such operation, and final deployment file set becomes number and is
The file deployment set
Step2: calculate virtual stream media server M
iNeed the file number h of deployment, obtaining current unlatching virtual stream media server number from the virtual machine server management node is M, and then h is
And the h minimum value is 1;
Step3: to virtual stream media server M
iCarry out following operation: from the file deployment set
A middle picked at random h files in stream media is disposed, and leaves out this h files in stream media from set;
Step4: each virtual stream media server is repeated the operation of Step3;
Step5:, select a virtual stream media server to dispose at random to remaining files in stream media in the last file deployment set.
Newly open virtual stream media server prefix deployment strategy
The high capacity of virtual stream media services cluster triggers virtual stream media server set-up mechanism; According to the real time access record of files in stream media, real-time focus files in stream media is deployed on the virtual stream media server of new establishment.
With reference to shown in Figure 9, described is the step of newly opening virtual stream media server prefix deployment strategy, specific as follows:
Step1:Hadoop storage cluster acquisition time T obtains files in stream media access times N constantly; If Hadoop storage cluster file access record format is:＜file mark θ
i, the file access time T
i; In time T constantly, file access total degree N is: satisfy T like this
iEqual T, the total number of file access record;
Step2: in time T constantly, obtain files in stream media θ from Hadoop storage cluster
iAccess times N
iIn time T constantly, for files in stream media θ
iAccess times N
iFor: satisfy T
iEqual T, file mark is θ
iThe bar number of file access record;
Step3: for each files in stream media θ
i, calculate real time access ratio N
iIf/N is N
i/ N jumps to Step4 greater than 5%; Otherwise check whether also have files in stream media not calculate,, to repeat step3, otherwise jump to Step5 if having;
Step4: with this files in stream media θ
jAdd real-time focus files in stream media data acquisition system { θ
j, summation is designated as k;
Step5: obtain current unlatching virtual machine number M from the virtual machine server management node, from real-time focus files in stream media data acquisition system, select at random
Individual being deployed to newly opened the virtual stream media server.
The interval cache policy
Utilize the correlation and the sequential of same the front and back request on the virtual stream media server, use the interval buffer memory to serve as the user.
With reference to shown in Figure 10, described is the interval cache policy of realizing on the virtual stream media server, does not consider the prefix part of files in stream media below, and the file content of the request of consideration is the prefix content afterwards of files in stream media:
Step1: on virtual stream media server M, for files in stream media θ, as request S
1When t arrived constantly, establishing it was first order request of θ, and virtual stream media server M serves to Hadoop storage cluster request msg;
Step2: as request S
2At t+t
1Arrive constantly, owing to there is S
1, just formed a coupling (S at interval
1, S
2), S
1At t+t
1Constantly beginning cache-time length is t
1Data, and the request S
2Read t from Hadoop storage cluster earlier
1After the data of time span, again from S
1Buffer memory in reading of data;
Step3: as request S
3Time t+t
1+ t
2Arrive S constantly
3Be t from Hadoop storage cluster time for reading length earlier
2The data of time span are again from S
2Buffer memory in reading of data.
(t-Δ t, t) in time, the time of each files in stream media is set to 1/10th of files in stream media playout length in the virtual stream media server interval cache responses time.Promptly when carrying out the interval coupling, maximum interval length match time is 1/10th of files in stream media playout length; When twice adjacent request time surpass the files in stream media playout length 1/10th the time, do not mate, promptly the Virtual Machine Manager node was thought do not have response to file in the Δ t time recently.
Dynamic cache policy at interval
With reference to shown in Figure 11, described is the dynamic interval cache policy of realizing on the virtual stream media server:
Step1: the virtual machine server management node is to Hadoop storage cluster request current service bandwidth occupancy rate φ, and Hadoop storage cluster accepts request, and returns current Hadoop storage cluster bandwidth occupancy rate φ.
Step2: the virtual machine server management node receives that Hadoop storage cluster returns bandwidth occupancy rate φ, if φ greater than 90%, step3 is jumped in the high capacity of Hadoop storage cluster service bandwidth; If φ is less than 10%, Hadoop storage cluster service bandwidth underload jumps to step4;
Step3: the virtual machine server management node sends to each virtual stream media server increases the message of cache size at interval; Each virtual stream media server is received increases buffered message at interval, checks current internal memory available quantity E, and buffer memory increases E/2 at interval;
Step4: the virtual machine server management node sends the message of initialization interval cache size to each virtual stream media server; Each virtual stream media server is received the initialization interval buffered message, and buffer memory is set to initial value I at interval.
Claims (7)
1. the stream media ordering method based on Hadoop and virtual stream media server cluster is characterized in that: comprise the steps:
(1) files in stream media is left on the Hadoop storage cluster, (2) streaming media server is structured on the Virtual Server Cluster, (3) virtual stream media server cluster directly obtains the files in stream media data from Hadoop storage cluster, and employing virtual stream media server colony dispatching and data cache method, realize the streaming media on demand function, wherein, the described virtual stream media server of step (3) colony dispatching method is, the virtual machine server management node is used for controlling the unlatching of virtual stream media server and closing, the virtual machine server management node with virtual stream media server cluster network bandwidth usage surpass 80% or memory usage to surpass 90% be foundation, trigger virtual stream media server set-up mechanism; With virtual stream media server cluster overall bandwidth occupancy less than 20% and memory usage less than 30% and to have user's request amount be that 0 virtual stream media server is a foundation, trigger virtual stream media server reclaim mechanism; The described virtual stream media server of step (3) cluster data cache method is, adopted the balanced deployment strategy of metadata cache of virtual stream media server cluster, carries out the prefix of file for the focus files in stream media and disposes; According to the real-time focus degree of files in stream media, real-time focus Documents Department is deployed on the virtual stream media server of new unlatching; And when carrying out the streaming media on demand service, consider the time order and function order and the relevance of request same stream media file, to the The data corresponding intervals buffer memory that obtains from Hadoop storage cluster; Read bandwidth according to virtual stream media server cluster to the file of Hadoop storage cluster, dynamically control the cache size on each virtual stream media server, trigger virtual machine scheduling policy.
2. according to right 1 described stream media ordering method based on Hadoop and virtual stream media server cluster, it is characterized in that: the concrete steps of described triggering virtual stream media server set-up mechanism are:
Step1: virtual machine server management node initialization;
Step2: the virtual machine server management node obtains the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
iObtain current unlatching virtual stream media server number N;
Step3: virtual machine server management node real-time update detects the network bandwidth α that each virtual stream media server takies
iAnd the β of committed memory
iIf virtual stream media server cluster network bandwidth usage
Surpass 80% or memory usage
Surpass 90%, then jump to Step4, otherwise continue to detect;
Step4: the virtual stream media server is reserved internal memory θ for the virtual stream media server that will open is provided with from assignable resource
N+1, network bandwidth μ
N+1, open the virtual stream media server and serve for user's program request.
3. according to right 1 described stream media ordering method, it is characterized in that the concrete steps of described triggering virtual stream media server reclaim mechanism are based on Hadoop and virtual stream media server cluster:
Step1: virtual machine server management node initialization;
Step2: the virtual machine server management node obtains the reservation internal memory θ that each virtual stream media server is provided with
iAnd the network bandwidth μ that reserves
iObtain current unlatching virtual stream media server number N;
Step3: the virtual machine server management node detects current each virtual stream media server and takies network bandwidth α
iAnd the β of committed memory
iIf the bandwidth usage of current virtual stream media server cluster
Less than 20% and memory usage
Less than 30% and exist the virtual stream media server to satisfy network bandwidth occupation rate β simultaneously
i/ μ
iBe 0%, then jump to Step4, otherwise continue to detect;
Step4: the virtual server management node, manage the internal memory α of this virtual stream media server to be recycled
L, network bandwidth β
LResource adds resource in the middle of the allowable resource, closes this virtual stream media server, reclaims virtual machine.
4. the stream media ordering method based on Hadoop and virtual stream media server cluster according to claim 1, it is characterized in that: the balanced deployment strategy of the metadata cache of described virtual stream media server cluster is deployed in each file prefix on the virtual stream media server cluster for the mode of disposing with prefix, and concrete steps are as follows:
Step1:, obtain the focus files in stream media θ of server in time T according to the record of Hadoop storage cluster convection current media file access
i, focus files in stream media θ
iDefinition be in time T, its access times N
iGreater than 5% of the total access times N of Hadoop storage cluster file;
Step2: according to focus files in stream media θ
iThe times N of visit
i, can obtain θ
iRelative temperature
Wherein, n is the number of whole focus files, N
jBe focus files in stream media θ wherein
jAccess times;
Step3: the virtual stream media server number of establishing current startup is M, according to focus files in stream media θ
iRelative temperature
Obtain θ
iNeed carry out the number ρ of the virtual server of prefix deployment
iFor
Step4: according to each focus files in stream media θ
iPrefix is disposed number ρ
i, from M virtual stream media server, choose ρ
iIndividual is file θ
iCarrying out the prefix of load balancing disposes.
5. the stream media ordering method based on Hadoop and virtual stream media server cluster according to claim 1 is characterized in that: described concrete steps at the new real-time focus file of opening of virtual stream media server deploy are:
Step1: obtain the visit situation of current Hadoop storage cluster convection current media file, establish file θ this moment
iThere is N
iIndividual request visit, the total request of Hadoop storage cluster response number is N;
Step2: for R
iSatisfy R
iThe file θ of 5%M
i, it is added real-time focus file set, the focus file set is { θ during document
i, θ
j..., θ
x, θ
y, and the focus file is k altogether;
Step3: establishing the virtual stream media server number that virtual stream media services this moment cluster opened is M, from focus file set { θ
i, θ
j..., θ
x, θ
yIn select at random
Individual, these focus files are carried out prefix dispose on the virtual stream media server of newly opening.
6. according to right 1 described stream media ordering method, it is characterized in that described interval buffer memory adopts following steps to realize based on Hadoop and virtual stream media server cluster:
Step1: when the t moment, the user asks q to arrive the virtual machine server management node, checks whether the files in stream media θ of current request is present in the middle of the prefix deployment of virtual stream media server cluster, if exist, jumps to Step2; Otherwise jump to Step3;
Step2: for the virtual stream media server set { S that exists files in stream media θ prefix to dispose
1, S
2..., S
n, the response request q of memory usage minimum serves the user in the selection virtual stream media server; When this moment this virtual stream media server had request q' to Hadoop storage cluster files in stream media θ, q' begins the data that buffer memory reads, and the data after the Hadoop storage cluster demand file θ prefix, the data of getting off up to the q' buffer memory can be request q service; If there is not the request to Hadoop storage cluster convection current media file θ in current virtual stream media server, then uses the prefix buffer memory to serve, and from Hadoop storage cluster, obtain the further part of θ prefix file;
Step3: check (whether t-Δ t t) exists identical file request to the virtual stream media server in time, if there is identical file request, jump to Step4 in the time to file θ; Otherwise jump to Step5;
Step4: the virtual stream media server m that will ask q to distribute to exist internal memory occupation rate minimum in the virtual stream media server of file θ request, m begins the data of cache file θ, this virtual stream media server m begins the file θ of request q request is served, the virtual stream media server uses the prefix buffer memory to serve, and beginning is to the further part of Hadoop storage cluster demand file θ prefix file, up to asking to begin data in buffer, utilize the interval data in buffer to serve at time t;
Step5: if there is no identical file request, then distribute the virtual stream media server of a memory usage minimum, begin to respond the service of this time asking q;
Step6: in the time of new request, be repeated to Step1, carry out Cyclic Service.
7. according to right 1 described stream media ordering method, it is characterized in that the cache size concrete steps on each virtual stream media server of described dynamic control are as follows based on Hadoop and virtual stream media server cluster:
Step1: the virtual machine server management node is obtained current Hadoop storage cluster service bandwidth occupation rate φ, if φ greater than 90%, jumps to Step2; If φ less than 10%, jumps to Step3;
Step2: the virtual machine server management node sends to each virtual stream media server increases the message of cache size at interval; Each virtual stream media server is received increases buffered message at interval, increases the interval buffer memory of each virtual stream media server, and size is half of each virtual stream media server free memory;
Step3: the virtual machine server management node sends the message of initialization interval cache size to each virtual stream media server; Each virtual stream media server is received the initialization interval buffered message, and interval cache size separately is set to initial value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110312612.0A CN102333126B (en) | 2011-10-15 | 2011-10-15 | Streaming media on demand method based on Hadoop and virtual streaming media server cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110312612.0A CN102333126B (en) | 2011-10-15 | 2011-10-15 | Streaming media on demand method based on Hadoop and virtual streaming media server cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102333126A CN102333126A (en) | 2012-01-25 |
| CN102333126B true CN102333126B (en) | 2013-07-31 |
Family
ID=45484727
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110312612.0A Expired - Fee Related CN102333126B (en) | 2011-10-15 | 2011-10-15 | Streaming media on demand method based on Hadoop and virtual streaming media server cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102333126B (en) |
Families Citing this family (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103164283B (en) * | 2012-05-10 | 2018-08-10 | 上海兆民云计算科技有限公司 | Virtualization resource dynamic dispatching management method and system in a kind of virtual desktop system |
| CN103885812B (en) * | 2012-12-21 | 2018-03-27 | 华为技术有限公司 | Virtual machine specification method of adjustment and device |
| CN103297431B (en) * | 2013-05-24 | 2016-07-13 | 南京邮电大学 | A kind of streaming media video video-on-demand duplicate hybrid buffer method based on Cloud Server group |
| CN103442034B (en) * | 2013-08-07 | 2016-08-10 | 中南民族大学 | A kind of stream media service method based on cloud computing technology and system |
| CN103678521B (en) * | 2013-11-30 | 2016-08-17 | 电子科技大学 | A kind of distributed document monitoring system based on Hadoop framework |
| CN103685492B (en) * | 2013-12-03 | 2017-01-25 | 北京智谷睿拓技术服务有限公司 | Dispatching method, dispatching device and application of Hadoop trunking system |
| CN111488975A (en) | 2014-03-07 | 2020-08-04 | 卡皮塔罗技斯Ip所有者有限责任公司 | System and method for allocating capital to trading strategies for big data trading in financial markets |
| CN104065738A (en) * | 2014-07-04 | 2014-09-24 | 云南电网公司 | Business system load balance method in intelligent automatic control |
| CN106161068B (en) * | 2015-04-15 | 2020-10-16 | 华为技术有限公司 | Recovery prompting and distributing method for network resources and controller |
| CN104853221A (en) * | 2015-05-22 | 2015-08-19 | 中山大学 | Multi-source stream video on demand system and multi-source stream video on demand method based on virtual server matrix |
| US10423800B2 (en) | 2016-07-01 | 2019-09-24 | Capitalogix Ip Owner, Llc | Secure intelligent networked architecture, processing and execution |
| CN107918617B (en) * | 2016-10-10 | 2021-11-30 | 北京京东尚科信息技术有限公司 | Data query method and device |
| CN106713021B (en) * | 2016-12-09 | 2020-02-11 | 北京奇虎科技有限公司 | Method and device for judging whether server in cluster needs to be recycled |
| US10387679B2 (en) | 2017-01-06 | 2019-08-20 | Capitalogix Ip Owner, Llc | Secure intelligent networked architecture with dynamic feedback |
| CN107395456B (en) * | 2017-07-18 | 2021-06-29 | 郑州云海信息技术有限公司 | Distributed file system direct current storage test method and platform |
| CN110399305B (en) * | 2019-07-31 | 2023-12-08 | 中国工商银行股份有限公司 | BTT module testing method and device |
| CN112486635A (en) * | 2020-12-09 | 2021-03-12 | 成都辰迈科技有限公司 | Cloud computing teaching method and system, computer equipment and storage medium |
| CN112584193B (en) * | 2020-12-24 | 2023-05-12 | 杭州米络星科技(集团)有限公司 | Method for constructing real-time streaming media cluster scheduling by utilizing UDP (user datagram protocol) characteristics |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101242338A (en) * | 2008-03-10 | 2008-08-13 | 清华大学 | Self-adapted adjusting method for P2P real time stream media buffer replacement time weight parameter |
| CN101697554A (en) * | 2009-09-27 | 2010-04-21 | 华中科技大学 | Method for scheduling P2P streaming media video data transmission |
-
2011
- 2011-10-15 CN CN201110312612.0A patent/CN102333126B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101242338A (en) * | 2008-03-10 | 2008-08-13 | 清华大学 | Self-adapted adjusting method for P2P real time stream media buffer replacement time weight parameter |
| CN101697554A (en) * | 2009-09-27 | 2010-04-21 | 华中科技大学 | Method for scheduling P2P streaming media video data transmission |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102333126A (en) | 2012-01-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102333126B (en) | Streaming media on demand method based on Hadoop and virtual streaming media server cluster | |
| JP5181031B2 (en) | Resilient service quality within a managed multimedia distribution network | |
| EP3382963B1 (en) | Method and system for self-adaptive bandwidth control for cdn platform | |
| CN102740159B (en) | Media file storage format and self-adaptation transfer system | |
| CN102439578B (en) | Dynamic variable rate media delivery system | |
| KR101490122B1 (en) | Method for distributing content data packages originated by users of a super peer-to-peer network | |
| CN100463516C (en) | An interactive network TV system | |
| CN101262490B (en) | Monitoring system | |
| CN107241384A (en) | A kind of content distribution service priority scheduling of resource method based on many cloud frameworks | |
| US9197687B2 (en) | Prioritized blocking of on-demand requests | |
| US20110078116A1 (en) | Method for controlling the distribution of data blocks and apparatus for the same | |
| CN101237429A (en) | Stream media living broadcasting system, method and device based on content distribution network | |
| CN107171839A (en) | A kind of bandwidth traffic cost control method | |
| CN101426024A (en) | Data flow controlling method, system and apparatus | |
| CN102164317A (en) | Internet protocol (IP) set-top box-oriented multi-server segmented downloading system | |
| CN101980505A (en) | 3Tnet-based video-on-demand load balancing method | |
| Wang et al. | PLVER: Joint stable allocation and content replication for edge-assisted live video delivery | |
| CN103685344A (en) | Synergetic method and system for multiple P2P (point-to-point) cache peers | |
| CN100576905C (en) | A kind of VOD frequency treating method and device thereof | |
| CN101695044A (en) | Stream media service node and load balancing method thereof | |
| Chen et al. | Zebroid: using IPTV data to support STB-assisted VoD content delivery | |
| CN112311826A (en) | Method, device and system for processing access request in content distribution system | |
| CN102017568A (en) | System for delivery of content to be played autonomously | |
| CN101562626B (en) | Method, system and device for medium distribution | |
| Pussep | Peer-assisted video-on-demand: cost reduction and performance enhancement for users, overlay providers, and network operators |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130731 Termination date: 20211015 |