CN105898384B - A kind of method and apparatus of streaming media video mixing frame per second control - Google Patents

A kind of method and apparatus of streaming media video mixing frame per second control Download PDF

Info

Publication number
CN105898384B
CN105898384B CN201610270343.9A CN201610270343A CN105898384B CN 105898384 B CN105898384 B CN 105898384B CN 201610270343 A CN201610270343 A CN 201610270343A CN 105898384 B CN105898384 B CN 105898384B
Authority
CN
China
Prior art keywords
streaming media
media video
video
weight
frame
Prior art date
Application number
CN201610270343.9A
Other languages
Chinese (zh)
Other versions
CN105898384A (en
Inventor
叶荣华
刘志聪
孙石平
张冲
Original Assignee
广州盈可视电子科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州盈可视电子科技有限公司 filed Critical 广州盈可视电子科技有限公司
Priority to CN201610270343.9A priority Critical patent/CN105898384B/en
Publication of CN105898384A publication Critical patent/CN105898384A/en
Application granted granted Critical
Publication of CN105898384B publication Critical patent/CN105898384B/en

Links

Abstract

The embodiment of the invention provides a kind of method and devices of streaming media video mixing frame per second control, one or more streaming media video is loaded into respectively in corresponding one or more buffer areas, the streaming media video includes video frame, and the method comprises determining that the corresponding weight of one or more described streaming media video;Respectively from one or more of buffer areas, one or more corresponding streaming media video is extracted frame by frame and is mixed;When there are buffer empty, the weights sum that statistics is stored with all streaming media videos in the buffer area of streaming media video is the first total weight value;Judge whether first total weight value is less than preset threshold value;If so, carrying out dilatation for the buffer area for being stored with streaming media video;If it is not, then carrying out interleave in the buffer empty.The present invention makes user experience compared to any method is used alone more preferably to find equalization point in fluency and real-time both demands.

Description

A kind of method and apparatus of streaming media video mixing frame per second control

Technical field

The present invention relates to technical field of video processing, more particularly to a kind of streaming media video mixing frame rate control method and A kind of streaming media video mixing frame per second control device.

Background technique

As media marketplace is more and more closely merged with network service, stream media broadcast and use are increasingly extensive, wherein In the MCU (Multipoint Control Unit, multipoint control unit) and safety defense monitoring system of video conferencing system In the systems such as divider, streaming media video hybrid technology, which starts to substitute original analog video mixer, to be widely applied.

By taking the MCU of video conferencing system as an example, the vision mixer (video mixer) in MCU obtains several terminals Streaming media video be mixed into the same video, and with fixed frame per second by the mixed video output carry out it is subsequent coding, The work such as transmission have saved bandwidth and terminal capabilities by the second-compressed mode of decoding, mixing, coding in practice.Existed in the past Divider in safety defense monitoring system mainly acts on the processing of analog signal, and with the networking of safety defense monitoring system, Streaming media video also as video conferencing system, is carried out second-compressed mode by tupe.

Vision mixer in video conferencing system and safety defense monitoring system based on network data source also referred to as flows matchmaker Volumetric video mixer.In streaming media video mixer, due to asymmetric (the Streaming Media view of each input of network jitter, frame per second Frequency with mix export video frame per second it is inconsistent) etc. reasons, streaming media video mixer must take frame per second control measure so that Mixer exports the mixed video of stable frame per second for subsequent coding.

There are mainly two types of frame rate control methods, a kind of method be it is preferential towards fluency, this method passes through setting one Biggish buffer area guarantees the fluency of video, but will lead to biggish delay.Another method be towards real-time most Excellent, this method is to reduce buffer area as far as possible, improves real-time, but the fluency that will lead to video is poor.

In streaming media video mixer, fluency and real-time be it is mutual restrict and conflicting, therefore, need at present A technical problem for wanting those skilled in the art urgently to solve is exactly: proposing a kind of streaming media video mixing frame per second control machine System finds equalization point in fluency and real-time both demands, and makes user experience any compared to being used alone Method is more preferably.

Summary of the invention

The embodiment of the present invention is used the technical problem to be solved is that a kind of streaming media video mixing frame rate control method is provided To find equalization point in fluency and real-time both demands, and make user experience compared to any one side is used alone Method is more preferably.

Correspondingly, the embodiment of the invention also provides a kind of streaming media videos to mix frame per second control device, on guaranteeing State the realization and application of method.

To solve the above-mentioned problems, the invention discloses a kind of streaming media video mixing frame per second control method, all the way or Multichannel streaming media video is loaded into respectively in corresponding one or more buffer areas, and the streaming media video includes video frame, institute The method stated includes:

Determine the corresponding weight of one or more described streaming media video;

Respectively from one or more of buffer areas, one or more corresponding streaming media video is extracted frame by frame and is mixed It closes;

When there are buffer empty, statistics be stored with all streaming media videos in the buffer area of streaming media video weight it With for the first total weight value;

Judge whether first total weight value is less than preset threshold value;If so, being stored with streaming media video for described Buffer area carries out dilatation;If it is not, then carrying out interleave in the buffer empty.

Preferably, the step of corresponding weight of one or more streaming media video described in the determination includes:

Read the layout parameter in preset configuration file;The layout parameter includes one or more described streaming media video Pixel ratio shared in preset total pixel;

Using the ratio as the weight of one or more streaming media video.

Preferably, the step of corresponding weight of one or more streaming media video described in the determination includes:

Receive additional parameter;It include the new weight of one or more described streaming media video in the additional parameter;

Using the new weight as the weight of one or more streaming media video.

Preferably, after the determination one or more streaming media video corresponding weight the step of, further includes:

The weights sum for calculating one or more streaming media video obtains the second total weight value;

Judge whether second total weight value is not equal to default value;

If so, calculating the weight of one or more streaming media video, and the ratio of default value and the second total weight value The product of value, the weight as one or more streaming media video.

Preferably, described the step of interleave is carried out in buffer empty, includes:

Replicate the previous mixed video frame of streaming media video corresponding to the buffer empty;

The video frame is inserted into the buffer empty.

Preferably, the preset threshold value is 0.7.

The embodiment of the invention also discloses a kind of device of streaming media video mixing frame per second control, one or more Streaming Medias Video is loaded into respectively in corresponding one or more buffer areas, and the streaming media video includes video frame, the device packet It includes:

Determining module, for determining the corresponding weight of one or more described streaming media video;

Mixing module, for from one or more of buffer areas, extracting one or more corresponding stream frame by frame respectively Media video is mixed;

Statistical module, for when there are buffer empty, statistics to be stored with all stream matchmakers in the buffer area of streaming media video The weights sum of volumetric video is the first total weight value;

Judgment module, for judging whether first total weight value is less than preset threshold value;If so, dilatation module is called, If it is not, then calling interleave module;

Dilatation module, for carrying out dilatation for the buffer area for being stored with streaming media video;

Interleave module, for carrying out interleave in the buffer empty.

Preferably, the determining module includes:

Reading submodule, for reading the layout parameter in preset configuration file;The layout parameter include it is described all the way Or the ratio that the pixel of multichannel streaming media video is shared in preset total pixel;

The first adjustment submodule, for using the ratio as the weight of one or more streaming media video.

Preferably, the determining module includes:

Receiving submodule, for receiving additional parameter;It include one or more Streaming Media view in the additional parameter Frequently new weight;

Second adjustment submodule, for using the new weight as the weight of one or more streaming media video.

Preferably, described device further include:

Computing module, the weights sum for calculating one or more streaming media video obtain the second total weight value;

Judgment module, for judging whether second total weight value is not equal to default value;If so, calling third adjustment Module;

Third adjusts module, for calculating the weight of one or more streaming media video, with default value and second The product of the ratio of total weight value, the weight as one or more streaming media video.

Compared with prior art, the embodiment of the present invention includes following advantages:

In embodiments of the present invention, weight is calculated according to conditions such as video content, layout parameter, user demands, works as network Shake leads to buffer area when being empty, the calculated result to weight as frame-skipping or is increased the foundation buffered, thus dynamically controls Buffer size processed, this adaptive process will be so that buffer area be finally set to a reasonable size.Due to being not required to Guarantee the smoothness of all videos, therefore can be less in required buffer area, ensure that video in the case of delay time is shorter Fluency and real-time, improve user experience effect.

Detailed description of the invention

Fig. 1 is a kind of groundwork process of streaming media video mixer;

Fig. 2 is the schematic diagram of video mix layout a a kind of;

Fig. 3 is the schematic diagram of video mix layout b a kind of;

Fig. 4 is a kind of step flow chart of streaming media video mixing frame rate control method embodiment 1 of the invention;

Fig. 5 is a kind of step flow chart of streaming media video mixing frame rate control method embodiment 2 of the invention;

Fig. 6 is the schematic diagram of video mix layout c a kind of;

Fig. 7 is a kind of schematic diagram of queue theory model of the invention;

Fig. 8 is a kind of schematic diagram of buffer area of the invention frame rate control method probability of demand corresponding with meeting;

Fig. 9 is a kind of total schematic diagram of buffer area of the invention frame rate control method probability of demand corresponding with meeting.

Figure 10 is a kind of structural block diagram of streaming media video mixing frame per second control device embodiment of the invention.

Specific embodiment

In order to make the foregoing objectives, features and advantages of the present invention clearer and more comprehensible, with reference to the accompanying drawing and specific real Applying mode, the present invention is described in further detail.

By taking the MCU of video conferencing system as an example, referring to Fig.1 shown in a kind of groundwork stream of streaming media video mixer Journey, detailed process are as follows:

In video conferencing system, the streaming media video of each terminal reaches multipoint control unit from different network links MCU controls (Frame rate control in frame per second by buffer area)Under effect, streaming media video is according to the regular hour Interval is decoded respectively using decoder, and decoded image brush is entered streaming media video mixer and carries out video mix generation New image is then encoded, encapsulates the new video flowing of composition, is distributed to each terminal.

In the concrete realization, when the input frame rate of streaming media video mixer and output frame rate are unequal, or due to net Network jitter caused by noise present in network or exchanges device capabilities problem and cause input reach it is unstable when, need Take frame per second control.For example, when input frame rate be 20 (frames/second), output frame rate be 30 (frame/second) when, theoretically, need every 2 frames are inserted into (duplication) 1 frame data and are mixed, it is contemplated that the arrival of streaming media video caused by network jitter is unstable, Then interleave may be carried out at random according to the case where streaming media video in buffer area, it could even be possible to can be because of excessive network Shake causes streaming media video to reach serious slow and frame-skipping (discarding) is taken to operate, to guarantee the same of each road streaming media video Step.

According to the frame rate control method that fluency is optimal, then when requiring network jitter, any frame is not skipped as far as possible, with It ensures the fluency of video, therefore this method needs biggish buffer area, causes delay longer.According to the frame that real-time is optimal Rate control method then requires to reduce buffer area as far as possible, real-time is improved, since lesser buffer area can not cope with network jitter Bring interference, therefore this method will lead to a large amount of frame-skippings, so that video fluency is bad.

In the frame rate control method generallyd use, each road streaming media video is treated substantially equally, and system needs to guarantee all The fluency of video causes system to need a buffer area more much larger than handling single video.On the contrary, if setting is one smaller Buffer area, due to fair play, the chance of each road streaming media video interleave as caused by network jitter or frame losing It is also identical.

In streaming media video mixer when carrying out video mix, streaming media video can be scaled, be cut, and according to Certain layout is arranged, and in the concrete realization, being bound in the video of final output, there are the views that a user most pays close attention to Frequency domain, that is, input video is that user most pays close attention to all the way for certain of Streaming Media mixer.Referring to a kind of video shown in Fig. 2 The schematic diagram of the schematic diagram of mixed layout a and a kind of video mix layout b shown in Fig. 3, it is mixed in the two videos of Fig. 2 and Fig. 3 It closes in schematic layout pattern, the video marked as 1 is the content that user most pays close attention to, remaining 234 are taken second place.If only needed preferential Guarantee the fluency and real-time of this video all the way of label 1, system then can be in the case where guaranteeing user experience, as far as possible Reduce and postpones and ensure fluency.

Exactly inventor herein based on the above situation, creatively proposes that one of the core concepts of the embodiments of the present invention exists In the preferential frame per second controlling mechanism of the attentinal contents of proposition is weighted each road streaming media video, and by weights sum conduct Thus frame-skipping or the foundation for increasing buffering dynamically control buffer size, this adaptive process will so that buffer area most It is set to a reasonable size eventually.It is needed compared to " fluency is optimal " due to being not required for guaranteeing that all videos are smooth The buffer area wanted is smaller.Compared to " real-time is optimal ", the fluency of main body ensure that.Reach the target of optimal user experience, Optimal equalization point is had found between real-time and fluency.

Referring to Fig. 4, the step of a kind of streaming media video of the invention mixes embodiment of the method 1 of frame per second control stream is shown Cheng Tu, one or more streaming media video can be loaded into respectively in corresponding one or more buffer areas, the Streaming Media view Frequency may include video frame, and the method can specifically include following steps:

Step 101, the corresponding weight of one or more described streaming media video is determined;

In the concrete realization, by RTSP, (real time streaming protocol, spreads streaming media video in real time Defeated agreement) etc. agreements access system, by decapsulation after obtain compressed video data.Wherein, the lattice of the compressed video data Formula can be h264, h263, wmv, mpeg2, vp8, jpeg etc..

When streaming media video mixer carries out video mix, by video scaling, cut and according to certain layout It is arranged, and can have the video area most paid close attention to of a user in the video of final output, that is, mixer is a certain Road input video is that user most pays close attention to.In order to guarantee user experience effect, the road Gu Duige streaming media video can be according to user Different weights is arranged in degree of concern.

Step 102, respectively from one or more of buffer areas, corresponding one or more Streaming Media view is extracted frame by frame Frequency is mixed;

In the concrete realization, Streaming Media mixer obtains each road streaming media video from buffer area as unit of frame, according to finger Fixed layout is zoomed in and out, is arranged, and is mixed into video data.Wherein, the video data being mixed into can be YUV (Luma and Chroma, a kind of colour coding method) data.Current main video is all based on YUV, YUV can also be subdivided into I420, The multiple formats such as YUY2, NV12.Certainly, mixed video can also be the data based on extended formattings such as RGB or RGBA.

It should be noted that making layout when the instruction for receiving user or other equipment (such as intelligent tracking system) and cutting When changing, when needing that streaming media video mixer is notified to change scaling, arrangement mode, and weight is recalculated, that is, needed again Execute step 101.

Step 103, when there are buffer empty, statistics is stored with all streaming media videos in the buffer area of streaming media video Weights sum be the first total weight value;

In practical applications, when due to factors such as network jitters, to lead to buffer area be empty, i.e. streaming media video mixer Certain can not currently be obtained all the way or when certain road streaming media videos, start policy control at this time, in conjunction with each road streaming media video Weight is obtained calculated result by calculating, and is determined the processing mode of streaming media video mixer using calculated result, for example be The size of no adjustment buffer area.

Step 104, judge whether first total weight value is less than preset threshold value;If so, 105 are thened follow the steps, if it is not, then Execute step 106;

Step 105, dilatation is carried out for the buffer area for being stored with streaming media video;

Step 106, interleave is carried out in the buffer empty.

In embodiments of the present invention, if there is buffer empty, then statistics is stored with institute in the buffer area of streaming media video There is the weights sum of streaming media video to calculate total weight value, and judges whether the total weight value is less than preset threshold value.Wherein, if being less than Preset threshold value then illustrates that the fluency of current video is unsatisfactory for desired value, should increase the buffer area for being stored with streaming media video; If more than preset threshold value, then illustrate that the fluency of current video meets desired value, in order to guarantee that streaming media video mixer works When per streaming media video all at least frame data all the way, therefore interleave operation is carried out in buffer empty.

In order to make original technical staff further appreciate that the embodiment of the present invention, it is illustrated below using specific example. Specifically, may include based on the preferential frame per second method of attentinal contents in a kind of preferable example of concrete application of the present invention Following module:

Input source management module: it is responsible for connecting input source by various real-time stream media protocols, obtains streaming media video number It is decoded after.

Buffer zone module: for buffering the image of decoded streaming media video.

Frame rate control module: the frame per second in control mixed process.

Weight computing module: the weight of each road streaming media video is set according to preset rules.

Strategic control module: determining situations such as network jitter, determines that Streaming Media mixer regards each road Streaming Media The processing mode of frequency.

Mixer module: the image of the streaming media video after multipath decoding is mixed into an image according to certain layout.

The switching command that instructor in broadcasting's module: receiving user or other equipment generate, control streaming media video mixer change cloth Office.

Coding module: the Video coding that streaming media video mixer is exported compresses.

Module is recorded in live streaming: going out or be recorded to hard disk for the net cast of compression.

It should be noted that due to that can not be broadcast live, record completely, therefore be also that can be shown only by display equipment Show, wherein this application for being only used for display is called image splitter.Buffer area can be set before the decoding, also can be set After decoding.

Referring to Fig. 5, the step of a kind of streaming media video of the invention mixes embodiment of the method 2 of frame per second control stream is shown Cheng Tu, one or more streaming media video can be loaded into respectively in corresponding one or more buffer areas, the Streaming Media view Frequency may include video frame, and the method can specifically include following steps:

Step 201, the layout parameter in preset configuration file is read;The layout parameter includes one or more described stream The pixel of media video ratio shared in preset total pixel;

In the concrete realization, layout parameter can be pre-stored in the configuration file of streaming media video mixer.Work as load It when entering configuration file, zooms in and out, arrange according to layout parameter, be mixed into video data.

Step 202, using the ratio as the weight of one or more streaming media video;

In embodiments of the present invention, the weight of each road streaming media video can be the ratio according to pixel shared by each road video Example.Such as streaming media video mixer is in the hybrid task of this processing road N streaming media video, the total pixel exported after mixing For S (the long * high resolution of resolution ratio), if certain streaming media video shared pixel in layout is S all the wayn, then the streaming media video Weight C in frame per second controln=Sn/S。

By taking the video mix of Fig. 1 layout as an example, it is assumed that the pixel of input source is 1280*720, can set Streaming Media view Frequency mixer output resolution ratio is 1706*720.Wherein for video 1 without scaling, occupancy pixel is 1280*720, video 2,3,4 It is scaled 426*240, the weight of video 1,2,3,4 is respectively 0.75,0.083,0.083,0.083 at this time.

In one preferred embodiment of the invention, the weight can also obtain in the following way:

Step S11 receives additional parameter;It include the new power of one or more described streaming media video in the additional parameter Value;

Step S12, using the new weight as the weight of one or more streaming media video.

User may want to specify weight manually, such as Fig. 1's In video mix layout, pixel shared by video 1,2,3 is equal, but user more pays close attention to video 1, for the ease of user configuration, Only needing user to set video 1 is " key frame ", sets a higher weight (such as 0.6) for video 1, remaining weight is again It is distributed according to pixel ratio.

It should be noted that carrying out weight computing, Huo Zhegen according to conditions such as video content, layout parameter, user demands It is calculated according to the comprehensive condition of certain of the above condition, obtains weight.Such as it in the concrete realization can also be according in video Degree of dynamism calculates (the high imparting of video content dynamic is compared with high weight), or present position calculates in layout according to video (different positions assigns different weights), the embodiment of the present invention to this with no restriction.

In one preferred embodiment of the invention, the method can also include:

The weights sum for calculating one or more streaming media video obtains the second total weight value;

Step S21, judges whether second total weight value is not equal to default value;If so, thening follow the steps S22;

Step S22 calculates the weight of one or more streaming media video, and the ratio of default value and the second total weight value The product of value, the weight as one or more streaming media video.

In embodiments of the present invention, the weight of each road streaming media video is by after primary distribution, it is also necessary to be verified.

Specifically, the checking procedure of weight can be and determine whether weights sum is equal to default value, if otherwise needing The adjustment of weight is carried out, so that weights sum is finally made to be equal to default value, also needs to carry out equal proportion scaling when necessary.

The pseudocode of checking procedure is as follows:

The detailed process of above-mentioned pseudocode are as follows:

1, the weights sum sum of each road streaming media video is calculated, and default value m=1 is set.

It should be noted that default value can be any reasonable value.For example, the video mix for Fig. 2 is laid out, it is defeated The road image Zhong Beige streaming media video out is all assigned, then default value m=1 can be set, and for the video of Fig. 3 Mixed layout, video 1 and video 2 are overlapped in the image of output, then it is the numerical value greater than 1 that default value m, which can be set, therefore In the embodiment of the present invention with no restriction to this.

2, judge whether each road streaming media video the sum of shared pixel in layout is defeated not equal to streaming media video mixer (i.e. weights sum sum is not equal to default value m) to total pixel, such as less than or greater than exporting total pixel out.Wherein for Less than the schematic diagram for being specifically referred to a kind of video mix layout c shown in fig. 6 the case where exporting total pixel.

If 3, weights sum sum is not equal to default value m, by each weight multiplied by m/sum times, for example, if m=1, Then each weight can be amplified 1/sum times, make the weights sum sum=m of final each road streaming media video.

It is of course also possible to use other modes verify weight, so that weights sum is finally made to be equal to default value, The embodiment of the present invention to this with no restriction.

Step 203, respectively from one or more of buffer areas, corresponding one or more Streaming Media view is extracted frame by frame Frequency is mixed;

Step 204, when there are buffer empty, statistics is stored with all streaming media videos in the buffer area of streaming media video Weights sum be the first total weight value;

Step 205, judge whether first total weight value is less than preset threshold value;If so, 206 are thened follow the steps, if it is not, then Execute step 207;

Step 206, dilatation is carried out for the buffer area for being stored with streaming media video;

Step 207, interleave is carried out in the buffer empty.

In one preferred embodiment of the invention, the step 207 may include following sub-step:

Sub-step S31 replicates the previous mixed video frame of streaming media video corresponding to the buffer empty;

The video frame is inserted into the buffer empty by sub-step S32.

In the concrete realization, due to network jitter may cause certain all the way or the buffer area on certain roads is empty situation. Judge at this time buffer area sky situation be caused by single network jitter or repeatedly network jitter cause.It is trembled if single network It is dynamic to cause, then temporarily think that the road streaming media video data has been off, then deletes stream matchmaker from streaming media video mixer Volumetric video;It is caused if multiple network jitter, then it is assumed that the network jitter of the streaming media video is larger, and needing to determine whether there is must Increase system buffer.

Decision process pseudocode is as follows:

The detailed process of above-mentioned pseudocode are as follows:

1, it is added up the weight of each road streaming media video of buffer area non-empty to obtain total weight value Csum, wherein

If 2, the total weight value is less than K, i.e. Csum< K, then it is assumed that current video mixed effect is unsatisfactory for after being weighted calculating Desired value, it should increase buffer area, improve fluency.

Wherein, (value range (0, m), such as 0.7, wherein the more high then fluency of K value is higher, then and in real time as K=0 by K Property precession effect is consistent, then consistent with fluency precession effect as K=m).

If 3, the total weight value is greater than K, i.e. Csum>=K, then it is assumed that current video mixed effect is weighted calculate after meet Desired value, per streaming media video all at least frame data all the way when in order to guarantee the work of streaming media video mixer, It is that empty streaming media video carries out interleave processing, for example can replicate the view that former frame carries out streaming media video mixing to buffer area Frequency frame data are put into buffer area.

In order to make it is further understood that the advantages of the embodiment of the present invention, below by the way of comparison pair It is described in detail towards the optimal frame rate control method of attentinal contents.

In the embodiment of the present invention the two kinds of frame rate control methods mainly used at present are ground make internal disorder or usurp on the basis of, propose one The preferential frame rate control method of kind attentinal contents.Using the correlation theory of queueing theory, three kinds of methods are modeled, are compared not The superiority and inferiority of various models in the case where with demand.

Queue theory model is established:

Since real time flow medium data are exported in source with constant rate of speed, and nonrandom arrival, admittedly it can not the direct row of applying Team theory model, but the noise or interference signal on network cause the process of the data packet active channel of network jitter can be by It is considered to reach at random and obeys Poisson distribution.And the input content stream in same hypothesis channel may be considered and believe with interference Number arrival process there are linear dependences.Frame per second control section in Fig. 1 in dotted line is converted into this hair as shown in Figure 7 A kind of schematic diagram of bright queue theory model.

If using input content stream as discuss object, need to calculate individual queue (i.e. buffer area) team leader be 0 probability and Its influence to system.But need be by model conversion take noise flowing as the model that object is discussed, after conversion shown in form and Fig. 7 It is constant, but input object is noise flowing, and computing object becomes the probability that individual queue is taken by noise and its influence to system.

If link bandwidth is w, input content bit rate is b, then is μ=w-b for the service rate of noise, is more than the rate Noise then will lead to content stream and can not reach buffer area, can be considered that noise occupies buffer area at this time.In order to simplify the process of argumentation, Assume that the bit rate of each input content stream is identical in this example, the shake intensity and bandwidth of each link are identical.Due to shaking intensity It is identical, i.e., each road noise sound arrival rate λ12...=λN=λ, similarly again since each road input content bit rate is identical with bandwidth, Then μ12...=μN=μ.If maximum team leader is N, number of queues K.Meet Poisson distribution due to reaching, services as fixed length point Cloth, queue (buffer area) is limited, and therefore, which is actually K M/D/1/N/ ∞ parallel connection concatenated mixed model again.Series connection Service rule according to three kinds of frame rate control methods institute towards demand difference and have respective difference.

1, the frame rate control method optimal towards fluency

The optimal frame rate control method of fluency requires each video pictures as smooth as possible, does not skip any number According to.The content stream for requiring each buffer area cannot be sky, and being equal to each buffer area cannot be taken by noise flowing.That is, if Any queue violates the optimal requirement of fluency there is a phenomenon where being taken by noise in system.

By M/D/1/N/ ∞ model calculation formula it is found that queue reaches the probability of longest limitation are as follows:

PNNP0

Wherein

Then

When then meeting the optimal demand of fluency, K queue does not reach the probability of the upper limit are as follows:

Assuming that bandwidth w=10 (100kb/s), input content bit rate b=8 (100kb/s), noise arrival rate λ=1 (100kb/s), then service rate μ=2 (100kb/s), if queue upper limit N=5 (100kb), total K=9 queue, going out for calculating be full The probability of sufficient demand are as follows:

P=0.865

I.e. when buffer area is 500kb, there is 86.5% probability to meet fluency demand in system operation.

2, the frame rate control method optimal towards real-time

The more demanding application of the optimal frame rate control method of real-time, reduces as far as possible in acceptable effective scope Buffering is even without buffering, to reach the requirement in minimum delay.Similarly, it is known that the probability of the queue arrival upper limit are as follows:

If it is desired to which the picture for occurring frame-skipping simultaneously is no more than 30%, that is, requires while the queue for reaching the upper limit is no more than k= K*30%.

When then meeting the optimal demand of real-time, probability that at least K-k queue is not taken by noise are as follows:

Under the same terms above, it is calculated:

P=0.88

I.e. when buffer area is 500kb, there is 88% probability to meet real-time demand in system operation.

3, the frame rate control method optimal towards attentinal contents

In the concrete realization, it can generally zoom in and out, arrange according to the demand of user when video mix, such as Fig. 2,3 institutes The mixed form shown.In Fig. 2 and Fig. 3, picture that the video of serial number 1 is indicated is the content that user most pays close attention to, then can be with It is reduced as far as possible delay time in the case where guaranteeing that the picture effect is optimal, and ignores other non-interesting within an acceptable range The frame-skipping situation of picture.Such method is exactly the frame rate control method optimal towards attentinal contents.

Because only considering that the effect of some video is optimal, only need to guarantee some queue when specified in system Interior is empty probability lower than limit value.So system can be regarded as the M/D/1/N/ of a single team by calculating process The system A of ∞ model, in addition a K-1 M/D/1/N/ ∞ model parallel connection concatenated system B again, the total buffer value of system take A, The maximum value of B two systems.

A system is calculated first.

For single group of system, the probability of meet demand:

Under the same terms above, it is calculated:

PA=0.984

B system is calculated again.What it is due to B system is the demand for weakening fluency and emphasizing real-time, and calculation method meets face The calculation method optimal to real-time, it may be assumed that

Under the same terms above, it is calculated:

PB=0.894

Then A, B system while the probability of meet demand are as follows:

P=PA*PB=0.879

I.e. when buffer area is 500kb, 87.9% probability meets attentinal contents priority requirement in system operation.

Three kinds of frame per second control modes are compared and analyzed:

The method for substituting into parameter by hypothesis condition in example, calculates under given conditions, various methods are in identical item The probability of meet demand under part.In order to further analyze situation of each method under different qualifications, drawn by matlab In each method caching N and cache meet demand probability P relationship, be specifically referred to one kind of the invention shown in Fig. 8 Buffer area meets the schematic diagram of corresponding frame rate control method probability of demand, and wherein a is the fluency mode of priority, and b is real-time The mode of priority, c are the attentinal contents mode of priority, and d is three's composograph.In order to further be convenient for comparing each method in identical item Three kinds of methods are plotted to same image by the difference under part, are specifically referred to a kind of caching of the invention shown in Fig. 9 and are expired Total schematic diagram of the corresponding frame rate control method probability of demand of foot.Be readily apparent in Fig. 9 (a), it is any under the same conditions, close The probability of content priority method meet demand is infused between other two kinds of frame rate control methods, reasoning is it is found that if meet probability In identical situation, buffer size needed for this method is also between other two methods, as shown in Fig. 9 (b) dotted line.It is comprehensive Above it is found that the attentinal contents mode of priority can be slow needed for reducing to a certain extent in the case where not reducing user experience Rush duration.

Two kinds of common frame rate control methods in the streaming media video hybrid technology discussed in this example, and at two kinds The frame rate control method optimal towards attentinal contents is proposed on the basis of common frame rate control method.By modeling, row is utilized The correlation theory of team theory, analyzes several method and has been compared.Compare the frame per second it can be found that optimal towards attentinal contents Control method can find suitable equalization point in fluency and two kinds of demands of real-time, and user experience effect is good.

It should be noted that for simple description, therefore, it is stated as a series of action groups for embodiment of the method It closes, but those skilled in the art should understand that, embodiment of that present invention are not limited by the describe sequence of actions, because according to According to the embodiment of the present invention, some steps may be performed in other sequences or simultaneously.Secondly, those skilled in the art also should Know, the embodiments described in the specification are all preferred embodiments, and the related movement not necessarily present invention is implemented Necessary to example.

Referring to Fig.1 0, show a kind of structural frames of the Installation practice of streaming media video mixing frame per second control of the present invention Figure, one or more streaming media video can be loaded into respectively in corresponding one or more buffer areas, the streaming media video It may include video frame, can specifically include following module:

Determining module 301, for determining the corresponding weight of one or more described streaming media video;

In one preferred embodiment of the invention, the determining module 301 may include:

Reading submodule, for reading the layout parameter in preset configuration file;The layout parameter include it is described all the way Or the ratio that the pixel of multichannel streaming media video is shared in preset total pixel;

The first adjustment submodule, for using the ratio as the weight of one or more streaming media video.

In one preferred embodiment of the invention, the determining module 301 may include:

Receiving submodule, for receiving additional parameter;It include one or more Streaming Media view in the additional parameter Frequently new weight;

Second adjustment submodule, for using the new weight as the weight of one or more streaming media video.

Mixing module 302, for respectively from one or more of buffer areas, extract frame by frame it is corresponding one or more Streaming media video is mixed;

Statistical module 303, for when there are buffer empty, statistics to be stored with all streams in the buffer area of streaming media video The weights sum of media video is the first total weight value;

Judgment module 304, for judging whether first total weight value is less than preset threshold value;If so, calling dilatation mould Block 305, if it is not, then calling interleave module 306;

Dilatation module 305, for carrying out dilatation for the buffer area for being stored with streaming media video;

Interleave module 306, for carrying out interleave in the buffer empty.

In one preferred embodiment of the invention, described device can also include:

Computing module, the weights sum for calculating one or more streaming media video obtain the second total weight value;

Judgment module, for judging whether second total weight value is not equal to default value;If so, calling third adjustment Module;

Third adjusts module, for calculating the weight of one or more streaming media video, with default value and second The product of the ratio of total weight value, the weight as one or more streaming media video.

For device embodiment, since it is basically similar to the method embodiment, related so being described relatively simple Place illustrates referring to the part of embodiment of the method.

All the embodiments in this specification are described in a progressive manner, the highlights of each of the examples are with The difference of other embodiments, the same or similar parts between the embodiments can be referred to each other.

It should be understood by those skilled in the art that, the embodiment of the embodiment of the present invention can provide as method, apparatus or calculate Machine program product.Therefore, the embodiment of the present invention can be used complete hardware embodiment, complete software embodiment or combine software and The form of the embodiment of hardware aspect.Moreover, the embodiment of the present invention can be used one or more wherein include computer can With in the computer-usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) of program code The form of the computer program product of implementation.

The embodiment of the present invention be referring to according to the method for the embodiment of the present invention, terminal device (system) and computer program The flowchart and/or the block diagram of product describes.It should be understood that flowchart and/or the block diagram can be realized by computer program instructions In each flow and/or block and flowchart and/or the block diagram in process and/or box combination.It can provide these Computer program instructions are set to general purpose computer, special purpose computer, Embedded Processor or other programmable data processing terminals Standby processor is to generate a machine, so that being held by the processor of computer or other programmable data processing terminal devices Capable instruction generates for realizing in one or more flows of the flowchart and/or one or more blocks of the block diagram The device of specified function.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing terminal devices In computer-readable memory operate in a specific manner, so that instruction stored in the computer readable memory generates packet The manufacture of command device is included, which realizes in one side of one or more flows of the flowchart and/or block diagram The function of being specified in frame or multiple boxes.

These computer program instructions can also be loaded into computer or other programmable data processing terminal devices, so that Series of operation steps are executed on computer or other programmable terminal equipments to generate computer implemented processing, thus The instruction executed on computer or other programmable terminal equipments is provided for realizing in one or more flows of the flowchart And/or in one or more blocks of the block diagram specify function the step of.

Although the preferred embodiment of the embodiment of the present invention has been described, once a person skilled in the art knows bases This creative concept, then additional changes and modifications can be made to these embodiments.So the following claims are intended to be interpreted as Including preferred embodiment and fall into all change and modification of range of embodiment of the invention.

Finally, it is to be noted that, herein, relational terms such as first and second and the like be used merely to by One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning Covering non-exclusive inclusion, so that process, method, article or terminal device including a series of elements not only wrap Those elements are included, but also including other elements that are not explicitly listed, or further includes for this process, method, article Or the element that terminal device is intrinsic.In the absence of more restrictions, being wanted by what sentence "including a ..." limited Element, it is not excluded that there is also other identical elements in process, method, article or the terminal device for including the element.

It is mixed to a kind of streaming media video mixing frame rate control method provided by the present invention and a kind of streaming media video above Close frame per second control device, be described in detail, specific case used herein to the principle of the present invention and embodiment into Elaboration is gone, the above description of the embodiment is only used to help understand the method for the present invention and its core ideas;Meanwhile for this The those skilled in the art in field, according to the thought of the present invention, there will be changes in the specific implementation manner and application range, In conclusion the contents of this specification are not to be construed as limiting the invention.

Claims (8)

1. a kind of method of streaming media video mixing frame per second control, which is characterized in that one or more streaming media video is distinguished It is loaded into corresponding one or more buffer areas, the streaming media video includes video frame, and the method includes:
Determine the corresponding weight of one or more described streaming media video;
Respectively from one or more of buffer areas, one or more corresponding streaming media video is extracted frame by frame and is mixed;
When there are buffer empty, the weights sum that statistics is stored with all streaming media videos in the buffer area of streaming media video is First total weight value;
Judge whether first total weight value is less than preset threshold value;If so, for the buffering for being stored with streaming media video Area carries out dilatation;If it is not, then carrying out interleave in the buffer empty;
The method also includes:
The weights sum for calculating one or more streaming media video obtains the second total weight value;
Judge whether second total weight value is not equal to default value;
If so, the weight of the streaming media video all the way is calculated, the product with default value and the ratio of the second total weight value, as The new weight of the streaming media video all the way;
Alternatively,
The weight for calculating the multichannel streaming media video, the product with default value and the ratio of the second total weight value, as described more The new weight of road streaming media video.
2. the method according to claim 1, wherein one or more streaming media video described in the determination is corresponding Weight the step of include:
Read the layout parameter in preset configuration file;The layout parameter includes the picture of one or more streaming media video Element ratio shared in preset total pixel;
Using the ratio as the weight of one or more streaming media video.
3. method according to claim 1 or 2, which is characterized in that one or more streaming media video described in the determination The step of corresponding weight includes:
Receive additional parameter;It include the new weight of one or more described streaming media video in the additional parameter;
Using the new weight as the weight of one or more streaming media video.
4. the method according to claim 1, wherein described the step of carrying out interleave in buffer empty, includes:
Replicate the previous mixed video frame of streaming media video corresponding to the buffer empty;
The video frame is inserted into the buffer empty.
5. the method according to claim 1, wherein the preset threshold value is 0.7.
6. a kind of device of streaming media video mixing frame per second control, which is characterized in that one or more streaming media video is distinguished It is loaded into corresponding one or more buffer areas, the streaming media video includes video frame, and the device includes:
Determining module, for determining the corresponding weight of one or more described streaming media video;
Mixing module, for from one or more of buffer areas, extracting one or more corresponding Streaming Media frame by frame respectively Video is mixed;
Statistical module, for when there are buffer empty, statistics to be stored with all Streaming Media views in the buffer area of streaming media video The weights sum of frequency is the first total weight value;
Judgment module, for judging whether first total weight value is less than preset threshold value;If so, dilatation module is called, if it is not, Then call interleave module;
Dilatation module, for carrying out dilatation for the buffer area for being stored with streaming media video;
Interleave module, for carrying out interleave in the buffer empty;
Described device further include:
Computing module, the weights sum for calculating one or more streaming media video obtain the second total weight value;
Judgment module, for judging whether second total weight value is not equal to default value;If so, third is called to adjust mould Block;
Third adjusts module, for calculating the weight of the streaming media video all the way, and the ratio of default value and the second total weight value The product of value, the new weight as the streaming media video all the way;
Alternatively,
For calculating the weight of the multichannel streaming media video, product with default value and the ratio of the second total weight value, as institute State the new weight of multichannel streaming media video.
7. device according to claim 6, which is characterized in that the determining module includes:
Reading submodule, for reading the layout parameter in preset configuration file;The layout parameter includes described all the way or more The pixel of road streaming media video ratio shared in preset total pixel;
The first adjustment submodule, for using the ratio as the weight of one or more streaming media video.
8. device according to claim 6 or 7, which is characterized in that the determining module includes:
Receiving submodule, for receiving additional parameter;It include that one or more described streaming media video is new in the additional parameter Weight;
Second adjustment submodule, for using the new weight as the weight of one or more streaming media video.
CN201610270343.9A 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control CN105898384B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Publications (2)

Publication Number Publication Date
CN105898384A CN105898384A (en) 2016-08-24
CN105898384B true CN105898384B (en) 2019-03-22

Family

ID=56701955

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Country Status (1)

Country Link
CN (1) CN105898384B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018045571A1 (en) * 2016-09-09 2018-03-15 深圳市大疆创新科技有限公司 Image coding method and system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1141106A (en) * 1994-12-19 1997-01-22 摩托罗拉公司 Multimedia multiplexing device and method using dynamic packet segmentation
CN1149374A (en) * 1994-04-20 1997-05-07 汤姆森消费电子有限公司 Multiplexer system using constant bit rate encoders
CN101527834A (en) * 2009-03-26 2009-09-09 浙江大华技术股份有限公司 Wireless narrowband network video transmission method
CN102427532A (en) * 2011-10-24 2012-04-25 成都虢电智能电力科技有限公司 Multipath video adapting coding transmission method under limited bandwidth and transmission system thereof

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7940843B1 (en) * 2002-12-16 2011-05-10 Apple Inc. Method of implementing improved rate control for a multimedia compression and encoding system
US7143170B2 (en) * 2003-04-30 2006-11-28 Akamai Technologies, Inc. Automatic migration of data via a distributed computer network
JP4249672B2 (en) * 2004-08-16 2009-04-02 Kddi株式会社 Video information multiple simultaneous encoding device
US8042140B2 (en) * 2005-07-22 2011-10-18 Kangaroo Media, Inc. Buffering content on a handheld electronic device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1149374A (en) * 1994-04-20 1997-05-07 汤姆森消费电子有限公司 Multiplexer system using constant bit rate encoders
CN1141106A (en) * 1994-12-19 1997-01-22 摩托罗拉公司 Multimedia multiplexing device and method using dynamic packet segmentation
CN101527834A (en) * 2009-03-26 2009-09-09 浙江大华技术股份有限公司 Wireless narrowband network video transmission method
CN102427532A (en) * 2011-10-24 2012-04-25 成都虢电智能电力科技有限公司 Multipath video adapting coding transmission method under limited bandwidth and transmission system thereof

Also Published As

Publication number Publication date
CN105898384A (en) 2016-08-24

Similar Documents

Publication Publication Date Title
Krasic et al. Quality-adaptive media streaming by priority drop
Hsu et al. Joint selection of source and channel rate for VBR video transmission under ATM policing constraints
Fitzek et al. MPEG-4 and H. 263 video traces for network performance evaluation
Lee et al. Scalable rate control for MPEG-4 video
US8107524B2 (en) Adaptive bandwidth footprint matching for multiple compressed video streams in a fixed bandwidth network
US7003794B2 (en) Multicasting transmission of multimedia information
CA2617893C (en) Video communication quality estimation device, method, and program
US20070024705A1 (en) Systems and methods for video stream selection
CA2289504C (en) Dynamic selection of media streams for display
EP1271954A2 (en) Method of transmitting layered video-coded information
KR100458040B1 (en) Transmitting/receiving method
JP2011172243A (en) Decoder and decode information processing method
JP2012135009A (en) Video encoding method, video encoder, video decoding method, and video decoder
Duffield et al. SAVE: An algorithm for smoothed adaptive video over explicit rate networks
JP2008507922A (en) Trick mode and speed transition
JP3852761B2 (en) Network system, content providing system, terminal device, content transmission method, and program
KR20090060274A (en) Techniques for managing visual compositions for a multimedia conference call
US7734104B2 (en) Image coding apparatus, image decoding apparatus and image processing system
US8661152B2 (en) Method and apparatus for reducing deterioration of a quality of experience of a multimedia service in a multimedia system
US8630306B2 (en) Fast channel change apparatus and method for IPTV
JP3095382B2 (en) Packet transmission method and a data processing system
Luan et al. Impact of network dynamics on user's video quality: Analytical framework and QoS provision
CN101411208B (en) Temporal quality metric for video coding
CN1037888C (en) Apparatus for hierarchically dividing video signals
US8428127B2 (en) Method of performing rate control for a compression system

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
COR Change of bibliographic data
GR01 Patent grant