CN105898384A - Method and apparatus for controlling streaming video mixing frame rate - Google Patents

Method and apparatus for controlling streaming video mixing frame rate Download PDF

Info

Publication number
CN105898384A
CN105898384A CN201610270343.9A CN201610270343A CN105898384A CN 105898384 A CN105898384 A CN 105898384A CN 201610270343 A CN201610270343 A CN 201610270343A CN 105898384 A CN105898384 A CN 105898384A
Authority
CN
China
Prior art keywords
video
streaming video
channels
streaming
buffer
Prior art date
Application number
CN201610270343.9A
Other languages
Chinese (zh)
Other versions
CN105898384B (en
Inventor
叶荣华
刘志聪
孙石平
张冲
Original Assignee
广州盈可视电子科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州盈可视电子科技有限公司 filed Critical 广州盈可视电子科技有限公司
Priority to CN201610270343.9A priority Critical patent/CN105898384B/en
Publication of CN105898384A publication Critical patent/CN105898384A/en
Application granted granted Critical
Publication of CN105898384B publication Critical patent/CN105898384B/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H04N21/23106Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365Multiplexing of several video streams
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/23805Controlling the feeding rate to the network, e.g. by controlling the video pump
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/266Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
    • H04N21/2662Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities

Abstract

An embodiment of the invention provides a method and apparatus for controlling the streaming video mixing frame rate. One or more channels of streaming video are respectively loaded into corresponding one or more buffer regions, the streaming video including video frames. The method includes the steps of determining weight values corresponding to the one or more channels of streaming video; extracting, respectively from the one or more buffer regions, the corresponding one or more channels of streaming video frame by frame for mixing; when an empty buffer region exists, conducting statistics of the sum of the weight values of the streaming video in the buffer regions stored with the streaming video as a first total weight value; determining whether the first total weight value is less than a preset threshold; if so, expanding the capacity for the buffer regions stored with the streaming video; and if not, inserting frames into the empty buffer region. The invention can find the balance between two demands of fluency and real time properties, and enables better user experience as compared with that using any one method alone.

Description

一种流媒体视频混合帧率控制的方法和装置 A method and apparatus for controlling the frame rate of the media stream video mixing

技术领域 FIELD

[0001] 本发明涉及视频处理技术领域,特别是涉及一种流媒体视频混合帧率控制方法和一种流媒体视频混合帧率控制装置。 [0001] The present invention relates to video processing technologies, and particularly to a hybrid video streaming frame rate control method and apparatus for mixing streaming video frame rate control.

背景技术 Background technique

[0002] 随着媒体市场与网络服务越来越紧密的融合,流媒体传播和使用日益广泛,其中, 在视频会议系统的MCU(Multipoint Control Unit,多点控制单元),以及安防监控系统的分割器等系统中,流媒体视频混合技术开始替代原有的模拟视频混合器被大量应用。 [0002] As the media market and increasingly close integration of network services, streaming media increasingly widespread dissemination and use, which, in the MCU video conferencing system (Multipoint Control Unit, multipoint control unit), and the division of security monitoring system system or the like, starts streaming video mixing technique to replace the original analog video mixer is a large number of applications.

[0003] 以视频会议系统的MCU为例,在Μ⑶中的视频混合器(video mixer)获取若干终端的流媒体视频混合到同一个视频中,并以固定的帧率将该混合视频输出进行后续的编码、 传输等工作,实际中通过解码、混合、编码的二次压缩模式节约了带宽和终端性能。 [0003] The MCU video conferencing system, for example, in the video mixer Μ⑶ (video mixer) obtaining a plurality of streaming video to the same terminal of a video mixing, and mixed at a fixed frame rate of the output video for subsequent coding, transmission, etc., and terminal capability saves bandwidth in the compressed mode by the second actual decoding, mixing, encoded. 以前在安防监控系统中的分割器主要作用于模拟信号的处理,而随着安防监控系统的网络化,其处理模式也与视频会议系统一样,将流媒体视频进行二次压缩模式。 Before major role in the division of the security monitoring system with network security monitoring system, the processing mode of the video conference system is also the same as in processing an analog signal, the second streaming video compression mode.

[0004] 在视频会议系统及安防监控系统中基于网络数据源的视频混合器又称之为流媒体视频混合器。 [0004] Based on the network data source, a video mixer and a mixer known as streaming video and video conferencing systems security monitoring systems. 在流媒体视频混合器中,由于网络抖动、帧率不对称(各个输入的流媒体视频与混合输出视频的帧率不一致)等原因,流媒体视频混合器必须采取帧率控制措施,使得混合器输出稳定的帧率的混合视频用于后续的编码。 In streaming video mixer, due to network jitter, the frame rate asymmetrical (each input frame rate streaming video and output video inconsistent mixing), etc., streaming video mixer frame rate control measures must be taken, so that the mixer stable mixed output video frame for subsequent encoding.

[0005] 帧率控制方法主要有两种,一种方法是面向流畅性优先的,该方法通过设定一个较大的缓冲区来保证视频的流畅性,但是会导致较大的延迟。 [0005] There are two main frame rate control method, a method is preferred for smooth, the method by setting a larger buffer to ensure smooth video, but can lead to large delay. 另一种方法是面向实时性最优,该方法是尽可能减小缓冲区,提高了实时性,但是会导致视频的流畅性较差。 Another method is optimal for real-time, which is to reduce the buffer zone as much as possible, improved real-time, but will lead to poor fluency video.

[0006] 在流媒体视频混合器中,流畅性和实时性是相互制约和相互矛盾的,因此,目前需要本领域技术人员迫切解决的一个技术问题就是:提出一种流媒体视频混合帧率控制机制,在流畅性和实时性这两种需求中找到平衡点,并使得用户体验相比单独使用任何一种方法更佳。 [0006] In the streaming video mixer, smooth and real-time constraints are mutually contradictory and, therefore, there needs to be urgently addressed skilled in the art of a technical question is: to provide a frame rate streaming video mixing control mechanism, to find a balance and fluency in both real-time demand, and makes the user experience better use alone compared to either method.

发明内容 SUMMARY

[0007] 本发明实施例所要解决的技术问题是提供一种流媒体视频混合帧率控制方法,用以在流畅性和实时性这两种需求中找到平衡点,并使得用户体验相比单独使用任何一种方法更佳。 [0007] The technical problem to be solved by embodiments of the present invention to provide a frame rate control method for streaming video mixing to find a balance and smooth both real-time demand, and so that the user experience in comparison to use alone either way better.

[0008] 相应的,本发明实施例还提供了一种流媒体视频混合帧率控制装置,用以保证上述方法的实现及应用。 [0008] Accordingly, embodiments of the present invention further provides a hybrid streaming video frame rate control means, to ensure the realization and application of the above method.

[0009] 为了解决上述问题,本发明公开了一种流媒体视频混合帧率控制的方法,一路或多路流媒体视频被分别载入对应的一个或多个缓冲区中,所述流媒体视频包括视频帧,所述的方法包括: [0009] In order to solve the above problems, the present invention discloses a method of streaming video frame rate control of mixing, one or more channels video streaming are respectively loaded with one or more corresponding buffer, the streaming video It comprises a video frame, the method comprising:

[0010] 确定所述一路或多路流媒体视频对应的权值; [0010] determining the one or more channels of video media streams corresponding weights;

[0011] 分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; [0011] from each of the one or more buffers, frame by frame corresponding to the extracted one or more channels are mixed streaming video;

[0012] 当存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; [0012] When there is an empty buffer, it is stored streaming video count in the weighted sum of all video streaming and the first total weight;

[0013] 判断所述第一总权值是否小于预置阈值;若是,则针对所述存储有流媒体视频的缓冲区进行扩容;若否,则在所述空缓冲区中进行插帧。 [0013] determining the first total weight value is smaller than a preset threshold; if yes, there are streaming video buffer for expansion for the storage; if not, then the interpolation frame in the empty buffer.

[0014] 优选地,所述确定所述一路或多路流媒体视频对应的权值的步骤包括: [0014] Preferably, the step of determining said one or more channels of the streaming video corresponding weight value comprises:

[0015] 读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; [0015] reads preset configuration file layout parameters; parameter includes a ratio of the pixel layout of the one or more channels of streaming video in the total of the preset pixel;

[0016] 将所述比值分别作为所述一路或多路流媒体视频的权值。 [0016] The weight ratio of each media streaming video as the one or more channels.

[0017] 优选地,所述确定所述一路或多路流媒体视频对应的权值的步骤包括: [0017] Preferably, the step of determining said one or more channels of the streaming video corresponding weight value comprises:

[0018] 接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; [0018] The additional parameters received; the additional parameters included in the new weight of the one or more channels streaming video;

[0019] 将所述新的权值作为所述一路或多路流媒体视频的权值。 [0019] The new weight as the weight of one or more channels of video media stream.

[0020] 优选地,在所述确定一路或多路流媒体视频对应的权值的步骤之后,还包括: [0020] Preferably, after the step of determining the weight of one or more channels corresponding to the streaming video, further comprising:

[0021 ]计算所述一路或多路流媒体视频的权值之和获得第二总权值; Right [0021] calculating the one or more channels of streaming video obtain a second total sum of weights;

[0022] 判断所述第二总权值是否不等于预设数值; [0022] determining whether the second total weight value is not equal to a preset value;

[0023] 若是,则计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 [0023] If, then calculating the weight of one or more channels streaming video media, the product with a preset value and the second value of the ratio of the total weight, as the weight of one or more channels of video media stream.

[0024] 优选地,所述在空缓冲区中进行插帧的步骤包括: Step [0024] Preferably, the interpolation frame in the free buffer comprises:

[0025] 复制所述空缓冲区所对应的流媒体视频前一进行混合的视频帧; [0025] Copy mixing a video frame front buffer space corresponding to the streaming video;

[0026] 将所述视频帧插入所述空缓冲区中。 [0026] The video frame is inserted into the empty buffer.

[0027] 优选地,所述K为0.7。 [0027] Preferably, the K is 0.7.

[0028] 本发明实施例还公开了一种流媒体视频混合帧率控制的装置,一路或多路流媒体视频被分别载入对应的一个或多个缓冲区中,所述流媒体视频包括视频帧,所述的装置包括: [0028] Example embodiments of the present invention also discloses a device for mixing streaming video frame rate control, video streaming one or more channels are respectively loaded with one or more corresponding buffer, the streaming video comprises a video frames, said apparatus comprising:

[0029] 确定模块,用于确定所述一路或多路流媒体视频对应的权值; [0029] determining means for determining one or more branches of the streaming video corresponding weight value;

[0030] 混合模块,用于分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; [0030] The mixing module, for respectively from the one or more buffers, frame by frame corresponding to the extracted one or more channels streaming video mixing;

[0031] 统计模块,用于在存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; [0031] Statistics module, configured to, when there is an empty buffer, statistics are stored weights of all of the streaming video streaming video buffer value and the first total weight;

[0032] 判断模块,用于判断所述第一总权值是否小于预置阈值;若是,则调用扩容模块, 若否,则调用插帧模块; [0032] determination means for determining the first total weight value is smaller than a preset threshold; if yes, the expansion module call, and if not, call the interpolation frame module;

[0033]扩容模块,用于针对所述存储有流媒体视频的缓冲区进行扩容; [0033] The expansion module for the expansion buffer for storing a streaming video;

[0034]插帧模块,用于在所述空缓冲区中进行插帧。 [0034] The interpolation frame module, for performing the interpolation frame in the empty buffer.

[0035]优选地,所述确定模块包括: [0035] Preferably, the determining module comprises:

[0036] 读取子模块,用于读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; [0036] The read sub-module for reading the layout parameters preset configuration file; parameter includes a ratio of the pixel layout of the one or more channels of streaming video in the total of the preset pixel;

[0037] 第一调整子模块,用于将所述比值分别作为所述一路或多路流媒体视频的权值。 [0037] The first adjusting sub-module, configured to stream the weight ratio of each video media as the one or more channels. [0038]优选地,所述确定模块包括: [0038] Preferably, the determining module comprises:

[0039]接收子模块,用于接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; [0039] receiving submodule configured to receive additional parameters; the additional parameters included in the new weight of the one or more channels streaming video;

[0040]第二调整子模块,用于将所述新的权值作为所述一路或多路流媒体视频的权值。 [0040] The second adjustment module for the new weight as the weight of one or more channels of video media stream. [0041 ] 优选地,所述装置还包括: [0041] Preferably, said apparatus further comprising:

[0042]计算模块,用于计算所述一路或多路流媒体视频的权值之和获得第二总权值; [0043]判断模块,用于判断所述第二总权值是否不等于预设数值;若是,则调用第三调整丰旲块; [0042] calculation means for calculating one or more branches of the right media streaming video obtain a second sum value of the total weight; [0043] determination means for determining whether the second value is not equal to the total weight of the pre set value; if the third adjustment is called abundance Dae block;

[0044] 第三调整模块,用于计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 [0044] The third adjustment module for calculating the weight of the one or more channels of streaming video, and a second predetermined value and the ratio of the total weight of the product, as the one or more channels of streaming video weights.

[0045] 与现有技术相比,本发明实施例包括以下优点: [0045] Compared with the prior art, embodiments of the present invention comprises the following advantages:

[0046] 在本发明实施例中,根据视频内容、布局参数、用户需求等条件计算权值,当网络抖动导致缓冲区为空时,将对权值的计算结果作为跳帧或增大缓冲的依据,由此来动态控制缓冲区大小,这种自适应的过程将使得缓冲区最终被设定为一个合理的大小。 [0046] In an embodiment of the present invention, weights are calculated according to the conditions of video content, layout parameters, user demand, when network jitter buffer is empty cause, the result will be calculated as a weight increase or skip frames buffered based, thereby to dynamically control the size of the buffer, this adaptive process such that the buffer will eventually be set to a reasonable size. 由于不需要保证全部视频的流畅,因此可以在所需缓冲区较少,延迟时间较短的情形下保证了视频的流畅性及实时性,提高用户体验效果。 Since no guarantee smooth all of the video, and therefore can require less buffer to ensure smooth video and real-time situation at a short delay time, improve the user experience.

附图说明 BRIEF DESCRIPTION

[0047] 图1是一种流媒体视频混合器的基本工作流程; [0047] FIG. 1 is a basic workflow media streaming video mixer;

[0048] 图2是一种视频混合布局a的不意图; [0048] FIG 2 is a layout of a video mixing is not intended;

[0049]图3是一种视频混合布局b的不意图; [0049] FIG. 3 is a video mixing arrangement is not intended to b;

[0050]图4是本发明的一种流媒体视频混合帧率控制方法实施例1的步骤流程图; [0050] FIG. 4 is a streaming video frame rate control according to the present invention, the mixing method of Example 1 Step flowchart embodiment;

[0051 ]图5是本发明的一种流媒体视频混合帧率控制方法实施例2的步骤流程图; [0051] FIG. 5 is a streaming video frame rate control according to the present invention, the mixing procedure of Example 2 is a flowchart of a method embodiment;

[0052]图6是一种视频混合布局c的不意图; [0052] FIG. 6 is a video mixing arrangement is not intended to c;

[0053]图7是本发明的一种排队论模型的示意图; [0053] FIG. 7 is a queuing model according to the present invention on a schematic diagram;

[0054] 图8是本发明的一种缓存区与满足对应的帧率控制方法需求概率的示意图; [0054] FIG. 8 is a schematic diagram of the probability of the buffer control method of the frame rate corresponding to meet the needs of the present invention;

[0055] 图9是本发明的一种缓存区与满足对应的帧率控制方法需求概率的总示意图。 [0055] FIG. 9 is a schematic view of the buffer area to the total demand for frame rate control method corresponding to the probability of satisfying the present invention.

[0056] 图10是本发明的一种流媒体视频混合帧率控制装置实施例的结构框图。 [0056] FIG. 10 is a streaming video frame rate according to the present invention, the mixing block diagram showing a control apparatus of the embodiment.

具体实施方式 Detailed ways

[0057] 为使本发明的上述目的、特征和优点能够更加明显易懂,下面结合附图和具体实施方式对本发明作进一步详细的说明。 [0057] For the above-described objects, features and advantages of the invention more apparent, the accompanying drawings and the following specific embodiments of the present invention will be further described in detail.

[0058]以视频会议系统的MCU为例,参照图1所示的一种流媒体视频混合器的基本工作流程,具体流程如下所示: [0058] The MCU video conferencing system as an example, with reference to FIG. 1 the basic workflow streaming media video mixer, the specific flow is as follows:

[0059] 在视频会议系统中,各终端的流媒体视频从不同的网络链路到达多点控制单元Μ⑶,经过缓冲区,在帧率控制(Frame rate control)作用下,流媒体视频按照一定的时间间隔采用解码器分别进行解码,将解码后的图像刷入流媒体视频混合器进行视频混合生成新的图像,再经过编码、封装组成新的视频流,分发给各个终端。 [0059] In the video conference system, streaming video reaches each terminal Μ⑶ multipoint control unit from a different network link, through the buffer under control of the frame rate (Frame rate control) action, according to a certain streaming video each time interval using the decoder to decode the decoded image brush inflow media video mixer mixing video image to generate a new, re-encoded, the package consisting of the new video stream, distributed to each terminal.

[0060] 在具体实现中,当流媒体视频混合器的输入帧率与输出帧率不相等,或者由于网络中存在的噪声或交换设备性能问题导致的网络抖动而导致输入到达不稳定的时候,需要采取帧率控制。 [0060] In a specific implementation, the input frame rate and the output frame rate when streaming video mixer are not equal, the network or due to the presence of noise or switching device performance problems caused by the network jitter caused when the input reaches unstable, the need for frame rate control. 例如,当输入帧率为20 (帧/秒),输出帧率为30 (帧/秒)时,理论上,需要每隔2帧就插入(复制)1帧数据进行混合,但是考虑到网络抖动导致的流媒体视频到达不稳定, 则可能会根据缓冲区内的流媒体视频的情况随机进行插帧,甚至有可能会因为过大的网络抖动导致流媒体视频到达严重迟缓而采取跳帧(丢弃)操作,来保证各路流媒体视频的同步。 For example, when the input frame rate is 20 (frames / sec), the output frame rate of 30 (frame / sec), is theoretically required is inserted every two (copying) data of a mixed, but taking into account network jitter resulting in streaming video instability arrive, you may randomly interpolated frames based on the case of streaming video in the buffer, there may even be due to excessive network jitter cause serious streaming video arrives slow and take skipping (discard ) operations to ensure the synchronization of streaming video from various quarters.

[0061] 若采用流畅性最优的帧率控制方法,则要求网络抖动时,尽可能不跳过任何帧,以保障视频的流畅性,因此该方法需要较大的缓冲区,导致延迟较长。 When [0061] the use of a smooth optimal frame rate control method, the network jitter requirements, without skipping any frame as possible, to ensure the smooth video, this method requires a larger buffer, resulting in a longer delay . 若采用实时性最优的帧率控制方法,则要求尽可能减小缓冲区,提高实时性,由于较小的缓冲区无法应对网络抖动带来的干扰,因此该方法会导致大量跳帧,使得视频流畅性不佳。 If the optimal use of real-time frame rate control method is required to reduce the buffer as possible, improve real-time, due to a smaller buffer can not cope with interference brought by network jitter, this method will lead to a lot of jump, making video fluency poor.

[0062] 在通常采用的帧率控制方法中,各路流媒体视频被平等对待,系统需要保证所有视频的流畅性,导致系统需要一个比处理单一视频大很多的缓冲区。 [0062] In the frame rate control method is generally employed, the brightest video streaming are treated equally, all the system needs to ensure the smooth video, the system takes a lot more than a single video processing's buffer. 相反,若设定一个较小的缓冲区,由于平等对待的原因,各路流媒体视频由于网络抖动导致的插帧或丢帧的机会也是相同的。 Conversely, if a smaller set of buffers, because of equal treatment, various streaming video due to the interpolated frames result in dropped frames or opportunistic network jitter is the same.

[0063] 在流媒体视频混合器在进行视频混合的时候,会将流媒体视频缩放、裁剪,并按照一定的布局进行排列,在具体实现中,最终输出的视频中一定会存在一个用户最关注的视频区域,也就是流媒体混合器的某一路输入视频是用户最关注的。 [0063] In the video mixer performing streaming video when mixed, will streaming video scaling, cropping, and arranged in a certain layout In a specific implementation, the final output video will be most concerned about the presence of a user the video area, that is, the flow path of a media mixer input video is user concern. 参照图2所示的一种视频混合布局a的不意图及图3所不的一种视频混合布局b的不意图,在图2及图3这两个视频混合布局示意图中,标号为1的视频是用户最关注的内容,其余的234则次之。 Referring to FIG. 2 A video shown in the layout of a hybrid is not intended to FIG. 3 and not a video mixing arrangement is not intended to b, 2 and 3 in both the video mixing schematic layout drawing, the reference numeral 1 video content users are most concerned about, and the rest 234 times. 如果只需要优先保证标号1这一路视频的流畅性和实时性,系统则可以在保证用户体验的情况下,尽可能的减小延迟并保障流畅性。 If you only need to ensure the number 1 priority this video fluency and real-time system can guarantee in the case of user experience, as much as possible to reduce the delay and to guarantee fluency.

[0064]正是本专利发明人基于上述情况,创造性地提出本发明实施例的核心构思之一在于,提出的关注内容优先的帧率控制机制,对各路流媒体视频进行加权,并由权值之和作为跳帧或增大缓冲的依据,由此来动态控制缓冲区大小,这种自适应的过程将使得缓冲区最终被设定为一个合理的大小。 [0064] The present patent disclosure is based on the above situation, the core concept is proposed creatively embodiment of the present invention is that the attention content priority frame rate control mechanism proposed for streaming video weighted brightest by weight based on the sum increases as skipping or buffered, thereby to dynamically control the size of the buffer, this adaptive process such that the buffer will eventually be set to a reasonable size. 相比"流畅性最优",由于并不要求保证全部视频流畅,因此需要的缓冲区更小。 Compared to "fluency optimal", as they do not guarantee that all requirements for smooth video, thus requiring a smaller buffer. 相比"实时性最优",保证了主体的流畅性。 Compared to "real-time best" to ensure the fluency of the body. 达到了最佳用户体验的目标, 在实时性和流畅性之间找到了最优的平衡点。 Reached the goal the best user experience, between real-time and fluency to find the optimal balance.

[0065] 参照图4,示出了本发明的一种流媒体视频混合帧率控制的方法实施例1的步骤流程图,一路或多路流媒体视频可以被分别载入对应的一个或多个缓冲区中,所述流媒体视频可以包括视频帧,所述的方法具体可以包括如下步骤: Step Method [0065] Referring to Figure 4, there is shown a streaming video frame rate control according to the present invention is mixed in the flow chart of Example 1, one or more channels streaming video may be loaded corresponding to a respective one or more buffer, the streaming video may include video frames, the method may specifically include the following steps:

[0066] 步骤101,确定所述一路或多路流媒体视频对应的权值; [0066] Step 101, determining that the one or more channels of video media streams corresponding to the weights;

[0067] 在具体实现中,流媒体视频通过RTSP(real time streaming protocol,实时流传输协议)等协议接入系统,经过解封装后得到压缩视频数据。 System access protocols [0067] In a specific implementation, streaming video via RTSP (real time streaming protocol, Real Time Streaming Protocol) or the like, obtained after decapsulation compressed video data. 其中,所述压缩视频数据的格式可以为11264、11263、丽1¥、11^)682、¥口8、」口68等等。 Wherein said compressed video data format may be 11264,11263, Li 1 ¥, 11 ^) 682, ¥ 8 port, "port 68 and the like.

[0068] 在流媒体视频混合器进行视频混合的时候,将视频缩放、裁剪并按照一定的布局进行排列,而最终输出的视频中会存在一个用户最关注的视频区域,也就是混合器的某一路输入视频是用户最关注的。 [0068] When the mixed video streaming video mixer, video scaling, cropping and arranged according to a certain layout of the final output video will exist in the video area are most concerned about a user, i.e. a mixer all the way to the user input video is most concerned about. 为了保证用户体验效果,故对各路流媒体视频可以根据用户关注程度设置不同的权值。 In order to ensure the user experience, it is possible to set different weights for streaming video from various quarters according to the degree of user attention.

[0069] 步骤102,分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; [0069] Step 102, from each of the one or more buffers, one or more ways to extract frame by frame corresponding to streaming video mixing;

[0070] 在具体实现中,流媒体混合器从缓冲区以帧为单位获取各路流媒体视频,按照指定的布局进行缩放、排列,混合成视频数据。 [0070] In a specific implementation, streaming media mixer from the buffer in units of frames acquired brightest streaming video, scaled by the specified layout, arranged and mixed into the video data. 其中,混合成的视频数据可以是YUV(Luma and Chroma,一种颜色编码方法)数据。 Wherein mixed into video data may be YUV (Luma and Chroma, a color coding) data. 目前主要的视频都是基于YUV,YUV还可以细分为1420、 YUY2、NV12等多种格式。 At present the main video are based on YUV, YUV can also be subdivided into 1420, YUY2, NV12 and other formats. 当然,混合视频还可以是基于RGB或RGBA等其他格式的数据。 Of course, hybrid video data may also be based on other formats of RGB or the like RGBA.

[0071] 需要说明的是,当接收用户或其他设备(例如智能跟踪系统)的指令,做出布局切换时,需要通知流媒体视频混合器改变缩放、排列方式时,以及重新计算权值,即需要重新执行步骤101。 [0071] Incidentally, when the instruction receiving user or other device (e.g. a smart tracking system), when making the switching arrangement needs to be notified to change the zoom streaming video mixer, when the arrangement, and recalculate the weights, i.e., need to re-execute step 101.

[0072] 步骤103,当存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; [0072] Step 103, when there is an empty buffer, the weights of all video streaming buffer count value is stored in the streaming video and a first total weight;

[0073] 在实际应用中,当由于网络抖动等因素导致缓冲区为空时,即流媒体视频混合器当前无法获取某一路或某些路流媒体视频时,此时启动策略控制,结合各路流媒体视频的权值,经过计算获得计算结果,并采用计算结果决定流媒体视频混合器的处理方式,比如是否调整缓冲区的大小。 When [0073] In practice, when due to network jitter and other factors cause a buffer is empty, that the current streaming video mixer can not get a road or some way streaming video, this time to start policy control, combined with the brightest weight of streaming video, the calculation result obtained through the calculation, and the calculation result using the processing mode determined streaming video mixer, such as whether to adjust the size of the buffer.

[0074] 步骤104,判断所述第一总权值是否小于预置阈值;若是,则执行步骤105,若否,则执行步骤106; [0074] Step 104, determining whether the first total weight value is smaller than a preset threshold; if yes, perform step 105, otherwise, execute step 106;

[0075] 步骤105,针对所述存储有流媒体视频的缓冲区进行扩容; [0075] Step 105, a buffer for expansion for the stored streaming video;

[0076] 步骤106,在所述空缓冲区中进行插帧。 [0076] Step 106, the interpolation frame in the empty buffer.

[0077] 在本发明实施例中,如果存在空缓冲区,则统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和计算总权值,并判断所述总权值是否小于预置阈值。 [0077] In an embodiment of the present invention, if there is an empty buffer, then the count value is stored to calculate the total weight of streaming video buffer value for all weights and streaming video, and determine whether the value is less than the total weight preset threshold. 其中,若小于预置阈值,则说明当前视频的流畅性不满足预期值,应当增大存储有流媒体视频的缓冲区; 若大于预置阈值,则说明当前视频的流畅性满足预期值,为了保证流媒体视频混合器工作的时候每一路的流媒体视频都至少有一帧数据,因此在空缓冲区中进行插帧操作。 Wherein, when less than a preset threshold, then the current flowing does not satisfy the expected value of the video should be increased is stored streaming video buffer; if greater than the preset threshold, then the current value of the video fluency meet expected, in order to streaming video mixer to ensure that when operating in each channel streaming video data have at least one, and therefore the frame interpolation operation in an empty buffer.

[0078] 为了使本来技术人员进一步了解本发明实施例,以下采用具体的示例进行说明。 [0078] In order to make the art would further understand the embodiments of the present invention, the following will be described using specific examples. 具体而言,在本发明具体应用的一种优选示例中,基于关注内容优先的帧率方法可以包括如下模块: Specifically, in a preferred example of the present invention the particular application, the preferred content of interest based on frame rate may include the following modules:

[0079] 输入源管理模块:负责通过各种实时流媒体协议连接输入源,获取流媒体视频数据后进行解码。 [0079] input source management module: is responsible for decoding an input source is connected through a variety of Real Time Streaming Protocol, acquiring streaming video data.

[0080] 缓冲区模块:用于缓冲解码后的流媒体视频的图像。 [0080] The buffer module: an image buffer after decoding streaming video.

[0081 ]帧率控制模块:控制混合过程中的帧率。 [0081] The control module frame rate: frame rate control during mixing.

[0082] 权值计算模块:按照预设规则设定各路流媒体视频的权值。 [0082] The weight calculation module: streaming video set brightest weight value according to a preset rule.

[0083] 策略控制模块:对网络抖动等情况进行判定,决定流媒体混合器对各路流媒体视频的处理方式。 [0083] The policy control module: the network jitter of the determination, determines the handling of streaming media mixer brightest streaming video.

[0084] 混合器模块:将多路解码后的流媒体视频的图像按照一定布局混合成一个图像。 [0084] The mixer module: the image decoding streaming video multiplexed into a mixed image in accordance with a certain layout.

[0085] 导播模块:接收用户或其他设备产生的切换指令,控制流媒体视频混合器改变布局。 [0085] directed module: receiving a handover command or other user equipment generates, streaming video mixer control to change the layout.

[0086] 编码模块:将流媒体视频混合器输出的视频编码压缩。 [0086] Encoding module: the coded media streaming video compression video output of the mixer.

[0087] 直播录制模块:将压缩的视频直播出去或录制到硬盘。 [0087] live recording module: the compressed live video or recorded to the hard disk out.

[0088]需要说明的是,由于完全可以不直播、录制,故也是可以仅仅通过显示设备进行显示,其中仅用于显示的这种应用叫做画面分割器。 [0088] Incidentally, since not completely live recording, it also may only be displayed on the display device, wherein this application is called Quad for display only. 缓冲区可以设置在解码前,也可以设置在解码后。 Buffer may be provided before the decoder, after decoding may be provided.

[0089] 参照图5,示出了本发明的一种流媒体视频混合帧率控制的方法实施例2的步骤流程图,一路或多路流媒体视频可以被分别载入对应的一个或多个缓冲区中,所述流媒体视频可以包括视频帧,所述的方法具体可以包括如下步骤: [0089] Referring to Figure 5, there is shown a streaming video frame rate control method of the present invention, the mixing procedure of Example 2 is a flowchart of embodiments, one or more channels may be loaded streaming video or a plurality of respectively corresponding buffer, the streaming video may include video frames, the method may specifically include the following steps:

[0090] 步骤201,读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; [0090] Step 201 reads preset configuration file layout parameters; parameter includes a ratio of the pixel layout of the one or more channels of streaming video in the total of the preset pixel;

[0091] 在具体实现中,布局参数可以预先存储于流媒体视频混合器的配置文件中。 [0091] In a specific implementation, layout parameters may be pre-stored in the streaming video mixer configuration file. 当载入配置文件时,按照布局参数进行缩放、排列,混合成视频数据。 When loading the configuration file, in accordance with the zoom layout parameters, arranged and mixed into the video data.

[0092] 步骤202,将所述比值分别作为所述一路或多路流媒体视频的权值; [0092] Step 202, the ratio of the weight of each media streaming video as the one or more channels;

[0093] 在本发明实施例中,各路流媒体视频的权值可以是按照各路视频所占像素的比例。 [0093] In an embodiment of the present invention, various weights may be streaming video in accordance with the ratio of pixels occupied by each channel of video. 例如流媒体视频混合器在本次处理N路流媒体视频的混合任务,混合后输出的总的像素为S (分辨率长*分辨率高),若某一路流媒体视频在布局中所占像素为Sn,则该流媒体视频在帧率控制时的权值C n= Sn/S。 E.g. streaming video mixer mixing task of this process N channels of streaming video, the total output is mixed pixel S (L * high resolution resolution), if a certain percentage of video streaming media pixel layout is Sn, the streaming video frame rate control right when the value of C n = Sn / S.

[0094] 以图1的视频混合布局为例,假设输入源的像素均为1280*720,可以设定流媒体视频混合器输出分辨率为1706*720。 Pixel [0094] In the video mixing arrangement of FIG. 1 as an example, assumed that the input source 720 are 1280 *, streaming video mixer can set an output resolution of 1706 * 720. 其中视频1无需缩放,其占用像素为1280*720,视频2、3、4 缩放为426*240,此时视频1、2、3、4的权值分别为0.75、0.083、0.083、0.083。 Wherein 1 without zooming the video, which is 1280 * 720 pixels occupied, 2,3,4 scaled video 240 * 426, then video weight 1,2,3,4 0.75,0.083,0.083,0.083 respectively.

[0095] 在本发明的一种优选实施例中,所述权值还可以通过如下方式获得: [0095] In one preferred embodiment of the present invention, the weighting values ​​may also be obtained by:

[0096] 步骤S11,接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; [0096] step S11, the additional parameters received; the additional parameters included in one or more ways of the new stream media video weight value;

[0097]步骤S12,将所述新的权值作为所述一路或多路流媒体视频的权值。 [0097] Step S12, the weights of the new value as the weight of one or more channels of video media stream.

[0098] 在实际应用中,比如在一些特定情况下用户可能希望手动指定权值,例如在图1的视频混合布局中,视频1、2、3所占像素相等,但是用户更多关注视频1,为了便于用户配置, 只需要用户设定视频1为"主画面",为视频1设定一个较高的权值(例如0.6),余下的权值再按照像素比例分配即可。 [0098] In practical applications, such as in certain cases the user may wish to manually specify weights, for example, in a layout view video mixing, video pixel 1,2,3 equal share, but more user attention video 1 for convenience of user profiles, the user only needs to set the video 1 "main screen", a video set a higher weight (e.g. 0.6), and then the rest of the weights allocated according to the ratio of the pixel.

[0099] 需要说明的是,根据视频内容、布局参数、用户需求等条件进行权值计算,或者根据以上某些条件的综合情况进行计算,取得权值。 [0099] Incidentally, according to the conditions of video content, layout parameters, user demand value calculating weights, or be calculated based on the consolidation of some of the above conditions, to obtain the right value. 比如在具体实现中还可以根据视频内的动态程度计算(视频内容动态高的赋予较高权值),或者根据视频在布局中所处位置计算(不同的位置赋予不同的权值),本发明实施例对此不作限制。 In a specific implementation example may also be calculated according to the degree of dynamics within the video (video with high dynamic content given a higher weight), or videos location in the layout is calculated in accordance with (different positions give different weights), the present invention It is not limited to this embodiment.

[0100] 在本发明的一种优选实施例中,所述的方法还可以包括: [0100] In one preferred embodiment of the present invention, the method may further comprise:

[0101 ]计算所述一路或多路流媒体视频的权值之和获得第二总权值; Right [0101] calculating the one or more channels of streaming video obtain a second total sum of weights;

[0102] 步骤S21,判断所述第二总权值是否不等于预设数值;若是,则执行步骤S22; [0102] step S21, the total weight is determined whether the second preset value is not equal; if yes, executing step S22;

[0103] 步骤S22,计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 [0103] Step S22, the flow of calculating the weight of one or more channels of video media, and a second predetermined value and the ratio of the total weight of the product, as the weight of one or more channels of video media stream.

[0104] 在本发明实施例中,各路流媒体视频的权值经过一次分配之后,还需要进行校验。 [0104] In an embodiment of the present invention, right after the brightest video streaming through the primary distribution value, but also for verification.

[0105] 具体而言,权值的校验过程可以是判定权值之和是否等于预设数值,若否则需要进行权值的调整,使得最终使权值之和等于预设数值,必要时还需要进行等比例缩放。 [0105] Specifically, the verification process can be determined weights and the sum of weights is equal to a preset value, or if necessary to adjust the weights, so that the final sum of weights equal to a preset value and, if necessary, the need for scaling.

[0106] 校验过程的伪代码如下所示: [0106] pseudo-code verification procedure is as follows:

[0107] [0107]

Figure CN105898384AD00101

[0108] 上述伪代码的具体过程为: [0108] The specific process for the pseudo-code:

[0109] 1、计算各路流媒体视频的权值之和sum,并设置预设数值m= 1。 [0109] 1, calculates weights of the streaming video brightest values ​​and sum, and setting preset value m = 1.

[0110]需要说明的是,预设数值可以是任意合理值。 [0110] Incidentally, the predetermined value can be any reasonable value. 比如,对于图2的视频混合布局,其输出的图像中被各路流媒体视频全部分配完毕,则可以设置预设数值m=l,而对于图3的视频混合布局,其输出的图像中视频1及视频2重叠,则可以设置预设数值m为大于1的数值,故在本发明实施例中对此不作限制。 For example, for video mixing arrangement of FIG. 2, the output image is the brightest of all video streaming distribution is completed, you can set preset value m = l, and for the video mixing arrangement of FIG. 3, the output of the video image video 1 and 2 overlap can be set to a value greater than 1, so in this embodiment of the present invention is not limited preset value m.

[0111] 2、判断各路流媒体视频在布局中所占像素之和是否不等于流媒体视频混合器输出总的像素(即权值之和sum不等于预设数值m),比如小于或大于输出总的像素。 [0111] 2, and the sum total of the pixel determines the pixel media video output of the mixer is not equal to share streaming video streaming brightest in the layout (i.e., the sum of the weights is not equal to a preset value sum m), such as less than or greater than the total pixel output. 其中对于小于输出总的像素的情况具体可以参照图6所示的一种视频混合布局c的示意图。 C layout diagram of a video mixing in the case wherein less than the total pixel output may specifically with reference to FIG. 6.

[0112] 3、若权值之和sum不等于预设数值m,则将每一个权值乘以m/sum倍,例如,若m= 1, 则可以将每一个权值放大1/sum倍,使最终各路流媒体视频的权值之和sum=m〇 [0112] 3, if the weighted sum is not equal to the predetermined value and the sum m, then each multiplied by a weight m / sum times, for example, if m = 1, it is possible to enlarge each of the weights 1 / sum times so that the final value of the brightest right streaming video and sum = m〇

[0113] 当然,也可以采用其他方式对权值进行校验,使得最终使权值之和等于预设数值, 本发明实施例对此不作限制。 [0113] Of course, other means may be used to verify the weight, so that the final weight value equals the preset value, embodiments of the present invention is not limited to this embodiment.

[0114] 步骤203,分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; [0114] Step 203, from each of the one or more buffers, frame by frame corresponding to the extracted one or more channels are mixed streaming video;

[0115] 步骤204,当存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; [0115] Step 204, when there is an empty buffer, the weights of all video streaming buffer count value is stored in the streaming video and a first total weight;

[0116]步骤205,判断所述第一总权值是否小于预置阈值;若是,则执行步骤206,若否,则执行步骤207; [0116] Step 205, determining whether the first total weight value is smaller than a preset threshold; if yes, step 206 is performed, otherwise, step 207 is executed;

[0117]步骤206,针对所述存储有流媒体视频的缓冲区进行扩容; [0117] Step 206, there are streaming video buffer for expansion for the storage;

[0118] 步骤207,在所述空缓冲区中进行插帧。 [0118] Step 207, the interpolation frame in the empty buffer.

[0119] 在本发明的一种优选实施例中,所述步骤207可以包括如下子步骤: [0119] In one preferred embodiment of the present invention, the step 207 may include the following sub-steps:

[0120] 子步骤S31,复制所述空缓冲区所对应的流媒体视频前一进行混合的视频帧; [0120] Sub-step S31, the duplication of the video frame prior to said mixing a buffer space corresponding to streaming video;

[0121 ]子步骤S32将所述视频帧插入所述空缓冲区中。 [0121] The sub-step S32 is inserted into the space of the video frame buffer.

[0122] 在具体实现中,由于网络抖动可能造成在某一路或某些路的缓冲区为空的情况。 [0122] In a specific implementation, since this could cause the network jitter buffer in a road or some channel is empty. 此时判断缓冲区空的情况是由单次网络抖动造成还是多次网络抖动造成。 At this time, it is determined that the buffer is empty jitter caused by a single network or multiple networks shake. 若为单次网络抖动造成,则暂时认为该路流媒体视频数据已经离线,则从流媒体视频混合器中删除该流媒体视频;若为多次网络抖动造成,则认为该流媒体视频的网络抖动较大,需要判定是否有必要加大系统缓冲区。 If it is caused by a single network jitter, for the time being considered the way streaming video data has been off-line, from streaming video mixer deletes the streaming video; if it is caused by multiple network jitter, streaming video network that is considered jitter is large, needs to decide whether it is necessary to increase the buffer system.

[0123] 判定过程伪代码如下: [0123] Pseudo code decision process:

[0124] j [0124] j

Figure CN105898384AD00111

[0125] 上述伪代码的具体过程为: [0125] The specific process for the pseudo-code:

[0126] [0126]

Figure CN105898384AD00112

〗、将缙冲区非空的各路流媒体视频的权值进行累加得到总权值C sum,其中 〗, The weights of non-null Jin brightest video streaming buffer zone is accumulated to give a total weight of C sum, wherein

[0127] 2、若该总权值小于K,即Csum〈K,则认为当前视频混合效果经过加权计算后不满足的预期值,应该增大缓冲区,提高流畅性。 [0127] 2, if the total weight is less than K, i.e. Csum <K, it is expected that the value of the current video weighted after mixing is not satisfied, the buffer should be increased to improve the fluidity.

[0128] 其中,K(取值范围(0,m),例如0.7,其中K值越高则流畅性越高,当K = 0时则和实时性优先效果一致,当K=m时则和流畅性优先效果一致)。 [0128] where, K (in the range (0, m), for example 0.7, where the higher the K value the higher the fluidity, and when K = the real-time effect of consistent priority 0 when the m and K = fluency priority consistent results).

[0129] 3、若该总权值大于K,即Csum彡K,则认为当前视频混合效果经过加权计算后满足的预期值,为了保证流媒体视频混合器工作的时候每一路的流媒体视频都至少有一帧数据, 对缓冲区为空的流媒体视频进行插帧处理,比如可以复制前一帧进行流媒体视频混合的视频帧数据放入到缓冲区中。 [0129] 3, if the total weight is greater than K, i.e. Csum San K, the value of the current video expected that after mixing weighted satisfied, in order to ensure that each channel of the mixer when streaming video streaming video is working at least one data buffers to empty the interpolation frame processing streaming video, such as video frame data can be copied for a previous frame is mixed into the streaming video to the buffer.

[0130] 为了使本领域技术人员进一步理解本发明实施例的优点,以下采用对比的方式对面向关注内容最优的帧率控制方法进行详细说明。 [0130] In order that those skilled in the art to further understand the advantages of the embodiments of the present invention, for the attention content to the optimal frame rate control method employed is described in detail by way of comparison.

[0131] 本发明实施例中在对目前主要采用的两种帧率控制方法研宄的基础上,提出了一种关注内容优先的帧率控制方法。 [0131] embodiment of the present invention on the basis of the two main frame rate control method used in the study based on the proposed priority attention content frame rate control method. 利用排队论的相关理论,对三种方法进行建模,比较在不同需求的情况下各种模型的优劣。 Using the theory of queuing theory, modeling the merits of the three methods, compare various models in the case of different needs.

[0132] 排队论模型建立: [0132] queuing theory model:

[0133] 由于实时流媒体数据在源端以恒定速率输出,并非随机到达,固无法直接套用排队论模型,但是网络上的噪声或者干扰信号引起网络抖动的数据包占用通道的过程可以被认为是随机到达服从泊松分布的。 [0133] Since the real-time streaming media data at a constant rate output at the source, not random arrival, it is unable to directly apply queuing theory models, but cause noise or interfering signals on the network during a data packet occupying channel network jitter may be considered Poisson distribution of random arrival. 而在同一假定通道内的输入内容流可以认为是跟干扰信号的到达过程存在线性相关性的。 Input content stream in the same channel can be assumed that a linear correlation exists with the interfering signal arrival process. 将图1中虚线内的帧率控制部分转换为如图7所示的本发明的一种排队论模型的示意图。 Queuing Theory schematic diagram of a model of the present invention is the frame rate in the dashed line in FIG 1 is converted into the control section 7 shown in FIG.

[0134] 若以输入内容流作为讨论对象,需要计算各队列(即缓冲区)的队长为0的概率及其对系统的影响。 [0134] In terms of the input content stream affect the probability of the object as discussed, need to be calculated for each queue (i.e., buffer) is 0 and the captain of the system. 但需要将模型转换为以噪声流为讨论对象的模型,转换后形式和图7所示不变,但输入对象为噪声流,计算对象变为各队列被噪声占满的概率及其对系统的影响。 But needs to be converted to the noise model to discuss the flow model of the object, the same form shown in FIG. 7 and FIG converted, but the input noise current object, the object becomes calculated probability of each queue is occupied, and the noise of the system influences.

[0135] 若链路带宽为w,输入内容比特率为b,则对于噪声的服务率为y=w_b,超过该速率的噪声则会导致内容流无法到达缓冲区,此时可视为噪声占用缓冲区。 [0135] If the link bandwidth w, the input content bit rate b, then the noise service was y = w_b, noise exceeding the rate will result in the content stream can not reach the buffer occupancy at this time can be considered as noise buffer. 为了简化论证过程, 本示例中假设各输入内容流的比特率相同,各链路的抖动强度及带宽相同。 To simplify the process demonstrate, the present example assuming the same bit rate of each input content stream, the same strength of each link jitter and bandwidth. 由于抖动强度相同,即各路噪声到达率. .=λΝ = λ,同理又由于各路输入内容比特率和带宽相同, 则μL = μ2...=μΝ = μ。 For the same intensity jitter, i.e. the brightest noise arrival rate.. = ΛΝ = λ, empathy and because the same content for each input bit rate and bandwidth, μL = μ2 ... = μΝ = μ. 设最大队长为Ν,队列数为Κ。 Set the highest Captain Ν, the number of queues Κ. 由于到达符合泊松分布,服务为定长分布,队列(缓冲区)有限,因此,该模型实际上是Κ个M/D/1/N/m并联再串联的混合模型。 Since the arrival Poisson distribution, distribution services for the fixed length, the queue (buffer) is limited, and therefore, this model is actually Κ a M / D / 1 / N / m in parallel and then in series hybrid model. 串联服务的规则根据三种帧率控制方法所面向的需求不同而有着相应差别。 Series of rules based on the service in three different frame rate control method for the demand with a corresponding difference.

[0136] 1、面向流畅性最优的帧率控制方法 [0136] 1, the optimal frame rate for smooth control method

[0137] 流畅性最优的帧率控制方法要求每一个视频画面都尽可能流畅,不跳过任何数据。 [0137] Optimal smooth frame rate control method requires a video picture are each as smoothly as possible, without skipping any data. 即要求各缓冲区的内容流不能为空,等同于各缓冲区不能被噪声流占满。 That requires the content of each stream can not be empty buffer, each buffer is equivalent to a noise current can not be filled. 也就是说,若系统中任意队列发生被噪声占满的现象则违背流畅性最优要求。 That is, if any of the system queues are filled by noise phenomenon is contrary to the requirements of optimum fluidity.

[0138] 由M/D/1/N/m模型计算公式可知,队列到达最长限制的概率为: [0138] apparent from M / D / 1 N / m is calculated / model, the probability of the queue reaches a maximum limit of:

[0139] ΡΝ=ρΝΡ〇 [0139] ΡΝ = ρΝΡ〇

Figure CN105898384AD00121

[0142] 则满足流畅性最优需求时,Κ个队列均未到达上限的概率为: When [0142] is the best meet the needs of fluency, neither the probability of reaching the upper limit of Κ queues as follows:

[0140] [0140]

[0141] [0141]

[0143] [0143]

Figure CN105898384AD00122

[0144] 假设带宽w=10(100kb/s),输入内容比特率b = 8(100kb/s),噪声到达率λ=1 (100kb/s),则服务率y = 2(100kb/s),若队列上限N=5(100kb),共Κ = 9个队列,计算的出满足需求的概率为: [0144] Suppose the bandwidth w = 10 (100kb / s), the input content bit rate b = 8 (100kb / s), the noise arrival rate λ = 1 (100kb / s), the service rate y = 2 (100kb / s) , if the probability needs queue limit N = 5 (100kb), total queue Κ = 9, the calculation is:

[0145] Ρ = 〇.865 [0145] Ρ = 〇.865

[0146] 即当缓冲区为500kb时,系统运行过程中有86.5%的概率满足流畅性需求。 [0146] That is, when the buffer is 500kb, the system is running with a 86.5% probability to meet the needs of fluency.

[0147] 2、面向实时性最优的帧率控制方法 [0147] 2, real-time frame rate control method for optimal

[0148]实时性最优的帧率控制方法要求较高的应用,在可接受的效果范围内尽可能减小缓冲甚至无缓冲,来达到最小延迟的要求。 [0148] real-time frame rate control method optimal demanding applications, reducing as much as possible within an acceptable cushioning effect or no buffering range, to achieve the minimum delay requirements. 同理,已知队列到达上限的概率为: Similarly, the probability of reaching the upper limit of the queue known as:

[0149] [0149]

Figure CN105898384AD00131

[0150] 若要求同时出现跳帧的画面不超过30%,即要求同时到达上限的队列不超过k = K*30%。 [0150] If the screen requires simultaneous skipping of not more than 30%, i.e., simultaneously arrive at the queue requires no more than the upper limit k = K * 30%.

[0151] 则满足实时性最优需求时,至少有Kk个队列未被噪声占满的概率为: When [0151] is the best real-time to meet demand, probability that at least one queue is not Kk noise filled as follows:

[0152] [0152]

Figure CN105898384AD00132

[0153] 在上文相同条件下,计算得出: [0153] Under the same conditions described above, calculated as follows:

[0154] P = 〇.88 [0154] P = 〇.88

[0155] 即当缓冲区为500kb时,系统运行过程中有88%的概率满足实时性需求。 [0155] That is, when the buffer is 500kb, the system is running 88% probability meet real-time requirements.

[0156] 3、面向关注内容最优的帧率控制方法 [0156] 3, for the follow what the best frame rate control method

[0157] 在具体实现中,视频混合时一般会按照用户的需求进行缩放、排列,例如图2,3所示的混合形式。 [0157] In a specific implementation, a video mixing typically scaled in accordance with the needs of users, are arranged in mixed form shown in FIG. 3, for example. 图2及图3中,序号为1的视频所标示的画面为用户最关注的内容,那么可以在保证该画面效果最优的情况下尽可能降低延迟时间,而在可接受范围内忽略其他非关注画面的跳帧情况。 In FIG. 2 and FIG. 3, No. 1 video picture is marked as the content users are most concerned, it can guarantee the best effect in the case where the screen as much as possible to reduce the delay time, while ignoring other non-within an acceptable range concerned about skipping the case of the screen. 这样的方法就是面向关注内容最优的帧率控制方法。 Such an approach is content-oriented focus on the best frame rate control method.

[0158] 因为只考虑某一个视频的效果最优,所以系统中只需要保证某一个队列在指定时间内的为空的概率低于限定值即可。 [0158] Since only one video effects considered optimal, so the system need only ensure that the probability of an empty queue within a certain specified time below the limit value. 那么计算过程可以将系统看作是一个单队的M/D/1/N/ °°模型的系统A,加上一个K-1个M/D/1/N/oo模型并联再串联的系统B,系统总的缓冲值取A、 B两个系统的最大值。 Then the calculation process can be viewed as a single system team M / D / 1 / N / A °° system model, plus a K-1 th M / D / 1 / N / oo model system in parallel and then in series the total buffer B, the system takes the value a, maximum value B of the two systems.

[0159] 首先计算A系统。 [0159] A first computing system.

[0160] 对于单队系统,其满足需求的概率: [0160] For a single team system to meet the needs of probability:

[0161] [0161]

Figure CN105898384AD00133

[0162] 在上文相同条件下,计算得出: [0162] Under the same conditions described above, calculated as follows:

[0163] PA=0.984 [0163] PA = 0.984

[0164] 再计算B系统。 [0164] B and then computing system. 由于B系统的是弱化流畅性强调实时性的需求,其计算方法符合面向实时性最优的计算方法,即: Since B system is weakened fluency emphasis on the needs of real-time, which is calculated in line with the best real-time calculation method for, namely:

[0165] [0165]

Figure CN105898384AD00134

[0166] 在上文相同条件下,计算得出: [0166] Under the same conditions described above, calculated as follows:

[0167] Pb = 0.894 [0167] Pb = 0.894

[0168]贝IJA、B系统同时满足需求的概率为: [0168] Tony IJA, the probability of B system while meeting the needs of:

[0169] P = Pa*Pb = 0.879 [0169] P = Pa * Pb = 0.879

[0170] 即当缓冲区为500kb时,系统运行中87.9%的概率满足关注内容优先需求。 [0170] That is, when the buffer is 500kb, the system is running 87.9% probability content to meet the priority needs attention.

[0171 ]对三种帧率控制方式进行对比分析: [0171] The three kinds of frame rate control methods were compared:

[0172]示例中通过假定条件代入参数的方法,计算了在特定条件下,各种方法在相同条件下满足需求的概率。 The method of the parameters of [0172] example, by substituting assumptions, the calculated probability to meet the needs of various methods under the same conditions under specific conditions. 为了进一步分析各方法在不同限定条件下的情况,通过matlab绘制了各方法中缓存N与缓存满足需求的概率P的关系,具体可以参照图8所示的本发明的一种缓存区满足对应的帧率控制方法需求概率的示意图,其中a为流畅度优先方法,b为实时性优先方法,C为关注内容优先方法,d为三者合成图像。 To further analyze each case under different conditions defined by the respective methods matlab drawn cache N relations probability P of the cache needs to satisfy, with respect to one specific cache can satisfy the present invention shown in FIG. 8 corresponds a schematic view of the frame rate control requirements of the probabilistic approach, wherein a first method of fluency, b is a real-time priority method, C is the preferred method of the attention content, d is three synthetic images. 为了进一步便于比较各方法在相同条件下的差异,将三种方法绘制到同一图像,具体可以参照图9所示的本发明的一种缓存与满足对应的帧率控制方法需求概率的总示意图。 To further facilitate the methods were compared under the same conditions, the same three methods to draw the image, specifically with respect to one cache of the present invention shown in FIG. 9 a schematic view of the overall frame rate control method corresponding to probability demand satisfied. 图9(a)中易于看出,在任何相同的条件下,关注内容优先方法满足需求的概率介于其他两种帧率控制方法之间,推理可知,若满足概率相同的情况下,该方法所需的缓冲区大小也介于其他两种方法之间,如图9(b)虚线所示。 FIG. 9 (a) is liable to be seen in any of the same conditions, the preferred method of the attention content to meet the needs of the probability is between the other two frame rate control method, the inference can be seen, if satisfied at the same probability, the method buffer size is also interposed between the other two methods, as shown in FIG. 9 (b) broken line. 综上可知,关注内容优先方法可以在不降低用户体验的情况下,在一定程度上减小所需的缓冲时长。 To sum up, when the length of the attention content priority method may reduce the required buffering to a certain extent without reducing the user experience.

[0173]本示例中讨论的流媒体视频混合技术中的两种常用的帧率控制方法,以及在两种常用帧率控制方法的基础上提出了面向关注内容最优的帧率控制方法。 [0173] Two common frame rate control method for streaming video mixing technique discussed in the present example, and made optimal for the attention content on the basis of the frame rate control method used on the two frame rate control method. 通过建模,利用排队论的相关理论,对几种方法进行了分析和比较。 Through modeling, using the theory of queuing theory, several methods are analyzed and compared. 比较可以发现,面向关注内容最优的帧率控制方法可以在流畅性与实时性两种需求中找到合适的平衡点,用户体验效果好。 Comparison can be found, for the follow what the best frame rate control method can find the right balance fluency and real-time two requirements, a good user experience.

[0174]需要说明的是,对于方法实施例,为了简单描述,故将其都表述为一系列的动作组合,但是本领域技术人员应该知悉,本发明实施例并不受所描述的动作顺序的限制,因为依据本发明实施例,某些步骤可以采用其他顺序或者同时进行。 [0174] Incidentally, the operation procedure of the method embodiment, for ease of description, it is described as a series combination of actions, those skilled in the art should know that, the described embodiments are not embodiments of the present invention by limiting, since the embodiment according to the present invention, some steps may be performed simultaneously or in other sequences. 其次,本领域技术人员也应该知悉,说明书中所描述的实施例均属于优选实施例,所涉及的动作并不一定是本发明实施例所必须的。 Secondly, those skilled in the art should also understand that the embodiments are described in the specification are exemplary embodiments, the operation is not necessarily related to the embodiment of the present invention must be implemented.

[0175] 参照图10,示出了本发明一种流媒体视频混合帧率控制的装置实施例的结构框图,一路或多路流媒体视频可以被分别载入对应的一个或多个缓冲区中,所述流媒体视频可以包括视频帧,具体可以包括如下模块: [0175] Referring to Figure 10, there is shown apparatus of the present invention is a streaming media frame rate video mixing control block diagram showing an embodiment, one or more channels, respectively, streaming video may be loaded with one or more buffers corresponding the streaming video may comprise a video frame, may include the following modules:

[0176] 确定模块301,用于确定所述一路或多路流媒体视频对应的权值; [0176] determining module 301 for determining one or more branches of the streaming video corresponding weight value;

[0177] 在本发明的一种优选实施例中,所述确定模块301可以包括: [0177] In one preferred embodiment of the present invention, the determining module 301 may include:

[0178] 读取子模块,用于读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; [0178] read sub-module for reading the layout parameters preset configuration file; parameter includes a ratio of the pixel layout of the one or more channels of streaming video in the total of the preset pixel;

[0179] 第一调整子模块,用于将所述比值分别作为所述一路或多路流媒体视频的权值。 [0179] The first adjustment module, the said ratio for each video streaming media weights as the one or more channels.

[0180] 在本发明的一种优选实施例中,所述确定模块301可以包括: [0180] In one preferred embodiment of the present invention, the determining module 301 may include:

[0181] 接收子模块,用于接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; [0181] receiving submodule configured to receive additional parameters; the additional parameters included in the new weight of the one or more channels streaming video;

[0182] 第二调整子模块,用于将所述新的权值作为所述一路或多路流媒体视频的权值。 [0182] The second adjustment module for the new weight as the weight of one or more channels of video media stream.

[0183] 混合模块302,用于分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; [0183] mixing module 302, for respectively from the one or more buffers, frame by frame corresponding to the extracted one or more channels are mixed streaming video;

[0184] 统计模块303,用于在存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; [0184] Statistics module 303 is used when there is an empty buffer, statistics are stored weights of all of the streaming video streaming video buffer value and the first total weight;

[0185] 判断模块304,用于判断所述第一总权值是否小于预置阈值;若是,则调用扩容模块305,若否,则调用插帧模块306; [0185] determining module 304, configured to determine the first total weight value is smaller than a preset threshold; if yes, the expansion module 305 is invoked, and if not, block 306 calls the interpolated frame;

[0186] 扩容模块305,用于针对所述存储有流媒体视频的缓冲区进行扩容; [0186] expansion module 305, a buffer for expansion for the stored streaming video;

[0187] 插帧模块306,用于在所述空缓冲区中进行插帧。 [0187] interpolation frame module 306 for performing interpolation frame in the empty buffer.

[0188] 在本发明的一种优选实施例中,所述装置还可以包括: [0188] In one preferred embodiment of the present invention, the apparatus may further comprise:

[0189] 计算模块,用于计算所述一路或多路流媒体视频的权值之和获得第二总权值; [0189] calculation means for calculating a second weight of the total weight of the one or more channels of streaming video and obtained values;

[0190] 判断模块,用于判断所述第二总权值是否不等于预设数值;若是,则调用第三调整丰旲块; [0190] determination means for determining whether the second total weight value is not equal to a preset value; if the third adjustment is called abundance Dae block;

[0191 ]第三调整模块,用于计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 [0191] The third adjustment module for calculating the weight of the one or more channels of streaming video, and a second predetermined value and the ratio of the total weight of the product, as the one or more channels of streaming video weights.

[0192] 对于装置实施例而言,由于其与方法实施例基本相似,所以描述的比较简单,相关之处参见方法实施例的部分说明即可。 [0192] For the apparatus of the embodiment, since the method of the embodiment which is substantially similar, the description of a relatively simple, some embodiments of the methods see relevant point can be described.

[0193] 本说明书中的各个实施例均采用递进的方式描述,每个实施例重点说明的都是与其他实施例的不同之处,各个实施例之间相同相似的部分互相参见即可。 [0193] In the present specification, various embodiments are described in a progressive way, differences from the embodiment and the other embodiments each of which emphasizes embodiment, various portions of the same embodiment of similarity between the embodiments refer to each other.

[0194] 本领域内的技术人员应明白,本发明实施例的实施例可提供为方法、装置、或计算机程序产品。 [0194] skill in the art should understand that the embodiment of the embodiment of the present invention may be provided as a method, apparatus or computer program product. 因此,本发明实施例可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。 Accordingly, the present invention embodiments may entirely hardware embodiment, an entirely software embodiment, or an embodiment in conjunction with the form of software and hardware aspects. 而且,本发明实施例可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。 Further, a computer program product according to embodiments may employ one or more of which comprises a computer usable storage medium having computer-usable program code (including but not limited to, disk storage, CD-ROM, optical memory, etc.) on the embodiment of the present invention.

[0195] 本发明实施例是参照根据本发明实施例的方法、终端设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。 [0195] Example embodiments of the present invention with reference to an embodiment of the method according to the present invention, the terminal apparatus (systems), and the flowchart and / or block diagrams of a computer program product described herein. 应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。 It should be understood and implemented by computer program instructions and block, and the flowchart / or block diagrams each process and / or flowchart illustrations and / or block diagrams of processes and / or blocks. 可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理终端设备的处理器以产生一个机器,使得通过计算机或其他可编程数据处理终端设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。 These computer program instructions may be provided to a processor a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal apparatus to produce a machine, such that the instructions executed by a computer or other programmable data processing terminal processor generating a flow diagram for implementing a process or apparatus or more processes and / or block diagram of a plurality of blocks in the block functions.

[0196] 这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理终端设备以特定方式工作的计算机可读存储器中,使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品,该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。 [0196] These computer program instructions may also be stored in a can direct a computer or other programmable data processing apparatus to function in a particular terminal computer readable memory, such that the instructions stored in the computer readable memory comprising instruction generating means for producing products, the instruction means implemented in a flowchart or more flows and / or block diagram block or blocks in a specified function.

[0197] 这些计算机程序指令也可装载到计算机或其他可编程数据处理终端设备上,使得在计算机或其他可编程终端设备上执行一系列操作步骤以产生计算机实现的处理,从而在计算机或其他可编程终端设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。 [0197] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal, so that a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer terminal implemented so that a computer or other instructions that execute on the device provide steps for implementing the programming terminal in a flowchart or more flows and / or block diagram block or blocks a function specified step.

[0198] 尽管已描述了本发明实施例的优选实施例,但本领域内的技术人员一旦得知了基本创造性概念,则可对这些实施例做出另外的变更和修改。 [0198] While the present invention has been described with preferred embodiments of the embodiment, but those skilled in the art from the underlying inventive concept can make further modifications and variations to these embodiments. 所以,所附权利要求意欲解释为包括优选实施例以及落入本发明实施例范围的所有变更和修改。 Therefore, the appended claims are intended to explain embodiments including the preferred embodiment and as fall within the scope of the embodiments of the present invention, all changes and modifications.

[0199] 最后,还需要说明的是,在本文中,诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来,而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。 [0199] Finally, it should be noted that, herein, relational terms such as first and second and the like are only used to distinguish one entity or operation from another entity or action without necessarily requiring or implying any such actual relationship or order between these entities or operations. 而且,术语"包括"、"包含"或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者终端设备不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者终端设备所固有的要素。 Further, the term "comprising", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, article, or terminal device not include only those elements but not explicitly listed further comprising the other elements, or elements of the process further comprising, method, article, or inherent to the terminal device. 在没有更多限制的情况下,由语句"包括一个......"限定的要素,并不排除在包括所述要素的过程、方法、物品或者终端设备中还存在另外的相同要素。 Without more constraints, by the wording "include a ......" defined does not exclude the existence of additional identical elements in the element comprising a process, method, article, or terminal device.

[0200] 以上对本发明所提供的一种流媒体视频混合帧率控制方法和一种流媒体视频混合帧率控制装置,进行了详细介绍,本文中应用了具体个例对本发明的原理及实施方式进行了阐述,以上实施例的说明只是用于帮助理解本发明的方法及其核心思想;同时,对于本领域的一般技术人员,依据本发明的思想,在具体实施方式及应用范围上均会有改变之处, 综上所述,本说明书内容不应理解为对本发明的限制。 [0200] mixing the above frame rate control method and a frame rate control streaming video mixing apparatus, described in detail herein through specific embodiments and principles of the present invention to one embodiment of the present invention, video streaming provided are set forth in the above described embodiments are only used to help understand the method and core idea of ​​the present invention; while those of ordinary skill in the art, according to the ideas of the present invention, in the embodiments and application scopes change place, Therefore, the specification shall not be construed as limiting the present invention.

Claims (10)

1. 一种流媒体视频混合帧率控制的方法,其特征在于,一路或多路流媒体视频被分别载入对应的一个或多个缓冲区中,所述流媒体视频包括视频帧,所述的方法包括: 确定所述一路或多路流媒体视频对应的权值; 分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进行混合; 当存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; 判断所述第一总权值是否小于预置阈值;若是,则针对所述存储有流媒体视频的缓冲区进行扩容;若否,则在所述空缓冲区中进行插帧。 A streaming video frame rate control of mixing process, wherein one or more channels video streaming are respectively loaded with one or more corresponding buffer, the streaming video comprising video frames, the the method comprising: determining weights of the way or channel corresponding to the video streaming; from each of the one or more buffers, frame by frame corresponding to the extracted one or more channels streaming video mixing; if there is an empty buffer time zone, the statistics are stored weights of all of the streaming video streaming video buffer value and the first total weight; determining the first total weight value is smaller than a preset threshold; if yes, for the storage there is a buffer for expansion streaming video; if not, then the interpolation frame in the empty buffer.
2. 根据权利要求1所述的方法,其特征在于,所述确定所述一路或多路流媒体视频对应的权值的步骤包括: 读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; 将所述比值分别作为所述一路或多路流媒体视频的权值。 2. The method according to claim 1, wherein said step of determining the one or more channels streaming video corresponding weight value comprises: reading the layout parameters preset configuration file; the layout parameters It includes a ratio of pixels of the one or more channels of streaming video in the total of the preset pixel; the weight ratio of each media streaming video as the one or more channels.
3. 根据权利要求1或2所述的方法,其特征在于,所述确定所述一路或多路流媒体视频对应的权值的步骤包括: 接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; 将所述新的权值作为所述一路或多路流媒体视频的权值。 3. The method of claim 1 or claim 2, wherein said step of determining the one or more channels streaming video corresponding weight value comprises: receiving additional parameter; the additional parameters include the way or the flow path of the new video media weight; the new weight value as the weight of one or more channels of video media stream.
4. 根据权利要求1所述的方法,其特征在于,在所述确定一路或多路流媒体视频对应的权值的步骤之后,还包括: 计算所述一路或多路流媒体视频的权值之和获得第二总权值; 判断所述第二总权值是否不等于预设数值; 若是,则计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 4. The method according to claim 1, wherein, after said step of determining the weight of one or more channels corresponding to the streaming video, further comprising: calculating the weight of one or more channels streaming video media obtaining a second sum total weight; determining whether the second total weight value is not equal to a preset value; if yes, calculating the weight of one or more channels of media streaming video with a preset value and the second total weight the ratio of the value of the product, as the weight of one or more channels of video media stream.
5. 根据权利要求1所述的方法,其特征在于,所述在空缓冲区中进行插帧的步骤包括: 复制所述空缓冲区所对应的流媒体视频前一进行混合的视频帧; 将所述视频帧插入所述空缓冲区中。 5. The method according to claim 1, characterized in that, in the empty buffer in the interpolated frame comprises the step of: mixing a copy of the video frame front buffer space corresponding to streaming video; and the video frame is inserted into the empty buffer.
6. 根据权利要求1所述的方法,其特征在于,所述K为0.7。 6. The method according to claim 1, wherein said K is 0.7.
7. -种流媒体视频混合帧率控制的装置,其特征在于,一路或多路流媒体视频被分别载入对应的一个或多个缓冲区中,所述流媒体视频包括视频帧,所述的装置包括: 确定模块,用于确定所述一路或多路流媒体视频对应的权值; 混合模块,用于分别从所述一个或多个缓冲区中,逐帧提取对应的一路或多路流媒体视频进彳丁混合; 统计模块,用于在存在空缓冲区时,统计存储有流媒体视频的缓冲区中所有流媒体视频的权值之和为第一总权值; 判断模块,用于判断所述第一总权值是否小于预置阈值;若是,则调用扩容模块,若否, 则调用插帧模块; 扩容模块,用于针对所述存储有流媒体视频的缓冲区进行扩容; 插帧模块,用于在所述空缓冲区中进行插帧。 7. - kind of streaming video frame rate control of the mixing device, characterized in that one or more channels video streaming are respectively loaded with one or more corresponding buffer, the streaming video comprising video frames, the the apparatus comprising: a determining module, for determining the weight of the one or more channels corresponding to the streaming video; mixing module, for respectively from the one or more buffers, frame by frame corresponding to the extracted one or more channels streaming video butoxy mixed into the left foot; counting module, configured to, when there is an empty buffer, statistics are stored weights of all of the streaming video streaming video buffer value and the first total weight; determining module, with to determining that the first total weight value is smaller than a preset threshold; if yes, the expansion module call, if not, the call module interpolated frame; expansion module for streaming video have for the memory buffer for expansion; interpolation frame module, for performing the interpolation frame in the empty buffer.
8. 根据权利要求7所述的装置,其特征在于,所述确定模块包括: 读取子模块,用于读取预置配置文件中的布局参数;所述布局参数包括所述一路或多路流媒体视频的像素在预置的总像素中所占的比值; 第一调整子模块,用于将所述比值分别作为所述一路或多路流媒体视频的权值。 8. The apparatus according to claim 7, wherein the determining module comprises: a reading sub-module for reading the layout parameters preset configuration file; the layout parameters include one or more branches streaming video pixels in the total pixels in a preset ratio; a first adjusting sub-module, configured to stream the weight ratio of each video media as the one or more channels.
9. 根据权利要求7或8所述的装置,其特征在于,所述确定模块包括: 接收子模块,用于接收额外参数;所述额外参数中包括所述一路或多路流媒体视频新的权值; 第二调整子模块,用于将所述新的权值作为所述一路或多路流媒体视频的权值。 9. The apparatus of claim 7 or claim 8, wherein the determining module comprises: a receiving sub-module for receiving extra parameters; the additional parameter comprises one or more branches of the new streaming video weight; a second adjusting sub-module, new weights for the weight value as the one or more channels of streaming video.
10. 根据权利要求7所述的装置,其特征在于,还包括: 计算模块,用于计算所述一路或多路流媒体视频的权值之和获得第二总权值; 判断模块,用于判断所述第二总权值是否不等于预设数值;若是,则调用第三调整模块; 第三调整模块,用于计算所述一路或多路流媒体视频的权值,与预设数值和第二总权值的比值之积,作为所述一路或多路流媒体视频的权值。 10. The apparatus according to claim 7, characterized in that, further comprising: calculating means for calculating the path all the way to the right or streaming video obtain a second total sum of weights; determining means for determining whether the second total weight value is not equal to a preset value; if the third adjustment module is invoked; third adjustment module, configured to calculate weights of the one or more channels of streaming video, the preset value and the volume ratio of the second total weight, as the weight of one or more channels of video media stream.
CN201610270343.9A 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control CN105898384B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Publications (2)

Publication Number Publication Date
CN105898384A true CN105898384A (en) 2016-08-24
CN105898384B CN105898384B (en) 2019-03-22

Family

ID=56701955

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610270343.9A CN105898384B (en) 2016-04-26 2016-04-26 A kind of method and apparatus of streaming media video mixing frame per second control

Country Status (1)

Country Link
CN (1) CN105898384B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107005703A (en) * 2016-09-09 2017-08-01 深圳市大疆创新科技有限公司 Image coding method and system

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1141106A (en) * 1994-12-19 1997-01-22 摩托罗拉公司 Multimedia multiplexing device and method using dynamic packet segmentation
CN1149374A (en) * 1994-04-20 1997-05-07 汤姆森消费电子有限公司 Multiplexer system using constant bit rate encoders
JP2006054801A (en) * 2004-08-16 2006-02-23 Kddi Corp Video information composition simultaneous encoding apparatus
CN101527834A (en) * 2009-03-26 2009-09-09 浙江大华技术股份有限公司 Wireless narrowband network video transmission method
US20110191449A1 (en) * 2003-04-30 2011-08-04 Akamai Technologies, Inc. Automatic migration of data via a distributed computer network
US20110243218A1 (en) * 2002-12-16 2011-10-06 Xiaochun Nie Method of implementing improved rate control for a multimedia compression and encoding system
CN102427532A (en) * 2011-10-24 2012-04-25 成都虢电智能电力科技有限公司 Multipath video adapting coding transmission method under limited bandwidth and transmission system thereof
US20120246689A1 (en) * 2005-07-22 2012-09-27 Genevieve Thomas Buffering content on a handheld electronic device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1149374A (en) * 1994-04-20 1997-05-07 汤姆森消费电子有限公司 Multiplexer system using constant bit rate encoders
CN1141106A (en) * 1994-12-19 1997-01-22 摩托罗拉公司 Multimedia multiplexing device and method using dynamic packet segmentation
US20110243218A1 (en) * 2002-12-16 2011-10-06 Xiaochun Nie Method of implementing improved rate control for a multimedia compression and encoding system
US20110191449A1 (en) * 2003-04-30 2011-08-04 Akamai Technologies, Inc. Automatic migration of data via a distributed computer network
JP2006054801A (en) * 2004-08-16 2006-02-23 Kddi Corp Video information composition simultaneous encoding apparatus
US20120246689A1 (en) * 2005-07-22 2012-09-27 Genevieve Thomas Buffering content on a handheld electronic device
CN101527834A (en) * 2009-03-26 2009-09-09 浙江大华技术股份有限公司 Wireless narrowband network video transmission method
CN102427532A (en) * 2011-10-24 2012-04-25 成都虢电智能电力科技有限公司 Multipath video adapting coding transmission method under limited bandwidth and transmission system thereof

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107005703A (en) * 2016-09-09 2017-08-01 深圳市大疆创新科技有限公司 Image coding method and system

Also Published As

Publication number Publication date
CN105898384B (en) 2019-03-22

Similar Documents

Publication Publication Date Title
US6989868B2 (en) Method of converting format of encoded video data and apparatus therefor
KR100557103B1 (en) Data processing method and data processing apparatus
JP4309098B2 (en) Method for transmitting hierarchical video coding information
US8019175B2 (en) Region-of-interest processing for video telephony
US8977063B2 (en) Region-of-interest extraction for video telephony
CN101507278B (en) Techniques and method for variable resolution encoding and decoding of digital video
US9635314B2 (en) Techniques for managing visual compositions for a multimedia conference call
US8107524B2 (en) Adaptive bandwidth footprint matching for multiple compressed video streams in a fixed bandwidth network
CN1324890C (en) Adaptive alternation-removing method of dynamic image and its equipment
US20120179833A1 (en) Method and apparatus for adapting media
EP2441265B1 (en) Motion based dynamic resolution multiple bit rate video encoding
JP2008533853A (en) Quasi-constant quality rate control with look-ahead
US9521411B2 (en) Method and apparatus for encoder assisted-frame rate up conversion (EA-FRUC) for video compression
JP3516585B2 (en) Data processing apparatus and data processing method
KR100471176B1 (en) Picture decoding device and sound decoding device
CN100359945C (en) Moving picture distribution system, moving picture distribution device and method, recording medium, and program
US7898950B2 (en) Techniques to perform rate matching for multimedia conference calls
JP2009533008A (en) Temporal quality metrics for video coding.
US9071841B2 (en) Video transcoding with dynamically modifiable spatial resolution
AU2006223416A1 (en) Content adaptive multimedia processing
KR20070114798A (en) Content-adaptive background skipping for region-of-interest video coding
BRPI0616426A2 (en) scalability techniques based on content information
WO2008007228A2 (en) System and method for video distribution and billing
KR20070063564A (en) A method and system for generating multiple transcoded outputs based on a single unit
JPH0837662A (en) Picture encoding/decoding device

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
COR Change of bibliographic data
GR01 Patent grant