GB2509313A

GB2509313A - Managing a queue of video frames using the current queue fill level

Info

Publication number: GB2509313A
Application number: GB1223407.6A
Authority: GB
Inventors: Mirko Gunther; Mirek Chalinski
Original assignee: Barco NV
Current assignee: Barco NV
Priority date: 2012-12-27
Filing date: 2012-12-27
Publication date: 2014-07-02
Also published as: GB201223407D0; BE1022040B1; WO2014102337A1

Abstract

The invention relates to a method for managing the queue (30) of video frames between a decoder (10) and a renderer (30), based on determining the current fill level (P) of the queue and adjusting the delivery to the renderer based on at least the current fill level. The method may also involve dropping and delaying frames, as well as repeating a previous frame. The method may also include adding information, such as a time indication, to the frames.

Description

A METHOD AND SYSTEM FOR MANAGING A QUEUE OF VIDEO FRAMES

Field of the invention

The invention relates to methods, systems and software for managing one or more queues of video frames.

Background

In control room and monitoring applications compressed video streams, from encoders of different vendors, are delivered as network packets across non-trivial network architectures. Expectation is that these video streams will be displayed in an evenly distributed manner over time so that the viewer perceives fluid constant movements.

Several network video standards provide so called presentation timestamps (P15) for each frame. Also frame rates may be encoded in the video streams. With the help of frame rate and the P15 information the video frame distribution over time can be reconstructed. Unfortunately in reality this is not always feasible because: (1) Not all encoder vendors support correct P15. This includes missing PISs or even wrong calculated PTSs.

(2) Not all encoders support valid frame rate information in their streams.

(3) Delays may happen on network switches, transmitters or receivers which introduce network jitter to the network packets of the video frames.

(4) Due to changing bandwidth requirements on CPUs during the encoding process it may also happen that jitter will be introduced to network stream.

Usually a queue is used to eliminate the jitter of the incoming stream. The frames are stored in the queue as they are received from the decoder. The incoming frames are stored in this queue during, for example, a predetermined amount of time. The frames can then be de-queued by the renderer at a constant output rate. When the average reception rate of the frames in the queue is equal to the constant output rate, then the queue allows de-queuing the frames at a constant output rate although the frames were received at an irregular rate. This eliminates the irregular arrival times, but introduces an additional delay and therefore increases the end-to-end latency. If the queue delay is set to a value large enough to cover the worst case jitter, then a regular delivery of frames to the renderer would be guaranteed. However in practical applications this would result in a long queue delay and in an unacceptable end-to-end latency.

In contrast to the above described fixed jitter queue management, several adaptive jitter queue management schemes were proposed. In 6,684,273 B2 a method is disclosed to adapt the queue size at the measured delay of the received packets. In 7,983,309 B2 a buffering time for one or more frames received by a frame buffer is determined based on a specific buffering time associated with a specific frame and on information representative of specific amount of data stored in the frame buffer.

Aim of the invention The aim of the invention is to presents methods, systems and software which requires neither presentation time stamps nor the frame rate information encoded in the video frames while enabling fluid and constant presentation of arbitrary video streams

Summary of the invention

The various aspects of the invention are described in the claims ito 38.

In a first aspect of the invention a method for managing a queue of video frames, typically between a decoder and a renderer ( processing unit for such decoded video frames) is disclosed wherein the method comprising determining the current fill level of the queue (e.g. counting the amount of video frame in the queue); and adjusting or adapting the delivery to the renderer by comparing the current fill level of the queue with a first predetermined fill level to hold the current fill level P as close as possible to this first predetermined fill level.

In an embodiment thereof the step of adjusting the delivery is operable only when the current fill level is larger than or equal to a second predetermined fill level. If adjusting the delivery is not operable then the delivery of the video frames is delayed, more preferably with a delay depending on the difference (e.g. proportional) between the current fill level and the second predetermined fill level.

In an embodiment thereof in case a reliable estimate of the video frame rate can be computed, time information is tagged to a video frame before the video frame is delivered to the renderer, more in particular such reliable estimate is deemed available when a predefined minimum amount of video frames has passed through the queue and/or when the video frame rate (as estimated) exceeds a predefined minimum frame rate, more preferably the estimate of the video frame rate is determined from the accumulated time of a predetermined amount of video frames (preferably equal to the predefined minimum amount).

The method of the first aspect of the invention and its embodiments which can be combined is part of handling video frames by a renderer, whereby the renderer drops the video frame if delivered when the renderer's video frame memory is full and/or those handling includes tagging a delivered video frame with a time indication; and if the renderer with its display rate cannot display the video frame such that the time indication tagged to the video frame falls within a predetermined time window, the renderer drops the video frame; and/or the renderer displays the video frame in accordance with the tagged time indication.

In a second aspect of the invention a controller suited for managing the queue of a system, comprising a decoder and a queue for temporarily storing a plurality of video frames and a renderer, wherein the controller comprises means (hardware or a combination of hardware and software) for executing any of the methods presented.

In a third aspect of the invention a system is provided, the system comprising a decoder, a queue for temporarily storing a plurality of video frames and a renderer, further comprising a controller for managing the queue, the controller comprising means for executing any of the methods presented.

In a fourth aspect of the invention a computer program product and related machine readable signal storage medium, comprising code segments that when executed on a suitable processing engine implement those steps in any of the methods presented.

Brief description of the drawings

Figure 1: Example queue fordecoded frames with Qmax = , Qmiu = 3 and P = 4 Figure 2 shows a flow chart of the method in accordance with an embodiment of the invention.

Figure 3 shows a schematic of the invented systems and subsystems.

Figure 4 shows a generic flowchart of the invented methods.

Detailed description of the invention

The invention provides for methods, systems and software for queue management, enabling amongst other evenly display in time of the video frames. A method is disclosed where the jitter of the incoming stream is eliminated for a minimum increase of the latency, with a constant queue size. The queue or frame buffer has a closed loop control system to keep the fill level of the queue constant. The average frame rate of the decoder output stream can be different from the display frame rate.

Note that the invented methods can be implemented in a dedicated ASIC or be executed as software on a general purpose unit or a graphic processing unit or a combination of those.

Embodiment In order to overcome missing or incorrect PTS (presentation time stamp) and frame rate information and obtain a fluid and even presentation of a video stream, we hereby apply a frame buffer or queue in a sort of closed loop control system as a solution to the above described problem. In the first part of the description it is assumed that the average frame rate of the decoder output is equal to the display frame rate although the invention is not limited thereto.

Frame rate calculation The average frame rate of the decoder's output stream is denoted f5. Since no in-band frame rate can be used, we calculate the average frame rate J by deriving it from the accumulated period of time Taccn of ii previously decoded frames. Preferably this will be calculated as a moving average. The average frame rate will be updated at the end of the decoding of each frame. The average frame duration T5 = will be updated according to f. The average frame rate is calculated over time intervals of for instance the least 60 frames which were received by the queue.

for n»=60 CC Cit As long as n C 60 frames, the frame rate will not be used to generate new frame rate and time information. If a frame rate below 10 frames per second is calculated, the frame rate is not used to generate new frame rate and time information too, because such a small frame rate indicates in most cases either an instable stream or a signal lost/found situation. If the stream has a frame rate below a certain value, for instance fps, because it's a thumbnail stream or similar, the practical experience showed, it's better to let the renderer to do its best to render such a stream.

Queue for decoded frames Decoded frames will be queued in the frame buffer. See Figure 1. The maximum queue size is denoted Qmax and the queue's minimum fill level is denoted Qmjn. The actual fill level is denoted P. In general: 0 «= Qntin <Qmax If the queue is full (P = Qma.3, new frames to be queued shall drop the oldest frames in the queue.

The choice of Qmax and is important. The memory size that is needed to store the frames in the queue is proportional with (Qmax + 1). The queue delay is proportional with (Qm + 1)because (Qm + 1) is the number of frames that must be kept in the queue. It is therefore important to find a minimum value of Qmax and Q that in practical situations will not result in queue overrun or under-run.

Closed loop to keep the queue fill level equal to Qmjn The current fill level of the queue is used as the closed loop control value. The task of the closed loop is to hold the current fill level P as close as possible to Qmn. After starting the stream, the queue does not start to deliver the frames to the renderer as long as the fill level P < Qm. Once the minimum fill level is reached, the queue will try to hold P = Qmjn by adjusting the delivery rate to the renderer.

There are two transient situations: (1] Initially the closed loop shall wait with delivery of frames to the renderer until = Qmin is reached.

(2] If P = Qm is reached or P > Qmjn, but no can yet be calculated, the next frames shall be delivered immediately. Neither time information nor any duration will be added to the frame's information; the renderer tries to work on the frame immediately. If the renderer's internal structures are full, the frame will be dropped.

There are two regime situations: (1) If P »= Qmin and f, can be derived, the next frame shall be delivered immediately. It will be tagged with the current time t110 and its duration T, derived from J. Usually the renderer is driven by the display frame rate V-Sync of the graphics device. If the renderer can't handle the frame before + T expired, then it will drop the frame.

(2) If the P < Qm the delivery of frames out of the queue will be delayed during a time period The delay time depends on the difference between the Qmn and P: tsleep = (Q7Th -P) (equation 1) The default value of Qmax = 4 and (condition 1) the default value of Qmjn = 3 (condition 2) Equation 1 and conditions 1 and 2 were both experimentally defined for jitter-free operation lowest latency and minimum memory size using a large number of encoders from different vendors and different network conditions.

This means that the queue should have 4 frames for optimal operation (P and E [LTS, T5, T5} for < 3 The average delay Taeiay caused by the queue is: Tactay = If the delivery of frame k to the renderer is delayed for an amount of time then the rendererwill repeat frame (k-i).

If the renderer is not able to process frames, due to the fact of expired t0 -E T5 or because of overruns in the queue, then some frames will be dropped.

Queue under-run If the queue runs out of frames because of a signal loss of the stream or a too huge jitter in the stream, then the renderer has to double the last frame.

Queue over-run If the queue runs over because of a burst in the stream or because it's not possible to deliver the frames as fast as needed, the queue drops the oldest frames in the queue to make room for the new one.

De-coupling of the queue from the renderer Accepting frames into the queue and delivering frames further to the renderer must be decoupled. The queue needs to run in a thread separate from the rendering task's thread(s). This avoids that the queue capability to accept further frames gets blocked by rendering delays.

Average frame rate of the incoming stream f is different from the display frame rate Li.

The algorithm is also working if the average frame rate L of the incoming stream is different from the display frame rate fa. The goal of the queue is to de-jitter the stream.

The queue is filled with frames from the decoder at an average rate equal to J. This means that the queue also has to deliver frames to the renderer at an average rate f.

(1) fs<fa If the average frame rate of the stream is lower as the frame rate of the display, the renderer has to repeat frames by its own.

(2) fS>f If the frame rate of the stream is larger than the frame rate of the display, then the queue runs over and drops frames, because the renderer doesn't accept more data as it can process.

A flow chart of an embodiment of the invented method is found in Figure 2.

Other embodiments of the invention The invention is useful for all cases where video frames must be presented, rendered or displayed in accordance with whatever timing requirement either in absolute time or relative with respect to each other or both, for instance timing requirements related to realizing a fluid presentation of a video stream, for instance by performing an evenly presentation in time of the video frames.

Further it is clear that the invention is equally applicable to situations where multiple queues are used (e.g. related to different type of video frames) and that the parameters such as minimum buffer fill level, maximum buffer fill level, delay computation parameters can be different per queue.

The invention manages one or more queues (denoted frame buffer(s)) for decoded frames, wherein decoded frames will be queued, more in particular the invented methods handles one or more of different situations such as (1) if the queue is full = Qmax) or nearly full, a replacement strategy (e.g. new frames to be queued shall drop the oldest frames in the queue or alternative strategies) or (2) other situations.

As presented above a closed loop to keep the queue fill level equal to Qmjn can be used but actually any target number between Qmjn and Qmax can be selected. The current fill level of the queue is used as the control value and any control strategy (for instance PID) can be used thereon (open loop, closed loop) based on the present current fill level P, but also earlier values or even predicted values., preferably with the task to hold the current fill level P as close as possible a target value (either fixed or also time varying) Naturally the invented methods are capable to recognize initialization or transient aspects to fill the queue.

In an embodiment of the invented methods computed or estimated frame rate data is used and the invented methods (either for queue management perspective or rendering management or both) are capable to operate even when such data is not (judged to be) reliable.

Further the invented methods provide a solution to avoid blocking due to rendering delays by decoupling the accepting of frames into the queue and delivering frames further to the renderer, by using multi-threading, hence running the various sub methods in separate threads. 11.

As indicate above the invented methods may rely on frame rate calculations, based on historic averaged data, preferably these methods are invoked only when they can produce reliable data.

The invented methods operate on variables such as amount of frames in the queue, frame rate (estimates), and hence perform operations such as counting, incrementing, calculating or re-calculating (using part of previous data), but also operations on queue level itself such as adding or removing frame therefrom. The control flow of such methods show the intelligence embedded therein and can comprise of one to more branches (nested) with one or more of those branches having feedback loops.

As conclusion it can be stated that the invention relates to a method for managing the queue of video frames between a decoder and the renderer, based on determining the current fill level of the queue and adjusting the delivery to the renderer based on at least the current fill level of the queue; and combination of said method with frame estimation and/or combination on renderer control.

Figure 3 shows a system with a first subsystem (e.g. a decoder) (10), a queue (20) and a second subsystem (e.g. a renderer)(30) , a first means (e.g. CPU, GPU, ASIC, FPGA) (40), measuring queue information (current and/or previous) and creating based thereon signals for accepting frames in the queue and/or transmitting frame outside the queue, and further second means (e.g. CPU, GPU, ASIC, FPGA) (50), generating steering signals for the second subsystem. Said means may communicate with each other, and may further receive first subsystem information. Note that (30)(40) may be one system but preferably the methods running for queue and second subsystem management (or even management of the first subsystem) run an own thread of control.

Figure 4 shows a general flow chart with a method (100) of determining a delivery rate (as part of managing the queue fill level), a method (120) of managing the rendering (as part of establishing quality display) and/or a method (130) of determining estimated frame rate information, more in particular the fill level (105) is inputted and leads to a delivery rate (115) while the outcome (125) is either displaying based on added time information (as based on an estimate frame rate) or displaying based on just repeating a frame or just dropping a frame. The input (135) relates to the time spend in a queue and based on the reliability of the estimate, to be tagged time information (145) is provided.

Claims

Claims 1. A method for managing the queue of video frames between a decoder and the renderer, comprising the steps of: determining the current fill level of the queue; and adjusting the delivery to the renderer by comparing at least the current fill level of the queue with a first predetermined fill level to hold the current fill level P as close as possible to this first predetermined fill level.
2. The method of claim 1, wherein said step of adjusting the delivery is operable only when the current fill level is larger than or equal too second predetermined fill level.
3. The method of claim 2, wherein when said step of adjusting the delivery is not operable, delivery of the video frames is delayed.
4. The method of claim 3, wherein the delay depends on the difference between the current fill level and the second predetermined fill level.
5. The method of claim 2, 3 or 4 wherein the first and second predetermined fill levels are equal.
6. The method of any of the preceding claims wherein if the current fill level exceeds a third predetermined fill level, new frames to be queued shall drop the oldest frames in the queue.
7. The method of any of the preceding claims, wherein in case a reliable estimate of the video frame rate can be computed, time information is tagged to a video frame before the video frame is delivered to the renderer.
8. The method of claim 7, wherein such reliable estimate is deemed available when a predefined minimum amount of video frames has passed through the queue.
9. The method of claim 7 or 8, wherein such reliable estimate is deemed available when the video frame rate (as estimated) exceeds a predefined minimum frame rate.
10. The method of claim 7, 8 or 9, wherein the estimate of the video frame rate is determined from the accumulated time of a predetermined amount of video frames (equal to the predefined minimum amount).
11. The method of claim 9 or 10, wherein this estimate is updated upon decoding of anewframe
12. The method of claim 9, 10, 11 wherein said estimate is computed as a moving average.
13. A method for handling video frames by a renderer, based on the method of any of the preceding claims for delivery of video frames from the queue between a decoder and the renderer; whereby the renderer drops the video frame if delivered when the renderer's video frame memory is full.
14. A method for handling video frames by a renderer, based on the method of any of the preceding claims for delivery of video frames from the queue between a decoder and the renderer; further comprising the steps of: tagging a delivered video frame with a time indication; and if the renderer with its display rate cannot display the video frame such that the time indication tagged tot the video frame falls within a predetermined time window, the renderer drops the video frame.
15. The method of any of the preceding claims for handling video frames by a renderer, wherein the renderer displays the video frame in accordance with the tagged time indication.
16. The method of any of the preceding claims wherein the time indication is based on the estimated video frame rate.
17. The method of any of the preceding claims for handling video frames by a renderer, wherein the renderer repeats displaying a previous video frame if other video frames are delayed.
18. A controller suited for managing the queue of a system, comprising a decoder, a queue for temporarily storing a plurality of video frames and a renderer, wherein the controller comprises means for determining the current fill level of the queue; and means for adjusting the delivery to the renderer by comparing at least the current fill level of the queue with a first predetermined fill level to hold the current fill level P as close as possible to this first predetermined fill level.
19, The controller of claim 18, wherein the means for adjusting is adapted to provide delivery only when the current fill level is larger than or equal to a second predetermined fill level.
20. The controller of claim 19, wherein the means for adjusting is adapted so that when the delivery is not operable, delivery of the video frames is delayed.
21. The controller of claim 20, wherein the means for adjusting is adapted so that the delay depends on the difference between the current fill level and the second predetermined fill level.
22. The controller of any of the claims 19 to 21 wherein the first and second predetermined fill levels are equal.
23. The controller of any of the claims 18 to 22 adapted so that if the current fill level exceeds a third predetermined fill level, new frames to be queued shall drop the oldest frames in the queue.
24. The controller of any of the claims 18 to 23, adapted so that in case a reliable estimate of the video frame rate can be computed, time information is tagged to a video frame before the video frame is delivered to the renderer.
25. The controller of claim 24, adapted so that such reliable estimate is deemed available when a predefined minimum amount of video frames has passed through the queue.
26. The controller of claim 24 or 25, adapted so that such reliable estimate is deemed available when the video frame rate (as estimated) exceeds a predefined minimum frame rate.
27. The controller of any of the claims 24 to 26, adapted so that the estimate of the video frame rate is determined from the accumulated time of a predetermined amount of video frames (equal to the predefined minimum amount).
28. The controller of claim 26 or 27, adapted so that this estimate is updated upon decoding of a new frame
29. The controller of any of claims 26, 27, 28 wherein said estimate is computed as a moving average.
30, A method for handling video frames by a renderer, based on the method of any of the claims 1 to 17 for delivery of video frames from the queue between a decoder and the renderer; whereby the renderer drops the video frame if delivered when the renderer's video frame memory is full.
31. A method for handling video frames by a renderer, based on the method of any of the claims 1 -1] or 30 for delivery of video frames from the queue between a decoder and the renderer; further comprising the steps of: tagging a delivered video frame with a time indication; and if the renderer with its display rate cannot display the video frame such that the time indication tagged tot the video frame falls within a predetermined time window, the renderer drops the video frame.
32. The method of any of the claims ito 17 or 30, 31 for handling video frames by a renderer, wherein the renderer displays the video frame in accordance with the tagged time indication.
33. The method of any of the preceding claims 1 to 17 or 30 to 32 wherein the time indication is based on the estimated video frame rate.
34. The method of any of the preceding claims 1 to 17 or 30 to 33 for handling video frames by a renderer, wherein the renderer repeats displaying a previous video frame if other video frames are delayed.
35. A system, comprising a decoder, a queue for temporarily storing a plurality of video frames and a renderer, and a controller for managing the queue, the controller being the controller of any of the preceding claims 18 to 29.
36. A computer program product comprising code segments that when executed on a suitable processing engine implement those steps in any of the methods of claims ito 17.
37. The computer program product, wherein the steps of the methods of claim 1 to 17 related to queue management and the steps of the methods of claim 1 to 17 related to renderer are separate threads.
38. A machine readable signal storage medium, storing the computer program product of claim 37.