WO2006035438A1

WO2006035438A1 - Media player and method for operating a media player

Info

Publication number: WO2006035438A1
Application number: PCT/IL2005/001040
Authority: WO
Inventors: Israel Safran; Lior Ben-Gigi; Yaniv Ben-Arie
Original assignee: Dvtel Inc.
Priority date: 2004-09-28
Filing date: 2005-09-28
Publication date: 2006-04-06

Abstract

The invention provides a media player comprising a controlled renderer for rendering media frames for outputting to one or more display devices. The renderer is configured to select frames from an input data stream according to one or more predetermined criteria for output to a display device. The invention also provides a method for operating a media player. The method comprises selecting frames from an input data stream according to one or more predetermined criteria for output to a display device.

Description

MEDIA PLAYER AND METHOD FOR OPERATING A MEDIA PLAYER

FIELD OF THE INVENTION

This invention relates to media player

BACKGROUND OF THE INVENTION

A key element in the modern networked environment is streaming of multi-media over LAN (Local Area Network), WAN (Wide Area Network), Internet, cellular and other networks. To achieve high performance while efficiently using resources, such as bit rate, that are inherently limited, sophisticated standards have been developed. For example, an important class of standards, including the H.261, H.263, H.264 and MPEG-4 standards, is used for streaming video. The compression algorithm used in these cases is based on the principle that temporal and spatial redundancy in motion pictures make up the majority of the visual information that humans perceive. By comparing changes from frame to frame and removing as much of the same (redundant) information as possible, the data indicative of a frame may be encoded in a way that substantially reduces the data storage and transfer requirements for the medium. Thus, the need to transmit data indicative of the full picture content for each frame is eliminated, enabling significant savings in certain parameters such as bit rates and data size. This method, however, necessitates processing power at the receiving end to decode, in near real time, the encoded data in order to reconstruct the picture to be displayed. Inadequate processing power of the media player to cope in real time with the incoming stream rate may result in degradation of the video quality. Fig. 1 shows a prior art multimedia player 100 that is used to handle an incoming data input stream 1. The player 100 comprises three basic components. A network interface 2 receives data packets of the input stream 1 from a source (not shown), and stores them in an associated buffer 16 for subsequent processing. A decoder 4 retrieves data packets from the buffer 16 and analyzes the retrieved data for information that is needed to prepare the data for presentation. This includes, for example, decoding the data required for generation of a picture to be displayed on a computer monitor. A Tenderer 6 uses the decoded data to generate data indicative of a frame and sends the data to a display device 18. For example, the decoded data may be RGB data of a video image to be displayed on a computer monitor.

When implemented as a software module, a player uses central processing unit (CPU) resources to execute its tasks. When the central processor performs other tasks simultaneously with operating the player, CPU utilization time is shared by several tasks under the control of the operating system, including the player. The frame rate of the media input stream to be displayed is determined by the encoding element of the source where the bit stream originated. Since the player has to process data that is continuously streamed over the network from an independent source, it should be allocated sufficient CPU time in order to properly process incoming data prior to the arrival of new data. If the CPU power allocated to the player at any moment is not sufficient for real time performance of all the tasks of the player, degradation in the quality of the displayed video occurs that may be evident in delays or artifacts. Examples of such phenomena are: Delayed presentation of video. This occurs when the network interface 2

(a high priority task) is capable of retrieving all data packets, but the subsequent stages (decoding, rendering) fail to process the data in a timely manner and thus the actual display of the picture is delayed. The delay may vary per picture frame (dependent upon the momentary CPU load) and thus a video stream may be displayed with a distorted or non-constant timing. Loss of data. - During periods of CPU overload, the network interface 2 might not be able to cope with the rate that the data packets are received, resulting in loss of data packets. In the case of video streaming, this results in a distorted picture in the form of artifacts, and it might take some time to regain a high quality picture when sufficient CPU resources are subsequently allocated to the player.

SUMMARY OF THE INVENTION

The present invention provides a media player and methods and systems for managing CPU power allocated to a media player. The player may be, for example, an audio player, a video player, or a multimedia player. The player receives data packets in a data stream from a source. The player is used together with a device, referred to herein generally as a "display device", "output device" or "window" for displaying frames. For example, if the player is a video player, the associated display device might be one or more monitor screens. In the case that the player is an audio player, the display device is one or more speakers. For a multimedia player, the display device might be a combination of one or more monitor screens and one or more speakers.

In its first aspect, the present invention provides a media player. The player of the invention has a controlled renderer that uses a selection criterion to select frames for display on the display device based on directives received from a renderer control. The selection criterion selects some of the frames, but not necessarily all of the frames, from the frames that were decoded from the input data stream for display on the display device so as to reduce the processing power required of the CPU while providing a continuous display of frames. In one preferred embodiment of the invention, the selection criterion selects frames at a predetermined spacing between consecutive frames that is larger than the spacing between frames in the initial input stream. In this embodiment, a "forecasted time for display" is calculated for each input frame, and the controlled renderer selects for display the frame having a forecasted time - A -

for display at or just before the time at which a frame is to be displayed. In a most preferred embodiment, the display rate of the displayed frames does not exceed a maximum rate that is at or slightly above the minimum display rate that is optimal for the specific application - which in some cases may be lower than the transmitted frame rate. For example, for a video player with a large monitor screen, a display rate of 30 frames per second is around the minimum rate that produces an impression of smooth and continuous display of the video frames. Thus, in an input data stream having an input rate significantly above 30 frames per second, data frames are discarded so as to produce a data stream for display having a frequency at or slightly above 30 frames per second.

The frame rate of the output stream to be displayed on one or more display devices or on one or more fields on a single display device, maybe dynamically changed based on criteria derived from prevailing conditions that may be internal or external to the application. Examples of such circumstances might include the size of a window or field on the display device, the relative importance of a window, the number of opened windows, etc. The frame rate of the output streams to be displayed may be changed dynamically to match the current CPU power allocated to the player. The displayed frame rate for one or more display devices is lowered when the CPU power allocated to the display device is low. When multiple copies of the same stream are to be output to two or more display devices, duplication of processing may be avoided by implementing a common decoder for all of the output devices while implementing a separate Tenderer for each display device.

According to the present invention, there is also provided a method for degradation of the quality of displayed video when the required CPU power exceeds the available power. This is achieved by raising the priority of the network interface and the decoding processes and lowering the priority of the rendering process. This reduces the chances of losing synchronization with the incoming stream data, and reduces the occurrence of serious artifacts. During periods of CPU power allotment to the player that is insufficient to process the input data stream in real time, large volumes of data may accumulate in the pre-renderer buffer waiting for further processing. In a preferred embodiment of the invention, the buffered data are selectively processed in order to reduce or eliminate the risk of a subsequent period of CPU power allotment that is insufficient to process all of the accumulated data.

It will also be understood that the system according to the invention may be a suitably programmed computer. Likewise, the invention contemplates a computer program being readable by a computer for executing the method of the invention. The invention further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing the method of the invention.

Thus, in its first aspect, the invention provides a media player comprising a controlled renderer rendering media frames for outputting to one or more display devices, the renderer being configured to select frames from an input data stream according to one or more predetermined criteria for output to a display device.

In its second aspect, the invention provides a method for operating a media player comprising rendering media frames for outputting to one or more display devices, the rendering comprising selecting frames decoded from an input data stream according to one or more predetermined criteria for output to a display device.

In its third aspect, the invention provides a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for operating a media player comprising rendering media frames for outputting to one or more display devices, the rendering comprising selecting frames from a decoded input data stream according to a predetermined criterion wherein at least one frame in the input stream may not be output to a display device. In its fourth aspect, the invention provides a computer program product comprising a computer useable medium having computer readable program code embodied therein for operating a media player the computer program product comprising rendering media frames for outputting to one or more display 5 devices, the rendering comprising selecting decoded frames from an input data stream according to a predetermined criterion wherein at least one frame in the input stream may not be output to a display device.

In its fifth aspect, the invention provides a computer program comprising computer program code means for performing all the steps of the method of the i o invention when said program is run on a computer.

In its sixth aspect, the invention provides a jitter filter for use in the media player of the invention, the jitter filter being configured to calculate, for at least one frame in an input stream, a forecasted time to display.

15 BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it may be carried out in practice, a preferred embodiment will now be described, by way of non- limiting example only, with reference to the accompanying drawings, in which:

Fig. 1 shows a simplified generalized block diagram of a prior art multi- 20 media player implemented as a software module; and

Fig. 2 shows a block diagram of a multi-media player in accordance with one embodiment of the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

25 Fig. 2 shows a media player 102 in accordance with one embodiment of the invention. The player may be, for example, an audio player, a video player, or a multimedia player. As in the prior art player 100 shown in Fig. 1, the player 102 has a network interface 2 that receives the input stream 1 and stores the data packets of the input stream 1 in the associated buffer 16. From the buffer 16 the undecoded frames 3 are routed to the decoder 4. The player 102 includes a jitter filter 8. The jitter filter 8 receives the undecoded frames from the buffer 16 and calculates a "forecasted time to display" 9 of each received frame. The forecasted time to display is obtained by adding to the "time received" (the time the undecoded frame 3 is received from the network interface module) the "average time between received pictures ", the later being continuously updated by the jitter filter 8 based on a predetermined number of the most recent frames. The forecasted time to display 9 generated by the jitter filter 8 is attached to the decoded picture 5 generated by the decoder 4 for the same frame. The forecasted time to display 9 and the decoded picture 5 are placed in a pre-renderer buffer 10 which is a cyclic buffer. The controlled renderer 14 selectively retrieves a selected frame 12 from the pre-renderer buffer 10, based on one or more selection criteria received through the renderer control 13, and sends each selected frame 12 to the display device 18 for display. In this respect, the pre- renderer buffer 10 acts as a "flexible joint" that is written to by the decoder 4 and read from by the renderer 14, each of these being a separate task that is executed independently of the other. The selection criteria select some of the frames for display, but not necessarily all of the frames, so as to reduce the processing power required of the CPU while providing a continuous display of frames. Examples

This example considers video data that are generated and input to the processor 102 at an input rate of 25 frames per second, and are to be rendered at a frame rate of 12.5 frames per second. The process which takes place in order to achieve this is described in Table 1. The "Time Received" column 15 lists the time attached to each frame in the "undecoded Frames " 3 stream. Note that, even though, originally, consecutive frames were spaced by exactly 0.04 sec, at the receiving end there is a "jitter " in the time difference, due to the properties of the network transmission. The "Time Between Frames " column 16 presents the time difference between a frame and the previous frame. Thus, as shown in Column 15, Frame 21 was received at time 0.001 sec, and Frame 22 was received at time 0.0039 sec. Hence, as shown in column 16, Frame 22 arrived 0.038 sec after Frame 21. The "Average Time Between Frames" column 17 presents the average time between frames for a predetermined number of the most recent frames. In the example of Table 1, the predetermined number is the 8 most recent frames (this is referred to herein as an "8-stage jitter filter"). For example, the average time between frames for Frames 21 to 28 is 0.0396 sec. The "Forecasted Time to Display" column 18 presents, for each input frame the time at which it would be displayed if no special treatment were to be carried out by the controlled renderer 14, and it is the sum of the "Time received" for the frame plus the "Average Time Between Frames " of the previous 8 frames. For example, the forecasted time to display of the Frame 28, (0.3176 sec) is the sum of the time Frame 28 was received (0.278 sec) and the average time between frames prior to Frame 28 (0.0396). The resulting "forecasted time to display" shown in Column 18 is attached to the decoded picture 5 and stored in the pre- renderer buffer 10.

TABLE 1 Overview of the possessing by the controlled Tenderer.

Since, in this example, the Tenderer 14 is required to render frames at 12.5 frames per second, it will request frames from the pre-rendering buffer 10 using a selection criterion that selects frames 0.08 sec apart. In response to each request, the controlled renderer 14 will receive a frame best fitting the request, which is the frame having a "Forecasted Time to Display " at or immediately prior to the time defined by the selection criteria 11. In the example of Table 1, this means that the Frame 8 which is associated in the pre-renderer buffer 10 with the time 0.3176 sec will be displayed at time 0.32 sec. The selected frames will be displayed at the times shown in the "Actual Time Displayed" column 19. In this example approximately 50% of the frames are discarded (as indicated in column 20) as would be expected when rendering a stream at half its input frame rate. The displayed frames are thus refreshed at a constant rate, unlike the jittery input stream.

Table 2 presents another example in which an input stream of 25 frames per second is displayed at 10 frames per second. (For simplicity, in Table 2 and all subsequent examples, no jitter is shown in the "Time Received" column).

TABLE 2 Handling of data frames when an input video stream of 25 frames per second is displayed at 10 frames per second.

The embodiments of the invention in the above examples provide control and management of CPU utilization in various aspects, such as:

1. Selectively displaying video at lower frames per second than the source frames per second.

A relevant scenario might be application directed - the application can specifically direct the renderer to generate the video output at a reduced rate relative to the source input. Examples for such cases are:

Small windows. It may be demonstrated that while 25-30 frames per second are required to show smooth motion on a large window, a frame rate of 10-12 frames per second is sufficient for perception by a human viewer if the same information is displayed on a small window. Thus upon resizing of a window the frame rate may be adjusted, reducing CPU utilization without harming the viewing experience.

Low priority display. In certain applications, some video windows may be assigned lower priority than others. In such cases, it is often possible to view the lower priority windows at a lower frame rate, thus reducing CPU utilization.

Another possible scenario is presented by cases in which the input frame rate from the source is higher than required for a human observer, so that the data stream contains redundant or excessive information. An example might be a situation in which video originally recorded at 25 frames per second is streamed for playback at double speed - namely a theoretic rate of 50 frames per second. As there is no added value in displaying video at rates higher than 30 frames per second, discarding part of the frames will not cause any operational degradation to the viewing experience. Table 3 presents this example.

Additionally, the frame rate for selected video windows may be determined to match the CPU power currently allocated to the player.

TABLE 3: Handling of data frames with a displayed video frame rate truncated to the minimum rate providing best usable value.

2. Improved handling of multiple copies of the same stream.

If multiple copies of the same stream are to be displayed on several windows, only the Tenderer module is separately implemented for each window. In addition to the elimination of redundant network interface and decoder modules, this also enables flexibility in setting a different frame rate for each copy of the displayed data. Table 4 presents a situation in which an input stream at 25 frames per second is displayed on two windows - one at 25 frames per sec. and the other at 12.5 frames per sec. TABLE 4: Handling of data frames to be displayed on two windows at different frame rates.

An example of a scenario in which this feature is useful is a security system in which video streams from many cameras are displayed in small windows, and one stream is also analyzed in more detail - displayed on a large widow.

3. Graceful recovery after periods of peak CPU utilization

When the CPU is overloaded, the network interface module is assigned higher priority, resulting in input data packets being buffered for later processing by the decoder and the renderer. When CPU power becomes available again, a typical scenario would result in the decoder and renderer having to cope with large quantities of buffered data, leading once again to overloading of the CPU. In a preferred embodiment of the invention, the player 102 deals with this problem, by discarding data from the buffer 16 while maintaining the time smoothness of the video stream.

An example of such a situation is presented in Table 5. As can be seen, when the CPU is overloaded, the "Time Between Frames " increases, and consequently also the "Average Time Between Frames ". Once adequate CPU power becomes available again, buffered data is made available at a high rate, resulting in short "Time Between Frames ", and consequently also the "Average Time Between Frames ". Processing all the data now made available to the renderer will result in a significant increase of CPU load. Additionally, the variability in the timing between the frames may result in a very jittery picture. The invention reduces or prevents overload of the CPU by discarding some frames from the pre- renderer buffer 10 even if the requested display frame rate is the same as the input frame rate, while maintaining and possibly improving the smoothness in time of the displayed pictures. TABLE 5: CPU overload scenario

Referring still to Table 5, it may be seen that until Frame 142, the "Actual Time Displayed" (0.48 sec) is the same as the "Forecasted Time to Display". However, for frame 143, the "Actual Time Displayed" should be 0.52 sec, but there is no new frame available with "Forecasted Time to Display" less than or equal to 0.52 sec (note that the last available frame, frame 142, was already displayed at time 0.48 sec). Therefore the renderer waits till a new frame is available - in this case Frame 143, since it has the "Forecasted Time to Display" of 0.54 sec, and the "Actual Time Displayed" is thus set to 0.54 sec. In order to try and maintain the desired display frame rate, as the last time in which a picture was displayed was 0.54 sec, the desired time to display of the next frame should be set to 0.58 sec (an increment of 0.04 sec). Table 6 summarizes this process, introducing the column "Desired Time to Display " which is what used to be in Table 5 "Actual Time Displayed", now dynamically modified to accommodate the gaps in available data. The same scenario (delayed "Actual Time Displayed" resulting in updating "Desired Time to Display") continues as long as the CPU overload causes delays in frames being forwarded for processing. The "Desired Time to Display " of Frame 144, 0.58 sec, is delayed until an "Actual Time Displayed" of 0.61 sec, which sets the "Desired Time to Display" of Frame 145 to 0.65 sec, and so on. Once processing power is regained, the renderer may find itself with "too many" new (not yet displayed) frames in the buffer. This may be seen at Frame 150 which has a "Desired Time to Display" of 0.96 sec, in which there are three new frames with "Forecasted Time to Display" of 0.96 sec or less. Only one of them is used, and the other two are discarded.

Claims

CLAIMS:

1. A media player comprising a controlled Tenderer rendering media frames for outputting to one or more display devices, the renderer being configured to select frames from an input data stream according to one or more predetermined criteria for output to a display device.

2. The media player according to Claim 1 being a video player, an audio player, or a multimedia player.

3. The media player according to Claim 1 or 2 wherein the one or more display devices includes a display screen or a speaker.

4. The media player according to any one of the previous claims wherein the predetermined criterion specifies that frames are output to a display device at a frame rate that is less than a frame rate of the input stream.

5. The media player according to any one of the previous claims further comprising a jitter filter configured to calculate, for at least one frame in the input stream, a forecasted time to display.

6. The media player according to Claim 5 wherein the predetermined criterion comprises:

(a) determining a time at which a frame is to be displayed;

(b) determining, for each of a plurality of frames in the input stream, a forecasted time to display; and .

(c) selecting a frame having a forecasted time to display not later than the determined time at which a frame is to be displayed.

7. The media player according to Claim 6 wherein the selected frame is the latest frame having a forecasted time to display not later than the determined time at which a frame is to be displayed.

8. The media player according to Claim 6 or 7 configured to display on a display device the selected frame at the determined time at which a frame is to be displayed.

9. The media player according to any one of Claims 5 to 8 wherein the forecasted time to display is calculated for at least one frame using an algorithm involving a time at which the fame was received and an average time between consecutive frames.

10. The media player according to Claim 9 the forecasted time to display is calculated for at least one frame as a sum of the time the frame was received and the average time between frames.

11. The media player according to any one of the previous claims wherein the criterion includes selecting frames for display at a frame rate calculated in a process involving a current CPU power allocated to the media player.

12. The media player according to Claim 11 wherein the display frame rate is lower the lower the current CPU power allocated to the media player.

13. The media player according to any one of the previous claims configured to discard a fraction of the decoded frames from the input stream prior to rendering of frames.

14. The media player according to Claim 13 wherein the fraction of discarded frames is determined based upon a CPU power allocated to the player.

15. The media player according to Claim 14 wherein the lower the current CPU power allocated to the media player the greater the fraction of discarded frames.

16. A method for operating a media player comprising rendering media frames for outputting to one or more display devices, the rendering comprising selecting frames decoded from an input data stream according to one or more predetermined criteria for output to a display device.

17. The method according to Claim 16 wherein the media player is a video player, an audio player, or a multimedia player.

18. The method according to Claim 16 or 17 wherein the one or more display devices includes a display screen or a speaker.

19. The method according to any one of Claims 16 to 18 wherein the predetermined criterion specifies that frames are output to a display device at a frame rate that is less than a frame rate of the input stream.

20. The method according to any one of Claims 16 to 19 further comprising calculating, for at least one frame in the input stream, a forecasted time to display.

21. The method according to Claim 20 wherein the predetermined criterion comprises:

(a) determining a time at which a frame is to be displayed;

(b) determining, for each of a plurality of frames in the input stream, a forecasted time to display; and

22. The method according to Claim 21 wherein the selected frame is the latest frame having a forecasted time to display not later than the determined time at which a frame is to be displayed.

23. The method according to Claim 21 or 22 further comprising displaying on a display device the selected frame at the determined time at which a frame is to be displayed.

24. The method according to any one of Claims 20 to 23 wherein the forecasted time to display is calculated for at least one frame using an algorithm involving a time at which the fame was received and an average time between consecutive frames.

25. The method according to Claim 24 the forecasted time to display is calculated for at least one frame as a sum of the time the frame was received and the average time between frames.

26. The method according to any one of Claims 16 to 25 wherein the criterion includes selecting frames for display at a frame rate calculated in a process involving a current CPU power allocated to the method.

27. The method according to Claim 26 wherein the display frame rate is lower the lower the current CPU power allocated to the player.

28. The method according to any one of Claims 16 to 27 further comprising discarding a fraction of the frames decoded from the input stream prior to rendering of frames.

29. The method according to Claim 28 wherein the fraction of discarded frames is determined based upon a CPU power allocated to the player.

30. The method according to Claim 29 wherein the lower the current CPU power allocated to the method the greater the fraction of discarded frames.

31. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for operating a media player comprising rendering media frames for outputting to one or more display devices, the rendering comprising selecting frames from a decoded input data stream according to a predetermined criterion wherein at least one frame in the input stream may not be output to a display device.

32. A computer program product comprising a computer useable medium having computer readable program code embodied therein for operating a media player the computer program product comprising rendering media frames for outputting to one or more display devices, the rendering comprising selecting decoded frames from an input data stream according to a predetermined criterion wherein at least one frame in the input stream may not be output to a display device.

33. A computer program comprising computer program code means for performing all the steps of any of Claims 16 to 30 when said program is run on a computer.

34. A computer program as claimed in Claim 33 embodied on a computer readable medium.

35. A jitter filter for use in the media player of any one of Claims 5 to 10, the jitter filter being configured to calculate, for at least one frame in an input stream, a forecasted time to display.