KR20080039892A - Feedback control for adaptive video delivery - Google Patents

Abstract

A method for controlling transmission of video data in a network, including: transmitting video data to a receiver via the network; receiving channel parameter information measured by the receiver; applying a predictive function to the channel parameter information to compensate for delay times in receiving the channel parameter information from the receiver, and generating a feedback function; and adjusting the video data to be transmitted in response to the feedback function to compensate for network conditions.

Description

Feedback control for adaptive video delivery {FEEDBACK CONTROL FOR ADAPTIVE VIDEO DELIVERY}

The present invention generally relates to data delivery, and more particularly to digital multimedia content delivery for use with a network comprising one or more wired or wireless links.

Delivering audiovisual multimedia content over wired and / or wireless networks presents many challenges. Compared to wired links, in wireless networks, the challenges for real-time video streaming include time-varying variations in wireless channel quality and high bit error rates. Adaptive multimedia content delivery on a network channel that employs feedback from the receiver may be feasible if the adaptation can strictly follow the channel change in time. However, because of delays in the feedback information caused by, for example, the limited bandwidth of the feedback channel, adaptation generally lags behind channel measurements. In control systems, this delay time is sometimes known as "dead time". Conventional video adaptation algorithms do not consider dead time. As a result, systems using adaptation can introduce unwanted overcompensation based on delayed feedback information.

Although the delay time in channel information feedback is a known problem, especially for content delivery on the network, no known solution has been proposed. Thus, in a feedback / adaptation loop that deals with feedback delay and dead time in a video delivery network, it is believed to provide a controller based on control theory.

An exemplary method according to the present invention is a method for controlling the transmission of video data in a network, comprising: transmitting video data to a receiver over the network; Receiving channel parameter information measured by the receiver; Applying a prediction function to the channel parameter information to compensate for delay time upon receiving the channel parameter information from the receiver, and generating a feedback function; And adjusting the video data to be transmitted in response to the feedback function to compensate for network conditions. An example system for implementing the method is also disclosed.

Understanding of the present invention will be facilitated in view of the following detailed description of the preferred embodiments of the present invention, taken in conjunction with the accompanying drawings, in which like reference numerals refer to like parts.

1 is a simplified block diagram of a feedback control system.

2 is a simplified block diagram of an end-to-end video delivery system.

3 is a simplified block diagram of a PID control system.

4 is a simplified block diagram of the Smith Predictor control system.

5 is a simplified block diagram of an end-to-end video delivery system.

6 is a flow diagram of a control process suitable for use with a wireless digital multimedia content delivery system.

The drawings and description of the present invention have been simplified to depict relevant elements for a clear understanding of the present invention, while many other elements found in common digital multimedia content delivery methods and systems have been ignored for clarity. However, as such elements are well known in the art, a detailed discussion of such elements is not provided herein. The disclosure herein is directed to all such variations and modifications known to those skilled in the art.

Digital multimedia content can be delivered to media consumers such as decoders / televisions or a combination thereof, for example on a wireless channel, over a network, characterized by time-varying variations in channel quality and high bit error rates. To cope with the time-varying nature of the channel, information relating to channel conditions may be input back from the receiver to the transmitter to adjust the video stream (s) generated at the transmitter end. For example, because the available bandwidth changes over time, the receiver can measure or estimate the bandwidth available between the transmitter and the receiver, send this information back to the transmitter, and then the transmitter is available. Instruct the video decoder to adjust one or more data parameters, such as quantization parameters, to produce the bit-stream at a rate appropriate for the bandwidth. Another adaptation is to change the amount of redundancy in the form of forward error correction (FEC) added to the underlying multimedia content transmitted to the receiver. With the possibility of increasing packet loss in the radio channel, more FEC redundancy can be applied, so that the receiver can recover lost packets, for example.

Referring now to FIG. 1, a block representation of an end-to-end video delivery system 10 is shown. In this figure, receiver 20 typically measures radio channel 30 parameters, such as available bandwidth, delay time, and packet loss probability, and feeds back channel parameter information to transmitter 40 over feedback channel 50. Let's do it. The elapsed time between the receiver 20 measuring channel parameters and the transmitter 40 receiving these parameters is the dead time 60 for the video delivery system 10. This dead time may vary depending on the bandwidth of the feedback channel. In unreliable feedback channels, some feedback information may even be lost. Based on the feedback information, the video encoder in the transmitter may take action to adapt to changing channel conditions (eg, to adapt the generated bit rate of the encoded video stream). The loop can be thought of as a simple proportional control loop. The error (eg difference) between the target and feedback is scaled by one factor (eg proportional value) to control the upcoming generated bit rate. If the dead time is very long compared to the channel variable duration, the action taken by the transmitter will be too late. The transmitter will already be overcompensated for the channel conditions well before it. If the channel conditions change, the action taken by the transmitter can cause a much larger error in the opposite direction. This phenomenon is known as overcompensation in control theory. As a result, the actual output may become unstable and may cause vibration.

Control theory proposes a number of solutions to compensate for dead time and overcompensation. Referring now to FIG. 2, a block representation of the feedback control system 100 is shown. The time between the measurement of the actual output 120 from the process 150 and the output of the combiner 130 fed back to the controller 140 is shown by the feedback delay (float time) 160. If process 150 is embodied as, for example, a chemical plant, controller 140 may be embodied as a PID (proportional, total, derivative) controller. The PID controller compares the measured value from the process with the reference set point value. The difference (or " error " signal) is then processed to calculate a new value for the processed process output, which sets the process measured value back to its required set value. The PID controller can adjust the process output based on the history and rate of change of the error signal. The correction of the PID controller is calculated from the error in three ways: directly canceling the current error {proportional}, the amount of time continued without the error being corrected (integral), and in time. Predicting future errors from the rate of change of errors {Derivative}.

Referring now to FIG. 3, shown is a block representation of a feedback control system 200 for distributing digital multimedia content. System 200 includes a PID controller 240. By design, the PID controller continues to ramp up the output of the controller as long as there is an error between the required output 220 from the channel 250 and the actual feedback signal 230. However, in the presence of dead time, the integrator K _i / s may overcompensate. Based on the estimated dead time caused by the time delay in the feedback channel 260 in the video delivery system introduced by the combiner 235, the PID controller 240 parameter is a PID controller to handle the dead time of D seconds. Can be tuned to essentially detun For the non-limiting purpose of further explanation, the PID control loop can be tuned by adjusting the control parameters (proportional, aggregate, derivative values) to obtain the required control response. There are a number of ways to tune the PID loop. One tuning method is to first set the total (I) and derivative (D) values to zero and increase the proportional (P) value until the output of the loop vibrates. Then, the I value can be increased until the vibration stops. And finally, the D value can be increased until the loop response is fast enough.

However, when applied to systems with very long dead times, PID type controllers have disadvantages. For example, in a video delivery system, when the channel enters a "bad" state (e.g., very low available bandwidth, very high burst packet loss), the feedback latency is much longer than Long dead time occurs. An alternative technique for controlling a system with a long dead time is described by O. J. M Smith, "A Controller to Overcome Floating Time." ISA Journal. 6, pages 28-33, Smith predictor, described in 1959. Smith predictors have been proposed for plant processes with long transport delays such as contact crackers and steel mills, but it is believed that this can be generalized to control processes with long loop delays. The Smith predictor overcomes the problem of delayed feedback by using the predicted future state of the output for control. It is believed that the controller incorporating the Smith predictor is well adapted to adapt the video stream within a wireless delivery system experiencing variable channel conditions.

Referring now to FIG. 4, a block representation of a feedback control system 300 incorporating a Smith predictor is shown. System 300 includes a conventional feedback loop that circulates back to controller 340 (from process or plant 350 output 320 through delayed feedback channel 360). System 300 also includes a second feedback loop (via process or plant model 370) that introduces two additional terms into actual feedback at combiner 330. The first item represents an estimate of the plant 350 output 320 (designated as the plant model element 374) in the absence of delay. The second item in the second feedback path is an estimate of the actual plant 350 output 320 expected in the presence of a feedback delay (specified by the delay model 372). If the second loop contains a precise representation of the process or plant delays (feedforward and feedback delays), the second loop serves to delay the output from the controller 340 to match the delay feedback from the peripherals. These two temporarily matched signals are generally canceled (because of combiners 376 and 378). The residual signal represents the actual output estimate from the conventional feedback loop (via delay 360) substantially without delay. In other words, if the model 370 is accurate in indicating the operation of the process or plant 350, its output will be a version without a dead time of the actual plant 350 output 320. Predictive control for dead-time compensation is implemented using both plant and delay models. In this way, a statistically accurate model allows compensation of the delay time caused in the feedback channel, effectively negating dead time errors in the controller.

In a video delivery system, model 370 is the same as the model of the communication channel that includes the feedback delay time. For the non-limiting purposes of the additional description, being predictable for the purposes herein means statistically accurate. Thus, wireless network conditions are not entirely unpredictable. Basically, a model of a wireless network, such as a two-step Markov-based channel model, or another more complex model, such as a stochastic channel model, can be used in a model 370 that integrates both channel and delay models. have.

A block diagram of an end-to-end video delivery system 400 that includes an embedded Smith predictor is shown in FIG. 5. System 400 generally includes video encoder 410, traffic shaping unit 420, controller 450 and model 470, all of which may be embedded within a server, for example. As used herein, the term “server” generally refers to a computing device connected to a network and managing network resources. A server may be a dedicated collection of computing hardware and / or software components, meaning that these components do no other work than the server task or that the server manages resources rather than the entire computing device. It may refer to hardware and / or software components. Servers generally include and / or use processors. As used herein, the term “processor” generally refers to a computing device that includes a central processing unit (CPU), such as a microprocessor. The CPU typically includes an arithmetic logic unit (ALU) that performs arithmetic logic operations, and a control unit that requests the ALU when needed to extract instructions (e.g., a computer program that integrates code) from memory, and decodes and executes the instructions. Include. As used herein, the term “memory” generally refers to one or more devices capable of storing data, such as in the form of chips, tapes, disks, or drives. The memory is merely a non-limiting example, such as random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), or electrically erasable programmable. It may take the form of one or more of a read only memory (EEPROM) chip. The memory may be internally or externally in an integrated unit such as, for example, an integrated circuit (IC) including a processor. The server may be implemented, for example, as an application specific integrated circuit (ASIC).

System 400 also generally includes a network 430 and a feedback channel 460, both of which may be implemented, for example, as IP networks. As used herein, the term “network” generally refers to a collection of two or more computing devices connected together, including wired and wireless networks.

System 400 also includes receiver 440, which may be implemented, for example, in a client. As used herein, the term “client” generally refers to an application that runs on a computing device and relies on a server to perform a particular operation. The client may be implemented as a processor integrated device, or an application specific integrated circuit (ASIC), for example.

In one non-limiting embodiment, the server may take the form of a head-end of a video distribution system, the network may take the form of a data transfer mechanism of a video distribution network, and the client may be a digital set-top box, smart-card. Or it may take the form of one or more applications that serve as consumers of a video distribution network, such as a personal computer (PC), all of which are non-limiting examples. As a further non-limiting example, where a video distribution network implements a cable system, the controller and channel model may be located at the cable network head-end by combining the encoder and the traffic shaper. If the video distribution network implements an xDSL network, they may be located in a telecommunications system control room.

With continued reference to FIG. 5, video encoder 410 receives digital multimedia content from source 405 and supplies this content to traffic shaping unit 420. Such a traffic shaping unit may be implemented, for example, as a leaky bucket algorithm executed by suitable computing resources. For example, traffic shaping implementations, such as leaky buckets, are used to control the rate at which traffic is sent to network 430.

Data from the format unit 420 is transmitted to the receiver 440 over the network 430 experiencing a time varying transmission condition. Receiver 440 then presents the multimedia content to a content consumer, such as a television. Receiver 440 also determines parameters associated with network 430 and provides these parameters to controller 450 via feedback channel 460. The network 430 may take the form of an IP network, for example, and the feedback information may be encapsulated in a real time control protocol such as real time streaming protocol (RTSP). Controller 450 uses predictor functionality, such as Smith predictor configuration, and integrates channel model 470 to provide adaptive feedback to video encoder 410 and traffic shaping unit 420. Encoder 410 may change, for example, a quantization parameter of the data output therefrom in response to the output of controller 450. Traffic shaping unit 420 may change error correction component (s), such as, for example, forward error correction (FEC) component (s) of data output therefrom in response to controller 450.

More advanced controller (s) can be used, such as nonlinear controllers and / or fuzzy logic controllers. The invention can be applied to both wired and wireless channels for video delivery.

Referring now to FIG. 6, a flow diagram of a process 500 suitable for controlling the transmission of video data in a network, such as a wireless network, is shown. Process 500 includes transmitting video data to a receiver over a network at block 510. Channel parameter information measured by the receiver is received at block 520. At least one prediction function is applied to the channel parameter information to compensate for the delay time upon receiving the channel parameter information from the receiver, and at least one feedback function is generated at block 530. Finally, the video data to be transmitted in response to the feedback function is adjusted at block 540 to compensate for network conditions. For example, one or more quantization parameters of the data encoder may be changed at block 540. Additionally or alternatively, the traffic shaper may change error correction component (s), such as forward error correction component (s) of data output therefrom, at block 540.

As discussed herein, dead time occurring in a video delivery system (eg, 480, FIG. 5) can result in a delayed control signal for the video transmitter. By using the controller within the loop and adapting the controller based on the communication channel model, this dead time can be overcome in the channel parameter feedback loop. Thus, this mechanism is optimal for video streams to cope with the time-varying nature of wireless communication channels, for example, in wireless local area networks (WLANs) that conform to the IEEE 802.11 standard or Hyperlan 2 (but not limited to these). To achieve the adaptation, there is provided a precise means for adaptively adjusting the video encoder and the traffic shaping device in the video transmitter.

It will be apparent to those skilled in the art that modifications and variations can be made in the apparatus and processes of the invention without departing from the spirit or scope of the invention. It is intended that the present invention cover modifications and variations of this invention provided that they come within the scope of the appended claims and their equivalents.

The present invention is generally available for data delivery, and more specifically for digital multimedia content delivery for use with a network comprising one or more wired or wireless links.

Claims (19)

A method for controlling the transmission of video data in a network, Transmitting video data to a receiver via the network; Receiving channel parameter information measured by the receiver; Applying a prediction function to the channel parameter information to compensate for delay time upon receiving the channel parameter information from the receiver, and generating a feedback function; And Adjusting the video data to be transmitted in response to the feedback function to compensate for network conditions. And a method for controlling the transmission of video data. The method of claim 1, wherein the prediction function is based on a statistical model of the network condition. The method of claim 1, wherein the network is a WLAN. 2. The method of claim 1, wherein the adjusting step includes adjusting a quantization parameter at the video encoder. The method of claim 1, wherein the step of adjusting comprises adjusting error correction data included in the video stream. A system for controlling the transmission of video data in a network, the system comprising: A transmitter for generating a video data stream and transmitting the video data stream over the network; A receiver for receiving the video stream on the network Including, The receiver measures channel parameter information associated with the transmission, sends the channel parameter information to the transmitter via a feedback channel, the transmitter including a controller for receiving the channel parameter information from the receiver, and the controller Applies a prediction function to the channel parameter information to produce a feedback function and adjusts the video data stream in response to the prediction function and channel parameter information. 7. The system of claim 6, wherein the prediction function is based on a statistical model of the network condition. The system of claim 6, wherein the network is a WLAN. 7. The system of claim 6, wherein the controller adjusts quantization parameters at the video encoder. 7. The system of claim 6, wherein the controller adjusts error correction data included in the video data stream. A transmitter for providing streaming video to a receiver over a network, An encoder having an output; A traffic shaper having an input coupled to the output of the encoder; A controller coupled to at least one of the encoder and the traffic shaper Including, The controller predictively modifies the operation of at least one of the encoder and the traffic shaper according to a channel model and at least one actual parameter of the network. 12. The transmitter of claim 11 wherein the controller modifies predictively based on a statistical model of the network condition. 13. The transmitter of claim 12 wherein the controller is coupled to both the encoder and the traffic shaper. The transmitter of claim 13, wherein the controller modifies the operation of both the encoder and the traffic shaper. 12. The transmitter of claim 11 wherein said at least one actual parameter of said network comprises at least one parameter indicating channel quality. 12. The transmitter of claim 11 wherein said at least one actual parameter of said network comprises at least one parameter indicative of a bit error rate. 12. The transmitter of claim 11 wherein the model comprises a plurality of models, one of the models representing a network feedforward delay and the other of the models representing a network feedback delay. 12. The transmitter of claim 11 further comprising a source of digital video data coupled to the encoder. 12. The transmitter of claim 11 wherein the network is a WLAN.

Images