WO2012018339A1

WO2012018339A1 - Application of unequal error protection rateless codes in multimedia streaming over multi-path networks

Info

Publication number: WO2012018339A1
Application number: PCT/US2010/044571
Authority: WO
Inventors: Zhenyu Wu
Original assignee: Thomson Licensing
Priority date: 2010-08-05
Filing date: 2010-08-05
Publication date: 2012-02-09

Abstract

A method and apparatus are described including retrieving a block of source data, organizing the block of source data into layers of source symbols, encoding the layers of source symbols using fountain codes, wherein the fountain codes have an unequal protection property such that the fountain encoded layers of source symbols have different protection strengths and streaming the fountain encoded source symbols to a multicast group of client devices over a path of any of a plurality of multiple paths in a wireless networks. Also described are a method and apparatus including receiving unequal error protection fountain encoded data from a path of any of a plurality of wireless networks, retrieving payload data from the received unequal error protection fountain encoded data to obtain payload data, adding the payload data to a current codeword and decoding the current codeword to obtain a block of source data.

Description

APPLICATION OF UNEQUAL ERROR PROTECTION RATELESS CODES IN MULTIMEDIA STREAMING OVER MULTI-PATH NETWORKS)

FIELD OF THE INVENTION

The present invention is directed to a multi-path network where the devices operating in the network use the unequal error protection (UEP) property of rateless (digital fountain) codes.

BACKGROUND OF THE INVENTION

In multicast and broadcast applications, data are transmitted from a server to multiple receivers over wired and/or wireless networks. A multicast system as used herein is a system in which a server transmits the same data to multiple receivers simultaneously, where the receivers form a subset of all the receivers up to and including all of the receivers. A broadcast system is a system in which a server transmits the same data to all of the receivers simultaneously. That is, a multicast system by definition can include a broadcast system.

As used herein a client includes (but is not limited to) a client device, a mobile terminal, a mobile station, a node, a station, an end device , a terminal device, a computer, a laptop, a dual mode smart phone or any other equivalent device.

Multi-path or multi-link networking has recently emerged as an effective solution to enhance network reliability and throughput by delivering service data over multiple independent network paths to a client.

In a multi-path multicast wireless network, a client can connect to a server via a number of network paths. Furthermore, as a group, clients may share network paths. Each network path can be uni-directional or bi-directional. For the same path, the network conditions for different clients may differ, for example, bandwidth, loss rate, etc. due to factors, such as wireless channel interference and signal strength attenuation. Furthermore, these connection paths can be dynamic due to possible client mobility.

As an example Fig. 1 shows clients A, B and C connected to the server through network paths Pi, P₂ and P3. In particular, clients A, B and C respectively have one, three and two paths connecting to the server. These network paths go through different networks, i.e., a 3GPP-LTE cellular network, a Wi-Fi network and a DVB-H network, in which the 3GPP-LTE path is bi-directional, whiles the other two paths are unidirectional. The shaded rectangles in Fig.l represent data packets going through these networks. As seen from Fig. 1, Client A only receives packets from Pi, while Client B has three data sources, and Client C receives packets from both P₂ and P3. So the clients may have different aggregated network bandwidths.

Another example of such type of networks is a Mobile Ad Hoc Network (MANET) as shown in Fig. 2. In such a network, a mesh topology connects clients to the server through multiple paths, and all the nodes (clients) in the network can serve as a router in order to build up a short-term dynamic network infrastructure. Multi-path networks also include other types, such as peer-to-peer (P2P) networks.

In the above multi-path wireless multicast networks, the server needs to address the data packet scheduling problem in order to efficiently utilize the network resources to stream multimedia content to the clients in a multicast group. The objective of the scheduling task is to jointly determine (find):

1) The optimal streaming rate for each client in the group so that its received QoS is maximized, and

2) The network path that should be used to transmit each data packet to each client, so that the QoS of the each client in the client group is maximized.

In other words, the server needs to determine the delivery path for each data packet and in turn control the data rate and flow allocation of each path, so that each client can achieve the best QoS that corresponds to its aggregated network bandwidth.

A straightforward design usually requires that the server be able to estimate and keep track of the details of each client's network paths. However, such an estimate may not be accurate or may involve large overhead. On the other hand, for paths with unidirectional based connection (e.g. UDP), or a multicast group with a large number of clients, it is impossible or very difficult for the server to do so. Furthermore, for dynamic and distributed networks such as MANET or P2P networks, the server does not have the full knowledge of the complete network's topology. As a result, the lack of such information can cause a waste of network resources.

As an example, consider the multi-path network shown in Fig. 1. Client B shares path Pi with Client A, and P₂, P3 with Client C. Suppose the bandwidth of each path is the same for all the three clients. To best utilize the bandwidth resources, the server needs to send the packets that are common to Clients A and B through Pi, and the packets that are shared by Clients B and C through P₂ and P₃, and rest of the packets that are exclusively for Client B through any path that has remaining bandwidth. This solution requires that the server constantly monitors the link connection status, and estimates the bandwidth of each client, which can be costly or inaccurate. It may not even be possible in the case of some networks such as for a DVB-H network since a DVB- H network does not have a feedback channel. Furthermore, even with accurate path information, the server still needs to perform the path selection and rate allocation process. The path selection and rate allocation tasks are generally not easy.

Many existing works in multi-path streaming rely on classic routing algorithms that find the best path (or sets of paths) given some established network metrics. They do not take into consideration the metrics of multimedia streaming applications. Thus though the selected paths may be optimal in terms of network utilization, the selected paths are or may be suboptimal from the viewpoint of client's QoS.

In one prior art study, a general framework for streaming multimedia data in multi-path networks is derived which encompasses both network and media aware metrics. The framework is based upon a set of models including multi-path network models and a video quality model. The prior art then casts and solves the rate allocation as an optimization problem. However, this approach only targets one client and requires detailed knowledge of the path conditions of that client. Hence, it is not applicable to multi-path streaming to a multicast group. Furthermore, the computational complexity involved in the work is high. SUMMARY OF THE INVENTION

The present invention is an application of rateless (digital fountain) codes with an unequal error protection (UEP) property used in scalable multimedia streaming to a multicast group within multi-path wireless networks. In the case of such use, clients may connect to a server with multiple paths, and each path can differ in terms of network type and conditions, which can change dynamically. As a result, it is difficult for the server to efficiently schedule data packet delivery to the clients, while at the same time fully utilize the individual bandwidth capacity of each client. The present invention exploits ratelessness and randomness properties of the digital fountain codes, so that the server can remain unconcerned about clients' detailed network information when it performs scheduling. Each client in a multicast group may have Quality of Service (QoS) requirements that are proportional to its aggregated network bandwidth. The present invention can reduce the packet scheduling overhead at the server and improve network resource utilization.

A method and apparatus are described including retrieving a block of source data, organizing the block of source data into layers of source symbols, encoding the layers of source symbols using fountain codes, wherein the fountain codes have an unequal protection property such that the fountain encoded layers of source symbols have different protection strengths and streaming the fountain encoded layers of source symbols to a multicast group of client devices over a path of any of a plurality of wireless networks. Also described are a method and apparatus including receiving unequal error protection fountain encoded data from a path of any of a plurality of wireless networks, retrieving payload data from the received unequal error protection fountain encoded data to obtain payload data, adding the payload data to a current codeword and decoding the current codeword to obtain a block of source data.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is best understood from the following detailed description when read in conjunction with the accompanying drawings. The drawings include the following figures briefly described below: Fig. 1 is a schematic diagram of a multi-path multicast wireless network.

Fig. 2 is a schematic diagram of a wireless mesh network.

Fig. 3 shows fountain codes with prioritized protection.

Fig. 4 shows prioritized fountain code decoding.

Fig. 5 is a flowchart of an exemplary implementation of the operation of the present invention from the perspective of a server.

Fig. 6 is a flowchart of an exemplary implementation of the operation of the present invention from the perspective of a client.

Fig. 7 is a block diagram of an exemplary device operating in a multi-path network using the unequal error protection property of rateless codes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Fountain codes are a relatively new class of codes originally designed for robust and scalable transmission of data over lossy computer networks. For a message to be delivered which includes a vector of input symbols (xj, ... ¾), a fountain code generates a potentially limitless stream of output symbols. Hereafter, the input symbols (data) are called "source symbols", the output symbols are called "encoded symbols". Source symbols and output symbols form a codeword. Each encoded symbol is generated independently by sampling from a distribution D, and adding the source symbols correspondingly to support the sampled vector.

There are two features of digital fountain codes that are exploited in the present invention. The first is that fountain codes can generate an arbitrary number of encoded symbols from a few to possibly infinite on the fly for each codeword, based on pseudorandom selections from a finite number of source symbols. Thus, unlike traditional FEC codes, the code rate for a fountain code is no longer fixed. Thus, digital fountain codes are also known as rateless codes. This property is beneficial for those network channels with dynamic or unknown loss rates, since the server can keep sending encoded symbols to feed (transmit to, send to, communicate to) the client(s) until they have received enough encoded symbols to recover the source symbols, or some network resource is depleted. The other feature of digital fountain codes is that, due to the random nature of the encoding process, as long as there is a sufficient number of source symbols in a codeword, there is high probability that the encoded symbols are linearly independent. Therefore, a digital fountain code decoder can recover the source symbols from an arbitrary set of encoded symbols of a codeword, when the received encoded symbols reach a sufficient amount. With this property, when the server sends encoded symbols through multiple network paths, a client can receive the encoded symbols from any of the multiple paths used by the server to transmit (communicate, send, forward). With high probability, there is no waste of the received encoded symbols in recovering source symbols.

In general, all the source symbols in a fountain codeword receive the same amount of protection from its encoded symbols. However, there is a variant of fountain codes called "unequal error protection" (UEP) fountain codes. With UEP fountain codes, the protection strength can be shifted among the source symbols. A common practice to utilize the UEP property is to allocate more protection to high-priority source symbols (data) and less protection for low-priority data. There are many different approaches to construct UEP fountain codes, such as those reported in the prior art.

Scalable multimedia coding is a coding paradigm in which a multimedia source is encoded into multiple layers, so that by decoding each successive layer, the reconstructed multimedia quality is improved in terms of resolution, fidelity, etc. In the present invention, videos that are encoded by SVC are specifically considered. However, the described invention can be applied to other scalable multimedia sources, such as scalable images and audio.

SVC (Scalable Video Coding) is an extension to the H.264/AVC video coding standard. It can provide scalability in spatial (resolution), temporal (frame rate) and SNR (quality/fidelity) dimensions. Its bit stream includes a base layer and one or more enhancement layers. In general, the base layer provides basic video quality, and with each additional enhancement layer added in order of its layer index, the video quality improves. Since the decoding of an enhancement layer depends on the base layer and all of lower-indexed enhancement layer(s), generally the base layer has the highest priority, followed by the enhancement layers with decreasing layer indices.

The present invention combines these components to address the server scheduling problem as described above in a multi-path video multicast streaming application.

At the server end, the server encodes a block of SVC video data with digital fountain codes and packetizes the encoded symbols output from the fountain encoder into network packets, and then sends these packets over the multiple links connects the server to the clients in a multicast group.

In Fig. 3 the symbols that are to be encoded and transmitted (communicated, sent out, forwarded) are indicated by boxes (squares). The shaded circles indicate high- priority (base layer) encoded symbols and the unshaded circles indicate low-priority (enhancement layer) encoded symbols. Fig. 3 illustrates the fountain encoding process at the server: the SVC bit stream data that corresponds to a media time interval T are included into a fountain codeword as source symbols for encoding. Suppose that the SVC bit stream includes a base and one or more enhancement layers. The layers are treated as high-priority and low-priority separately, as shown in Fig. 3. The included bit stream data from the two layers are treated as high-priority and low-priority separately. Specifically, every T bytes of the base layer data are grouped to form a high-priority source symbol (data), and likewise, every T bytes of enhancement layer data are grouped to form a low- priority source symbol (data).

According to the encoding rules for digital fountain codes, a random number of degrees d is chosen first for each encoding symbol. Then d source symbols are randomly chosen and XORed to form the encoded symbol. For different UEP fountain code construction methods, the random source symbol selection process is often altered so that more connections between source and encoded symbols come from high-priority source symbols, and vice versa. As a result, the high-priority source symbols get more protection. Then the server packetizes the encoded symbols from the fountain encoder output into a path that has remaining bandwidth. It continues sending these packets until the time interval T corresponding to the current codeword expires, or it is informed by some means that all the clients have received enough data. Then the server moves to fetch the next block of data from the SVC bit stream and repeats the above process.

Therefore, with the present invention, the server does not need to perform complicated path selection and rate allocation among multiple paths. Moreover, it does not have to have the specific knowledge about the path information of each of its client groups.

Fig. 5 is a flowchart of an exemplary implementation of the operation of the present invention from the perspective of a server. At 505 the server retrieves a block of data from an SVC bit stream corresponding to time interval T. At 507 the server organizes the SVC bit stream into layers. At 510 the server organizes the SVC data as source symbols in a fountain codeword. At 515 the server applies UEP coding to different SVC layers in the fountain codeword. At 520 the server generates encoded layer of symbols and transmits (sends, communicates, forwards) the fountain encoded layers of symbols over a path of any of a plurality of wireless networks. The paths are selected based on remaining bandwidth. A test is performed at 525 to determine if every client has received enough data (fountain encoded symbols) to properly decode the data. Feedback can be sent through network connections that have a feedback capability, e.g. 3G network as shown in Fig. 1. The server continues sending data within DVB-H bandwidth capacity until the playback time for the chunk of data expires. If every client has received enough data (fountain encoded symbols) to properly decode the data then processing proceeds to 505. If every client has not received enough data (fountain encoded symbols) to properly decode the data then a test is performed at 525 to determine if the time interval has expired. If the time interval has expired, then processing proceeds to 505. If the time interval has not expired, then processing proceeds to 520.

At the client-side, each client receives data packets from the path(s) that connects the client to the server. The client first retrieves the encoded symbols from each packet payload, and then forwards them to the fountain decoder to decode. For the current codeword, once the decoder has accumulated enough encoding symbols to recover the desirable amount of source symbols in the codeword, or the time interval T corresponding to the current codeword expires, the fountain decoder moves to receiving encoded symbols for the next codeword.

For a particular client, it may receive only a portion of the encoded symbols that originated from the server. This is because the server sends the encoded symbols over all the available links, but for a particular client, it may only have a subset of the connections. Another reason is that some packets may get dropped during delivery. Thus, the amount of received encoded symbols should be proportional to the client's effective aggregated bandwidth.

In Fig. 4 the symbols that were not able to be recovered are shown with a dotted boundary marking the circle. All other circles indicate symbols that were able to be recovered. The shaded circles indicate high-priority (base layer) symbols and the unshaded circles indicate low-priority (enhancement layer) symbols. As shown in Fig. 4, suppose a client only receives some of the encoded symbols of a codeword and there are still some symbols missing. Due to the UEP construction of the fountain codes, as long as there is a sufficient number of encoded symbols, all of the high-priority source symbols can be recovered, though not all the low-priority encoded symbols may be recovered. Hence, the client can reconstruct the video using the recovered base layer data to have basic video quality. On the other hand, if another client has more paths, or has better channel conditions, it can receive more encoded symbols and thus have a higher probability of recovering the additional enhancement layer(s). So the client is able to enjoy a higher video QoS.

Based upon the above descriptions of the server-side and the client-side operations, it is evident that using the present invention, the server does not need to have any path information of its clients' multicast groups and also avoids the complicated path selection and rate allocation processes. Meanwhile, each client in the multicast group(s) can reconstruct the amount of useful service data that is proportional to its effective aggregated bandwidth. With different UEP fountain code construction methods, different amounts of encoded symbols are needed to recover source symbols assigned to different priorities. Therefore, it is possible for the server to estimate the available aggregated bandwidth profile of the multicast group(s), and adjust the fountain codes construction to match the profile, in order to achieve the best QoS for all the clients.

Fig. 6 is a flowchart of an exemplary implementation of the operation of the present invention from the perspective of a client. At 605 the client starts to work on a new codeword. At 610 the client receives data packets from any of the connecting paths of any of a plurality of wireless networks available between itself and the server. At 615 the client retrieves the payload data (fountain encoded symbols) and adds the retrieved data to the current fountain encoded symbols (codeword) that it is working on (that the client may have received from another path (link, channel). At 620 the client attempts to decode the fountain encoded codewords. At 625 a test is performed to determine if the client has been able to recover the source symbols. If the client has been able to recover the source symbols, then at 635 a test is performed to determine if the client device is able to give the server feedback. If the client device is able to give the server feedback then at 640 the client device transmits feedback to the server that it is finished because the client device has received enough unequal error protected fountain encoded symbols to properly decode the source symbols. If the client device is not able to give the server any feedback then processing proceeds to 605. If the client has not been able to recover the source symbols, then a test is performed at 630 to determine if the time interval for the data has expired. If the time interval for the data has expired, then processing proceeds to 635. If the time interval for the data has not expired, then processing proceeds to 610.

Fig. 7 is a block diagram of an exemplary device operating in a multi-path network using the unequal error protection property of rateless codes. The device can be either a server or a client. The exemplary device (apparatus) shown in Fig. 7 may be either a server or a client. The transceiver actually transmits and receives data and any control signals and the control logic performs all other functions as described below. Specifically, when the device is behaving as a server, the control logic includes means for retrieving a block of source data, means for organizing the block of source data into layers of source symbols and means for encoding the layers of source symbols using fountain codes, wherein the fountain codes have an unequal protection property such that the fountain encoded layers of source symbols have different protection strengths. Each of the multiple paths may or may not have a feedback capability. Each block of source data is a scalable video coded bit stream coded by a scalable video coder. Different protection strengths are applied to different layers of the source symbols of the scalable video coded bit stream such that a number of the fountain encoded layers of source symbols necessary to recover different scalable video coded layers are different, wherein a base layer has a smallest amount of encoded source symbols to recover and a higher enhancement layer has more encoded source symbols to recover. The control logic also includes means for adjusting the fountain codes construction to adjust an amount of encoded layers of source symbols necessary to recover source symbols with different priorities to match a bandwidth profile of each client device in the multicast group. The transceiver includes means for streaming the fountain encoded layers of source symbols to a multicast group of client devices over a path of a plurality of wireless networks. The means for streaming includes means for streaming the fountain encoded source symbols at a full capacity of each of the multiple paths. Specifically, when the device is behaving as a client device the control logic includes means for retrieving payload data from received unequal error protection fountain encoded data to obtain payload data, means for adding the payload data to a current codeword and means for decoding the current codeword to obtain a block of source data. The control logic also includes means for repeatedly executing the means for receiving, retrieving, adding and decoding until the block of source data has been recovered or a time interval for usefulness of the block of source data has expired. The plurality of wireless networks include different paths between a client device and a server, which is a source of the block of source data. The client device is a member of a multicast group having a plurality of client devices as members, wherein the plurality of client devices may share a plurality of common paths. Each of the plurality of client devices receives the unequal error protection fountain encoded data. The block of source data corresponds to a base layer and an enhancement layer of a scalable video coded bit stream. The transceiver includes means for receiving unequal error protection fountain encoded data from a path from any of a plurality of wireless networks. The transceiver also includes means for sending said server feedback.

It is to be understood that the present invention may be implemented in various forms of hardware, software, firmware, special purpose processors, or a combination thereof. Preferably, the present invention is implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage device. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (CPU), a random access memory (RAM), and input/output (I/O) interface(s). The computer platform also includes an operating system and microinstruction code. The various processes and functions described herein may either be part of the microinstruction code or part of the application program (or a combination thereof), which is executed via the operating system. In addition, various other peripheral devices may be connected to the computer platform such as an additional data storage device and a printing device.

It is to be further understood that, because some of the constituent system components and method steps depicted in the accompanying figures are preferably implemented in software, the actual connections between the system components (or the process steps) may differ depending upon the manner in which the present invention is programmed. Given the teachings herein, one of ordinary skill in the related art will be able to contemplate these and similar implementations or configurations of the present invention.

Claims

CLAIMS:

1. A method, said method comprising: retrieving a block of source data; organizing said block of source data into layers of source symbols; encoding said layers of source symbols using fountain codes, wherein said fountain codes have an unequal protection property such that said fountain encoded layers of source symbols have different protection strengths; and streaming said fountain encoded layers of source symbols to a multicast group of client devices over a path of any of a plurality of wireless networks.

2. The method according to claim 1, wherein each said path may or may not have a feedback capability.

3. The method according to claim 1, wherein each said block of source data is a scalable video coded bit stream coded by a scalable video coder.

4. The method according to claim 3, wherein said different protection strengths are applied to different layers of said source symbols of said scalable video coded bit stream such that a number of said fountain encoded layers of source symbols necessary to recover different scalable video coded layers are different, wherein a base layer has a smallest amount of encoded source symbols to recover and a higher enhancement layer has more encoded source symbols to recover.

5. The method according to claim 1, wherein said streaming step streams said fountain encoded layers of source symbols at a full capacity of each said path.

6. The method according to claim 1, further comprising adjusting said fountain codes construction to adjust an amount of encoded layers of source symbols necessary to recover source symbols with different priorities to match a bandwidth profile of each said client device in said multicast group.

7. A method, said method comprising: receiving unequal error protection fountain encoded data from a path of any of a plurality of wireless networks; retrieving payload data from said received unequal error protection fountain encoded data to obtain payload data; adding said payload data to a current codeword; and decoding said current codeword to obtain a block of source data.

8. The method according to claim 7, further comprising repeating said steps of receiving, retrieving, adding and decoding until said block of source data has been recovered or a time interval for usefulness of said block of source data has expired.

9. The method according to claim 7, wherein said plurality of wireless networks include different paths between a client device and a server, which is a source of said block of source data.

10. The method according to claim 9, wherein said client device is a member of a multicast group having a plurality of client devices as members, wherein the plurality of client devices may share a common path.

11. The method according to claim 9, wherein each of said plurality of client devices receives said unequal error protection fountain encoded data.

12. The method according to claim 7, wherein said block of source data corresponds to a base layer and an enhancement layer of a scalable video coded bit stream.

13. The method according to claim 9, further comprising sending said server feedback.

14. An apparatus, comprising: means for retrieving a block of source data; means for organizing said block of source data into layers of source symbols; means for encoding said layers of source symbols using fountain codes, wherein said fountain codes have an unequal protection property such that said fountain encoded layers of source symbols have different protection strengths; and means for streaming said fountain encoded layers of source symbols to a multicast group of client devices over a path of any of a plurality of wireless networks.

15. The apparatus according to claim 14, wherein each said path may or may not have a feedback capability.

16. The apparatus according to claim 14, wherein each said block of source data is a scalable video coded bit stream coded by a scalable video coder.

17. The apparatus according to claim 16, wherein said different protection strengths are applied to different layers of said source symbols of said scalable video coded bit stream such that a number of said fountain encoded layers of source symbols necessary to recover different scalable video coded layers are different, wherein a base layer has a smallest amount of encoded source symbols to recover and a higher enhancement layer has more encoded source symbols to recover.

18. The apparatus according to claim 14, wherein said means for streaming includes means for streaming said fountain encoded layers of source symbols at a full capacity of each said path.

19. The apparatus according to claim 14, further comprising means for adjusting said fountain codes construction to adjust an amount of encoded layers of source symbols necessary to recover source symbols with different priorities to match a bandwidth profile of each said client device in said multicast group.

20. An apparatus, comprising: means for receiving unequal error protection fountain encoded data from a path of any of a plurality of wireless networks; means for retrieving payload data from said received unequal error protection fountain encoded data to obtain payload data; means for adding said payload data to a current codeword; and means for decoding said current codeword to obtain a block of source data.

21. The apparatus according to claim 20, further comprising means for repeatedly executing said means for receiving, retrieving, adding and decoding until said block of source data has been recovered or a time interval for usefulness of said block of source data has expired.

22. The apparatus according to claim 20, wherein said plurality of wireless networks include different paths between a client device and a server, which is a source of said block of source data.

23. The apparatus according to claim 22, wherein said client device is a member of a multicast group having a plurality of client devices as members, wherein the plurality of client devices may share a common path.

24. The apparatus according to claim 22, wherein each of said plurality of client devices receives said unequal error protection fountain encoded data.

25. The apparatus according to claim 20, wherein said block of source data corresponds to a base layer and an enhancement layer of a scalable video coded bit stream.

26. The apparatus according to claim 22, further comprising means for sending said server feedback.