CN107872735B

CN107872735B - Method and data transmitter for transmitting video

Info

Publication number: CN107872735B
Application number: CN201710880815.7A
Authority: CN
Inventors: 拉明·穆巴舍; 亚利尔·卡迈利; 格雷戈里·W·库克; 大卫·萨莫拉
Original assignee: Samsung Display Co Ltd
Current assignee: Samsung Display Co Ltd
Priority date: 2016-09-26
Filing date: 2017-09-26
Publication date: 2022-04-29
Anticipated expiration: 2037-09-26
Also published as: KR20180035140A; EP3300277A1; KR102326312B1; US10616383B2; TWI766883B; EP3920445A1; JP2018056994A; TW201820857A; CN107872735A; US20180091634A1; JP7071080B2

Abstract

The invention relates to a method and a data transmitter for transmitting video. In a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electrical communication with a receiver, the method comprising: receiving, by a transmitter, a frame of video data from a data source; recombining, by the transmitter, the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames; generating, by the transmitter, a tag for each of the plurality of data packets to tag each of the plurality of data packets, the tag corresponding to a different relative importance level of the plurality of data packets; performing, by the transmitter, a different protection technique for each of the plurality of data packets based on the tag corresponding to each of the plurality of data packets; and transmitting, by the transmitter, the plurality of data packets and the tag to the receiver for display on the display panel such that each data packet of the plurality of data packets is transmitted according to a different protection technique based on the tag to which the data packet corresponds.

Description

Method and data transmitter for transmitting video

Technical Field

An aspect of one or more example embodiments of the present invention relates to a system and method for electronic data communication.

Background

This application claims priority and benefits from U.S. provisional patent application No. 62/400,042 entitled "A SYSTEM AND METHOD FOR CROSS Layer Image Optimization (CLIO) FOR WIRELESS VIDEO TRANSMISSION OVER MULTI GIGABIT CHANNELS" (system and METHOD FOR CROSS Layer Image Optimization (CLIO)) filed by the united states patent and trademark office on 9, 26/2016, the entire contents of which are incorporated herein by reference.

The demand for wireless video transmission is increasing due to the emergence of new applications and new use cases. In recent years, technological advances in high-resolution display screens and the advent of high-quality video (HD, FHD, UHD, etc.) have resulted in increased bandwidth requirements for high-throughput transmissions. For example, uncompressed Ultra High Definition (UHD) video requires 12Gbps of bandwidth.

In addition to the limitation of high data rates, wireless video transmission is also time/delay sensitive. When provided at 60 frames per second, the inter-frame time is 1/60 ═ 16.6 ms. Therefore, any portion of the frame that is not received within 16.6 milliseconds must be discarded so that the display can begin rendering the next frame, and data retransmission is generally not a viable option. In addition to high bandwidth and latency requirements, wireless channels may be susceptible to interference, which may cause the quality of the wireless channel to vary unpredictably over time. Accordingly, it may be difficult to provide a guaranteed quality of service (QoS) for transmitting high quality video data over a wireless channel.

The above information discussed in this background section is only for enhancement of understanding of the background of the described technology and therefore it may contain information that does not form the prior art that is known to a person of ordinary skill in the art.

Disclosure of Invention

Aspects of one or more example embodiments of the present invention relate to systems and methods for electronic data communication.

According to some example embodiments, in a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprises: receiving, by a transmitter, a data signal from a data source; receiving, by the transmitter, a return signal from the receiver; selecting, by the transmitter, a profile from a plurality of profiles based on at least one of channel quality, video quality, codec requirements, and data rate requirements, each profile of the plurality of profiles including one or more parameters for transmitting the data signal to the receiver, the plurality of profiles including one or more profiles corresponding to transmission of uncompressed video data and one or more profiles corresponding to transmission of compressed video data; and transmitting, by the transmitter, the data signal to the receiver for display on the display panel according to the selected profile.

According to some embodiments, the data signal received by the transmitter is a video data signal.

According to some embodiments, the method further comprises selecting the profile from a plurality of profiles based on a return signal, wherein the return signal comprises an indicator of the quality of the wireless communication channel measured by the receiver.

According to some embodiments, the method further comprises selecting the profile from a plurality of profiles based on a return signal, wherein the return signal comprises an indicator of the visual quality measured by the receiver.

According to some embodiments, the method further comprises: identifying a display device corresponding to the receiver based on the return signal; and selecting, by the transmitter, the profile based on the display device corresponding to the receiver.

According to some embodiments, the plurality of profiles comprises one or more of: defining a first profile of parameters for transmission of uncompressed video data, wherein the channel quality exceeds a first threshold level, such that if pixel data from any pixel in a group of pixels is lost at a receiver after transmission, the receiver recalculates the lost pixel data by averaging the values of surrounding pixels; defining a second profile of parameters for transmitting uncompressed video data, wherein the channel quality is below the first threshold level but above the second threshold level such that pixel data from one pixel in a group of pixels is not transmitted to the receiver and the receiver recalculates the pixel data from this one pixel by averaging the values of surrounding pixels; defining a third profile of parameters for transmitting uncompressed video data, wherein the channel quality is below the first and second threshold levels but above the third threshold level, such that least significant bits of pixels in the data packets are not transmitted to the receiver and the receiver compensates for the least significant bits; fourth, fifth, sixth and seventh profiles defining parameters for transmitting compressed video data, the compressed video data being compressed using a layer-based compression method that generates a plurality of data layers, the plurality of data layers including a highest layer having a first compression rate and a lowest layer having data with a second compression rate lower than the first compression rate if added to the highest layer and any intermediate layers, wherein: each data layer is transmitted to the receiver according to a fourth profile; transmitting the subset of data layers to the receiver according to a fifth profile; transmitting a single one of the data layers to the receiver according to a sixth profile; and, according to the seventh profile, only a portion of the one or more data layers is transmitted to the receiver; and defining an eighth profile of parameters for transmitting video data having different priorities based on a display device for displaying the video data.

According to some embodiments, the method further comprises transmitting, by the transmitter, information about the selected profile to the receiver through bit selection in a header packet of the data signal.

According to some embodiments, the information about the selected profile includes at least one of decompression, reverse layering, decoding, and error correction information corresponding to the selected profile.

According to some example embodiments of the present invention, in a transmitter transmitting data for a display panel to a receiver through a wireless communication channel, the transmitter is configured to: receiving a data signal from a data source; receiving a return signal from a receiver; selecting a profile from a plurality of profiles based on at least one of channel quality, video quality, codec requirements, and data rate requirements, each profile of the plurality of profiles comprising one or more parameters for transmitting the data signal to a receiver, the plurality of profiles comprising one or more profiles corresponding to transmission of uncompressed video data and one or more profiles corresponding to transmission of compressed video data; and, transmitting the data signal to the receiver according to the selected profile for display on the display panel.

According to some embodiments, the transmitter is further configured to select the profile from a plurality of profiles based on a return signal, wherein the return signal comprises an indicator of the quality of the wireless communication channel measured by the receiver.

According to some embodiments, the transmitter is further configured to: identifying a display device corresponding to the receiver based on the return signal; and, based on the display device corresponding to the receiver, a profile is selected.

According to some embodiments, the plurality of profiles comprises one or more of: defining a first profile of parameters for transmission of uncompressed video data, wherein the channel quality exceeds a first threshold level such that if pixel data from any pixel in a group of pixels is lost at a receiver after transmission, the receiver recalculates the lost pixel data by averaging the values of surrounding pixels; defining a second profile of parameters for transmitting uncompressed video data, wherein the channel quality is below the first threshold level but above the second threshold level such that pixel data from one pixel in a group of pixels is not transmitted to the receiver and the receiver recalculates the pixel data from this one pixel by averaging the values of surrounding pixels; defining a third profile of parameters for transmitting uncompressed video data, wherein the channel quality is below the first and second threshold levels but above the third threshold level, such that least significant bits of pixels in the data packets are not transmitted to the receiver and the receiver compensates for the least significant bits; defining fourth, fifth, sixth and seventh profiles of parameters for transmitting compressed video data, the compressed video data being compressed using a layer-based compression method that generates a plurality of data layers, the plurality of data layers including a highest layer having a first compression rate and a lowest layer having data with a second compression rate lower than the first compression rate if added to the highest layer and any intermediate layers, wherein: each data layer is transmitted to the receiver according to a fourth profile; transmitting the subset of data layers to the receiver according to a fifth profile; transmitting a single one of the data layers to the receiver according to a sixth profile; and, according to the seventh profile, only portions of the one or more data layers are transmitted to the receiver; and defining an eighth profile of parameters for transmitting video data having different priorities based on a display device for displaying the video data.

According to some embodiments, the transmitter is further configured to transmit information about the selected profile to the receiver through bit selection in a header packet of the data signal, wherein the information about the selected profile includes at least one of decompression, inverse layering, decoding, and error correction information corresponding to the selected profile.

According to some embodiments of the invention, in a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprises: receiving, by a transmitter, a data signal from a data source; receiving, by the transmitter, a return signal from the receiver; selecting, by a transmitter, a profile from a plurality of profiles, each profile of the plurality of profiles comprising one or more parameters for transmitting the data signal to the receiver, the plurality of profiles comprising one or more profiles corresponding to transmission of uncompressed video data and one or more profiles corresponding to transmission of compressed video data; and transmitting, by the transmitter, the data signal to the receiver for display on the display panel according to the selected profile.

According to some embodiments, the method further comprises: a profile is selected based on the channel quality.

According to some embodiments, the method further comprises selecting a profile based on the video quality measured at the receiver.

According to some embodiments, the method further comprises selecting a profile based on codec requirements.

According to some embodiments, the method further comprises selecting a profile based on the data rate requirements.

According to some embodiments, the method further comprises: identifying a display device corresponding to the receiver based on the return signal; and selecting, by the transmitter, the profile based on a display device corresponding to the receiver.

According to some embodiments, the method further comprises: monitoring, by the transmitter, a return signal from the receiver; and selecting, by the transmitter, another profile for subsequent transmission based on a change in at least one of the channel quality and the video quality.

According to some example embodiments of the present invention, in a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprises: receiving, by a transmitter, a frame of video data from a data source; grouping, by a transmitter, bits corresponding to a frame of video data into a plurality of groups each corresponding to a plurality of importance levels; recombining, by the transmitter, the plurality of groups in order of importance to generate a recombined data frame having a group, the group corresponding to the plurality of data packet headers, having a highest importance level and arranged as a first of the plurality of groups; inserting, by the transmitter, a value indicating a length of a data bit corresponding to each data packet header before each data packet header; and transmitting, by the transmitter, bits corresponding to the frame of video data to the receiver for display on the display panel such that each of the plurality of groups is transmitted according to a different protection technique based on the importance levels corresponding to the plurality of groups.

According to some embodiments, each group has a different Modulation and Coding Scheme (MCS) value based on the group's corresponding importance level.

According to some embodiments, each group has a different forward error correction coding rate based on the group's corresponding importance level.

According to some embodiments, a decoder at a receiver is configured to reconstruct a frame of video data.

According to some embodiments, reconstructing the frame of video data comprises: each data packet header is moved to the original relative position of each data packet header.

According to some embodiments, reconstructing the frame of video data comprises: the value indicating the length of the data bit is removed.

According to some embodiments, the method further comprises: monitoring, by the transmitter, a return signal from the receiver; and adjusting, by the transmitter, the different protection techniques for subsequent transmissions.

According to some embodiments, the method further comprises: different protection techniques are selected by the transmitter depending on the type of display device corresponding to the receiver.

According to some example embodiments of the present invention, in a transmitter for transmitting data for a display panel to a receiver through a wireless communication channel, the transmitter is configured to: receiving a frame of video data from a data source; grouping bits corresponding to a frame of video data into a plurality of groups each corresponding to a plurality of importance levels; recombining the plurality of groups in order of importance to generate a recombined data frame having a group, the group corresponding to the plurality of data packet headers, having a highest importance level and arranged as a first of the plurality of groups; inserting a value indicating a length of a data bit corresponding to each data packet header before each data packet header; and transmitting bits corresponding to the frame of video data to a receiver for display on a display panel such that each of the plurality of groups is transmitted according to a different protection technique based on the importance levels corresponding to the plurality of groups.

According to some embodiments, a decoder at the receiver is configured to reconstruct the frame of video data by moving each data packet header to the original relative position of each data packet header.

According to some embodiments, the transmitter is further configured to: monitoring for a return signal from the receiver; and adjusting the different protection techniques for subsequent transmissions.

According to some example embodiments of the present invention, in a method of transmitting video for a display panel by a transmitter, the transmitter in electronic communication with a receiver via a wireless communication channel, the method comprising: receiving, by a transmitter, a frame of video data from a data source; grouping, by a transmitter, bits corresponding to a frame of video data into a plurality of groups each corresponding to a plurality of importance levels; recombining, by the transmitter, the plurality of groups in order of importance to generate a recombined data frame having a group, the group corresponding to the plurality of data packet headers, having a highest importance level and arranged as a first of the plurality of groups; and transmitting, by the transmitter, bits corresponding to the frame of video data to the receiver for display on the display panel such that each of the plurality of groups is transmitted according to a different protection technique based on the importance levels corresponding to the plurality of groups.

According to some embodiments, the method further comprises inserting, by the transmitter, before each data packet header, a value indicating a length of data bits corresponding to each data packet header.

According to some embodiments, the method further comprises: monitoring, by the transmitter, a return signal from the receiver; and adjusting, by the transmitter, a different protection technique for subsequent transmission based on a change in at least one of the channel quality and the video quality.

According to some embodiments, the method further comprises: depending on the type of display device corresponding to the receiver, different protection techniques are selected by the transmitter for data corresponding to different regions of the image.

According to some example embodiments of the present invention, in a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprises: receiving, by a transmitter, a frame of video data from a data source; recombining, by the transmitter, the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames; generating, by the transmitter, a tag for each of the data packets to tag each of the plurality of data packets, the tag corresponding to a different relative importance level of the data packet; performing, by the transmitter, a different protection technique for each of the data packets based on the label corresponding to each of the data packets; and transmitting, by the transmitter, the data packets and the tags to a receiver for display on a display panel such that each data packet is transmitted according to a different protection technique based on the tag to which the data packet corresponds.

According to some embodiments, marking each of the data packets comprises adding a bit pattern to a header of the data packet indicating the relative importance level of the data packet.

According to some embodiments, the tag of each packet corresponds to a Modulation and Coding Scheme (MCS) value based on the packet's corresponding importance level.

According to some embodiments, more important data packets have a lower MCS value than less important data packets.

According to some embodiments, the tag of each data packet corresponds to a forward error correction coding rate based on the corresponding importance level of the data packet.

According to some embodiments, the method further comprises: packing, by a transmitter, a data packet into a plurality of aggregated MAC protocol data unit (A-MPDU) subframes; and transmitting, by the transmitter, the a-MPDU sub-frames to the receiver.

According to some embodiments, the method further comprises: tagging, by a transmitter, a plurality of Physical Layer Convergence Procedure (PLCP) protocol data unit (PPDU) frames using a same tag; packing, by a transmitter, a-MPDU sub-frames having the same importance level into a PPDU frame; and transmitting, by the transmitter, the PPDU frame to the receiver.

According to some embodiments, the method further comprises selecting, by the transmitter, different protection techniques for different regions of an image of the frame of video data according to a type of display device corresponding to the receiver.

According to some embodiments, in a transmitter transmitting data for a display panel to a receiver through a wireless communication channel, the transmitter is configured to: receiving a frame of video data from a data source; recombining the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames; generating a tag for each of the data packets to tag each of the plurality of data packets, the tag having a different relative importance level to the data packet; performing a different protection technique for each of the data packets based on the label corresponding to each of the data packets; and transmitting the data packets and the tags to a receiver for display on a display panel such that each data packet is transmitted according to a different protection technique based on the tag to which the data packet corresponds.

According to some embodiments, the transmitter is further configured to: packetizing the data packets into a plurality of aggregated MAC protocol data unit (A-MPDU) subframes; and transmitting the a-MPDU sub-frames to a receiver.

According to some embodiments, the transmitter is further configured to: marking a plurality of Physical Layer Convergence Procedure (PLCP) protocol data unit (PPDU) frames; packaging the A-MPDU sub-frames with the same importance level into a PPDU frame; and transmitting the PPDU frame to a receiver.

According to some example embodiments of the present invention, in a method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprises: receiving, by a transmitter, a frame of video data from a data source; recombining, by the transmitter, the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames; generating, by a transmitter, a tag for each of the data packets to tag each of the data packets, the tag corresponding to a different relative importance level of the data packet; and transmitting, by the transmitter, the data packets and the tags to a receiver for display on a display panel such that each data packet is transmitted according to a different protection technique based on the tag to which the data packet corresponds.

According to some embodiments, the method further comprises performing, by the transmitter, a different protection technique for each of the data packets based on the tag corresponding to each of the data packets.

According to some embodiments, the method further comprises: monitoring, by the transmitter, a return signal from the receiver; and adjusting, by the transmitter, the different protection techniques based at least on a change in at least one of the channel quality and the video quality.

Drawings

A more complete understanding of the present invention, as well as many additional features and aspects thereof, will become apparent by reference to the following detailed description when considered in conjunction with the accompanying drawings in which like reference numbers refer to similar components, and wherein:

fig. 1 is a block diagram illustrating a wireless data transmission system showing a high-level overview illustration of a cross-layer optimization system, in accordance with some example embodiments of the present invention;

FIG. 2 illustrates an example layer-based compression scheme used in accordance with some example embodiments of the invention;

FIG. 3 is a block diagram illustrating further details of an example architecture and some components of a cross-layer optimization system, according to some example embodiments of the invention;

FIG. 4 illustrates an example of segmenting pixels into different types, according to some example embodiments of the invention;

FIG. 5 shows an example of a field of view for one eye, according to some example embodiments of the invention;

FIG. 6 illustrates an example of reassembly of a packet or layer structure, according to some example embodiments of the present invention;

FIG. 7 illustrates an example head structure according to some example embodiments of the invention;

FIG. 8 illustrates an example reorganization of data, according to some example embodiments of the present invention;

FIG. 9 illustrates an example reorganization of pixel bits and data packets, according to some example embodiments of the present invention;

FIG. 10 illustrates available header fields under a wireless communication standard; and

FIG. 11 is a flow chart illustrating a process for cross-layer image optimization according to some example embodiments of the invention.

Detailed Description

Aspects of exemplary embodiments of the present invention relate to systems and methods for electronic data communication.

The detailed description set forth below in connection with the appended drawings is intended as a description of exemplary embodiments of a system and method for cross-layer image optimization (CLIO) for wireless video transmission over multi-gigabit channels in accordance with the present invention and is not intended to represent the only form in which the present invention may be constructed or utilized. This description sets forth the features of the invention in connection with the illustrated embodiments. It is to be understood, however, that the same or equivalent functions and structures may be accomplished by different embodiments that are also intended to be encompassed within the spirit and scope of the invention. As shown elsewhere herein, the same reference numbers are intended to refer to the same elements or features.

The future of display technologies includes a world full of inexpensive displays fed by various wireless streaming devices (cell phones, set-top boxes, projectors, etc.). High quality video transmission over wireless links presents challenges. Wireless devices are non-stationary, and wireless links have insufficient bandwidth and are susceptible to a variety of noise. The delay can also be high and variable, which is particularly detrimental to video. Due to the strict requirements for video transmission, a general design method in which different layers, such as an Application (APP) layer, a Medium Access Control (MAC) layer, and a Physical (PHY) layer, are independently designed, is disadvantageous for high data rate wireless data transmission. Thus, embodiments of the present invention provide a cross-layer approach in which information on one layer is used to change parameters on a different layer. This flexibility allows fast adaptation to fast changes in the radio link.

The IEEE802.11ad standard is capable of providing the required bit rate for uncompressed Full High Definition (FHD) wireless video. IEEE802.11ad operates in the 60GHz band using channels with a 2.16GHz bandwidth and provides a bandwidth of up to 4.6Gbps on the physical layer (PHY) using a single carrier, which is sufficient for uncompressed FHD video transmission. However, IEEE802.11ad can only obtain the maximum bandwidth in a particular deployment. For example, IEEE802.11ad requires that the transmitter and receiver be located within a short distance of each other and within line of sight (LoS). Accordingly, embodiments of the present invention provide an improved method for wireless data transmission.

According to several embodiments, various features of the present invention may be used to improve and guarantee QoS for video streams over wireless networks, including solutions across a Physical (PHY) layer, a Media Access Control (MAC) layer, or an Application (APP) layer. Thus, information from one layer (e.g., the MAC layer) may be used to optimize parameters in another layer (e.g., the APP layer). For example, in video streaming, the APP layer may use information about channel quality in rate control (network awareness). The lower layer may also be configured to use information about video traffic characteristics. According to various embodiments, the system employs dynamic profile partitioning, dynamic tagging of packets, unequal error protection for different layers in layer-based compression techniques, and importance level aware modulation and modulation/coding selection, packetization, and bit or pixel pruning.

According to one embodiment, a cross-layer approach may be used to optimize the perceptual quality in delay-constrained scalable video transmission. In addition, Unequal Error Protection (UEP) may be employed for each video layer according to packet loss visibility (packet loss visibility) in the PHY layer. Buffer aware source adaptation also exists in the APP layer. Based on the IEEE802.11 Media Independent Handover (MIH) framework, a rate adaptation scheme is used for the QoS-driven seamless handover scheme. To control the rate, the Quantization Parameter (QP) is applied to single layer coding (AVC/h.264) and the enhancement layer is discarded for scalable coding (SVC/h.264). Rate and traffic adaptation and admission control and automatic layer management for optimizing QoS in terms of the number of sessions allowed in wireless video transmission and video quality are also included. Traffic management, path selection and frame filtering are included as cross-layer optimization techniques for video streaming over UDP/RTP in cellular networks. The cross-layer framework includes video codec optimization, slice optimization for layer coding transmission, and adaptive Modulation and Coding Scheme (MCS) to optimize user QoS level and energy efficiency of the wireless multimedia broadcast receiver in consideration of display size and energy constraints.

Compression, in essence, removes redundancy from the source and is therefore inherently more sensitive to transmission errors. Compression systems may use several methods to mitigate the effects of transmission errors in video streams. If the video stream has quality, spatial or temporal scalability, we can use the concept of Unequal Error Protection (UEP) to provide a higher level of protection for the more important information bits. On the receiver side, if the video stream has error recovery properties, error propagation is minimized and a large percentage of errors can be ignored. After decoding, error concealment can be applied to the decoded output. One technique for using time domain error concealment is to save the previous frame and replay it when the current frame is corrupted. Another possibility is to use surrounding pixels of the error region to predict the current pixel. Typically, error concealment is used as the last resort if other methods fail.

Fig. 1 is a block diagram illustrating a wireless data transmission system 100 according to some example embodiments of the invention. The wireless data transmission system 100 includes a transmitter 102, the transmitter 102 configured to receive data (e.g., uncompressed video data) at an input 103 from a data source 104 (e.g., an external data source such as a video card computer system). The wireless data transmission system 100 further comprises a receiver 106. The receiver 106 may be incorporated into a user device or electronic device 108 (e.g., an electronic display device, a smart phone, a television, a tablet, etc.) having a display panel 110 configured to display images and video. The display panel 110 includes a plurality of pixels, each of which may include a plurality of different color components (or sub-pixels) (e.g., each of which may include a red component, a green component, and a blue component).

In the case of a video, each frame of the video is displayed (e.g., momentarily) as an image on the display panel 110. In some embodiments, the display panel 110 may be included as part of any electronic device 108 with an integrated display, such as a television, monitor, cellular phone, tablet computer, wearable device, Augmented Reality (AR) headset, or Virtual Reality (VR) headset, for example.

Further, the transmitter 102 is in wireless data communication with the receiver 106 via

radio stations

112 and 114 incorporated in the transmitter 102 and the receiver 106, respectively. Accordingly, the transmitter 102 and receiver 106 are configured to transmit data to and from each other using any suitable wireless data spectrum and standard, such as the Institute of Electrical and Electronics Engineers (IEEE)802.11 standard. For example, the transmitter 102 is configured to transmit a data signal along with parameters or characteristics of the cross-layer image optimization scheme to the receiver 106 over a direct (e.g., data) channel 130 (e.g., a wireless communication channel), and the receiver 106 is configured to transmit feedback data or a return signal (e.g., including channel or visual quality data) to the transmitter 102 over a return channel 132.

Transmitter 102 further includes, among other elements, an Application (APP) layer 116 (including, for example, a display protocol and video encoder module), a Medium Access Control (MAC) layer 118, and a Physical (PHY) layer 120. In accordance with embodiments of the present invention, various aspects of cross-layer image optimization may be accomplished or performed in the APP layer 116, the MAC layer 118, and the PHY layer 120. Similarly, the receiver 106 includes an APP layer 122, a MAC layer 124, and a PHY layer 126.

The wireless data transmission system 100 may further include a cross-layer image optimization (CLIO) module (or CLIO engine) 134 operating as part of the transmitter 102 or in coordination with the transmitter 102 to control the system and method of cross-layer image optimization. The CLIO module 134 may include, for example, a processor and a memory coupled to the processor that stores instructions that, when executed by the processor, cause the processor to perform various operations of the cross-layer image optimization systems and methods described herein. For example, the CLIO module 134 may be in electronic communication with the APP layer 116, the MAC layer 118, and the PHY layer 120 to exchange data and commands back and forth that implement the cross-layer image optimization systems and methods described herein.

Thus, as shown in fig. 1, the wireless data transmission system 100 is configured to receive data at the transmitter 102 and transmit data to the receiver 106 after implementing various mechanisms of cross-layer image optimization in the APP layer 116, the MAC layer 118, and the PHY layer 120.

A video compression/decompression system or codec can be evaluated in essentially six different ways: compression, quality (distortion), complexity, latency, quality scalability and error resilience.

According to some example embodiments of the present invention, a layer-based compression architecture may be used for data or video compression, where a video or data stream is parsed or divided into multiple quality layers, resolution layers, spatial locations, or color components. In related art layer-based compression architectures (e.g., JPEG2000), a data stream starts with a header followed by a series of headers and bits. However, the data streams may be ordered out of order of importance, especially for real-time UHD codecs.

In one embodiment, a layer-based compression system (such as or similar to the JPEG2000 standard) is used as the layer-based compression as part of a cross-layer image optimization system and method (e.g., the cross-layer image optimization system and method performed by CLIO module 134). A layer-based encoder according to an embodiment of the present invention may employ the following steps: color conversion of the components, wavelet conversion on each color channel, quantization of the wavelet coefficients, and final arithmetic coding of rectangular regions of wavelet coefficients called code blocks. Such encoders provide flexibility in the structure of the encoded data by allowing various sequences of these code blocks based on color components, resolution level, spatial block position, or layer. Individual code blocks may be independently decodable, but the headers of these code blocks may depend on the previous header and thus may be decoded in sequence.

A layer-based stream of compressively-encoded data used in accordance with an embodiment of the present invention may start with a frame header followed by a series of data packet headers and data bits. As described above, the data stream may be divided into multiple quality layers, resolution levels, spatial locations, and color components. However, depending on the particular encoding system used (e.g., JPEG2000), the data streams may be ordered out of order of importance, particularly for real-time UHD codecs. The example in fig. 2 shows an example of a layer-based compressed encoded data stream generated in layers; for example, the first layer may correspond to a compression ratio of 8: 1; at the end of the second layer (all of the first layer packets have been decoded), the compression rate is set to 4: 1. Finally, decoding the third layer packet may be equivalent to transmitting the data stream at a 2:1 compression rate. Embodiments of the present invention are not limited to the compression ratios or the number of layers described above. Rather, depending on the design and functionality of the cross-layer image optimization system and method, embodiments of the present invention may use any suitable number of layers having any suitable compression rate, such that the first or highest layer has the highest compression rate, the next layer has data that when added to the first layer results in a compression rate lower than that of the first layer, and so on, until the lowest or last layer has data that when added to each of the aforementioned layers results in a relatively low compression rate, is visually lossless, or is approximately visually lossless for the viewer (e.g., 1:1, 1.5:1, 2: 1).

According to some embodiments, in the case of VR display, when a stereoscopic image is transmitted to a receiver, an error in data transmission may affect both left and right images even if the error is only on one side. Thus, in some embodiments of the present invention, a cross-layer image optimization system and method (e.g., the CLIO module 134 and/or the video encoder module of the APP layer 116) may perform compression on the left image plus right image differences and then ensure that the respective left image plus right image difference slices are encoded into a multiplexed data stream. Such a process may reduce (e.g., by half) the number of instances in which errors in the data transfer affect the visibility of the product.

Embodiments of the present invention in a cross-layer algorithm for improving video stream quality may further use one or more features of the PHY layer, such as dynamic Adaptive Modulation and Coding (AMC), where the Modulation and Coding Scheme (MCS) is changed based on channel quality indicators in the PHY layer. Lower MCS values provide lower bit error rates, but lower throughput may also be achieved. The MAC layer may inform the APP layer of the MCS to adjust the compression rate based on the MCS.

The different bits and packets lost have different effects on video quality and embodiments of the present invention operate to protect the more important bits in video reception using UEP methods, including more protection of the Most Significant Bit (MSB) in uncompressed/compressed video, more protection of the packet header in JPEG2000, or more protection of the I-frame in MPEG-4. Protection may be achieved by using a more robust MCS, retransmission, or additional error correction codes like Reed-Solomon.

Fig. 3 is a block diagram illustrating an example architecture and some further details of some components of a cross-layer optimization system of a wireless data transmission system 100, including control schemes at the APP, MAC and PHY layers, according to some example embodiments of the present invention. The measurement feedback block is used to estimate conditions such as bit error rate, video quality, and available bandwidth for certain service classes. Models in the APP and MAC layers can then minimize distortion in video quality by optimal bit allocation, reduce the number of bits required for forward error correction, and determine the priority of packets based on the impact of packet loss.

In some embodiments, as shown in fig. 3, in the APP layer, the system may enable/disable optical compression (light compression) based on feedback and measurements from the receiver (e.g., based on data received from the receiver 106 on the return channel 132). The feedback may for example be based on video quality and the measure of video quality may for example be based on a signal-to-noise ratio (SNR) or a peak signal-to-noise ratio (PSNR) reported by the receiver.

The MAC layer in fig. 3 may include many functions, modules or procedures, including, for example, pixel segmentation, retransmission and multiple CRC checks, packetization with features like MSB/LSB separation, and different buffers. Based on feedback received from the PHY layer or parameters in the APP layer, the system (e.g., CLIO module 134) may change parameters for these functions in the MAC layer and determine how these functions, modules, or programs should work. For example, the system may determine whether to perform pixel segmentation and, if so, how the segmentation is divided into data packets. Further, the system may be configured to determine or select whether retransmission is enabled.

According to some embodiments, a system (e.g., CLIO module 134) may use these features in UEP implementations. For example, some packets may have these characteristics (e.g., retransmissions), or the characteristics may have different parameters for different packets (e.g., different MCSs or MSB/LSB separation).

As shown in fig. 3, PHY layers, such as PHY layer 302 (with corresponding PHY layer 126 in the receiver), and PHY layer 300 (with corresponding PHY layer 120 in the transmitter) may also be incorporated into the various layers for implementation of UEP. Different data packets may be sent through different layers with different MCS values. Different layers may also be associated with different buffers having different priorities. According to some example embodiments, a system (e.g., CLIO module 134) may associate different packets to different layers and buffers according to video quality and measurement feedback.

According to some example embodiments of the present invention, the first step in the cross-layer image optimization for uncompressed or compressed transmission is layering or segmentation. Embodiments of the present invention utilize a plurality of predefined hierarchical and/or split profiles as described in more detail below. As part of a data transmission session, one of the different hierarchical and/or segmentation profiles is selected based on one or more predefined parameters including, for example, the type of display device, different criteria or indicators of video visual quality, including peak signal-to-noise ratio (PSNR), channel quality, codec requirements or selections, MCS and data rate requirements, etc. Each profile indicates or is defined by a different set of procedures in the transmitter and receiver including, for example, different error correction techniques, compression or packetization.

Furthermore, embodiments of the invention may use Unequal Error Protection (UEP) methods as part of cross-layer image optimization for ordered bitstreams of different layers. The ordering of the bitstream or data packets may be performed on layer-based compressed video or uncompressed video. Different layers use different MCS values according to channel quality.

According to some example embodiments of the invention, dynamic marking of packets (e.g., packets in IEEE802.11 ad) may be used as part of the cross-layer image optimization systems and methods of the invention. Because the importance level awareness procedure is implemented in the MAC packetizer for the transmitter and the depacketizer at the receiver, the importance levels and corresponding information can be transmitted or signaled to the receiver.

As described above, in accordance with embodiments of the present invention, the system may use a dynamic layering and segmentation profile as part of the cross-layer image optimization system and method of the present invention. Both the transmitter and the receiver (e.g., transmitter 102 and receiver 106) may use or agree to a number of predefined hierarchical/split profiles. These profiles may be selected based on different standards for video visual quality (including PSNR), channel quality, codec requirements or choices, MCS, data rate requirements, etc. The receiver then selects the respective algorithm to run to reverse the layering method and/or further correct any transmission errors using the error concealment algorithm corresponding to the selected profile.

In one embodiment, although the system uses 7 profiles for various channel conditions, the number of profiles is not limited thereto, and any suitable number of profiles may be used according to the design and function of the wireless data transmission system.

For example, profile "00" may be defined for when a channel operates under ideal or near ideal conditions, while profile "06" may be defined for when a channel operates under very poor conditions. Then, a profile between "00" and "06" is defined for each intermediate condition.

For example, one profile (e.g., profile "00" or ideal/near ideal condition profile) may be defined for uncompressed video transmission at channel quality. For such a profile, the pixel segmentation technique may assume a high probability that pixels in close proximity to each other contain very similar content. If any pixel in a group of 4 pixels is missed, it can be recalculated from the other 3 captured pixels. As shown in FIG. 4, profile "00" may be selected by dividing the image into 4 types of pixels. Pixels of three colors of one type are packed into 1 packet. All four types of packets are transmitted to the receiver. A receiver that knows that profile "00" is transmitted can improve image quality. For example, if pixel data for any pixel (or any color in a pixel) is missing, the receiver runs an averaging method to replace the missing pixel or color by averaging groups of 4 or 8 surrounding pixels (the number of averaged pixels may be dynamic). Image compression methods also rely on the idea that pixels near each pixel will have a correlation value. This method can be implemented more simply and more robust to bit errors than entropy coding compression algorithms.

The second profile (e.g., profile "01") may be selected by dividing the image into 4 types of pixels similar to the first profile (profile "00"). However, three types of pixels are selected for transmission. The system calculates whether the distortion due to dropping one type of pixel is below a threshold. If the profile is signaled to the receiver, the receiver will estimate the missing pixel data value based on the average of three (or more) other pixels within each of the four pixel types. The threshold may be selected based on different criteria including, for example, visual quality (e.g., PSNR). The selection of the type of pixels not transmitted and the threshold may also be signaled.

The third profile (e.g., profile "02") may operate (e.g., be selected accordingly) on a smaller bandwidth than the first two profiles (e.g., profiles "00" and "01") described above. For example, the third configuration file may not transmit the Least Significant Bits (LSBs) of the pixels in one or all of the data packets. The averaging algorithm can then compensate for these bits in the receiver. A third profile (e.g., profile "02") for uncompressed video may be selected in consideration of another threshold level for distortion rather than the first or second profiles described above (e.g., profiles "00" or "01").

When the quality of the channel is low and the PHY layer signals a lower value for the MCS, other profiles may be considered for compressing the video. For example, multiple quality hierarchies may be used to define a new profile for wireless video transmission. In layer-based compression, as described above, multiple layers may be generated. According to embodiments of the invention, the highest layer may include data from the plurality of layers having the highest compression rate, and the lowest layer may include data that, if added to the higher layers, would form an image having a relatively low (e.g., 1:1 or 1:1.5, or any suitable or visually lossless compression rate). For example, in a three-layer compression scheme (e.g., as shown in FIG. 2), layer 1 may include data for a compressed image having a compression rate of 4: 1. Layer 2 may include compressed image data that, if added to layer 1, would result in a compression ratio of 2: 1. Layer 3 may comprise a bit stream, and if added to

layers

1 and 2, the resulting image has a compression ratio of 1:1 (or 1.5:1 or any other visually lossless compression ratio).

The transmitter selects which layer to send based on channel quality, visual quality, codec requirements, etc. We can define a fourth profile (e.g., profile "03") to send layer 1, layer 2, and layer 3 (or all layers), a fifth profile (e.g., profile "04") to send layer 1 and layer 2 (or a subset of layers), and a sixth profile (e.g., profile "05") to send only layer 1 or the layer with the highest compression ratio. For the seventh profile (e.g., profile "06"), the transmitter may transmit a portion of one of the layers, e.g., layer 1 and layer 2 and only a portion of layer 3. The receiver, by knowing that the seventh profile (e.g., profile "06") is sent, will look for missing data information or run additional correction algorithms. The transmitter may indicate the length of the original data packet and the length of the transmitted data packet in the data packet header. Thus, the receiver knows that a portion of the layer has not been received and can infer the correct size of the received data packet. Likewise, the receiver sends an alert to the decoder that the portion of the data packet was not received, and the receiver may request the transmitter to resend or retransmit the additional information.

Additional profiles may be established based on the type of data being transmitted, the nature of the display on which the image is to be displayed, or the importance of a particular portion of an image relative to other portions of the same image. For example, according to some embodiments, a particular portion or region of an image may be more important than other portions of the image, meaning that errors in the data transmission of bits corresponding to one portion of the image may be more perceptible to a viewer than errors in the data transmission of bits corresponding to another portion of the image.

According to some embodiments, the cross-layer image optimization systems and methods may be deployed in the context of a VR display system, for example where the display (e.g., display panel 110) is a VR display. In this case, because users tend to move their heads to focus on different portions of the image, a large portion (e.g., 90%) of the viewer's perception of the image is focused at and around the center of the image (e.g., the center 15 degree field of view). Thus, as shown in fig. 5, the cross-layer image optimization system and method (e.g., CLIO module 134) may prioritize data corresponding to a first or central region of an image (e.g., a 15 degree field of view, a region labeled with label "a" in fig. 5) over a second region of the image (e.g., a region labeled with label B in fig. 5, e.g., between a second field of view outside the first region and the first region, or between a 15 degree and 20 degree field of view). Further, both the first and second regions of the image may be prioritized over a third region of the image (e.g., the region labeled "C" in fig. 5, e.g., outside of the first and second regions, or outside of the 20 degree field of view).

Another method of identifying areas of an image having a higher importance or priority is described in U.S. patent application No. 15/288,977, which is incorporated herein by reference in its entirety. For example, according to some embodiments, data corresponding to an area of an image having a higher importance may be designated as having a higher importance level or a higher degree of error protection than data corresponding to an area of an image having a lower importance. The area of the image with higher importance may vary depending on the type of display device corresponding to the receiver, e.g., as discussed herein with reference to fig. 5.

Because different regions of an image may be prioritized differently, a cross-layer image optimization system and method (e.g., CLIO module 134) may select a profile (e.g., a layering/segmentation profile as discussed above) or UEP scheme (described in more detail below) such that data in a first region has the lowest error level, a second region of the image has a second lowest error level, and so on, as compared to data in other regions of the image. Further, according to some embodiments, the cross-layer image optimization systems and methods (e.g., CLIO module 134) may segment the data such that errors from the outer regions do not propagate to the highest priority (e.g., center) region of the image.

According to some example embodiments, the cross-layer image optimization system may use various profiles (e.g., first through seventh profiles), such as those discussed above, to prioritize different portions of an image for transmission, such that a smaller number of profiles (e.g., a less error prone profile) is selected for higher priority portions of the image and a larger number of profiles (e.g., a more error prone profile) is selected for lower priority portions of the image. According to some embodiments, the individual profiles may be specifically predefined according to the type of display device (e.g., television, VR display, AR display, wireless medical display, etc.) so that data is transmitted in different programs in transmission (e.g., compressing, encoding, packetizing, dropping or deleting bits) or reception (e.g., decompressing, decoding, error correcting, depacketizing or retransmitting) according to the priority of the portion of image data being exchanged.

In one embodiment, the selection criteria for a layering or partitioning profile may be based on video visual quality (e.g., PSNR) and/or channel quality (e.g., SNR). The hierarchy identifies whether uncompressed video should be sent, uncompressed video with some packets lost, or compressed video with different compression rates. Different layers may handle different processing depending on prioritization, delay tolerance, error protection, and/or reliability. Accordingly, by selecting a profile or receiving a profile number, the transmitter and receiver may enable different procedures prior to or as part of data transmission (e.g., compressing, encoding, packetizing, dropping or deleting bits) or reception (e.g., decompressing, decoding, error correcting or depacketizing).

According to some embodiments, the selection of the configuration file may be transmitted to the receiver by bit selection in a MAC header packet or a PHY header packet. Based on the bit selection, the receiver may identify the selected hierarchical method. In addition to the profile number, any other information required by the selected profile may be transmitted to the receiver, including, for example, the number of layers compressed and/or the number of packets/bytes in each layer. The additional information may be sent via a bit pattern in the data packet header or via a secure channel, or may be predefined.

Accordingly, one or more example embodiments of the present invention include systems and methods for selecting a set of predefined hierarchical and/or partitioned profiles for transmitting data (e.g., uncompressed video data) from a transmitter to a receiver over a wireless data communication channel. The profile may be selected based on various conditions or parameters including, for example, channel quality, video quality, codec requirements or selections, MCS and data rate requirements, the nature or characteristics of the data being transmitted, the nature or characteristics of a display device at the receiver end (e.g., the receiver end coupled to the receiver), etc. For example, the selection of the profile may be transmitted from the transmitter to the receiver through bit selection in the MAC layer header packet and/or the PHY layer header packet. Depending on the selected profile, additional information may also be transmitted from the transmitter (e.g., in MAC layer header packets and/or PHY layer header packets) to the receiver indicating one or more parameters for decompression, decoding, error correction, etc. Based on information (e.g., bit selection) transmitted from the transmitter to the receiver, the receiver may be enabled to identify a method and/or protocol used at the transmitter for data layering and/or segmentation and/or compression, and thereby select a corresponding algorithm or program to perform, to reverse layering and/or segmentation and/or compression, and also to correct any errors in the transmission using an error concealment algorithm or program corresponding to the selected profile.

Embodiments of the present invention may further use priority-based Unequal Error Protection (UEP) to protect (e.g., facilitate proper transmission) more important bits, e.g., more important bits for video decoding purposes. For example, in layer-based compression, bits associated with or corresponding to a higher compression rate (e.g., a higher compression rate corresponding to the base layer or layer 1) are more important in video decoding and therefore require more protection. In addition, bit errors in the header or header of the data packet result in the video decoder being unable to decode subsequent data bits. In one embodiment of the cross-layer image optimization system and method, the packet header is moved to the beginning of each frame, as shown in FIG. 6. In addition, the data packet header may be identified by a unique start value and end value, as shown in fig. 7. When the header is moved to the beginning of the frame, the identifiers are also moved to separate the headers from each other. According to some example embodiments, the ending identifier may also be removed. The last end identifier may be reserved to distinguish between the end of the header information and the data information in the next layer.

According to some example embodiments, cross-layer image optimization systems and methods (e.g., CLIO module 134) may use different protection techniques for those headers and base layers to resist loss or errors in the wireless medium. For example, different MCS values may be used for different layers, or different Forward Error Correction (FEC) algorithms may be used at different coding rates, according to some embodiments. In addition, unequal error protection operations may be performed at the MAC layer, thereby enabling embodiments of the present invention to improve or optimize the reassembly engine and also apply the reassembly engine to different codecs. For example, when the APP layer passes the entire data to the MAC layer, the MAC layer has more flexibility to perform the various cross-layer image optimization operations described herein. Furthermore, by performing UEP in the MAC layer, the system is relatively more dynamic because, as described herein, different functions may be modified using different parameters according to the particular situation. Furthermore, embodiments of the present invention are thus able to perform only partial reassembly based on the state of the wireless channel.

In addition to layers of different compression rates, the compression techniques used in accordance with embodiments of the present invention may provide additional segmentation techniques by providing different data components as shown in FIG. 8. For example, various similar pixel bits (e.g., pixel bits having the same or similar MCS values) may be reassembled in a data packet to be grouped together. Any suitable compression technique that can also separate its output into multiple layers of different priorities can be used. The same method of moving the head and data may be used. Different MCS values or FECs with different coding rates may be used for the header or partition of the data.

In one embodiment, the header includes information about the length of the data packet in the frame that comes after the header. In some cases, the cross-layer image optimization systems and methods may transmit a portion of the data bits in each layer or partition according to channel conditions, MCS conditions, bandwidth conditions, and the like. Therefore, the length of the data stream that can be transmitted can also be appended to the header information and transmitted to the receiver. For example, a fixed or predetermined number of bytes (N bytes) may be reserved to identify the length. According to some example embodiments, the length of the data stream may be at the end of each header for ease of identification.

Thus, systems and methods according to some example embodiments of the present invention may be configured to re-segment or re-group bits in a layer or in a layer-based compression system according to a priority, delay tolerance, and/or protection level or reliability level of transmissions over a wireless data communication channel. A frame of data includes frame header bits and packet header bits that require a higher level of protection than the data bits. The data bits in a frame may also be partitioned into different priority levels, protection levels, and/or reliability levels. The header may be identified according to a predefined or known bit string, and the data bits may be ordered according to an order of importance. According to some embodiments, bits of higher priority or smaller delay tolerance may be moved or reassembled to the beginning of a frame of data. For example, a frame of data may begin with a header, followed by bits of data (e.g., bits of data in order of priority or importance). The data packet header may be separated or identified by a predefined or predetermined bit string. In addition, according to some embodiments, the length of the data bit stream may be indicated by inserting an indicator of the length corresponding to each data packet header into the data packet header, before inserting the data packet, or after inserting the data packet. The decoder at the receiver reconstructs the frame, for example by moving the data packet header and other data bits to their original positions (before reassembly), and removing any additional inserted data such as data length bits, but uses the additional data to reconstruct the frame.

Thus, embodiments of the present invention may provide implementations of unequal error protection for any layer and/or level-based compression technique for cross-layer image optimization. Headers, layers, and/or data partitions may be reassembled according to importance or priority levels, and such reassembly may be performed in the MAC layer of the transmitter. According to a cross-layer image optimization scheme, some portion of the data from each packet may be removed or dropped and information about the data may be incorporated into the header information. Furthermore, embodiments of the present invention may use unequal error protection, e.g., to protect information with different levels of importance by using different MCS values for forward error correction, or in different ways. As part of unequal error protection according to some example embodiments of the invention, because all headers may be reassembled to come together (e.g., at one location), all headers may be subjected to an unequal error protection algorithm rather than, for example, using multiple small forward error corrections for each of the headers of a frame individually.

According to some example embodiments of the present invention, because different data packets may be protected differently using UEP and layer/partition-based compression, the loss of different data packets affects visual quality differently. If some packets in the base layer or header are lost, for example, the system may not be able to reconstruct the video at the receiver side. On the other hand, some packet loss may cause PSNR degradation. Further, as discussed above, depending on the type of display device used in the system, different portions of the image data may have a higher priority in minimizing errors than other portions of the image data. Accordingly, embodiments of the present invention may further use dynamic UEP and importance awareness tagging, as described in more detail below.

As described above, the cross-layer image optimization system and method according to embodiments of the present invention operates based on priority, delay tolerance, protection, or reliability to segment data packets into different levels of importance. Each importance level may be handled differently in transmission and/or reception.

In one embodiment, each frame is divided into a plurality of video data packets having different levels of importance. The system then marks each packet with a corresponding importance level, and the corresponding bits are packed in an aggregated MAC protocol data unit (a-MPDU) sub-frame and a Physical Layer Convergence Procedure (PLCP) protocol data unit (PPDU) frame. In one example, a PPDU may be tagged with an importance level and dynamically pack data packets and a-MPDU sub-frames having the same importance level. If retransmission is enabled, the A-MPDU sub-frames that need to be retransmitted are required to be queued and placed in the appropriate PPDU with the appropriate level of importance.

According to another embodiment, each a-MPDU sub-frame may be marked according to an importance level. In this case, all PPDUs are considered equal in importance level and are packed with a-MPDUs of different importance levels. In delay sensitive scenarios where retransmission is required, it may be advantageous to flag a-MPDU sub-frames instead of PPDU packets. While marking a-MPDU sub-frames instead of PPDU packets may use more overhead for tag information, this approach may more easily manage packets of multiple importance levels.

According to various embodiments, a bit pattern may be added to the header of an A-MPDU or PPDU to mark the packet. The bit pattern may be M bits, with 0.. 0 representing the highest importance and 1.. 1 representing the lowest importance. According to some embodiments, the value M may be predefined, or the value M may be signaled as part of the header information. According to some embodiments, the transmitter and receiver may define 2 for each level^MAn importance level and different programs.

For guaranteed quality of service for wireless video transmission through IEEE802.11ad or IEEE802.11 ay, according to some example embodiments of the present invention, a cross-layer image optimization system and method may implement UEP for each marked importance level through dynamic importance level aware AMC. Different importance levels may be modulated using different MCS values, or different importance levels may be encoded with different Forward Error Correction (FEC) values, or using the same FEC values with different coding rates.

In a related art system, the PHY may recommend an MCS index suitable for the current channel situation. However, according to some embodiments of the present invention, the cross-layer image optimization systems and methods may adapt the recommended MCS for most of the bit stream. For example, the system may decrease the MCS index for more important layers or packets and increase the MCS index for less important layers or packets. Fig. 9 shows an example for uncompressed video transmission, where the system selects different MCS values for the MSB/LSB portions of the bits representing the RGB colors in the pixel. The four MSB bits are transmitted at MCS-1, the next 2 bits are transmitted at MCS, and the last 2 LSB bits are transmitted at MCS + 1. The example also includes recombining bits into the packet/a-MPDU sub-frames such that bits having the same MCS index are grouped into the same packet/a-MPDU sub-frame.

For example, for compressed video transmission, after layer-based compression is performed and bits are reassembled according to the importance level of the transmission, as discussed above with respect to fig. 6 and 8, header and base layer 1 are sent at MCS-1, layer 2 is sent at MCS, and finally layer 3 may be sent at MCS or MCS + 1. Thus, the header and layer 1 are more protected and transmitted with almost no errors. Since these information bits are sent at MCS-1, the required bandwidth may exceed the supported bandwidth of the channel with the MCS. To compensate, portions of layer 3 may be dropped and/or layer 3 bits may be modulated with MCS + 1.

In IEEE802.11ad, the MCS index is signaled only for each PPDU packet of 262143 bytes. Thus, if the PPDU is marked with an importance level, embodiments of the present invention may use an IEEE802.11ad procedure to use a recommended MCS index or a modified MCS index for each packet. According to some embodiments, signaling in IEEE802.11ad may be modified to signal MCS for each a-MPDU subframe. For example, currently reserved bits or unused information in the header of each packet may be used. According to the IEEE802.11ad specification, the header includes the fields in FIG. 10. As an example, in a peer-to-peer connection, embodiments of the present invention use "address 4" to signal the MCS. Embodiments of the present invention may additionally add new bytes to the header to signal the MCS for each a-MPDU.

Thus, cross-layer image optimization systems and methods according to some example embodiments of the invention may operate to segment data (e.g., pixel data, packets, etc.) of a video frame into different importance categories based on, for example, priority, delay tolerance, and/or protection or reliability. Each video frame may be divided into multiple importance levels, and corresponding bits may be packed into a-MPDU sub-frames. Different data packets (e.g., a-MPDU sub-frames, PPDU sub-frames, etc.) may be marked according to the importance level, and according to some embodiments, different data packets may be transmitted to the receiver according to the order of their importance. Different packets (e.g., a-MPDU sub-frames) may be grouped according to their importance levels (e.g., into PPDU packets). A value or label indicating the importance level may be added to the header of the data packet. At the receiver end, the receiver may be enabled to determine the tag level and thereby further enable different error recovery and/or concealment procedures depending on the marking. Furthermore, data having different importance levels may be protected differently by selecting different Modulation and Coding Scheme (MCS) values for each data packet. In addition to the tag, a corresponding MCS value may be inserted into the header of the packet.

Thus, embodiments of the present invention may enable dynamic segmentation and/or tagging of data packets (e.g., in IEEE Multi-Gigabit 802.11) and significance awareness procedures, including wireless cross-layer image optimized AMC, in transmitters and receivers for wireless data transmission. Embodiments of the present invention may enable tagging each small packet within an a-MPDU (data) packet using multiple content driven tags. The tag may be transmitted and signaled to a receiver. Different programs or parameters may be associated with tags incorporating various aspects of the cross-layer image optimization mechanisms discussed herein. In addition, embodiments of the present invention may use unequal error protection based tags, such as AMC (MCS adaptation for tags).

According to various embodiments of the invention, the system may continuously monitor various parameters and factors for cross-layer image optimization and make adjustments accordingly. For example, as described above, the system may monitor the return signal from the receiver for subsequent data transmissions and adjust the cross-layer image optimization accordingly as the environment changes, e.g., by selecting different profiles, compression or error correction techniques, etc.

FIG. 11 is an example flow diagram illustrating various operations in a method for cross-layer image optimization according to some example embodiments of the invention. However, the number and order of operations shown in FIG. 11 are not limited to the flow chart shown. For example, according to some embodiments, the order of the operations may be modified (unless otherwise specified), or the process may include additional operations or fewer operations, without departing from the spirit and scope of the present invention. For example, various other features and operations described in this disclosure and the corresponding figures may be incorporated into the process or certain operations may be omitted in accordance with some example embodiments of the invention.

The process begins and at 1200 a system (e.g., wireless data transmission system 100 and/or transmitter 102) receives data (e.g., a video data signal including, for example, uncompressed video data) from a data source. At 1202, return data is received from a receiver (e.g., receiver 106), the return data including various information regarding data transmission to the receiver including, for example, channel quality, visual quality, and type of display device connected to the receiver. Then, at 1204, the system can identify the type of display device. At 1206, a layering/partitioning profile is selected based on one or more predefined parameters, and then transmitted, the one or more predefined parameters including, for example, a type of display device, different criteria or indicators of video visual quality including peak signal-to-noise ratio (PSNR), channel quality, codec requirements or selections, MCS and data rate requirements, and the like.

At 1208, layer or level based compression can be performed on the data, e.g., according to the selected layering/segmentation profile. Then, at 1210, the layers, partitions, packets, or data bits are reassembled into groups or packets according to the importance levels, as described above.

At 1212, the data may be marked to indicate an importance level of the data and/or data packets of the data. Then, at 1214, unequal error protection may be performed to transmit the data to the receiver based on the different types of importance levels of the data. At 1216, the data is transmitted to a receiver, e.g., for display on a display panel.

A wireless data transmission system and/or any other relevant devices or components according to embodiments of the invention described herein may use any suitable hardware, firmware (e.g., application specific integrated circuits), software, or suitable combination of software, firmware and hardware. For example, the various components of the wireless data transmission system may be formed on one Integrated Circuit (IC) chip or on separate IC chips. In addition, the respective components of the display device may be implemented on a flexible printed circuit film, a Tape Carrier Package (TCP), a Printed Circuit Board (PCB), or formed on the same substrate as the display device. Further, the various components of the wireless data transmission system may be processes or threads running on one or more processors in one or more computing devices, executing computer program instructions and interacting with other system components to perform the various functions described herein. The computer program instructions are stored in a memory that can be implemented in a computing device using standard memory devices, such as Random Access Memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media, such as CD-ROMs, flash drives, etc. In addition, those skilled in the art will recognize that the functionality of the various computing devices may be combined or integrated into a single computing device, or that the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the scope of the exemplary embodiments of this invention.

While the present invention has been described in particular embodiments, those skilled in the art will readily devise variations of the described embodiments that do not depart from the scope and spirit of the invention. Moreover, the invention itself will suggest solutions for other tasks and be suitable for other applications for those skilled in the various arts. It is the applicants' intention to cover by the claims all such uses of the invention and those changes and modifications which have been selected for purposes of disclosure to be made to the embodiments of the invention herein without departing from the spirit and scope of the invention. The present embodiments of the invention are, therefore, to be considered in all respects as illustrative and not restrictive, and the scope of the invention is indicated by the appended claims and their equivalents rather than the foregoing description.

Claims

1. A method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprising:

receiving, by the transmitter, a frame of video data from a data source;

recombining, by the transmitter, the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames;

receiving, by the transmitter, a return signal from the receiver operating on the display panel;

generating, by the transmitter, a tag for each of the plurality of data packets to tag each of the plurality of data packets, a plurality of the tags corresponding to different relative levels of importance of the plurality of data packets, wherein the tag of each data packet corresponds to a modulation and coding scheme value based on the corresponding level of importance of the data packet and the return signal;

marking, by the transmitter, each of the plurality of data packets with a corresponding importance level and corresponding bits are packed in an aggregate MAC protocol data unit subframe and a physical layer convergence process protocol data unit frame, wherein the marking is for the aggregate MAC protocol data unit subframe;

performing, by the transmitter, a different protection technique for each of the plurality of data packets based on the tag corresponding to each of the plurality of data packets; and

transmitting, by the transmitter, the plurality of data packets and the plurality of tags to the receiver for display on the display panel such that each of the plurality of data packets is transmitted according to the different protection technique based on the tag to which the data packet corresponds.

2. The method of claim 1, wherein marking each of the plurality of data packets comprises adding a bit pattern to a header of the data packet indicating a relative importance level of the data packet.

3. The method of claim 1, wherein more important data packets have lower modulation and coding scheme values than less important data packets.

4. The method of claim 1, wherein the label for each data packet corresponds to a forward error correction coding rate based on the corresponding importance level of the data packet.

5. The method of claim 1, further comprising:

transmitting, by the transmitter, the aggregated MAC protocol data unit subframe to the receiver.

6. The method of claim 1, further comprising: selecting, by the transmitter, the different protection techniques for different regions of an image of the frame of video data according to a type of display device corresponding to the receiver.

7. A transmitter for transmitting data for a display panel to a receiver over a wireless communication channel, the transmitter configured to:

receiving a frame of video data from a data source;

recombining the video data frames into a plurality of data packets according to the importance levels of the bits of the video data frames;

receiving a return signal from the receiver operating on the display panel;

generating a tag for each of the plurality of data packets to tag each of the plurality of data packets, a plurality of the tags corresponding to different relative levels of importance for the plurality of data packets, wherein the tag for each data packet corresponds to a modulation and coding scheme value based on the corresponding level of importance for that data packet and the return signal;

marking each of the plurality of data packets with a corresponding importance level and corresponding bits are packed in an aggregate MAC protocol data unit subframe and a physical layer convergence process protocol data unit frame, wherein the marking is for the aggregate MAC protocol data unit subframe;

performing a different protection technique for each of the plurality of data packets based on the label corresponding to each of the plurality of data packets; and

transmitting the plurality of data packets and the plurality of tags to the receiver for display on the display panel such that each of the plurality of data packets is transmitted according to the different protection technique based on the tag to which that data packet corresponds.

8. The transmitter of claim 7, wherein marking each of the plurality of data packets comprises adding a bit pattern to a header of the data packet indicating a relative importance level of the data packet.

9. The transmitter of claim 7, wherein the tag for each packet corresponds to a forward error correction code rate based on the packet's corresponding importance level.

10. The transmitter of claim 7, wherein the transmitter is further configured to:

transmitting the aggregated MAC protocol data unit subframe to the receiver.

11. A method of transmitting video for a display panel over a wireless communication channel by a transmitter in electronic communication with a receiver, the method comprising:

receiving, by the transmitter, a frame of video data from a data source;

marking, by the transmitter, each of the plurality of data packets with a corresponding importance level and corresponding bits are packed in an aggregate MAC protocol data unit subframe and a physical layer convergence process protocol data unit frame, wherein the marking is for the aggregate MAC protocol data unit subframe; and

transmitting, by the transmitter, the plurality of data packets and the plurality of tags to the receiver for display on the display panel such that each of the plurality of data packets is transmitted according to a different protection technique based on the tag to which the data packet corresponds.

12. The method of claim 11, wherein marking each of the plurality of data packets comprises adding a bit pattern to a header of the data packet indicating a relative importance level of the data packet.

13. The method of claim 11, further comprising:

14. The method of claim 11, further comprising:

monitoring, by the transmitter, a return signal from the receiver; and

adjusting, by the transmitter, the different protection techniques based on a change in at least one of channel quality and video quality.