WO2025215015A1

WO2025215015A1 - Rate control with reference picture resampling (rpr)

Info

Publication number: WO2025215015A1
Application number: PCT/EP2025/059588
Authority: WO
Inventors: Martin Pettersson; Lukasz LITWIC; Per-Erik Brodin; Jack ENHORN
Original assignee: Telefonaktiebolaget LM Ericsson AB
Current assignee: Telefonaktiebolaget LM Ericsson AB
Priority date: 2024-04-11
Filing date: 2025-04-08
Publication date: 2025-10-16
Anticipated expiration: 2026-10-11

Abstract

A sending device (200) encodes pictures to a bitstream (110). More particularly, the sending device determines a first and second resolutions for first and second pictures, respectively. The first resolution is different from the second resolution and at least one of the first and second resolutions is determined based on a decision to maintain a specified bitrate, a decision to satisfy a certain level of quality, a decision to satisfy a specified latency requirement, a decision to allocate resources differently, and/or one or more features of one or both of the first and second pictures. So determined, the sending device encodes the first and second pictures at the first and second resolutions, respectively, to the bitstream. The second picture follows the first picture in decoding order and is encoded as an inter-coded picture predicted from the first picture.

Description

RATE CONTROL WITH REFERENCE PICTURE RESAMPLING (RPR)

TECHNICAL FIELD

This application relates generally to video encoding and decoding techniques, and more particularly to a system and method for changing resolution to optimize video compression and video quality.

BACKGROUND

Using a video codec, an encoder compresses a video stream into a coded video bitstream prior to transmitting the bitstream to a receiving device over a network. This so-called compression process is generally referred to as “encoding.” Upon receipt, the receiving device uses the video codec to decompress the coded video bitstream before forwarding the video for further processing and/or display to a user. The process of decompressing the encoded video bitstream is generally referred to as “decoding.” Generally, video streams can be encoded and decoded using any video codec desired. However, some examples of widely deployed video codecs include those that are configured to operate according to H.264Z Advanced Video Coding (AVC), H.265/ High Efficiency Video Coding (HEVC), and/or H.266/ Versatile Video Coding (WC), each of which was developed and standardized jointly by the Moving Picture Experts Group (MPEG) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T).

Regardless of the particular standard, however, video codecs typically support both intra-coded and inter-coded pictures. An intra-coded picture may only predict from samples of the same picture, whereas inter-coded pictures may also predict from previously decoded pictures referred to as “reference pictures.” Inter-coded pictures may further be divided into “P- pictures,” which may only predict from one reference picture at a time for each coding block, and bi-directional “B-pictures,” which may predict from up to two reference pictures simultaneously for each coding block.

During the encoding process, conventional video codecs determine a difference between the pixel data of a first image (e.g., a reference image) and the pixel data of a second, subsequent image that used the first image as a reference image. This difference is typically referred to as the “residual,” and is transformed into the frequency domain by the encoder. The transformed residual is then quantized and entropy-coded before being transmitted to a decoder together with one or more necessary entropy-coded prediction parameters such as prediction mode and motion vectors. The level of quantization is controlled with a quantization parameter (QP) value. The decoder performs the reverse process. That is, the decoder applies inverse quantization and inverse transformation to a received bitstream to obtain the residual. So obtained, the decoder adds the residual to an intra-prediction or inter-prediction to reconstruct the picture. SUMMARY

The present disclosure provides a method and corresponding apparatus for using RPR in a rate controller of a video encoder to change the resolution in a bitstream to optimize the video compression and achieve good quality for a given bitrate. The logic for changing the resolution is based on one of more of a decision to maintain a certain bitrate, reach a certain latency requirement, or allocate resources differently. Additionally, or alternatively, the logic for changing the resolution is based on one or more features of the pictures in the video. Such features may include, but are not limited to, the level of complexity of a picture, an indication of the content of a picture, a change in motion of an object in the picture, a level of detail, or a sudden change in pixel values between the first and second picture.

Additionally, in at least one embodiment, the present disclosure also uses periodic IRAP pictures. However, in this embodiment, the periodic IRAP pictures are encoded at a resolution that is lower than the resolution of the inter-coded pictures to achieve a more uniform bit allocation between the pictures in the bitstream.

Accordingly, in a first aspect, the present disclosure provides a method, implemented by a sending device comprising an encoder, for encoding pictures to a bitstream. In this embodiment, the method comprises the sending device determining a first resolution for a first picture, and a second resolution for a second picture. The first resolution is different from the second resolution and at least one of the first and second resolutions is determined based on a decision to maintain a specified bitrate, a decision to satisfy a certain level of quality, a decision to satisfy a specified latency requirement, a decision to allocate resources differently, and/or one or more features of one or both of the first and second pictures. So determined, the method comprises the sending device encoding the first picture to the bitstream at the first resolution, and encoding the second picture to the bitstream at the second resolution. The second picture follows the first picture in decoding order and is encoded as an inter-coded picture predicted from the first picture.

In a second aspect, the present disclosure provides a sending device comprising an encoder for encoding pictures to a bitstream. In this aspect, the sending device is configured to determine a first resolution for a first picture, and a second resolution for a second picture. The first resolution is different from the second resolution and at least one of the first and second resolutions is determined based on a decision to maintain a specified bitrate, a decision to satisfy a certain level of quality, a decision to satisfy a specified latency requirement, a decision to allocate resources differently, and/or one or more features of one or both of the first and second pictures. So determined, the method of the second aspect comprises the sending device encoding the first picture to the bitstream at the first resolution, and encoding the second picture to the bitstream at the second resolution. The second picture follows the first picture in decoding order and is encoded as an inter-coded picture predicted from the first picture. In a third aspect, the present disclosure provides a sending device comprising an encoder for encoding pictures to a bitstream. In this aspect, the sending device comprises communications circuitry configured to communicate with a receiving device, and processing circuitry operatively connected to the communications circuitry. In this aspect, the processing circuitry is configured to determine a first resolution for a first picture, and a second resolution for a second picture. Additionally, the first resolution is different from the second resolution and at least one of the first and second resolutions is determined based on a decision to maintain a specified bitrate, a decision to satisfy a certain level of quality, a decision to satisfy a specified latency requirement, a decision to allocate resources differently, and/or one or more features of one or both of the first and second pictures. The processing is further continued to encode the first picture to in the bitstream at the first resolution and encode the second picture to the bitstream at the second resolution. The second picture follows the first picture in decoding order and is encoded as an inter-coded picture predicted from the first picture.

In a fourth aspect, the present disclosure provides a computer program comprising instructions that, when executed on processing circuitry of a sending device comprising an encoder, causes the sending device to perform the method according to the first aspect.

A fifth aspect of the present disclosure provides a non-transitory computer-readable storage medium comprising a computer program stored thereon. The computer program comprises executable instructions that, when executed by processing circuitry in a sending device comprising an encoder, causes the sending device to perform the method according to the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a functional block diagram illustrating a communications network configured according to embodiments of the present disclosure.

Figure 2 illustrates periodic Intra-Random Access Point (IRAP) pictures in a bitstream according to embodiments of the present disclosure.

Figure 3A is a graph illustrating the results of not using reference picture resampling (RPR) for intra-coded pictures according to embodiments of the present disclosure.

Figure 3B is a graph illustrating the results of using RPR for intra-coded pictures according to embodiments of the present disclosure.

Figure 3C is a graph illustrating the results of using a 2-step RPR approach for intra- coded pictures according to embodiments of the present disclosure.

Figures 4A-4D are graphs illustrating different methods for handling the size of an intra frame according to embodiments of the present disclosure.

Figure 5 illustrates periodic IRAP pictures in a bitstream in which the IRAP pictures are encoded at a lower resolution than the inter-coded pictures between them according to embodiments of the present disclosure. Figure 6 illustrates a stepwise approach to encoding periodic IRAP pictures to a bitstream at a lower resolution than the inter-coded pictures disposed between them, and where an inter-coded picture appearing immediately after an IRAP picture is encoded to the bitstream at an intermediate resolution between those of the IRAP picture and at least one of the other inter-coded pictures in the bitstream according to embodiments of the present disclosure.

Figure 7 is a flow diagram illustrating a method for encoding pictures into a bitstream according to embodiments of the present disclosure.

Figure 8 is a functional block diagram illustrating some exemplary components of a sending device (e.g., comprising an encoder) configured according to embodiments of the present disclosure.

Figure 9 is a functional block diagram illustrating some exemplary components of a receiving device (e.g., comprising a decoder) configured according to embodiments of the present disclosure.

DETAILED DESCRIPTION

This application claims the benefit of U.S. Provisional Application No. 63/632649 filed 11 April 2024, the entire disclosure of which is hereby incorporated herein by reference.

The present disclosure provides a method and corresponding apparatus for using RPR in a rate controller of a video encoder to change the resolution in a bitstream to optimize the video compression and achieve good quality for a given bitrate. The logic for changing the resolution is based on one of more of a decision to maintain a certain bitrate, reach a certain latency requirement, allocate resources differently. Additionally, or alternatively, the logic for changing the resolution is based on one or more features of the pictures in the video. Such features may include, but are not limited to, the level of complexity of a picture, an indication of the content of a picture, a change in motion of an object in the picture, a level of detail, or a sudden change in pixel values between the first and second picture.

Turning now to the drawings, Figure 1 is a functional block diagram illustrating a communications network 10 configured according to one embodiment of the present disclosure. As seen in Figure 1 , network 10 comprises an access network 12 communicatively connecting a client device 300 (e.g., a HMD, a mobile phone, or a computer) with a network node 200 (e.g., a server node) disposed in a cloud network 14. In some embodiments, a computing device 20 may be disposed between client device 300 and server node 200 and is configured to perform at least some of the processing functions of client device 300. The access network 12 may be any type of communications network (e.g., Wireless Fidelity (WiFi), ETHERNET, Wireless Local Area Network (WLAN), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), etc.), and functions to connect subscriber devices, such as client device 300, to one or more service provider nodes, such as network node 200. The cloud network 14 provides such subscriber devices with “on-demand” availability of computer resources (e.g., memory, data storage, processing power, etc.) without requiring the user to directly, actively manage those resources. According to embodiments of the present disclosure, such resources include, but are not limited to, one or more conversational services, cloud gaming applications, or XR applications being executed on network node 200. The XR applications, which are described in more detail below, may comprise, for example, those used in connection with gaming applications and/or simulation applications.

Video Coding

As previously stated, video codecs are used to compress (i.e., encode) pictures to a bitstream. In the context of Figure 1 , for example, a video codec executing on network node 200 may encode images for a video (e.g., pictures generated by a game engine) to send to client device 300 for decoding and display to a user. Some of the more widely known and used video codecs were developed and standardized by the MPEG and ITU-T, and include, but are not limited to, H.264/AVC, H.265/HEVC, and H.266/WC.

Pictures in AVC, HEVC, and WC are identified by their picture order count (POC) values. The POC determines the output order of decoded pictures. Additionally, according to the H.264/AVC, H.265/HEVC video coding standards, a picture is typically divided into one or more slice Network Abstraction Layer (NAL) units. A slice may be a full picture or a part of a picture. Further, each NAL unit is essentially a data packet, thereby facilitating the packetization of the NAL units into network packets. In this disclosure, the terms “picture” and “frame” are used interchangeably.

Random Access Point (RAP) Pictures

In AVC, HEVC, and WC, as well as in most other modern video codecs, random access point pictures, which are commonly referred to by those of ordinary skill in the art as “key pictures,” are typically used in a video bitstream for three reasons.

1. as a start of a bitstream;

2. to refresh a video; or

3. to let a decoder tune into the bitstream, such as when switching channels in broadcast TV scenario or when moving to a different part of the video in a video- on-demand (VOD) scenario.

RAP pictures are typically intra-coded, which means that a RAP picture predicts only from itself and not from other pictures. Such prediction, however, results in a much lower compression efficiency compared to scenarios where inter-picture prediction is allowed. Therefore, intrarandom access point (IRAP) pictures tend to produce undesirable bitrate spikes in the bitstream, which can result in increased jitter and overall latency.

WC in particular supports two types of IRAP pictures. These are:

1. instantaneous decoding refresh (IDR) pictures; and

2. clean random access (CRA) pictures.

IRAP pictures may have “trailing pictures” that follow the associated IRAP picture in both decoding and output order, and “leading pictures” that follow the associated IRAP picture in decoding order but precede the IRAP picture in output order. There are two types of leading pictures in WC, random access decodable leading (RADL) pictures that is correctly decoded when starting the decoding at the associated IRAP picture, and random access skipped leading (RASL) pictures that may not be correctly decoded when starting the decoding at the associated IRAP picture. IDR pictures may have associated RADL pictures but not RASL pictures and it is thus ensured that all pictures in the bitstream following the IDR picture in decoding order can be decoded correctly. However, CRA pictures may also have RASL pictures. In some cases, e.g., when decoding begins at the CRA picture, such leading pictures may need to be dropped.

Gradual Decoding Refresh

In some cases, gradual decoding refresh (GDR) pictures may be used as an alternative to using IRAP pictures. GDR pictures, which are recommended for low-latency video scenarios, refresh only a portion of a video at a time over a range of pictures. This allows for a much smother bitrate, but at the cost of a slightly decreased compression efficiency and longer tune-in time. Another drawback of using GDR for real-time streaming is that recovery from packet loss can be delayed. This is because the refresh is spread across multiple frames (i.e., pictures). Additionally, the overall compression efficiency is reduced when using GDR since periodic intracoded blocks are inserted in the bitstream to allow for the video to be refreshed. Despite these drawbacks, however, GDR is optional to support in H.264/AVC and H.265/HEVC but made mandatory for decoders to support in H.266/WC.

Coded video sequence

In WC and HEVC, a coded video sequence (CVS) is a sequence of access units (AUs) followed by zero or more AUs up to, but not including, the AU starting the next CVS in decoding order. In WC, the AU starting a CVS may include an IRAP picture, such as an IDR or a CRA picture, for example, or it may comprise a GDR picture. In HEVC, however, the AU starting the CVS is an IDR picture. Regardless of the particular standard, however, a CVS in the context of this disclosure is broadly defined as a sequence of coded video frames (i.e., pictures) in a bitstream.

Reference picture resampling (RPR)

A new feature in WC is reference picture resampling (RPR). RPR allows the spatial resolution (i.e., the width and height) of the pictures being encoded to a bitstream (e.g. encoded to a CVS in the bitstream) to change in the middle of the bitstream without having to encode an additional IRAP picture into the bitstream. In WC, the spatial resolution of a picture may be modified on a picture level and is therefore signaled in the picture parameter set (PPS). In comparison, both AVC and HEVC signal the spatial resolution of a picture on the sequence level in the sequence parameter set (SPS). Notably, neither AVC nor HEVC currently support RPR functionality.

When the current picture and a reference picture have different spatial resolutions, RPR enables the encoder to use the reference picture to predict the current picture by scaling the reference picture to the same spatial resolution as the current picture before prediction occurs. This scaling is done on the block level.

As defined in the context of this disclosure, RPR functionality is not limited to the RPR functions of WC. Rather, RPR is defined more generically as reference picture resampling according to codecs, such as WC and AV1 , as well as reference picture resampling functionality that may be provided by future video codecs.

Low latency video coding

Low latency video coding is a key requirement for many applications of video, including conversational services, cloud gaming, and services for extended reality (XR). As defined in this disclosure, XR comprises real-time virtual reality (VR), augmented reality (AR), and mixed reality (MR). There are various factors that can affect the latency of a coded video bitstream. Among these factors are those related to reference pictures and those related to rate control and buffer management.

Reference Pictures

Video for broadcast TV and video on demand (VOD) streaming is typically encoded with a group of pictures (GOP) structure. With such structures, pictures reference both past and future pictures to achieve a sufficient level of prediction with high-level of compression. To accomplish this, encoders often use bidirectional predicted pictures (i.e., B pictures) that predict from two reference pictures at a time. However, while such functionality may help with prediction and compression, using future pictures for prediction also increases latency. Therefore, it is not recommended to use reference future pictures for low latency video coding. Similarly, it is common for professional encoders used in broadcasting and VOD scenarios to provide a look- ahead feature to look ahead to the pictures that are coming. However, this feature also significantly increases latency, and thus, is also not recommended for low latency video coding.

Rate Control and Buffer Management

The number of bits needed to compress video pictures is highly dependent on the content of the video. Low motion scenes with little detail are relatively easy to compress, while scenes with lots of motion and detailed structures are more complex and require many more bits. Therefore, broadcasting and VOD services typically use a variable bitrate (VBR) to maintain a constant quality of the video. To handle variable bitrate, the encoding and decoding buffers must be sufficiently large. However, this adds latency. To achieve lower latency with smaller buffer sizes, the video can be encoded with constant bitrate (CBR). CBR has a lower compression efficiency than VBR but is not recommended for use with broadcasting and VOD services. Rather, CBR is recommended for use with real-time video services.

Rate control algorithms typically modify the QP value per picture, per slice, or per block to achieve desired bitrates. Alternatively, or additionally, the lambda value associated with the rate control algorithm can be changed.

Video Streaming

MPEG-DASH

MPEG-Dynamic Adaptive Streaming over HTTP (MPEG-DASH), specified in ISO/IEC 23009-1 dated August 2022 and which is incorporated herein by reference in its entirety, is an adaptive bitrate (ABR) streaming technology. With MPEG-DASH, a multimedia file is partitioned into one or more segments and delivered to a client using Hypertext Transfer Protocol (HTTP), typically over Transmission Control Protocol (TCP). An MPEG-DASH session is set-up using a media presentation description (MPD) that describes segment information including timing, Uniform Resource Locator (URL), and media characteristics such as video resolution and bit rates. MPDs, which are Extensible Markup Language (XML)-based, can be static (e.g. for movies) or dynamic (e.g., for live content).

Commonly, VOD services using ABR modify both the frame rate and the resolution of a video in chunks or segments, such as in MPEG-DASH, to achieve appropriate bitrates depending on the available network bandwidth. Particularly, each segment starts with an intracoded picture to facilitate the switch between different resolutions. For each segment, a plurality of different bitrates are typically pre-encoded where each bitrate corresponds to a certain quality in a bitrate ladder. The following is an example of a bitrate ladder.

Table 1

Segments are typically in the order of 6 to 20 seconds long. Depending on the available network bandwidth at a given segment time, a quality for the segment is requested/chosen with a bitrate that matches the available bandwidth. Additionally, the different qualities Q1-Q5 may be encoded using different codecs.

Low Latency Video Streaming

When streaming video with low latency, it is important to use a transport protocol that meets the latency requirements. A popular transport protocol for achieving latencies below 500ms is the Real-time Transport Protocol (RTP), which is typically run over the User Datagram Protocol (UDP). The standard for RTP, i.e., RFC 3550 entitled “RTP: A Transport Protocol for Real-Time Applications and dated July 2003, is expressly incorporated herein by reference. TCP is normally not recommended as it requires the retransmission of lost packets, which rapidly increases the latency. AVC, HEVC, and WC all have RTP payload formats to properly pack the elementary bitstream that was compressed using the codec in RTP.

A congestion control algorithm is essential to achieve low latency even when the conditions of the network vary. Congestion occurs when the transmitted bitrate is higher than the available bitrate capacity over a given transmission path. Applications for video streaming may employ congestion control to achieve robust performance and avoid congestion collapse (i.e., a state where the network is stable but there is low throughput).

Packet loss and packet loss concealment

Nodes in currently existing networks often have large “jitter” buffers. This helps the nodes avoid having to drop packets when faced with congestion. However, this is often referred to as “buffer-bloat.” For real-time streaming applications, this means that the most common form of packet loss is “late-loss.” Such late-loss occurs when packets successfully arrive at the receiver but took longer than an acceptable delay period to arrive at the receiver. In these cases, the “late-loss” packets are simply dropped by the jitter buffer.

Different real-time video applications have different strategies for dealing with packet loss. For example, consider video conference applications. In such applications, it is generally agreed that dropping all frames until an error-free frame can be decoded will provide the best user experience. This is because the video may not exhibit much motion. Therefore, having a video freeze during a video conference is generally preferrable over obtaining decoding artifacts that, when displayed, can distort an object in the video, such as a person’s face. On the other hand, a cloud gaming application usually has a lot of motion. In such cases, game “freezes” can negatively affect the game play. Therefore, the most common strategy for dealing with such freezes is to allow for decoding with errors, and then letting the decoder conceal the errors as much as possible.

However, packet loss concealment methods differ between decoder implementations with some packet loss concealment methods being better able to cope with errors than others. For example, hardware-based decoding is most often preferred in battery constrained devices. While this prolongs battery life, it also constrains the application into using whatever concealment methodologies the hardware decoder can provide for that specific hardware platform.

Recovering from Loss

Full Intra Reguest

Decoders are unable to decode an error-free video in cases where packets are lost, and thus, need time to recover. Therefore, decoders in real-time video streaming applications commonly recover from packet loss by requesting an encoder to produce an intra-picture. In applications using RTP, this is achieved by the receiving device sending an Real-Time Transport Control Protocol (RTCP) feedback message comprising a Negative Acknowledged, Picture-Loss-Indication (NACK-PLI) or a NACK-Slice-Loss-lndication (NACK-SLI). Once the decoder has received the intra-picture, it will be able to resume decoding without errors.

However, the requested intra-picture will typically create a spike in the number of transmitted packets. As such, there is a risk of additional packet loss from the intra-picture. The additional packet loss can lead the decoder to issue a second intra-picture request to the encoder, which in turn, could cause additional packet loss, thereby leading the decoder to send even more requests for intra-pictures from the encoder. This can create a negative spiral of requesting intra-pictures to correct for packet loss, which only leads to additional packet loss.

L4S (Low Latency, Low Loss, Scalable)

L4S is specified in RFC 3168 “The Addition of Explicit Congestion Notification (ECN) to IP” September 2001 , which is expressly incorporated herein by reference in its entirety. As described in RFC 3168, L4S is a network feature for early network congestion indication. It is a technology used to reduce queue delay to provide low latency to IP flows and provide higher performance throughput. L4S uses Explicit Congestion Notification (ECN) marking in order to indicate congestion in the network to hosts and nodes.

Periodic Intra-Pictures and Gradual Decoding Refresh

A video bitstream, such as bitstream 30 illustrated in Figure 2, for example, may include periodic intra-coded pictures in order to tune into a bitstream or recover from lost or corrupted frames. Those of ordinary skill in the art typically refer to such periodic intra-coded pictures as “key frames.” In AVC, HEVC, and WC, such pictures are called intra-random access point (IRAP) pictures 32. As seen in Figure 2, a bitstream 30 typically includes a sequence of intercoded pictures 34 between periodic IRAP pictures 32.

AVC, HEVC, and WC also support tuning into the bitstream at an inter-coded picture using gradual decoding refresh (GDR). With GDR, each picture from the tune-in point typically refreshes a new area of the picture by coding that area with intra-coded blocks until the entire picture has been refreshed a few pictures later. One main advantage using GDR is that it allows a more even bitrate to be achieved compared to using periodic intra-pictures, such as intra- coded blocks, which typically require significantly more bits than inter-coded blocks for a given quality. GDR is supported normatively in WC using the GDR picture type, and optionally supported in AVC and HEVC using the recovery point supplemental enhancement information (SEI) message. The pictures in the video bitstream from an IRAP/GDR picture to the next IRAP/GDR picture in WC is referred to as a sequence.

Typically, periodic intra-pictures are not used in real-time video streaming applications. This is because intra-pictures generally require more data to be sent unnecessarily, even in cases where there is no packet loss. Moreover, recovery from packet loss is delayed compared to using forced intra-pictures since the decoder will have to wait for the next periodic intrapicture.

Resolution change

Typical VOD services use ABR streaming to control quality delivered to an end-user while simultaneously mitigating the effects of a varying bandwidth in the transmission channel. In ABR streaming, multiple renditions of the same content are produced at the server (e.g., network nodes, content delivery networks (CDNs), etc.). However, the renditions differ in terms of codec resolution and bitrates. In some cases, a decision on which particular resolution is fetched for playback depends on the client device. In other cases, such as in broadcast scenarios where the content is encoded at a chosen resolution, the encoder needs to select whichever resolution that is optimal for coding performance. The same is for low latency applications that use a 1 -to-1 distribution model (i.e., where content served only to a user is specific to that user and is not redistributed to other users). An example of this is cloud gaming content where each gaming session will differ depending on user interactions (e.g., input).

In all these video distribution cases, where video resolution changes based on available bandwidth (either by client device or by encoding device), video can be seamlessly decoded on the client device, but only in situations where the resolution changes in the resulting bitstream do not impact the decoding process. In most cases, this requires the encoder to insert an intrapicture to signal the decoder that subsequent pictures are coded at a new resolution. However, intra-pictures are not always needed. By way of example only, a WC RPR feature allows for resolution changes without inserting an intra-picture.

While useful, the existing state-of-the-art techniques for changing resolution in the middle of a bitstream are problematic. For example, with ABR services, each segment is encoded with an intra-picture to cause the decoder to switch to a new resolution for a new segment. However, it is more inefficient to encode intra-coded pictures than it is to encode intercoded pictures. This inefficiency only increases the overall bitrate, and thus, may cause undesirable bitrate spikes in the bitstream.

Current solutions require encoders to use either full intra-pictures, which generates large bitrate spikes, or GDR, which as described previously, splits an intra-picture over several pictures. The latter provides ultra-low latency capability at the expense of longer tune-in time. For several applications, however, GDR may be too excessive, and a softer control of intrapicture size could be more beneficial.

RPR was developed for WO in order to facilitate a change in resolution of a picture in a bitstream without having to insert a new intra-coded frame. However, there is no apparent guidance in existing literature that describes how to determine RPR usage, especially in the case of low latency video, its impact on management of rate control buffers under low latency constraints, and its impact on video coding performance and quality. For cases dealing specifically with video coding performance and quality, there is no apparent discussion or solution on how to mitigate the effects of a delayed increase in bits when intra-pictures are encoded at a lower resolution. Nor does there appear to be any discussion on the impact that this has on the coding performance of the pictures that reference such an intra-picture. Additionally, existing state-of-the-art solutions do not appear to sufficiently handle refreshing a video after a packet loss or video corruption has occurred.

In more detail, existing solutions configure a decoder to send a message to an encoder requesting that it refresh a video by sending an intra-coded picture (e.g., IRAP picture). However, a problem with this technique is that the intra-coded pictures in the bitstream are typically much larger in size when compared to the inter-coded pictures of the bitstream. Such large IRAP pictures, as stated above, may cause whatever delay exists to be extended, as well as network congestion. The bitrate spikes from the IRAP pictures may be mitigated by encoding the IRAP pictures at a lower quality by increasing a quantization parameter (QP) value of the codec. However, encoding the IRAP pictures in this manner will typically have a negative impact on the quality.

Another issue can occur during sudden changes in the content where temporal redundancy between pictures is reduced such that efficient inter-picture prediction is prohibited. Such changes occur, for example, during scene changes where video frames from different scenes are adjacent to each other. In such cases, even if the encoder does not choose to insert intra-frame pictures, inter-frame pictures may be coded with a substantial proportion of intracodec blocks. However, this would still cause the same or similar effect as with using intra frame pictures. That is, a large frame that causes a spike in the bitrate.

Accordingly, embodiments of the present disclosure address these and other issues. Particularly, the present embodiments use RPR in a rate controller of a video encoder to change resolution in a bitstream. In doing so, the embodiments of the present disclosure optimize video compression and achieve good quality for a given bitrate.

In accordance with the present disclosure, whether to change the resolution is a decision that may be based on a variety of desirable outcomes. Among these include, but are not limited to:

• a decision to maintain a certain bitrate;

• a decision to reach a certain latency requirement; and • a decision to allocate resources differently.

Additionally, or alternatively, the decision on whether to change the resolution is based on:

• one or more features of the pictures in the video including level of complexity;

• an indication of the content of a picture;

• the change in motion

• the level of detail; or

• a sudden change in pixel values between the first and second picture.

In another embodiment, the present disclosure encodes periodic IRAP pictures at a first resolution and one or more inter-coded pictures following the IRAP pictures at a second resolution, where the first resolution is lower than the second resolution. This allows a more uniform bit allocation between the pictures in the bitstream.

Accordingly, the present disclosure addresses the need to sufficiently handle bitrate spikes on a per-picture basis for large pictures - regardless of whether those large pictures are intra-coded or inter-coded. Such bitrate spikes may, for instance, occur from:

• a need to provide a tune-in option such as by inserting periodic intra-coded pictures in the bitstream;

• a need to efficiently handle a sudden break of temporal redundancy, which can lead to inter-coded pictures being encoded with multiple intra blocks; and

• a need to change the resolution to provide a better quality tradeoff given the available bandwidth.

Additionally, the present embodiments disclose using RPR to predict from a different resolution picture, thereby addressing the issues identified above. Particularly, according to the present embodiments:

• Intra-coded pictures are encoded at a resolution that is different (i.e., lower) than the resolution used to encode the inter-coded picture(s) that reference them in order to reduce bitrate spikes; and

• RPR is used to change the resolution of a picture when needed. For example, there may be a need to change the resolution of the pictures in a bitstream in order to maintain a desired number of bits per picture. Other reasons for such changes in resolution include, but are not limited to, allowing a temporary change in resolution due to available bandwidth, content complexity, and/or the availability of computing resources.

In one embodiment, according to the present disclosure, changing to low resolution for encoding intra-coded pictures to reduce bitrate spikes is done only in cases where the generated bits-quality make better trade off. In other cases, however, QP offset may be applied in addition to, or in lieu of, RPR. This may provide a more robust solution. When using RPR for intra-coded pictures there may still be bitrate spikes in consecutive pictures. However, the present embodiments address this issue by applying QP offsets to deal with the bitrate spikes, and/or to introduce one or more additional intermediate resolutions for the inter-coded pictures that follow the “low-resolution” intra-coded picture.

The embodiments of the present disclosure provide advantages and benefits that conventional encoding/decoding systems and methods do not or cannot provide. For example, when compared to the state-of-the-art techniques, one advantage is that the present embodiments provide the ability to avoid bitrate spikes resulting from large intra-pictures or inter-pictures with a large proportion of intra-codec blocks. As stated above, and as explained in more detail below, this can be accomplished by reducing the resolution of selected pictures. This functionality is important because it allows applications to have more flexible control of the buffers in both the encoder and the decoder, which are required for low latency applications.

In another advantage, the techniques provided by the present embodiments provide a more robust solution to encoders. More particularly, encoders can use the resolution of a picture or group of pictures as an additional parameter (e.g., to QP, lambda, scaling matrices) to control the bits that are generated for each picture.

Embodiment 1 - Rate control with RPR

In a first embodiment of the present disclosure, a rate controller of a video encoder uses RPR to optimize the video compression and achieve good quality for a given bitrate. In one version of the embodiment, other adaptive rate control tools may be used together with the change in resolution to adapt the rate of the pictures. Some examples of these other adaptive rate control tools include, but are not limited to, changing QP (e.g., either on a block, slice, or picture level), changing lambda values, and reducing the frame rate.

Generally, conventional VOD services change the resolution of a video by inserting a new intra-coded picture into the beginning of a segment. In contrast to these conventional services, however, changes in the resolution of a video according to this embodiment does not require the insertion of a new intra-coded picture into the bitstream. Rather, the present embodiments are configured to change the resolution of the video at inter-coded frames using RPR.

In one embodiment, the decision to change the resolution in the bitstream is based on a decision to maintain a certain bitrate. For example, a sudden change in the complexity of the video (e.g., a sudden change in motion, details, etc.) may cause the bitrate to spike or the quality of the video to deteriorate. To prevent that from happening, the present embodiments configure the rate controller of the encoder to decide to lower the resolution to maintain a similar bitrate and similar quality level.

In one embodiment, the rate controller decides to change the resolution in the bitstream based on a decision to meet certain latency requirements. For example, in a low latency video scenario, it is critical to have the video pictures encoded in time to avoid increasing latency. Conventionally configured video encoders accomplish this by simply dropping the video frames that it does not have time to encode. However, that may cause visible frame jitter and reduce the frame rate. Therefore, when the rate controller of an encoder configured according to the present embodiments realizes that maintaining current latency requirements will be problematic (e.g., due to increased complexity in a scene or an otherwise increased burden on the processor), it decides to lower the resolution to maintain encoding. Once the burden to encode subsides, the rate controller may once again increase the resolution.

In another embodiment, a rate controller configured according to the present disclosure decides to change the resolution in the bitstream based on a decision to make better use of the available resources. By way of example only, the rate controller in this embodiment lowers the resolution during the encoding process instead of using processing power and memory to keep encoding at a high resolution. In doing so, the rate controller is freed to perform a variety of actions. For example, the rate controller can spend the resources it saved by lowering the resolution on more compression-efficient, but more complex, coding tools and methods that it would otherwise not be able to afford to use in terms of resources or latency requirements. In another example, thanks to the additional available CPU cycles, the rate controller of the present disclosure can increase the rate distortion optimization (RDO). Some examples of the more compression-efficient, more complex tools, and RDO decisions include, but are not limited to, SAO, neural network-based coding tools, a wider motion vector search, and testing of more coding modes (such as intra-coding modes, for example).

In one embodiment, a rate controller configured according to the present disclosure may change the resolution of a video stream in a “stepwise” manner over a number of pictures. Such a stepwise increase in resolution, as seen in more detail below, makes the change of resolution less noticeable to a user viewing the pictures. Additionally, a stepwise approach to increasing the resolution helps to maintain a predetermined bitrate per picture.

For instance, consider a situation where the rate controller of an encoder configured according to the present embodiments changes (in this case, lowers) the resolution of a video stream to half the current resolution. Such may be done, for example, due to a sudden increase in scene complexity or from the insertion of an intra-coded picture. In these cases, the prediction for an inter-coded picture following the intra-coded picture (e.g. IRAP picture) in the decoding order would be worse than if the inter-coded picture and the intra-coded picture had the same resolution. This is because the intra-coded picture, at half the resolution of the following intercoded pictures, does not have a sufficient number of bits to sufficiently support predicting the following inter-coded pictures. Therefore, the present embodiments employ a “stepwise” approach to changing the resolution. Specifically, the intra-coded picture is still encoded at the lower resolution. However, the inter-coded picture that immediately follows the intra-coded picture in decoding order is encoded at an intermediate resolution that is between the resolution of the intra-coded picture and a third inter-coded picture appearing after the first and second pictures in the decoding order. By stepwise changing the resolution, as provided by the present embodiments, the relative predictions will be sufficiently accurate. Moreover, the step back to the original resolution (i.e., from the resolution of the inter-coded picture immediately following the intra-coded picture to the resolution of the next inter-coded picture in the decoding order) can be made without significantly diverting from a desired bitrate.

This is illustrated by graphs 40, 50, and 60 in Figures 3A-3C, which show three different cases of encoding. Pictures are indicated by rectangles labeled “I” for intra-coded picture (e.g., an IRAP picture) and “B” for bi-directional predicted pictures (e.g., inter-coded pictures). The size of the rectangle illustrates the relative resolution of a picture, and the bars immediately above the “I” and “B” rectangles illustrate a bitrate for the corresponding picture.

In Figure 3A, the “I” picture has the same resolution as the “B” pictures that follow it in the decoding order. As previously described, this can cause a spike in the bitrate, as indicated by bar 42. In Figure 3B, the “I” picture is encoded at half the resolution of the “B” pictures that follow it. This approach makes it possible to encode the “I” picture with a QP and bitrate that is similar to those used to encode the “B” pictures that preceded it, as indicated by bar 52. However, the “B” picture immediately following the “I” picture in Figure 3B would be unable to accurately predict from the “I” picture. This also causes a bitrate spike for this picture, albeit smaller than the bitrate spike of Figure 3A where the “I” and “B” pictures had the same resolution.

In Figure 3C, the “I” picture has half the resolution of the “B” pictures that precede it in the decoding order. However, rather than encode the “B” picture 64 immediately following the “I” picture at the same resolution of the preceding “B” pictures, embodiments of the present disclosure employ the previously described “stepwise” approach and encode this “intermediate B picture” 64 at a resolution that is higher than the resolution used to encode the “I” picture, but lower than the resolution used to encode next “B” picture 66 in the decoding order. Because the “I” picture and the “intermediate B” picture 64 are closer in resolution, and because the intermediate “B” picture 64 and the next “B” picture 66 are closer in resolution, prediction from the “I” picture is enhanced. Specifically, bitrate spikes do not occur at all, or are at least significantly reduced, as shown by bar 62. Then, because of the better prediction and smaller resolution, the bitrate becomes about the same as the first B-pictures.

Figures 4A-4D are graphs 70, 80, 90, and 100 illustrating how the present embodiments configure a device to control the bits-per-picture using RPR. In more detail, graph 70 in Figure 4A shows an anchor, which uses a typical QP offset such as used in Joint Video Experts Team (JVET) Common Test Conditions (CTC) for low delay configuration. As seen in Figure 4A, the size of the intra-coded picture (POC 64) is significantly larger than the other pictures (i.e., POC 60, 62, 66, 68). This may require the provisioning of a larger rate control buffer. Graph 80 in Figure 4B shows how QP offset +6 helps to reduce the size of an intracoded picture by 2 but leads to increase in a consecutive picture size, which compensates for the worse prediction. Figure 4C illustrates a situation that is similar to that shown in Figure 4B, but with the intra-coded picture being coded with a reduced resolution (by 2x in both dimensions). Similar to Figure 4B above, graph 90 in Figure 4C illustrates an increase in a consecutive picture size which compensates for inadequate prediction. Figure 4D illustrates a graph 100 showing how gradual resolution changes applied on an intra-coded picture and the inter-coded pictures following it maintains the low size of the pictures and avoids the large bitrate peaks illustrated in Figures 4B and 4C.

In one embodiment, the change in resolution is used together with other adaptive rate control tools to adapt the rate of the pictures. Such other adaptive rate control tools may be configured to change a QP (e.g., either on the block, slice, or picture level), change lambda values, and reduce a frame rate.

Accordingly, an encoder configured according to the present disclosure may perform all or a subset of the following functions to encode pictures to a bitstream.

• Determine a first resolution A;

• Encode a first picture to the bitstream, where the first picture is encoded with the first resolution A;

• Determine a second resolution B, where resolution B is different from resolution A;

• Encode a second picture to the bitstream, where the second picture follows the first picture in decoding order, is encoded as an inter-coded picture predicted from the first picture, and further, is encoded with the second resolution B. The second picture may be encoded using RPR.

• Determine a third resolution C, where resolution C is different from both resolutions A and B;

• Encode a third picture to the bitstream, where the third picture follows the first and second pictures in decoding order, is encoded as an inter-coded picture predicted from one or both of the first and second pictures, and is encoded with the third resolution C. The third picture may also be encoded using RPR.

In accordance with the present disclosure, determining at least one of the first resolution A, second resolution B, and third resolution C can be based on a decision to maintain a certain bitrate, reach a certain latency requirement, and/or allocate resources differently. Additionally, or alternatively, determining at least one of the resolutions may be based on one or more features of the video.

In one preferred embodiment, at least one of the height and width of the first picture with resolution A is smaller than the corresponding height and width of the second picture with resolution B. For instance, consider an example where resolution A is full HD (i.e., 1920x1080 pixels), while resolution B is 4K (i.e., 3840x2160 pixels). Thus, resolution A is less than resolution B. In another version, at least one of the height and the width of the first picture with resolution A is larger than the corresponding height and width of the second picture with resolution B. For instance, resolution A may be 4K, 3840x2160 pixels, while resolution B may be full HD, 1920x1080 pixels.

Embodiment 2 - Rate Control with Resolution Change Based on Content Dependency

In this embodiment, an encoder uses a rate control algorithm that applies resolution change to achieve a desired bitrate. In these embodiments, the decision to change the resolution may be based on the content of a picture, a part of a picture, and/or a sequence of pictures. Additionally, this embodiment may be based on the previous embodiment using RPR, but could also be applied when changing the resolution without using RPR, such as for codecs that are not configured to support RPR.

The rate controller of the video encoder may, for instance, be triggered to change the resolution by one or more of the following:

1 . A change in the motion of an object in a scene. Such a change may be triggered, for example, by a measure of motion vector distances or by an analysis of the motion between frames. For instance, an increase in motion of the object may cause the rate controller to lower (i.e., decrease) the resolution. Similarly, a change to a more “static” scene where the object exhibits little or no movement may trigger an increase in the resolution. Static scenes with high level of spatial detail may benefit from full resolution reference frame.

2. The level of details of a picture or part of picture. For example, a scene with a low level of details, such as unfocused scenes, skies with clouds, etc., may cause the rate controller to lower the resolution during the encoding process. On the other hand, a scene with a higher level of details may cause the rate controller to increase the resolution during the encoding process. It should be noted, however, that a picture having a high level of detail may cause the opposite effect. For instance, consider a situation where a high level of detail is not vital for the scene. In such instances, the details of the scene, such as the trees in a forest or a large crowd of people, may not be important or vital to the scene. In such cases, therefore, it may be advantageous for the rate controller of an encoder configured according to the present embodiments to lower the resolution instead of increasing the resolution. This would facilitate the encoder’s ability to achieve a better quality overall for the video stream. Further, the level of details may be based on the block partitioning of the encoder, the amount or size of the quantization coefficients, or an analysis of the picture. 3. A sudden change in pixel values in the picture or parts of the picture. Such changes may occur in scene changes. For example, a scene change may happen from frame-to-frame or over a number of frames, such as the case for fade-ins and fade- outs. Another example is a sudden change in the brightness of the content due to a camera flash or a flash of lighting, which changes the brightness of the entire picture all at once. In these cases, it may be advantageous for the rate controller of the encoder to temporarily lower the resolution of a given scene to cope with the extra complexity.

Periodic intra refresh with RPR

As stated above, embodiments of the present disclosure mitigate bitrate spikes. For example, when an inter-coded picture is lost (e.g., one or more incoming packets at the are dropped and/or corrupted), a rate controller of an encoder configured according to the present embodiments refreshes the video at the next intra-coded picture. The sending device can accomplish this function either autonomously or in response to receiving an indication from a receiving device or a network node. However, to achieve successful prediction from the intracoded pictures, at least some of the inter-coded pictures use RPR.

For example, in one embodiment, an encoder configured according to the present disclosure periodically encodes one or more intra-pictures (e.g., IRAP pictures) to the bitstream. Such pictures may be encoded at regular timed intervals (e.g., every 10 seconds), autonomously as needed, and/or at variable positions in the bitstream (e.g. at scene cuts). Regardless of the periodicity, however, the encoder in this embodiment encodes the intra-coded pictures at a resolution that is lower than the resolution at which it encodes the inter-coded pictures to the bitstream.

Figure 5 illustrates one such embodiment. As seen in Figure 5, bitstream 110 comprises a plurality of IRAP pictures 112 and one or more inter-coded pictures 114 disposed between the IRAP pictures 112. Further, as indicated by the different sizes of the pictures 112, 114, the IRAP pictures 112 are encoded with a resolution that is lower than the resolution at which it encodes the inter-coded pictures 114.

In one embodiment, at least one of the intra-coded pictures (e.g., IRAP pictures 112) are decoded by the receiving device but are never displayed. That is, the decoder at receiving device only uses the intra-coded pictures in the bitstream as reference pictures for prediction, but does not output the decoded intra-coded pictures for further processing and/or display. However, because the lower resolution video (i.e., the lower resolution intra-coded pictures such as IRAP pictures 112) are not displayed, the present embodiments improve the overall quality of the video.

Additionally, as seen in Figure 6, for example, the encoder may be configured to encode an inter-coded picture 116 appearing immediately after an IRAP picture 112 in bitstream 110 at a reduced resolution. By way of example only, the encoder at the sending device may encode an IRAP picture 112 at half the normal resolution of the inter-coded pictures 114. Then, the encoder may encode an inter-coded picture appearing immediately after the IRAP picture 112 at an intermediate resolution that is higher than that of IRAP picture 112 and lower than that of an inter-coded picture 114. As stated above, this “stepwise” approach to changing the resolutions of the pictures encoded to the bitstream helps mitigate spikes in the bitrate.

Figure 7 is a flow diagram illustrating a method 120 for encoding pictures to a bitstream according to embodiments of the present disclosure. For illustrative purposes only, method 120 is implemented by an encoder at a sending device, such as a network node in cloud 14. However, those of ordinary skill in the art will readily appreciate that method 120 may be implemented by an encoder at devices other than a network node, such as computing device 20 seen in Figure 1 , for example.

As seen in Figure 7, the encoder implementing method 120 determines a first resolution for a first picture, and a second resolution for a second picture (box 122). The first resolution is different from the second resolution, and further, at least one of the first and second resolutions is determined based on:

• a decision to maintain a specified bitrate;

• a decision to satisfy a certain level of quality;

• a decision to satisfy a specified latency requirement;

• a decision to allocate resources differently; and/or

• one or more features of one or both of the first and second pictures.

So determined, the encoder implementing method 120 then encodes the first picture to the bitstream at the first resolution (box 124), and encodes the second picture to the bitstream at the second resolution (box 126). In this embodiment, the second picture follows the first picture in decoding order and is encoded as an inter-coded picture predicted from the first picture. In one embodiment, the bitstream comprises a CVS and the first picture and the second picture are encoded to the CVS in the bitstream.

Next, the encoder implementing method 110 determines a third resolution for a third picture (box 128). In this embodiment, the third resolution is different from the second resolution. The encoder then encodes the third picture to the bitstream at the third resolution (box 130). According to this embodiment, the third picture follows the first and second pictures in the decoding order and is encoded as an inter-coded picture predicted from one or both of the first and second pictures. In one embodiment, the bitstream comprises the CVS. Further, in addition to the first picture and the second picture, the third picture is encoded to the CVS in the bitstream. Once the pictures are decoded to the bitstream, the encoder (e.g., the sending device) sends the bitstream to a receiving device. In one embodiment, the second picture is encoded as an inter-coded picture predicted from the first picture using Reference Picture Resampling (RPR).

In one embodiment, the resolution for the first picture is determined to be lower than the resolution of the second picture based on at least one of the following:

• to maintain the specific bitrate;

• to satisfy a certain level of quality;

• to reach a certain latency requirement;

• to decrease a resource allocation for encoding the first and/or the second picture; and

• to increase a rate distortion optimization (RDO).

In one embodiment, the resolution for the second picture is determined to be lower than the resolution of the first picture based on at least one of the following:

• to maintain the specific bitrate;

• to satisfy a certain level of quality;

• to reach a certain latency requirement;

• to decrease a resource allocation for encoding the second picture; and

• to increase a rate distortion optimization (RDO).

In one embodiment, the first resolution is incrementally increased or incrementally decreased over a number of pictures to reach the second resolution.

In one embodiment, the one or more features comprises a complexity of the first and/or second pictures.

In one embodiment, the one or more features comprises one or more of:

• a content of one or both of the first and second pictures;

• a change in motion of an object in at least one of the first and second pictures;

• a level of detail; and

• a sudden change in pixel values between the first and second picture.

In one embodiment, the level of detail for the at least one of the first and second pictures is based on one or more of:

• a block partitioning of the encoder;

• an amount or size of one or more quantization coefficients; and

• an image analysis of the at least one of the first and second pictures.

In one embodiment, the sudden change in pixel values between the first and second picture is caused by one of:

• a scene change;

• a sudden change in brightness; and

• a sudden change in a scene associated with the first and second pictures.

In one embodiment, the sudden change in pixel values occurs from frame-to-frame. In one embodiment, the sudden change in pixel values occurs over a number of frames.

In one embodiment, the first resolution of the first picture is incrementally increased or incrementally decreased over the second resolution of the second picture to reach the third resolution of the third picture.

In one embodiment, the third picture is encoded as an inter-coded picture predicted from one or both of the first and second pictures using RPR.

In one embodiment, the first picture is encoded as an inter-coded picture.

In one embodiment, the first picture is encoded as an intra-coded picture.

In one embodiment, the bitstream uses periodic intra-refresh to refresh an image.

In one embodiment, the first picture is an Intra Random Access Point (IRAP) picture.

In one embodiment, the first picture is periodically encoded as an IRAP picture.

In one embodiment, the first picture is encoded as an IRAP picture at a variable position in the bitstream.

In one embodiment, a resolution of the IRAP picture is lower than a resolution of an inter-coded picture.

In one embodiment, one or more inter-coded pictures encoded in the bitstream use RPR to predict from their associated IRAP picture.

In one embodiment, the IRAP pictures are configured as reference pictures for prediction use only at the receiving device.

In one embodiment, a resolution of each of one or more consecutive pictures following their associated IRAP picture in the decoding order is incrementally increased.

In one embodiment, the first resolution is lower than the second resolution.

In one embodiment, at least one of a height and a width of the first picture at the first resolution is smaller than a corresponding height and width of the second picture at the second resolution.

In one embodiment, a change from the first resolution to the second resolution when encoding the first and second pictures adapts a bitrate or quality of at least one of the first picture, the second picture and one or more subsequent pictures.

In one embodiment, the bitrate or the quality of the at least one of the first picture, second picture and one or more subsequent pictures is further adapted based on one or more adaptive rate control tools.

In one embodiment, the one or more adaptive rate control tools comprise:

• changing a quantization parameter (QP) on one of a block level, a slice level, and a picture level;

• changing one or more lambda values;

• reducing a frame rate; and/or

• increasing the frame rate. • In one embodiment, the receiving device comprises a decoder.

• In one embodiment, the receiving device comprises a network node.

• In one embodiment, the bitstream is a coded video stream (CVS).

An apparatus can perform any of the methods herein described by implementing any functional means, modules, units, or circuitry. In one embodiment, for example, the apparatuses comprise respective circuits or circuitry configured to perform the steps shown in the method figures. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and/or one or more microprocessors in conjunction with memory. For instance, the circuitry may include one or more microprocessors or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory may include program instructions for executing one or more telecommunications and/or data communications protocols as well as instructions for carrying out one or more of the techniques described herein, in several embodiments. In embodiments that employ memory, the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

Figure 8 is a functional block diagram illustrating some exemplary components of a sending device 200 configured according to embodiments of the present disclosure. As seen in Figure 8, sending device 200 comprises a physical computing node having physical circuitry and can be configured to implement one or more of the encoding functions, as previously described. In some cases, there are multiple sending devices 200 configured to implement one or more of the encoding functions. Thus, according to the present disclosure, there may be multiple sending devices 200 distributed throughout the communication network (e.g., the cloud) configured to execute the previously described functions.

As seen in Figure 8, sending device 200 comprises communication circuitry 202, processing circuitry 204, an encoder 206, and memory 208. The communication circuitry 202 comprises the hardware required for communicating directly or indirectly with one or more other network nodes, as well as with a receiving device 300, such as a user device. In this regard, the communication circuitry 202 may comprise a Network Interface Circuit (NIC) (e.g., an ETHERNET or similar interface). In some embodiments, the sending device 200 may be configured for wireless communications, and thus, communication circuitry 202 would comprise the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, sending device 200 may also be coupled to one or more antenna (not shown). In other embodiments, however, sending device 200 is configured to communicate via a wire interface. The processing circuitry 204 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the sending device 200. In accordance with the present disclosure, processing circuitry 204 may comprise the encoder 208 and can be configured by software to perform one or more of the methods herein described including method 120 seen in Figure 7.

Memory 208 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 204 for operation. Memory 208 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 208 stores a computer program 210 comprising executable instructions that configure the processing circuit 204 in the sending device 200 to perform one or more of the methods herein described including method 120 seen in Figure 7. A computer program 210 in this regard may comprise one or more code modules corresponding to the means or units described above.

In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 210 for configuring the processing circuitry 204 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 210 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. A computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

Embodiments of the present disclosure further include a carrier containing such a computer program 210. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of an apparatus, cause the apparatus to perform as described above.

Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by sending device 200. This computer program product may be stored on a computer readable recording medium. Figure 9 is a functional block diagram illustrating some exemplary components of a receiving device 300 configured according to embodiments of the present disclosure. As seen in Figure 9, receiving device 300 comprises a physical computing node having physical circuitry and can be configured to implement one or more of the decoding functions, as previously described. In some cases, there are multiple receiving devices 300 configured to implement one or more of the decoding functions. Thus, according to the present disclosure, there may be multiple receiving devices 300 distributed throughout the communication network (e.g., the cloud) configured to execute the previously described decoding functions.

As seen in Figure 9, receiving device 300 comprises communication circuitry 302, processing circuitry 304, a decoder 306, and memory 308. The communication circuitry 302 comprises the hardware required for communicating directly or indirectly with one or more other network nodes, as well as with a sending device 200. In this regard, the communication circuitry 302 may comprise a Network Interface Circuit (NIC) (e.g., an ETHERNET or similar interface). In some embodiments, the receiving device 300 may be configured for wireless communications, and thus, communication circuitry 302 would comprise the radio frequency (RF) circuitry needed for transmitting and receiving signals over a wireless communication channel. In these cases, receiving device 300 may also be coupled to one or more antenna (not shown). In other embodiments, however, receiving device 300 is configured to communicate via a wire interface.

The processing circuitry 304 comprises one or more microprocessors, hardware, firmware, or a combination thereof that controls the overall operation of the receiving device 300. In accordance with the present disclosure, processing circuitry 304 may comprise the decoder 306 and can be configured by software to decode the pictures sent in the bitstream from the sending device 200.

Memory 308 comprises both volatile and non-volatile memory for storing computer program code and data needed by the processing circuitry 304 for operation. Memory 308 may comprise any tangible, non-transitory computer-readable storage medium for storing data including electronic, magnetic, optical, electromagnetic, or semiconductor data storage. Memory 308 stores a computer program 310 comprising executable instructions that configure the processing circuit 304 in the receiving device 300 to decode the pictures sent in the bitstream from the sending device 200. A computer program 310 in this regard may comprise one or more code modules corresponding to the means or units described above.

In general, computer program instructions and configuration information are stored in a non-volatile memory, such as a ROM, erasable programmable read only memory (EPROM) or flash memory. Temporary data generated during operation may be stored in a volatile memory, such as a random-access memory (RAM). In some embodiments, computer program 310 for configuring the processing circuitry 304 as herein described may be stored in a removable memory, such as a portable compact disc, portable digital video disc, or other removable media. The computer program 310 may also be embodied in a carrier such as an electronic signal, optical signal, radio signal, or computer readable storage medium.

Those skilled in the art will also appreciate that embodiments herein further include corresponding computer programs. As stated above, a computer program comprises instructions which, when executed on at least one processor of an apparatus, cause the apparatus to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

Embodiments of the present disclosure further include a carrier containing such a computer program 310. This carrier may comprise one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

Embodiments further include a computer program product comprising program code portions for performing the steps of any of the embodiments herein when the computer program product is executed by receiving device 300. This computer program product may be stored on a computer readable recording medium.

The present embodiments may, of course, be carried out in other ways than those specifically set forth herein without departing from characteristics described herein. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, and all changes coming within the meaning and equivalency range of the appended embodiments are intended to be embraced therein.

Claims

1 . A method (120), implemented by a sending device (200) comprising an encoder, for encoding pictures to a bitstream, the method comprising: determining (122) a first resolution for a first picture, and a second resolution for a second picture, wherein the first resolution is different from the second resolution and wherein at least one of the first and second resolutions is determined based on: a decision to maintain a specified bitrate; a decision to satisfy a certain level of quality; a decision to satisfy a specified latency requirement; a decision to allocate resources differently; and/or one or more features of one or both of the first and second pictures; encoding (124) the first picture to the bitstream at the first resolution; and encoding (126) the second picture to the bitstream at the second resolution, wherein the second picture follows the first picture in decoding order and is encoded as an intercoded picture predicted from the first picture.

2. The method of claim 1 further comprising sending (132) the bitstream to a receiving device (300).

3. The method of any of claims 1-2, wherein the second picture is encoded as an inter-coded picture predicted from the first picture using Reference Picture Resampling (RPR).

4. The method of any of claims 1-3, wherein the resolution for the first picture is determined to be lower than the resolution of the second picture based on at least one of the following: to maintain the specific bitrate; to satisfy a certain level of quality; to reach a certain latency requirement; to decrease a resource allocation for encoding the first and/or the second picture; and to increase a rate distortion optimization (RDO).

5. The method of any of claims 1-3, wherein the resolution for the second picture is determined to be lower than the resolution of the first picture based on at least one of the following: to maintain the specific bitrate; to satisfy a certain level of quality; to reach a certain latency requirement; to decrease a resource allocation for encoding the second picture; and to increase a rate distortion optimization (RDO).

6. The method of any of claims 1-5, wherein the first resolution is incrementally increased or incrementally decreased over a number of pictures to reach the second resolution.

7. The method of any of claims 1-6, wherein the one or more features comprises a complexity of the first and/or second pictures.

8. The method of any of claims 1-7, wherein the one or more features comprises one or more of: a content of one or both of the first and second pictures; a change in motion of an object in at least one of the first and second pictures; a level of detail; and a sudden change in pixel values between the first and second picture.

9. The method of claim 8, wherein the level of detail for the at least one of the first and second pictures is based on one or more of: a block partitioning of the encoder; an amount or size of one or more quantization coefficients; and an image analysis of the at least one of the first and second pictures.

10. The method of any of claims 8-9, wherein the sudden change in pixel values between the first and second picture is caused by one of: a scene change; a sudden change in brightness; and a sudden change in a scene associated with the first and second pictures.

11 . The method of any of claims 8-10, wherein the sudden change in pixel values occurs from frame-to-frame.

12. The method of any of claims 8-10, wherein the sudden change in pixel values occurs over a number of frames.

13. The method of any of claims 1-12, further comprising: determining (128) a third resolution for a third picture, wherein the third resolution is different from the second resolution; and encoding (130) the third picture to the bitstream at the third resolution, wherein the third picture follows the first and second pictures in the decoding order and is encoded as an inter-coded picture predicted from one or both of the first and second pictures.

14. The method of claim 13, wherein the first resolution of the first picture is incrementally increased or incrementally decreased over the second resolution of the second picture to reach the third resolution of the third picture.

15. The method of any of claim 13-14, wherein the third picture is encoded as an inter-coded picture predicted from one or both of the first and second pictures using RPR.

16. The method of any of claims 1-15, wherein the first picture is encoded as an inter-coded picture.

17. The method of any of claims 1-16, wherein the first picture is encoded as an intra-coded picture.

18. The method of any of claims 1-17, wherein the bitstream uses periodic intra-refresh to refresh an image.

19. The method of any of the preceding claims, wherein the first picture is an Intra Random Access Point (IRAP) picture.

20. The method of any of the preceding claims, wherein the first picture is periodically encoded as an IRAP picture.

21 . The method of claim 20, wherein the first picture is encoded as an IRAP picture at a variable position in the bitstream.

22. The method of any of claims 20-21 , wherein a resolution of the IRAP picture is lower than a resolution of an inter-coded picture.

23. The method of any of claims 20-22, wherein one or more inter-coded pictures encoded in the bitstream use RPR to predict from their associated IRAP picture.

24. The method of any of claims 20-23, wherein the IRAP pictures are configured as reference pictures for prediction use only at the receiving device.

25. The method of any of claims 20-24, wherein a resolution of each of one or more consecutive pictures following their associated IRAP picture in the decoding order is incrementally increased.

26. The method of any of the preceding claims, wherein the first resolution is lower than the second resolution.

27. The method of any of the preceding claims, wherein at least one of a height and a width of the first picture at the first resolution is smaller than a corresponding height and width of the second picture at the second resolution.

28. The method of any of the preceding claims, wherein a change from the first resolution to the second resolution when encoding the first and second pictures adapts a bitrate of at least one of the first picture, the second picture and one or more subsequent pictures.

29. The method of claim 28, wherein the bitrate of the at least one of the first picture, second picture and one or more subsequent pictures is further adapted based on one or more adaptive rate control tools.

30. The method of claim 29, wherein the one or more adaptive rate control tools comprise: changing a quantization parameter (QP) on one of a block level, a slice level, and a picture level; changing one or more lambda values; reducing a frame rate; and/or increasing the frame rate.

31 . The method of any of claims 1-30, wherein the receiving device comprises a decoder (306).

32. The method of any of claims 1-30, wherein the receiving device comprises a network node (20).

33. The method of any of the previous claims wherein the bitstream comprises a coded video stream (CVS).

34. A sending device (200) comprising an encoder for encoding pictures to a bitstream (110), the sending device configured to: determine (122) first resolution for a first picture, and a second resolution for a second picture, wherein the first resolution is different from the second resolution and wherein at least one of the first and second resolutions is determined based on: a decision to maintain a specified bitrate; a decision to satisfy a certain level of quality; a decision to satisfy a specified latency requirement; a decision to allocate resources differently; and/or one or more features of one or both of the first and second pictures; encode (124) the first picture to the bitstream at the first resolution; and encode (126) the second picture to the bitstream at the second resolution, wherein the second picture follows the first picture in decoding order and is encoded as an intercoded picture predicted from the first picture.

35. The sending device of claim 34, wherein the sending device is further configured to perform the method according to any one of claims 2-33.

36. A sending device (200) comprising an encoder for encoding pictures to a bitstream (110), the sending device comprising: communications circuitry (202) configured to communicate with a receiving device (300); and processing circuitry (204) operatively connected to the communications circuitry and configured to: determine (122) a first resolution for a first picture, and a second resolution for a second picture, wherein the first resolution is different from the second resolution and wherein at least one of the first and second resolutions is determined based on: a decision to maintain a specified bitrate; a decision to satisfy a certain level of quality; a decision to satisfy a specified latency requirement; a decision to allocate resources differently; and/or one or more features of one or both of the first and second pictures; encode (124) the first picture to the bitstream at the first resolution; and encode (126) the second picture to the bitstream at the second resolution, wherein the second picture follows the first picture in decoding order and is encoded as an intercoded picture predicted from the first picture.

37. The sending device of claim 36, wherein the processing circuitry is further configured to perform the method according to any one of claims 2-33.

38. A computer program comprising instructions that, when executed on processing circuitry of a sending device, causes the sending device to perform the method according to any of claims 1-33.

39. A carrier comprising the computer program of claim 38, wherein the carrier is one of an electronic signal, optical signal, radio signal, or computer readable storage medium.

40. A non-transitory computer-readable storage medium (208) comprising a computer program (210) stored thereon, the computer program comprising executable instructions that, when executed by processing circuitry in a sending device, causes the sending device to perform the method of any of claims 1-33.