JP2024050857A - Encoder tuning to improve the trade-off between latency and video quality in cloud gaming applications - Google Patents

Abstract

A method for cloud gaming is provided to provide a high quality experience to users of a cloud gaming service. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes encoding the plurality of video frames at an encoder bitrate, and transmitting the compressed plurality of video frames from a streamer on the cloud gaming server to a client. The method also includes measuring a maximum receiving bandwidth of the client, monitoring the encoding of the plurality of video frames at the streamer, and dynamically adjusting parameters of the encoder based on the monitoring of the encoding. [Selected Figure] FIG.

Description

The present disclosure relates to streaming systems configured to stream content over a network, and more specifically to high performance encoders and decoders for cloud gaming systems, and streaming systems configured for encoder adjustments that are aware of network transmission speed and reliability, as well as overall latency targets.

In recent years, there has been a continuous push for online services that allow online or cloud gaming in a streaming format between cloud gaming servers and clients connected over a network. The streaming format has become increasingly popular due to the availability of game titles on demand, the ability to network between players for multiplayer gaming, sharing of assets between players, sharing of instant experiences between players and/or spectators, allowing friends to watch friends play video games, having friends join in on-going gameplay, etc.
Unfortunately, the demands are also pushing up against the capacity of network connections and limitations of processing performed on the server and client with sufficient responsiveness to render high quality images when delivered to the client. For example, the results of all game activity performed on the server need to be compressed and transmitted back to the client with low millisecond latency for the best user experience. Round-trip latency can be defined as the overall time between a user's controller input and the display of a video frame at the client, which may include the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the use of that input at the server to generate a video frame responsive to the input, the processing and transfer of the video frame to an encoding unit (e.g., scan-out), the encoding of the video frame, the transmission of the encoded video frame back to the client, the receipt and decoding of the video frame, and either the processing or staging of the video frame prior to display.
One-way latency may be defined as a portion of the round-trip latency consisting of the time from the start of the transfer (e.g., scan-out) of a video frame to an encoding unit at the server to the start of the display of the video frame at the client. A portion of the round-trip latency and one-way latency is associated with the time it takes for a data stream to be transmitted from the client to the server and back across a communication network. Another portion is associated with the processing at the client and server, and improvements in these operations, such as advanced strategies for decoding and displaying frames, can result in a substantial reduction in the round-trip latency and one-way latency between the server and the client, providing a higher quality experience to users of the cloud gaming service.

The embodiments of the present disclosure have been made against this background.

Embodiments of the present disclosure relate to a streaming system configured to stream content (e.g., games) over a network, and more particularly to a streaming system configured to provide encoder adjustments to improve the tradeoff between one-way latency and video quality in a cloud gaming system, where the encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and/or number of video frames exceeding a target frame size, where the adjusted parameters may include encoder bitrate, target frame size, maximum frame size, and quantization parameter (QP) value, and where high performance encoders and decoders serve to reduce the overall one-way latency between the cloud gaming server and the client.

Embodiments of the present disclosure disclose a method for cloud gaming. The method includes generating a plurality of video frames when executing a video game at a cloud gaming server. The method includes encoding the plurality of video frames at an encoder bit rate, where the compressed plurality of video frames are transmitted from a streamer at the cloud gaming server to a client. The method includes measuring a maximum receive bandwidth of the client. The method includes monitoring encoding of the plurality of video frames at the streamer. The method includes dynamically adjusting parameters of the encoder based on the monitoring of the encoding.

In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating a plurality of video frames when executing a video game at a cloud gaming server. The computer readable medium includes program instructions for encoding the plurality of video frames at an encoder bit rate, and the compressed plurality of video frames are transmitted from a streamer at the cloud gaming server to a client. The computer readable medium includes program instructions for measuring a maximum receive bandwidth of the client. The computer readable medium includes program instructions for monitoring encoding of the plurality of video frames at the streamer. The computer readable medium includes program instructions for dynamically adjusting parameters of the encoder based on the monitoring of the encoding.

In yet another embodiment, a computer system includes a processor and a memory coupled to the processor having instructions stored therein, the instructions, when executed by the computer system, causing the computer system to perform a method of cloud gaming. The method includes generating a plurality of video frames when executing a video game at a cloud gaming server. The method includes encoding the plurality of video frames at an encoder bit rate, the compressed plurality of video frames being transmitted from a streamer at the cloud gaming server to a client. The method includes measuring a maximum receive bandwidth of the client. The method includes monitoring encoding of the plurality of video frames at the streamer. The method includes dynamically adjusting parameters of the encoder based on the monitoring of the encoding.

In yet another embodiment, a method for cloud gaming is disclosed. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes predicting a scene change for a first video frame of the video game, the scene change being predicted before the first video frame is generated. The method includes generating a scene change hint, where the first video frame is a scene change. The method includes sending the scene change hint to an encoder. The method includes delivering the first video frame to the encoder, where the first video frame is encoded as an I-frame based on the scene change hint. The method includes measuring a maximum receive bandwidth of a client. The method includes determining whether to encode or not encode a second video frame received at the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

In another embodiment, a non-transitory computer readable medium storing a computer program for cloud gaming is disclosed. The computer readable medium includes program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server. The computer readable medium includes program instructions for predicting a scene change in a first video frame of the video game, the scene change being predicted before the first video frame is generated. The computer readable medium includes program instructions for generating a scene change hint, the first video frame being a scene change. The computer readable medium includes program instructions for sending the scene change hint to an encoder. The computer readable medium includes program instructions for delivering the first video frame to the encoder, the first video frame being encoded as an I-frame based on the scene change hint. The computer readable medium includes program instructions for measuring a maximum receive bandwidth of a client. The computer readable medium includes program instructions for determining whether to encode or not encode a second video frame received at the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

In yet another embodiment, a computer system includes a processor and a memory coupled to the processor having instructions stored therein that, when executed by the computer system, cause the computer system to perform a method of cloud gaming. The method includes generating a plurality of video frames when executing a video game on a cloud gaming server. The method includes predicting a scene change for a first video frame of the video game, the scene change being predicted before the first video frame is generated. The method includes generating a scene change hint, where the first video frame is a scene change.
The method includes sending a scene change hint to an encoder. The method includes delivering a first video frame to the encoder, the first video frame being encoded as an I-frame based on the scene change hint. The method includes measuring a maximum receive bandwidth of a client. The method includes determining whether to encode or not encode a second video frame received at the encoder based on the maximum receive bandwidth of the client and a target resolution of the client display.

Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the present disclosure.

The present disclosure can be best understood by referring to the following description in conjunction with the accompanying drawings.

FIG. 2 is a diagram of a VSYNC signal at the beginning of a frame period according to one embodiment of the present disclosure. FIG. 2 is a diagram of the frequency of a VSYNC signal according to one embodiment of the present disclosure. FIG. 1 is a diagram of a system for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations according to one embodiment of the disclosure, where the VSYNC signals can be synchronized and offset to reduce one-way latency. FIG. 1 is a diagram for providing gaming between two or more peer devices according to one embodiment of the present disclosure, where the VSYNC signals can be synchronized and offset to achieve optimal timing of reception of controllers and other information between the devices. 1 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. 1 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. 1 illustrates the variation in one-way latency between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure. 1 illustrates a network configuration including a cloud gaming server and clients when streaming video frames generated from a video game running on the server, where the VSYNC signals between the server and clients are synchronized and offset, allowing overlapping of operations at the server and clients and reducing one-way latency between the server and clients. FIG. 1 is a flow diagram illustrating a method for cloud gaming, according to one embodiment of the present disclosure, in which encoding of video frames includes adjusting encoder parameters with awareness of network transmission speed and reliability, as well as overall latency goals. FIG. 1 illustrates measurement of a client's bandwidth by a streamer component operating at the application layer according to one embodiment of the present disclosure, where the streamer is configured to monitor and adjust the encoder so that compressed video frames can be transmitted at a rate within the client's measured bandwidth. FIG. 1A illustrates an embodiment of the present disclosure showing setting an encoder's quantization parameter (QP) to optimize quality and buffer utilization at the client; and FIG. 1B illustrates an embodiment of the present disclosure showing adjusting target frame size, maximum frame size, and/or QP (e.g., minimum QP and/or maximum QP) encoder settings to reduce the occurrence of I-frames that exceed the true target frame size supported by the client. FIG. 1 is a flow diagram illustrating a method for cloud gaming according to one embodiment of the present disclosure, where encoding a video frame includes determining whether to skip a video frame or to delay the encoding and transmission of a video frame if the encoding is long, such as when encoding an I-frame, or if the video frame being generated is large. 1A illustrates a sequence of video frames being compressed by an encoder according to an embodiment of the present disclosure, where the encoder drops the encoding of a video frame after encoding an I-frame when the client bandwidth is low relative to the target resolution of the client's display; FIG. 1B illustrates a sequence of video frames being compressed by an encoder according to an embodiment of the present disclosure, where in each sequence a video frame is encoded as an I-frame and a subsequent video frame is also encoded after a delay in encoding an I-frame when the client bandwidth is medium or high relative to the target resolution of the client's display; and FIG. 1C illustrates a sequence of video frames being compressed by an encoder according to an embodiment of the present disclosure, where in each sequence a video frame is encoded as an I-frame and a subsequent video frame is also encoded after a delay in encoding an I-frame when the client bandwidth is medium or high relative to the target resolution of the client's display. 1 illustrates components of an example device that can be used to implement aspects of various embodiments of the present disclosure.

Although the following detailed description includes many specific details for purposes of illustration, one of ordinary skill in the art will appreciate that many variations and modifications to the following details are within the scope of the disclosure. Accordingly, the aspects of the disclosure described below are set forth without loss of generality to, and without imposing limitations on, the claims that follow this description.

Generally speaking, various embodiments of the present disclosure describe methods and systems configured to reduce latency and/or latency instability between a source device and a target device when streaming media content (e.g., streaming audio and video from a video game). Latency instability can occur in one-way latency between the server and the client due to the additional time required to generate a complex frame (e.g., a scene change) at the server, the increased time it takes to encode/compress the complex frame at the server, variable communication paths through the network, and the increased time it takes to decode the complex frame at the client. Latency instability can also be introduced by clock differences at the server and the client that cause drift between the server and the client's VSYNC signals.
In embodiments of the present disclosure, one-way latency between a server and a client can be reduced in cloud gaming applications by providing high performance encoding and decoding. When decompressing streaming media (e.g., streaming video, movies, clips, content), it is possible to buffer a significant amount of the decompressed video, and therefore, when displaying the streamed content, it is possible to depend on the average decoding capabilities and metrics (e.g., to support 4K media at 60 Hz, it depends on the average amount of decoding resources).
However, in cloud gaming, the longer it takes to perform encoding and/or decoding operations (even for a single frame), the higher the one-way latency will be. Therefore, for cloud gaming, it is beneficial to provide powerful decoding and encoding resources that may seem unnecessary compared to the needs of streaming video applications, and the resources need to be optimized for the time to process the frames that require longer or the longest processing. In other embodiments of the present disclosure, encoder tuning may be performed to improve the tradeoff between latency and video quality in cloud gaming applications.
Encoder adjustments are made with knowledge of the network transmission speed and reliability, and overall latency goals. In an embodiment, if the encoding is taking long or the data generated is large (e.g., both conditions can occur with compressed I-frames), methods are implemented to determine whether to delay the encoding and transmission of subsequent frames or to skip them.
In an embodiment, adjustments to the quantization parameter (QP) value, the target frame size, and the maximum frame size are performed based on the available network speed to the client, e.g., if the network speed is faster, the QP may be reduced.
In other embodiments, monitoring of the I-frame occurrence rate is performed and used to set the QP. For example, if the frequency of I-frames is low, the QP can be reduced (e.g., resulting in higher encoding accuracy or higher quality of encoding) such that encoding of video frames can be skipped to keep one-way latency low while sacrificing video playback quality. Thus, high performance encoding and decoding, and encoder adjustments performed to improve the tradeoff between latency and video quality for cloud gaming applications, can lead to reduced one-way latency between cloud gaming servers and clients, smoother frame rates, and more reliable and/or consistent one-way latency.

With a general understanding of the various embodiments discussed above, exemplary details of the embodiments will now be described with reference to the various drawings.

Throughout the specification, references to a "game" or a "video game" or a "gaming application" are meant to represent any type of interactive application that is directed through the execution of input commands. For illustrative purposes only, interactive applications include applications for games, word processing, video processing, video game processing, and the like. Furthermore, the terms introduced above are intended to be interchangeable.

Cloud gaming involves running a video game on a server to generate game-rendered video frames, which are then transmitted to a client for display. The timing of operations on both the server and the client may be associated with respective vertical synchronization (VSYNC) parameters. When the VSYNC signals are properly synchronized and/or offset between the server and/or client, operations performed on the server (e.g., generating and transmitting video frames over one or more frame periods) are synchronized with operations performed on the client (e.g., displaying video frames on a display at a display frame or refresh rate corresponding to the frame periods). In particular, a server VSYNC signal generated on the server and a client VSYNC signal generated on the client may be used to synchronize operations on the server and the client. That is, when the server and client VSYNC signals are synchronized and/or offset, the server generates and transmits video frames in sync with the way the client will display those video frames.

The VSYNC signal and vertical blanking interval (VBI) are incorporated to generate and display video frames when streaming media content between the server and the client. For example, the server will attempt to generate rendered video frames for a game at one or more frame periods defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7 ms, generating a video frame every frame period results in 60 Hz operation, while generating one video frame every two frame periods results in 30 Hz operation), and then encode and transmit the video frames to the client. At the client, the received encoded video frames are decoded and displayed, and the client displays each video frame rendered for display beginning with the corresponding client VSYNC.

For illustration purposes, FIG. 1A shows how a VSYNC signal 111 may indicate the start of a frame period, where various operations may be performed during the corresponding frame period at the server and/or client. When streaming media content, the server may use the server VSYNC signal to generate and encode video frames, and the client may use the client VSYNC signal to display the video frames. The VSYNC signal 111 is generated at a specified frequency corresponding to a specified frame period 110, as shown in FIG. 1B. Additionally, the VBI 105 defines the period between when the last raster line is drawn on the display for the previous frame period and when the first raster line (e.g., top) is drawn on the display. As shown, after the VBI 105, the video frame rendered for display is displayed via raster scan line 106 (e.g., raster line by raster line from left to right).

Furthermore, various embodiments of the present disclosure are disclosed for reducing one-way latency and/or latency stability between a source device and a target device, such as when streaming media content (e.g., video game content). For purposes of illustration only, various embodiments for reducing one-way latency and/or latency stability are described in a server-client network configuration. However, it is understood that the various techniques disclosed for reducing one-way latency and/or latency stability may be implemented in other network configurations and/or on peer-to-peer networks, such as those shown in Figures 2A-2D. For example, various embodiments disclosed for reducing one-way latency and/or latency stability may be implemented between one or more server and client devices in various configurations (e.g., server-client, server-server, server-multiple clients, server-multiple servers, client-client, client-multiple clients, etc.).

FIG. 2A is a diagram of a system 200A for providing games between one or more cloud gaming networks 290 and/or servers 260 and one or more client devices 210 over a network 250 in various configurations, where according to one embodiment of the disclosure, server and client VSYNC signals can be synchronized and offset, and/or dynamic buffering can be performed on the client, and/or encoding and transmission operations on the server can be duplicated, and/or receiving and decoding operations on the client can be duplicated, and/or decoding and display operations on the client can be duplicated to reduce one-way latency between the server 260 and the client 210.
In particular, according to one embodiment of the present disclosure, the system 200A provides games via a cloud gaming network 290, the games being executed remotely from the client devices 210 (e.g., thin clients) of corresponding users playing the games. The system 200A can provide game control to one or more users playing one or more games through the cloud gaming network 290 via the network 250, in either single player or multiplayer mode. In some embodiments, the cloud gaming network 290 can include multiple virtual machines (VMs) executing on a hypervisor of a host machine, the one or more virtual machines being configured to execute a game processor module utilizing hardware resources available to the hypervisor of the host machine. The network 250 can include one or more communication technologies. In some embodiments, the network 250 can include a fifth generation (5G) network technology having an advanced wireless communication system.

In some embodiments, communication may be facilitated using wireless technology. Such technologies may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. 5G networks are digital cellular networks, where the service area covered by a provider is divided into small geographic areas called cells. Analog signals representing sound and images are digitized by the telephone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices within a cell communicate over the airwaves using a local antenna array and a low-power automatic transceiver (transmitter and receiver) within the cell over a frequency channel assigned by the transceiver from a pool of frequencies reused by other cells. The local antennas are connected to the telephone network and the Internet by high-bandwidth optical fiber or wireless backhaul connections. As with other cellular networks, mobile devices moving from one cell to another are automatically transferred to the new cell. It should be understood that 5G networks are merely one exemplary type of communication network, and that embodiments of the present disclosure may utilize previous generations of wireless or wired communication, as well as subsequent generations of wired or wireless technologies following 5G.

As shown, cloud gaming network 290 includes game server 260 that provides access to multiple video games. Game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, game server 260 may manage virtual machines that support game processors that instantiate instances of games for users. In this manner, multiple game processors of game server 260 associated with multiple virtual machines are configured to run multiple instances of one or more games associated with gameplay for multiple users.
In that manner, the backend server supports streaming of gameplay media (e.g., video, audio, etc.) for multiple game applications to multiple corresponding users. That is, the game servers 260 are configured to stream data (e.g., rendered images and/or frames of corresponding gameplay) back to corresponding client devices 210 over the network 250. In that manner, computationally complex game applications may be executed on the backend servers in response to controller inputs received and forwarded by the client devices 210. Each server may render images and/or frames that may then be encoded (e.g., compressed) and streamed to corresponding client devices for display.

For example, multiple users may access cloud gaming network 290 over communications network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, client devices 210 may be configured as thin clients that interface with backend servers (e.g., game servers 260 of cloud gaming network 290) configured to provide computing functionality (e.g., including game title processing engine 211).
In another embodiment, client device 210 may be configured with a game title processing engine and game logic for at least some local processing of a video game, and may be further utilized to receive streaming content generated by the video game running in a back-end, or for other content provided by back-end server support. For local processing, the game title processing engine includes basic processor-based functionality for running the video game and services related to the video game. The game logic is stored on the local client device 210 and is used to run the video game.

In particular, a client device 210 of a corresponding user (not shown) is configured to request access to the game over a communications network 250, such as the Internet, and to render display images generated by the video game executed by a game server 260, the encoded images being delivered to the client device 210 for display in association with the corresponding user.
For example, a user may interact, via a client device 210, with an instance of a video game running on a game processor of a game server 260. More specifically, the instance of the video game is executed by a game title processing engine 211. Corresponding game logic (e.g., executable code) 215 implementing the video game is stored in and accessible via a data store (not shown) and is used to execute the video game. The game title processing engine 211 may support multiple video games using multiple game logics, each of which is selectable by a user.

For example, client device 210 is configured to interact with game title processing engine 211 in connection with a corresponding user's gameplay, such as via input commands used to drive the gameplay. In particular, client device 210 can receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc.
Client device 210 may be any type of computing device having at least a memory and a processor module that can connect to game server 260 via network 250. Backend game title processing engine 211 is configured to generate rendered images that are distributed over network 250 for display on corresponding displays associated with client device 210.
For example, through a cloud-based service, rendered images for a game may be delivered by an instance of the corresponding game running on a game execution engine 211 of the game server 260. That is, the client device 210 is configured to receive encoded images (e.g., encoded from game rendered images generated through the execution of a video game) and display the rendered images for display 11. In one embodiment, the display 11 includes an HMD (e.g., displaying VR content). In some embodiments, the rendered images may be streamed wirelessly or wired to a smartphone or tablet directly from the cloud-based service or via the client device 210 (e.g., PlayStation® Remote Play).

In one embodiment, the game server 260 and/or the game title processing engine 211 include basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. Additionally, the game application services include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with friends' social networks, communication channels, text messaging, instant messaging, chat support, etc.

In one embodiment, cloud gaming network 290 is a distributed game server system and/or architecture. In particular, a distributed game engine that executes game logic is configured as a corresponding instance of a corresponding game. In general, a distributed game engine takes each function of the game engine and distributes them for execution by multiple processing entities. Individual functions may be further distributed across one or more processing entities. The processing entities may be configured in a variety of configurations, including physical hardware, and/or as virtual components or virtual machines, and/or as virtual containers, which differ from virtual machines because containers virtualize instances of game applications running on a virtualized operating system.
The processing entities may utilize and/or rely on servers and their underlying hardware on one or more servers (computing nodes) of the cloud gaming network 290, which may be located in one or more racks. Coordination, allocation, and management of the execution of these functions to the various processing entities is performed by a distributed synchronization layer. As such, the execution of those functions is controlled by the distributed synchronization layer to generate media (e.g., video frames, audio, etc.) for the game application in response to controller inputs by the player. The distributed synchronization layer allows these functions to be efficiently executed (e.g., through load balancing) across the distributed processing entities, such that critical game engine components/functions are distributed and restructured for more efficient processing.

The game title processing engine 211 includes a central processing unit (CPU) and a group of graphics processing units (GPUs) that may be configured to perform multi-tenancy GPU functions. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

2B is a diagram for providing a game between two or more peer devices, according to one embodiment of the present disclosure, where the VSYNC signals can be synchronized and offset to achieve optimal timing of the reception of controllers and other information between the devices. For example, a head-to-head game can be performed using two or more peer devices connected directly via network 250 or via peer-to-peer communications (e.g., Bluetooth, local area networking, etc.).

As shown, the game is running locally on each of the client devices 210 (e.g., game consoles) of the corresponding users playing the video game, and the client devices 210 communicate via peer-to-peer networking. For example, an instance of the video game is executed by a game title processing engine 211 of the corresponding client device 210. Game logic 215 (e.g., executable code) implementing the video game is stored on the corresponding client device 210 and is used to run the game. For purposes of illustration, the game logic 215 may be distributed to the corresponding client device 210 via a portable medium (e.g., optical medium) or over a network (e.g., downloaded from a game provider via the Internet).

In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for executing services related to games and game applications. For example, the processor-based functionality includes 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, and the like. Additionally, the game application services include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, social friend management, communication with friends' social network, communication channels, text messaging, instant messaging, chat support, and the like.

Client device 210 can receive input from various types of input devices, such as a game controller, a tablet computer, a keyboard, gestures captured by a video camera, a mouse, a touchpad, etc. Client device 210 can be any type of computing device having at least a memory and a processor module, and is configured to generate rendered images that are executed by game title processing engine 211 and display the rendered images on a display (e.g., display 11, or display 11 including a head mounted display (HMD), etc.).
For example, the rendered images may be associated with an instance of a game running locally on client device 210, implementing the corresponding user's gameplay, such as via input commands used to drive the gameplay. Some examples of client device 210 include a personal computer (PC), a game console, a home theater device, a general purpose computer, a mobile computing device, a tablet, a phone, or any other type of computing device capable of running an instance of a game.

2C illustrates various network configurations, including those illustrated in FIGS. 2A-2B, that benefit from proper synchronization and offsetting of VSYNC signals between source and target devices in accordance with embodiments of the present disclosure. In particular, the various network configurations benefit from proper alignment of the frequency of the server and client VSYNC signals and timing offsets of the server and client VSYNC signals to reduce one-way latency and/or latency variability between the server and client. For example, one network device configuration includes a cloud gaming server (e.g., source) to client (target) configuration.
In one embodiment, the client may include a web RTC client configured to provide audio and video communication within a web browser. Another network configuration includes a client (e.g., source) to server (target) configuration. Yet another network configuration includes a server (e.g., source) to server (e.g., target) configuration. Another network device configuration includes a client (e.g., source) to client (target) configuration, where each client may be, for example, a game console for providing head-to-head gaming.

In particular, alignment of the VSYNC signals may include synchronizing the frequencies of the server VSYNC signal and the client VSYNC signal, and may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal to maintain an ideal relationship between the server and client VSYNC signals for the purposes of eliminating drift and/or reducing one-way latency and/or latency variability.
To achieve proper alignment, in one embodiment, the server VSYNC signal may be adjusted to enforce proper alignment between the server 260 and client 210 pair.
In another embodiment, the client VSYNC signal may be adjusted to effect proper alignment between a server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from one another by a timing offset that can be adjusted at any time.
In another embodiment, alignment of the VSYNC signals may include synchronizing the frequency of the VSYNC of two clients, adjusting the timing offset between their VSYNC signals to eliminate drift, and/or achieving proper timing of receipt of controller and other information, and either VSYNC signal may be adjusted to achieve this alignment.
In yet another embodiment, alignment may include synchronizing the frequency of VSYNC of multiple servers, and may also include synchronizing the frequency of the server VSYNC signal and the client VSYNC signal and adjusting the timing offset between the client VSYNC signal and the server VSYNC signal, for example, for head-to-head cloud gaming. In server-to-client and client-to-client configurations, alignment may include both synchronizing the frequency between the server VSYNC signal and the client VSYNC signal and providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a server-to-server configuration, alignment may include synchronizing the frequency between the server VSYNC signal and the client VSYNC signal without setting a timing offset.

2D illustrates a multi-tenancy configuration between a cloud gaming server 260 and one or more clients 210 that benefits from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. In a server-to-client configuration, alignment may include both synchronizing the frequency between the server VSYNC signal and the client VSYNC signal, and providing a proper timing offset between the server VSYNC signal and the client VSYNC signal. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is adjusted at each client 210 to achieve proper alignment between the server 260 and client 210 pair.

For example, the graphics subsystem may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem may be the implementation of the graphics and/or rendering pipeline for multiple games. That is, the graphics subsystem is shared between multiple games being executed. In particular, the game title processing engine may include a CPU and GPU group configured to perform multi-tenancy GPU functions, and in one embodiment, one CPU and GPU group may be the implementation of the graphics and/or rendering pipeline for multiple games. That is, the CPU and GPU group is shared between multiple games being executed. The CPU and GPU group may be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on a corresponding CPU.

3 illustrates a general process for running a video game on a server to generate rendered video frames for the game and send those video frames to clients for display. Traditionally, some operations on the game server 260 and clients 210 are performed within a frame period defined by their respective VSYNC signals. For example, the server 260 seeks to generate rendered video frames for the game at 301 in one or more frame periods as defined by a corresponding server VSYNC signal 311.
The video frames are generated by the game in response to either control information delivered from an input device (e.g., a user's input command) in operation 350, or game logic that is not driven by control information. Transmission jitter 351 may be present when sending control information to the server 260, where jitter 351 measures the variation in network latency from the client to the server (e.g., when sending an input command).
As shown, the thick arrow indicates the current delay in sending the control information to the server 260, but due to jitter there may be a range of arrival times for the control information at the server 260 (e.g., the range enclosed by the dotted arrow). At flip time 309, the GPU arrives at a flip command indicating that the corresponding video frame has been fully generated and placed in the frame buffer of the server 260. The server 260 then performs a scan-out/scan-in (operation 302, where the scan-out may be aligned with the VSYNC signal 311) on that video frame over the subsequent frame period defined by the server VSYNC signal 311 (VBI omitted for clarity). The video frame is then encoded (operation 303) (e.g., encoding begins after the occurrence of the VSYNC signal 311, and the end of encoding may not be aligned with the VSYNC signal) and transmitted (operation 304, where the transmission may not be aligned with the VSYNC signal 311) to the client 210.
At client 210, the encoded video frames are received (operation 305, where reception may not be aligned with client VSYNC signal 312), decoded (operation 306, where decoding may not be aligned with client VSYNC signal 312), buffered, and displayed (operation 307, where start of display may not be aligned with client's VSYNC signal 312). In particular, client 210 displays each video frame rendered for display beginning with the corresponding occurrence of client VSYNC signal 312.

One-way latency 315 may be defined as the latency from the start of the transfer of a video frame to an encoding unit (e.g., scanout 302) at the server to the start of display of the video frame at the client 307. That is, one-way latency is the time from server scanout to client display, taking into account client buffering. An individual frame has a latency from the start of scanout 302 to the completion of decode 306, and this latency may vary from frame to frame due to high variability in server operations such as encoding 303 and transmission 304, network transmission with jitter 352 between the server 260 and client 210, and client receive 305.
As shown, the solid arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter 352, there may be a range of arrival times of the video frames at the client 210 (e.g., the ranges demarcated by the dotted arrows). Because one-way latency needs to be relatively stable (e.g., fairly consistent) to achieve a good playback experience, traditionally, buffering 320 is performed with the result that the display of individual frames with low latency (e.g., from the start of scanout 302 to the completion of decode 306) is delayed for several frame periods. That is, in the presence of network instability or unpredictable encoding/decoding times, additional buffering is required to ensure that one-way latency remains constant.

According to an embodiment of the present disclosure, one-way latency between a cloud gaming server and a client may vary due to clock drift when streaming video frames generated from a video game running on the server. That is, differences in frequency between the server VSYNC signal 311 and the client VSYNC signal 312 may result in the client VSYNC signal drifting relative to the frames arriving from the server 260. This drift may be due to very slight differences in the crystals used in each of the respective clocks at the server and the client. Additionally, embodiments of the present disclosure reduce one-way latency by performing one or more of synchronization and offsetting of the VSYNC signals for alignment between the server and the client, by providing dynamic buffering at the client, by overlapping the encoding and transmission of video frames at the server, by overlapping the receiving and decoding of video frames at the client, and by overlapping the decoding and display of video frames at the client.

Further, during encoding of the video frames (operation 303), in the previous technique, the encoder determines how much change there is between the current video frame being encoded and one or more previously encoded frames to determine whether there is a scene change (e.g., a complex image of the corresponding generated video frame). That is, a scene change hint can be inferred from the difference between the current frame being encoded and the previous frames already encoded. When streaming content from a server to a client over a network, the encoder at the server can decide to encode the video frames that are detected as a scene change with complexity. Otherwise, the encoder encodes the video frames that are not detected as a scene change with lower complexity.
However, scene change detection in the encoder may take up to one frame period (e.g., adding jitter) such that a video frame is initially encoded (in a first frame period) with lower complexity, but then re-encoded (in a second frame period) with higher complexity if it is determined that there is a scene change. Similarly, scene change detection may be unnecessarily triggered (such as via a small burst in the image) because the difference between the currently encoded video frame and the previously encoded video frame may exceed a threshold difference value even if there is no scene change. Thus, when a scene change is detected in the encoder, additional latency due to jitter is introduced in the encoder to accommodate performing scene change detection and re-encoding the video frame with higher complexity.

FIG. 4 illustrates the flow of data through a network configuration including a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server, where, according to an embodiment of the present disclosure, overlapping of server and client operations reduces one-way latency, and synchronization and offsetting of VSYNC signals between the server and client reduces one-way latency while at the same time reducing the variability of one-way latency between the server and client.
In particular, Figure 4 illustrates a desired alignment between the server VSYNC signal and the client VSYNC signal. In one embodiment, adjustments to the server VSYNC signal 311 are performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a network configuration of servers and clients.
In another embodiment, adjustment of the client VSYNC signal 312 is performed to obtain proper alignment between the server VSYNC signal and the client VSYNC signal, such as in a multi-tenant server to multiple client network configuration. For purposes of illustration, adjustment of the server VSYNC signal 311 is described in FIG. 4 for purposes of synchronizing the frequency of the server and client VSYNC signals and/or adjusting the timing offset between corresponding client and server VSYNC signals, but it will be understood that the client VSYNC signal 312 may also be used for adjustment.
In the context of this patent application, "synchronization" should be interpreted to mean adjusting the signals so that their frequencies match but their phases may differ, and "offset" should be interpreted to mean the time delay between the signals, such as the time between when one signal reaches its maximum value and when the other signal reaches its maximum value.

As illustrated, Figure 4 illustrates an improved process for executing a video game at a server to generate rendered video frames and send those video frames to a client for display in an embodiment of the present disclosure. The process is illustrated with respect to the generation and display of a single video frame at the server and the client. In particular, the server generates rendered video frames for the game at 401. For example, the server 260 includes a CPU (e.g., game title processing engine 211) configured to execute the game. The CPU generates one or more draw calls for the video frames, which include commands that are placed in a command buffer for execution by a corresponding GPU of the server 260 in a graphics pipeline.
The graphics pipeline may include one or more shader programs for vertices of objects in a scene to generate texture values that are rendered into a video frame for display, with this operation being performed in parallel across the GPU for efficiency. At flip time 409, the GPU reaches a flip command in the command buffer, indicating that the corresponding video frame has been completely generated and/or rendered and placed into a frame buffer at the server 260.

At 402, the server performs a scanout of the rendered video frame for the game to the encoder. In particular, the scanout is performed scanline by scanline or in groups of consecutive scanlines, where a scanline refers to the display of a single horizontal line, for example, from edge of the screen to edge of the screen. These scanlines or groups of consecutive scanlines are sometimes referred to as slices, and are referred to herein as screen slices. In particular, the scanout 402 may include several processes, including modifying the rendered frame for the game and overlaying it with another frame buffer or shrinking it to surround it with information from another frame buffer. During the scanout 402, the modified video frame is then scanned to the encoder for compression. In one embodiment, the scanout 402 is performed at the occurrence 311a of the VSYNC signal 311. In other embodiments, the scanout 402 may be performed before the occurrence of the VSYNC signal 311, such as at flip time 409.

At 403, the rendered video frame for the game (possibly modified) is encoded in an encoder, one encoder slice per encoder slice, to generate one or more encoded slices, where the encoded slices are independent of scanlines or screen slices. In this manner, the encoder generates one or more encoded (e.g., compressed) slices. In one embodiment, the encoding process begins before the scanout 402 process is fully completed for the corresponding video frame.
Additionally, the start and/or end of the encode 403 may or may not be aligned with the server VSYNC signal 311. The boundaries of an encoded slice are not limited to a single scan line, but may consist of a single scan line, or multiple scan lines. Additionally, the end of an encoded slice and/or the start of the next encoder slice may not necessarily occur at the edge of the display screen (e.g., may occur in the middle of the screen or the middle of a scan line), and an encoded slice need not traverse completely across the display screen. As illustrated, one or more encoded slices may be compressed and/or encoded, including a compressed "encoded slice A" with hash marks.

At 404, the encoded video frame is transmitted from the server to the client, and this transmission may be done on an encoded slice-by-encoded slice basis, with each encoded slice being a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 is fully completed for the corresponding video frame. Furthermore, the start and/or end of the transmission 404 may or may not be aligned with the server VSYNC signal 311. As shown, compressed encoded slice A is transmitted to the client independently of the other compressed encoder slices for the rendered video frame. The encoder slices may be transmitted one at a time or in parallel.

At 405, the client receives the compressed video frames, again for each encoded slice. Additionally, the start and/or end of reception 405 may or may not be aligned with the client VSYNC signal 312. As shown, compressed encoded slice A is received by the client. Transmission jitter 452 may exist between the server 260 and the client 210, where jitter 452 measures the variation in network latency from the server 260 to the client 210.
A lower jitter value indicates a more stable connection. As shown, the thick straight arrow indicates the current latency in transmitting the corresponding video frame to the client 210, but due to jitter there may be a range of arrival times for the video frames at the client 210 (e.g., the range enclosed by the dotted arrow). The variation in latency may also be due to one or more operations at the server, such as encoding 403 and transmitting 404, as well as network issues that introduce latency in transmitting the video frames to the client 210.

At 406, the client again decodes the compressed video frame for each encoded slice to generate a decoded slice A (shown without hash marks) that is now ready for display. In one embodiment, the decoding process 406 begins before the receiving process 405 is fully completed for the corresponding video frame. Furthermore, the start and/or end of the decoding 406 may or may not be aligned with the client VSYNC signal 312. At 407, the client displays the decoded rendered video frame on a display at the client. That is, the decoded video frame is placed in a display buffer that is streamed out to a display device, e.g., scanline by scanline.
In one embodiment, the display process 407 (i.e., streaming out to a display device) begins after the decode process 406 is fully completed for the corresponding video frame, i.e., after the decoded video frame is fully resident in the display buffer.
In another embodiment, the display process 407 starts before the decode process 406 is fully completed for the corresponding video frame. That is, the stream out to the display device starts at an address in the display buffer at a time when only a portion of the decoded frame buffer resides in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, such that the display buffer updates are performed before those portions are streamed out to the display. Additionally, the start and/or end of display 407 is aligned with the client VSYNC signal 312.

In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time between when scanout 402 begins and when display 407 begins. Embodiments of the present disclosure may align (e.g., synchronize frequency and adjust offset) the VSYNC signals between the server and client to reduce the one-way latency between the server and client and reduce the variability of the one-way latency between the server and client.
For example, embodiments of the present disclosure can calculate an optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312, so that even with near worst-case times required for server processing such as encoding 403 and transmitting 404, near worst-case network latency between the server 260 and the client 210, and near worst-case client processing such as receiving 405 and decoding 406, the decoded and rendered video frames are available in time for the display process 407. In other words, it is not necessary to determine the absolute offset between the server VSYNC and the client VSYNC, it is sufficient to adjust the offset so that the decoded and rendered video frames are available in time for the display process.

In particular, the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 may be adjusted by synchronization. The synchronization may be achieved by adjusting the server VSYNC signal 311 or the client VSYNC signal 312. For illustrative purposes, the adjustment is described with respect to the server VSYNC signal 311, but it is understood that the adjustment may be performed on the client VSYNC signal 312 instead. For example, as shown in FIG. 4, the server frame period 410 (e.g., the time between two occurrences 311c and 311d of the server VSYNC signal 311) is substantially equal to the client frame period 415 (e.g., the time between two occurrences 312a and 312b of the client VSYNC signal 312), which indicates that the frequencies of the server VSYNC signal 311 and the client VSYNC signal 312 are also substantially equal.

To maintain synchronization of the frequencies of the server and client VSYNC signals, the timing of the server VSYNC signal 311 may be manipulated. For example, the vertical blanking interval (VBI) of the server VSYNC signal 311 may be increased or decreased over a period of time, for example, to check for drift between the server VSYNC signal 311 and the client VSYNC signal 312. Manipulation of the vertical blanking interval (VBLANK) in the VBI provides for adjusting the number of scan lines used for VBLANK for one or more frame periods of the server VSYNC signal 311. A decrease in the number of scan lines of VBLANK shortens the corresponding frame period (e.g., time interval) between two occurrences of the server VSYNC signal 311. Conversely, an increase in the number of scan lines of VBLANK lengthens the corresponding frame period (e.g., time interval) between two occurrences of the VSYNC signal 311. In that way, the frequency of the server VSYNC signal 311 is adjusted so that the frequencies between the client and server VSYNC signals 311 and 312 are substantially the same frequency. Also, the offset between the server and client VSYNC signals can be adjusted by increasing or decreasing the VBI for a short period of time before returning it to its original value.
In one embodiment, the server VBI is adjusted. In another embodiment, the client VBI is adjusted. In yet another embodiment, instead of two devices (server and client), there may be multiple connected devices, each of which may have a corresponding VBI that is adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple devices may include one or more server devices and/or one or more client devices arranged in one or more server/client architectures, a multi-tenant server/client(s) architecture, or some combination thereof.

Alternatively, the server's pixel clock (e.g., located in the southbridge of the server's northbridge/southbridge core logic chipset, or in the case of a discrete GPU, generating the pixel clock itself using its own hardware) can, in one embodiment, be manipulated to perform coarse and/or fine adjustments to the frequency of the server VSYNC signal 311 over a period of time to bring the frequency synchronization between the server VSYNC signal 311 and the client's VSYNC signal 312 back into alignment.
Specifically, the pixel clock of the server southbridge may be overclocked or underclocked to adjust the overall frequency of the server VSYNC signal 311. In that way, the frequency of the server VSYNC signal 311 is adjusted so that the frequencies between the client and server VSYNC signals 311 and 312 are substantially the same frequency. The offset between the server and client VSYNC can be adjusted by briefly increasing or decreasing the client server pixel clock before restoring the pixel clock back to its original value. In one embodiment, the server pixel clock is adjusted.
In another embodiment, the client pixel clocks are aligned. In yet another embodiment, instead of two devices (a server and a client), there may be multiple connected devices, each of which may have a corresponding pixel clock that is aligned. In one embodiment, each of the multiple connected devices may be an independent peer device (e.g., without a server device). In another embodiment, the multiple connected devices may include one or more server devices and one or more client devices arranged in one or more server/client architectures, a multi-tenant server/client(s) architecture, or some combination thereof.

In one embodiment, high performance codecs (e.g., encoders and/or decoders) may be used to further reduce one-way latency between the cloud gaming server and the client. In traditional streaming systems involving streaming of compressed media (e.g., streaming movies, television shows, videos, etc.), when the streaming media is decompressed at the end target (e.g., client), a significant amount of the decompressed video may be buffered at the client to deal with variability in the encoding operation (e.g., longer encoding times), jitter that invades the transmission quality, and in the decoding operation (e.g., longer decoding times). Thus, traditional streaming systems may rely on average decoding capabilities and metrics (e.g., average decoding resources) as the decoded content deals with latency variability, allowing video frames to be displayed at the desired speed (e.g., supporting 4K media at 60 Hz or displaying video frames every time a client VSYNC signal occurs).

However, in cloud gaming environments, buffering is highly limited (e.g., moving to zero buffering) to allow real-time gaming to be realized. As a result, any variability introduced into the one-way latency between the cloud gaming server and the client can negatively impact downstream operations. For example, longer encoding and decoding of complex frames will result in correspondingly longer one-way latency (even for a single frame), ultimately resulting in longer response times to the user and negatively impacting the user's real-time experience.

In one embodiment, for cloud gaming, it is beneficial to provide more powerful decoding and encoding resources that may be unnecessary when compared to the needs of streaming video applications. Furthermore, as described in more detail below, the encoder resources need to be optimized for the time to process frames that are long or require the longest processing. That is, in an embodiment where the encoder can be adjusted to improve the tradeoff between one-way latency and video quality in the cloud gaming system, the encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and/or number of video frames that exceed a target frame size, and the adjusted parameters may include the encoder bitrate, target frame size, maximum frame size, and quantization parameter (QP) value, where high performance encoders and decoders help reduce the overall one-way latency between the cloud gaming server and the client.

With detailed descriptions of the various client devices 210 and/or cloud gaming network 290 (e.g., within game server 260) of FIGS. 2A-2D, flow diagram 500 of FIG. 5 illustrates a method of cloud gaming according to one embodiment of the present disclosure, where encoding video frames includes adjusting encoder parameters with awareness of network transmission speed and reliability, as well as overall latency goals. The cloud gaming server is configured to stream content to one or more client devices over a network. This process results in smoother frame rates and more reliable latency, and one-way latency between the cloud gaming server and the client is reduced and made more consistent, thereby improving the smoothness of the client's display of the video.

At 510, multiple video frames are generated when executing a video game on a cloud gaming server. Typically, the cloud gaming server generates rendered video frames for multiple games. For example, the game logic of the video game is built based on a game engine or a game title processing engine. The game engine includes core functions that can be used by the game logic to build a game environment of the video game. For example, some functions of the game engine may include a physics engine for simulating physical forces and collisions for objects in the game environment, a rendering engine for 2D or 3D graphics, collision detection, sound, animation, artificial intelligence, networking, streaming, etc. In that way, the game logic does not need to build from scratch the core functions provided by the game engine.

The game logic in combination with the game engine is executed by the CPU and GPU, which may be configured in an Accelerated Processing Unit (APU), such that the CPU and GPU, along with shared memory, may be configured as a rendering pipeline for generating rendered video frames for the game, such that the rendering pipeline outputs the rendered images for the game, including corresponding color information for each pixel of a targeted and/or virtualized display, as video or image frames suitable for display.
In particular, the CPU may be configured to generate one or more draw calls for a video frame, with each draw call including a command stored in a corresponding command buffer that is executed by the GPU in the GPU pipeline. In general, the graphics pipeline may perform shader operations on vertices of objects in a scene to generate texture values for pixels of a display. In particular, the graphics pipeline receives input geometry (e.g., vertices of objects in a gaming environment), and vertex shaders construct the primitives or polygons that make up the objects. The vertex shader programs may perform lighting, shading, shadowing, and other operations on the primitives.
Depth or Z-buffering is performed to determine which objects are visible when rendered from the corresponding viewpoint. Rasterization is performed to project objects in the 3D game environment onto the 2D plane defined by the viewpoint. Pixel-sized fragments are generated for the objects, and one or more fragments may contribute to the color of a pixel in the image. The fragments may be merged and/or blended to determine the combined color of each pixel of the corresponding video and may be stored in a frame buffer. Subsequent video frames are generated and/or rendered for display using a similarly configured command buffer, and multiple video frames are output from the GPU pipeline.

At 520, the method includes encoding the video frames at an encoder bit rate. In particular, the video frames are scanned into an encoder where they are compressed before streaming to the client using a streamer operating at the application layer. In one embodiment, each of the rendered video frames for the game can be composited and blended with additional user interface features into a corresponding modified video frame and then scanned into an encoder, which compresses and streams the modified video frames to the client.
For brevity and clarity, the method of adjusting encoder parameters disclosed in FIG. 5 is described with respect to encoding a plurality of video frames, but it is understood that it supports encoding modified video frames. The encoder is configured to compress a plurality of video frames based on the described format. For example, when streaming media content from the cloud gaming server to the client, the Motion Picture Experts Group (MPEG) or H.264 standard may be implemented. In particular, the encoder may perform compression by video frame or by encoder slicing of video frames, and as mentioned above, each video frame may be compressed as one or more encoded slices. In general, when streaming media, video frames are compressed as I frames (intra frames) or P frames (predicted frames), each of which may be divided into encoded slices.

At 530, the client's maximum receive bandwidth is measured. In one embodiment, the maximum bandwidth experienced by the client is determined by a feedback mechanism from the client. Figure 6 illustrates measuring the bandwidth of a client 210 by a streamer of a cloud gaming server, according to one embodiment of the disclosure, where the streamer 620 is configured to monitor and adjust the encoder 610 such that compressed video frames can be transmitted at a rate within the client's measured bandwidth.
As shown, compressed video frames, encoded slices, and/or packets are delivered from an encoder 610 to a buffer 630 (e.g., first in, first out - FIFO). The encoder delivers the compressed video frames at an encoder fill rate 615. For example, the buffer may be filled as fast as the encoder can generate compressed video frames, encoded slices 650, and/or packets of encoded slices 655. Additionally, the compressed video frames are drained from the buffer at a buffer drain rate 635 for delivery over network 250 to client 210.
In one embodiment, the buffer drain rate 635 is dynamically adjusted to the client's measured maximum receive bandwidth. For example, the buffer drain rate 635 may be adjusted to be approximately equal to the client's measured maximum receive bandwidth. In one embodiment, the encoding of packets is performed at the same rate that they are transmitted, with both operations dynamically adjusted to the maximum available bandwidth available to the client.

In particular, the streamer 620 operating at the application layer measures the maximum bandwidth of the client 210, such as by using a bandwidth tester 625. The application layer is used in the User Datagram Protocol/Internet Protocol (UDP/IP) suite of protocols used to interconnect network devices over the Internet. For example, the application layer specifies the communication protocols and interface methods used to communicate between devices over an IP communication network. During testing, the streamer 620 provides additional buffered packets 640 (e.g., forward error correction (FEC) packets) to allow the buffer 630 to stream packets from a predefined bit rate, such as the maximum bandwidth tested.
In one embodiment, the client returns as feedback 690 to the streamer 620 the number of packets received over a range of incremental sequence identifiers (IDs), such as a range of video frames. For example, the client may report something like 145 of 150 video frames received at sequence IDs 100-250 (e.g., 150 video frames). In this way, the streamer 620 at the server 260 can calculate packet loss, and because the streamer 620 knows the amount of bandwidth that was transmitted (e.g., tested) during that sequence of packets, the streamer 620 can dynamically determine what the client's maximum bandwidth is at a particular point in time.
The measured maximum bandwidth of the client may be delivered as control information 627 from the streamer 620 to the buffer 630 such that the buffer 630 can dynamically transmit packets at a rate approximately equal to the client's maximum bandwidth. In this manner, the transmission rate of compressed video frames, encoded slices, and/or packets may be dynamically adjusted depending on the currently measured maximum bandwidth of the client.

At 540, the encoding process is monitored by the streamer, i.e., the encoding of multiple video frames is monitored. In one embodiment, the monitoring is performed by the client 210 and feedback and/or adjustment control signals are provided back to the encoder. In another embodiment, the monitoring is performed by the streamer 620, etc., at the cloud gaming server 260. For example, monitoring of the encoding of the video frames may be performed by a monitoring and adjustment unit 629 of the streamer 620. Various encoding characteristics and/or operations can be tracked and/or monitored.
For example, in one embodiment, the rate of occurrence of I-frames within a plurality of video frames may be tracked and/or monitored. Additionally, in one embodiment, the rate of occurrence of scene changes within a plurality of video frames may be tracked and/or monitored. Also, in one embodiment, the number of video frames that exceed a target frame size may be tracked and/or monitored. Also, in one embodiment, the encoder bit rate used to encode one or more video frames may be tracked and/or monitored.

At 550, the encoder parameters are dynamically adjusted based on monitoring of the encoding of the video frames. That is, monitoring of the encoding of the video frames affects how the encoder operates when compressing current and future video frames received at the encoder. In particular, the monitoring and adjustment unit 629 is configured to determine which encoder parameters to adjust in response to monitoring the encoding of the video frames and analysis performed on the monitored information. Control signals 621 are transmitted back to the encoder 610 from the monitoring and adjustment unit 629 that are used to configure the encoder. Encoder parameters for adjustment include quantization parameters (QPs) (e.g., minimum QP, maximum QP) or quality parameters, target frame size, maximum frame size, etc.

The adjustments are made with awareness of the network transmission speed and reliability, as well as the overall latency goal. In one embodiment, smoothness of video playback is prioritized over low latency or image quality. For example, skipping the encoding of one or more video frames is disabled. Specifically, the balance between image resolution or image quality (e.g., 60 Hz) and latency is adjusted using various encoder parameters. In particular, the VSYNC signals at the cloud gaming server and the client can be synchronized and offset, so that one-way latency between the cloud gaming server and the client can be reduced, thereby reducing the need to skip video frames to promote low latency. Synchronization and offsetting of the VSYNC signals also provides overlapping operations at the cloud gaming server (scanout, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and/or overlapping operations at the cloud gaming server and the client, all of which facilitate reduced one-way latency, reduced variability in one-way latency, real-time generation and display of video content, and consistent video playback at the client.

In one embodiment, the encoder bitrate is monitored taking into account the upcoming frames and their complexity (e.g., predicted scene changes) to predict the demand for client bandwidth, where the encoder bitrate can be adjusted according to the predicted demand. For example, when prioritizing smoothness of video playback, the encoder monitoring and adjustment unit 629 can be configured to determine that the encoder bitrate used exceeds the measured maximum reception bandwidth. In response, the encoder bitrate can be reduced and the frame sizing can also be reduced.
If smoothness is a priority, it is desirable to use an encoder bit rate lower than the maximum receiving bandwidth (e.g., an encoder bit rate of 10 Mbits/sec for a maximum receiving bandwidth of 15 Mbits/sec). That way, even if the encoded frames spike above the maximum frame size, they can still be transmitted within 60 Hz (Hertz). In particular, the encoder bit rate can be translated into a frame size. A given bit rate and target speed of a video game (e.g., 60 frames per second) is translated into an average size of an encoded video frame. For example, with an encoder bit rate of 15 Mbits/sec and a given target speed of 60 frames/sec, 60 encoded frames share 15 Mbits, such that each encoded frame has approximately 250k encoded bits.
In this way, controlling the encoder bit rate also controls the frame size of the encoded video frames, so that increasing the encoder bit rate results in more bits for encoding (more precision) and decreasing the encoder bit rate results in fewer bits for encoding (less precision). Similarly, when the encoder bit rate used to encode a group of video frames is within the maximum received bandwidth being measured, the encoder bit rate can be increased and the frame size can also be increased.

In one embodiment, when smoothness of video playback is a priority, the encoder monitoring and adjustment unit 629 may be configured to determine whether the encoder bitrate used to encode a group of video frames from a plurality of video frames exceeds the measured maximum receive bandwidth. For example, the encoder bitrate may be detected to be 15 Megabits per second (Mbps), while the maximum receive bandwidth may currently be 10 Mbps. In this way, the encoder pushes out more bits than can be sent to the client without increasing one-way latency.
As mentioned above, if smoothness is a priority, it may be desirable to use an encoder bitrate lower than the maximum receiving bandwidth. In the above example, it may be acceptable to have the encoder bitrate set at 10 Mbit/s or less for the 10 Mbit/s maximum receiving bandwidth introduced above. That way, if the encoded frames spike above the maximum frame size, they can still be transmitted within 60 Hz. The QP value can be adjusted accordingly, with or without a reduction in the encoder bitrate, with the QP controlling the precision used when compressing the video frames.
That is, the QP controls the amount of quantization that is performed (e.g., compressing a variable range of values in a video frame to a single quantum value). In H.264, the QP ranges from "0" to "51". For example, a QP value of "0" means less quantization, less compression, more precision, and higher quality. For example, a QP value of "51" means less quantization, less compression, more precision, and higher quality. Specifically, the QP value can be increased such that the encoding of the video frame is performed with less precision.

In one embodiment, when prioritizing smoothness of video playback, the encoder monitoring by the monitoring and adjusting unit 629 may be configured to determine that the encoder bit rate used to encode a group of video frames from the plurality of video frames is within the maximum reception bandwidth. As previously introduced, when prioritizing smoothness, it may be desirable to use an encoder bit rate that is lower than the maximum reception bandwidth. In this manner, there is excess bandwidth available when transmitting the group of video frames. The excess bandwidth may be determined. Accordingly, a QP value may be adjusted, where the QP controls the precision used when compressing the video frames. In particular, the QP value may be reduced based on the excess bandwidth, such that the encoding is performed more precisely.

In another embodiment, the characteristics of an individual video game are taken into account when determining I-frame processing and QP settings, especially when prioritizing smoothness of video playback. For example, if "scene changes" in a video game are infrequent (e.g., only camera cuts), it may be desirable to have larger I-frames (lower QP or higher encoder bitrate). That is, the number of video frames identified as having a scene change within a group of video frames from the multiple video frames being compressed is determined to be less than a threshold number of scene changes. That is, the streaming system can handle the number of scene changes for the current conditions (e.g., measured client bandwidth, required latency, etc.). The QP value can be adjusted accordingly, where the QP controls the precision used when compressing the video frames. In particular, the QP value can be reduced so that the encoding is performed with greater precision.

On the other hand, if a video game experiences frequent "scene changes" during gameplay, it may be desirable to keep the I-frame size small (e.g., higher QP or lower encoder bitrate). That is, the number of video frames identified as having scene changes within a group of video frames from the plurality of video frames being compressed is determined to meet or exceed a threshold number of scene changes. That is, the video game is generating too many scene changes for the current conditions (e.g., measured client bandwidth, required latency, etc.). In response, the QP value can be adjusted, where the QP controls the precision used when compressing the video frames. In particular, the QP value can be increased so that encoding is performed with less precision.

In another embodiment, encoding patterns may be taken into account when determining I-frame processing and QP settings, especially when prioritizing smoothness of video playback. For example, if the encoder generates I-frames less frequently, it may be desirable to have larger I-frames (lower QP or higher encoder bitrate). That is, the number of video frames compressed as I-frames within a group of video frames from multiple video frames being compressed is within or below a threshold number of I-frames. That is, the streaming system can handle the number of I-frames for the current conditions (e.g., measured client bandwidth, required latency, etc.). The QP value may be adjusted accordingly, where the QP controls the precision used when compressing the video frames. In particular, the QP value may be reduced so that the encoding is performed with higher precision.

If the encoder generates I-frames frequently, it may be desirable to keep the I-frame size small (e.g., higher QP or lower encoder bitrate). That is, within a group of video frames from multiple video frames being compressed, the number of video frames compressed as I-frames is within or exceeds a threshold number of I-frames. That is, the video game is generating too many I-frames for the current conditions (e.g., measured client bandwidth, required latency, etc.). The QP value can be adjusted accordingly, where the QP controls the precision used when compressing the video frames. In particular, the QP value can be increased so that the encoding is performed with less precision.

In another embodiment, encoding patterns may be taken into consideration when spinning the encoder, especially when smoothness of video playback is a priority. For example, if the encoder frequently falls short of a target frame size, it may be desirable to make the target frame size larger. That is, it is determined that within a group of video frames from a plurality of video frames to be compressed and transmitted at the transmission rate, a number of video frames falls below a threshold. Each of the number of video frames falls within the target frame size (i.e., is equal to or smaller than the target frame size). At least one of the target frame size and the maximum frame size is increased accordingly.

On the other hand, if the encoder frequently exceeds the target frame size, it may be desirable to reduce the target frame size. That is, it is determined whether within a group of video frames from a plurality of video frames to be compressed and transmitted at the transmission rate, a number of video frames meets or exceeds a threshold value. Each of the number of video frames exceeds the target frame size. At least one of the target frame size and the maximum frame size is reduced accordingly.

7A illustrates setting the quantization parameter (QP) of an encoder to optimize quality and buffer utilization at a client, according to one embodiment of the present disclosure. Graph 720A illustrates the vertical frame size (in bytes) for each generated frame, shown horizontally. The target frame size and maximum frame size are static. In particular, line 711 illustrates the maximum frame size, and line 712 illustrates the target frame size, which is larger than the target frame size. As shown in graph 720A, there are multiple peaks that include compressed video frames that exceed the target frame size of line 712. Video frames that exceed the target frame size may take multiple frame periods to encode and/or transmit from the cloud gaming server, thus risking playback jitter (e.g., increased one-way latency).

Graph 700B shows the encoder response after the QP is set based on the target frame size, maximum frame size, and QP range (e.g., minimum QP and maximum QP) to optimize encoding quality and buffer utilization at the client. For example, the QP may be adjusted and/or tuned based on encoder monitoring of encoder bitrate, frequency of scene changes, and frequency of I-frame generation, as discussed above. Graph 700B shows the frame size (in bytes) vertically for each frame generated, as shown horizontally.
The target frame size at line 712 and the maximum frame size at line 711 remain in the same position as they were in graph 700A. After the QP adjustment and/or tuning, the number of peaks containing compressed video frames exceeding the target frame size at line 712 is reduced as compared to graph 700A. That is, the QP has been adjusted to optimize the encoding of video frames (i.e., to fit within the target frame size) for the current conditions (e.g., measured client bandwidth, required latency, etc.).

FIG. 7B illustrates the adjustment of target frame size, maximum frame size, and/or QP (e.g., minimum QP and/or maximum QP) encoder settings to reduce the occurrence of I-frames that exceed the true target frame size supported by the client, according to one embodiment of the present disclosure. For example, the QP may be adjusted and/or tuned based on encoder monitoring of the encoder bitrate, frequency of scene changes, and frequency of I-frame generation, as described above.

Graph 720A shows the vertical frame size (in bytes) for each generated frame, as shown horizontally. For illustrative purposes, 720A of FIG. 7B and graph 700A of FIG. 7A may reflect similar encoder conditions and are used for encoder tuning. In graph 720A, the target frame size and maximum frame size are static. In particular, line 711 shows the maximum frame size and line 712 shows the target frame size, where the maximum frame size is larger than the target frame size.
As shown in graph 720A, there are multiple peaks that include compressed video frames that exceed the target frame size at line 712. Video frames that exceed the target frame size may take multiple frame periods to encode and/or transmit from the cloud gaming server, and therefore risk introducing playback jitter (e.g., increased one-way latency). For example, the peaks that reach the maximum frame size at line 711 may be I-frames that take 16 milliseconds or more to transmit to the client, which creates playback jitter by increasing the one-way latency between the cloud gaming server and the client.

Graph 720B shows the encoder response after at least one of the target frame size and/or maximum frame size has been adjusted to reduce the occurrence of I-frames that exceed the true target frame size supported by the client. The true target frame size may be adjusted based on measured client bandwidth and/or encoder monitoring, including monitoring the encoder bit rate, frequency of scene changes, and frequency of I-frame generation, as described above.

Graph 720B shows the vertical frame size (in bytes) for each generated frame, as shown horizontally. In comparison to graph 720A, the target frame size for line 712' and the maximum frame size for line 711' have lower values. For example, the target frame size for line 712' is a smaller value than that for line 712, and the maximum frame size for line 711' is a smaller value than that for line 711.
After adjusting the target frame size and/or maximum frame size, the maximum size of the peaks of compressed video frames exceeding the target frame size 712' has been reduced for better transmission. Additionally, the number of peaks containing compressed video frames exceeding the target frame size 712' has also been reduced when compared to graph 700A. For example, only one peak is shown in graph 720B. That is, the target frame size and/or maximum frame size has been adjusted to optimize the encoding of video frames (i.e., to fit within the target frame size) for the current conditions (e.g., measured client bandwidth, required latency, etc.).

In conjunction with detailed descriptions of the various client devices 210 and/or cloud gaming network 290 (e.g., within game server 260) of FIGS. 2A-2D, flow diagram 800 of FIG. 8 illustrates a method of cloud gaming according to one embodiment of the disclosure, where encoding a video frame includes determining when to skip a video frame or when to delay the encoding and transmission of a video frame if the encoding is long or if the resulting video frame is large (e.g., when encoding an I-frame).
In particular, the decision to skip video frames is made taking into account network transmission speed and reliability, and overall latency targets. This process results in smoother frame rates and more reliable latency, making one-way latency between the cloud gaming server and the client reduced and more consistent, thereby improving the smoothness of the client's display of video.

At 810, a plurality of video frames are generated when executing a video game at a cloud gaming server operating in streaming mode. Typically, a cloud gaming server generates rendered video frames for a plurality of games. For example, the generation of rendered video frames for a game is described at 510 in FIG. 5 and is applicable to the generation of the video frames in FIG. 8.
For example, game logic for a video game is built on a game engine or game title processing engine. The game logic in combination with the game engine is executed by a CPU and a GPU, which together with a shared memory may be configured as a rendering pipeline for generating rendered video frames for the game, such that the rendering pipeline outputs the rendered images for the game as video or image frames suitable for display, including color information corresponding to each of the pixels of a target and/or virtualized display.

At 820, a scene change is predicted for a first video frame of the video game, the scene change being predicted before the first video frame is generated. In one embodiment, the game logic can cause the CPU to recognize a scene change while the video game is being executed. For example, the game logic or add-on logic can include code (e.g., scene change logic) that predicts a scene change when generating video frames, such as predicting that a range of video frames will include at least one scene change or predicting that a particular video frame is a scene change.
In particular, game logic or add-on logic configured for scene change prediction analyzes game state data collected during the execution of a video game to determine and/or anticipate and/or predict when there will be a scene change, such as within the next X number of frames (e.g., a range), or for an identified video frame, etc. For example, a scene change can be predicted when a character moves from one scene to another in a virtualized game environment, or when a character exits a level and transitions to another level in a video game, transition between two video frames (e.g., a scene cut in a cinematic sequence, or the start of interactive gameplay after a series of menus), and the scene change may be represented by a video frame that includes a large and complex scene of the virtualized game world or environment.

Game state data may define the state of the game at that time and may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, a character's location within the game world of the player's gameplay, a gameplay scene or game environment, a game application level, a character's assets (e.g., weapons, tools, bombs, etc.), loadouts, a character's skill set, a game level, character attributes, a character's location, number of lives remaining, total number of lives available, armor, trophies, time counter values, and other asset information, etc.

At 830, a scene change hint is generated and sent to the encoder, the hint indicating that the first video frame is a scene change. In this manner, notification of the upcoming scene change may be provided to the encoder, which may adjust its encoding operations when compressing the identified video frame. The notification provided as a scene change hint may be distributed via an API used to communicate between components or between applications running on components of cloud gaming server 260.
In one embodiment, the API may be a GPU API, for example, the API may be running on or called by game logic and/or add-on logic configured to detect scene changes to communicate with the encoder.
In one embodiment, the scene change hints may be provided as data control packets formatted such that all components receiving the data control packets can understand what type of information is contained in the data control packet and can understand the appropriate reference to the corresponding rendered video frame.
In one embodiment, the communication protocol used for the API, the format for the data control packets may be defined in a corresponding software development kit (SDK) for the video game.

At 840, the first video frame is delivered to an encoder. As previously described, the game-generated video frame may be composited and blended with additional user interface features resulting in a modified video frame that is scanned into the encoder. The encoder is configured to compress the first video frame based on a desired format, such as the MPEG or H.264 standards used for streaming media content from the cloud gaming server to the client.
When streaming, a video frame is encoded as a P-frame, and the next video frame is then encoded as another I-frame, until there is a scene change or the currently encoded frame cannot reference a key frame (e.g., a previous I-frame), in which case the first video frame is encoded as an I-frame based on a scene change hint, and the I-frame may be encoded without referencing any other video frame (e.g., standalone as a key image).

At 850, the maximum receive bandwidth of the client is measured. As previously described, the maximum bandwidth experienced by the client may be determined by means of a feedback mechanism from the client, as shown in operation 530 of FIGS. 5 and 6. In particular, a streamer of the cloud gaming server may be configured to measure the bandwidth of the client.

At 860, the encoder receives a second video frame, i.e., the second video frame is received after a scene change and compressed after the first video frame is compressed, and a decision is made by the encoder to either not encode the second video frame (or subsequent video frames) or to delay encoding the second video frame (or subsequent video frames). This decision is made based on the maximum receiving bandwidth of the client and the target resolution of the client display. That is, the decision to skip or delay encoding takes into account the bandwidth available to the client.
In general, if the current bandwidth experienced by the client is sufficient, a video frame generated and encoded for the client's target display can quickly return to low one-way latency after incurring a latency hit (e.g., generating a large I-frame for a scene change), and the second video frame (and/or subsequent video frames) may still be encoded with a delay.
On the other hand, if the current bandwidth experienced by the client is not sufficient, the second video frame (and/or subsequent video frames) may be skipped during the encoding process and not delivered to the client. In this way, it is possible to have fewer skipped frames (and lower latency) when the bandwidth to the client exceeds the bandwidth required to support the target resolution of the display at the client.

In one embodiment, compressed video frames are transmitted from the server to the client at a rate based on the maximum bit rate or bandwidth available over the network at a particular time. In this way, the transmission rate of encoded slices of compressed video frames and/or packets of encoded slices is dynamically adjusted according to the currently measured maximum bandwidth. Video frames can be transmitted as they are encoded, such that transmission occurs as soon as encoding is complete, without waiting for the next occurrence of a server VSYNC signal and without waiting for the entire video frame to be encoded.

Furthermore, in one embodiment, the encoding of packets is performed at the same rate as they are transmitted, with both operations dynamically adjusting to the maximum available bandwidth available to the client. Also, the encoder bitrate can be monitored taking into account the next frame and its complexity (e.g., predicted scene changes) to predict client bandwidth demand, and the encoder bitrate can be adjusted according to the predicted demand. Furthermore, the encoder bitrate can be communicated to the client, so that the client can adjust its decoding speed accordingly to match the encoder bitrate.

In one embodiment, the second video frame is skipped by the encoder when the transmission rate to the client is low relative to the target resolution of the client display. That is, the second video frame is not encoded. In particular, the transmission rate to the client for a group of compressed video frames exceeds the maximum receive bandwidth. For example, the transmission rate to the client may be 15 megabytes per second (Mbps), but the measured receive bandwidth of the client may currently be 5-10 Mbps. Thus, if every video frame is continually pushed to the client, the one-way latency between the cloud gaming server and the client increases. To promote low latency, the second and subsequent video frames may be skipped by the encoder.

9A illustrates a sequence 900A of video frames being compressed by an encoder that drops the encoding of a second video frame 920 after encoding a first I-frame 905 when the client bandwidth is low relative to the target resolution of the client's display, according to one embodiment of the present disclosure. The encoding and transmission blocks of video frames are shown in relation to a VSYNC signal 950. In particular, when extra bandwidth is not available, an I-frame that takes longer to encode causes one or more skipped frames to keep one-way latency low, which may include the time to display a video frame at the client.
As shown, skipping one or more video frames after the I-frame allows for a quick return to low one-way latency (e.g., within one or two frame periods), whereas not skipping the encoding of video frames would otherwise require several frame periods to return to low one-way latency.

For example, a sequence of video frames 900A includes one encoded I-frame 905, with the remaining frames encoded as P-frames. For purposes of illustration, blocks 901 and 902 are encoded as P-frames before block 905, which is encoded as an I-frame.
The encoder then compresses the video frames as P frames until the next scene change or until the video frame cannot reference a previous key frame (such as an I frame). In general, the encoding time for an I frame block may take longer than a P frame block. For example, the encoding time for an I frame block 905 may exceed one frame period. In some cases, the encoding time between a P frame and an I frame may generally be approximately the same, especially when using a high-powered encoder.

However, the transmission times between I and P frames are significantly different. As shown, the various transmission times are shown relative to the corresponding encoded video frames. For example, the transmission block 911 of the encoded P frame block 901 is shown with low latency, such that the encoding block 901 and the transmission block 911 can be performed within one frame period. Also, the transmission block 912 of the encoded P frame block 902 is shown with low one-way latency, such that the encoding block 902 and the transmission block 912 can also be performed within one frame period.

On the other hand, the transmission block 915A of the encoded I-frame block 905 is shown with a higher one-way latency, such that the encoding block 905 and the transmission block 915A occur over several frame periods, thereby introducing jitter into the one-way latency between the cloud gaming server and the client. To provide a real-time experience to the user with less one-way latency, a buffer at the client may not be used to correct for the jitter. In that case, the encoder may decide to skip encoding one or more video frames after the I-frame is encoded.
For example, video frame 920 is dropped by the encoder, in which case the transmission of the encoded video frames reverts to one of low one-way latency around highlighted region 910 as after five subsequent video frames have been encoded as P-frames and transmitted to the client, i.e., the fourth or fifth P-frame encoded after I-frame block 905 is encoded will also be transmitted to the client within the same frame period, thereby returning to low one-way latency between the cloud gaming server and the client.

In one embodiment, if the transmission rate to the client is high relative to the target resolution of the client display, the second video frame is still compressed by the encoder after the delay (i.e., waiting until the I-frame is encoded). In particular, the transmission rate to the client of the group of compressed video frames is within the maximum receive bandwidth. For example, the transmission rate to the client may be 13 megabytes per second (Mbps), but the client's measured receive bandwidth may currently be 15 Mbps. In this way, there is no increase in one-way latency between the cloud gaming server and the client, since there is no delay in receiving the encoded video frames at the client.

Furthermore, because the VSYNC signals at the cloud gaming server and client can be synchronized and offset, one-way latency between the cloud gaming server and client can be reduced, thereby compensating for latency variability caused by jitter at the server or at the client during transmission over the network.
Additionally, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and/or overlapping operations at the cloud gaming server and client, all of which facilitate compensation for latency variability introduced by server or network or client jitter, reduction in one-way latency, reduction in one-way latency variability, real-time generation and display of video content, and consistent video playback at the client.

FIG. 9B shows a sequence 900B of video frames being compressed by an encoder that takes into account the bandwidth available to the client, such that, according to one embodiment of the present disclosure, it is possible to have lower latency while at the same time having no or fewer skipped frames when the bandwidth exceeds the bandwidth required to support the target resolution of the client display.
In particular, in sequence 900B, a video frame is encoded as an I-frame, and subsequent video frames are also encoded normally, but after a delay in encoding the I-frame, when the client bandwidth is moderate to moderate for the target resolution of the client display. Because there is moderate bandwidth availability, a moderate amount of excess bandwidth is available to compensate for latency variability (e.g., jitter) between the cloud gaming server and the client, such that frame skipping can be avoided and a return to low one-way latency can be achieved relatively quickly (e.g., within two to four frame periods). The encoding and transmission blocks of the video frames are shown in relation to the VSYNC signal 950.

The sequence of video frames 900B includes one encoded I-frame 905, with the remaining frames being encoded as P-frames. For illustration purposes, encode block 901 and encode block 902 as P-frames before encode block 905 is encoded as an I-frame. The encoder then compresses the video frames as P-frames until the next scene change or until the video frames cannot reference a previous key frame (such as an I-frame). In general, the encoding time of an I-frame block may take longer than a P-frame block, and the transmission of an I-frame may take longer than one frame period. For example, the encoding time and transmission time for I-frame block 905 exceeds one frame period.
Also shown are various transmission times relative to the corresponding encoded video frames. For example, the encoding and transmission of the video frame prior to I-frame block 905 is shown with a low one-way latency such that the corresponding encoding and transmission blocks can be performed within one frame period. However, transmission block 915B of encoded I-frame block 905 is shown with a higher one-way latency such that encoding block 905 and transmission block 915B occur over more than one frame period, thereby introducing jitter into the one-way latency between the cloud gaming server and the client.
As mentioned above, the encoding time can be further reduced by adjusting one or more encoder parameters (e.g., QP, target frame size, maximum frame size, encoder bit rate, etc.) That is, the second or subsequent video frames after an I-frame are encoded with less precision when the transmission rate to the client is medium relative to the target resolution of the client display, and are encoded with less precision when the transmission rate is high relative to the target resolution.

After the I-frame block 905, the encoder continues compressing the video frames, although they may be temporarily delayed due to the encoding of the I-frame. Again, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and/or overlapping operations at the cloud gaming server and client, all of which facilitate compensation for variability in one-way latency introduced by server or network or client jitter, reduction in one-way latency, reduction in variability in one-way latency, real-time generation and display of video content, and consistent video playback at the client.

Because the client bandwidth is moderate with respect to the target resolution of the client display, the transmission of the encoded video frame returns to one of low one-way latency around highlighted region 940, such as after two or three subsequent video frames have been encoded as P-frames and transmitted to the client. Within region 940, P-frames encoded after I-frame block 905 is encoded are also transmitted to the client within the same frame period, thereby returning to low one-way latency between the cloud gaming server and the client.

9C illustrates a sequence 900C of video frames being compressed by an encoder that takes into account the bandwidth available to the client, such that, according to one embodiment of the present disclosure, if the bandwidth exceeds the bandwidth required to support the target resolution of the client display, it is possible to have no or fewer skipped frames while still having lower one-way latency. In particular, in sequence 900C, a video frame is encoded as an I-frame, and subsequent video frames are also encoded normally, and after a delay in encoding the I-frame, if the client bandwidth is high for the target resolution of the client display.
Because of high bandwidth availability, a large amount of excess bandwidth is available to compensate for variability (e.g., jitter) in one-way latency between the cloud gaming server and the client, so that frame skipping can be avoided and a return to low one-way latency can be achieved quickly (e.g., within one to two frame periods). The encoding and transmission blocks of a video frame are shown in relation to the VSYNC signal 950.

Similar to Figure 9B, the sequence of video frames 900C of Figure 9C includes one encoded I-frame 905, with the remaining frames encoded as P-frames. For illustration purposes, encode block 901 and encode block 902 as P-frames before encode block 905 is encoded as an I-frame. The encoder then compresses the video frames as P-frames until the next scene change or until the video frames cannot reference a previous key frame (such as an I-frame). In general, the encoding time for an I-frame block may take longer than a P-frame block. For example, the encoding time for I-frame block 905 may exceed one frame period.
Also shown are various transmission times associated with corresponding encoded video frames. For example, the encoding and transmission of the video frame preceding the I-frame block 905 is shown with a low latency such that the corresponding encoding and transmission blocks can be performed within one frame period. However, the transmission block 915C of the encoded I-frame block 905 is shown with a higher latency such that the encoding block 905 and the transmission block 915C occur over more than one frame period, thereby introducing jitter into the one-way latency between the cloud gaming server and the client. As previously discussed, the encoding time can be further reduced by adjusting one or more encoder parameters (e.g., QP, target frame size, maximum frame size, encoder bitrate, etc.).

After I-frame block 905, the encoder continues compressing video frames, although they may be temporarily delayed due to the encoding of the I-frame. Again, the synchronization and offset of the VSYNC signal provides for overlapping operations at the cloud gaming server (scan out, encode, and transmit), overlapping operations at the client (receive, decode, render, display), and/or overlapping operations at the cloud gaming server and client, all of which facilitate compensation for latency variability introduced by server or network or client jitter, reduction in one-way latency, reduction in one-way latency variability, real-time generation and display of video content, and consistent video playback at the client.
Because the client bandwidth is high relative to the target resolution of the client display, the transmission of the encoded video frames reverts to one of low one-way latency around highlighted region 970, such as after one or two subsequent video frames are encoded as P-frames and transmitted to the client. Within region 970, P-frames encoded after I-frame block 905 is encoded are also transmitted to the client within one frame period (although they straddle both sides of the occurrence of the VSYNC signal), thereby returning to low one-way latency between the cloud gaming server and the client.

10 illustrates components of an exemplary device 1000 that may be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 10 illustrates an exemplary hardware system suitable for streaming media content and/or receiving streamed media content, including providing encoder adjustments to improve the tradeoff between one-way latency and video quality in a cloud gaming system for the purpose of reducing latency and providing more consistent latency between clouds, and to improve smoothness of client display of video, where the encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and/or number of video frames exceeding a target frame size, where the adjusted parameters may include encoder bitrate, target frame size, maximum frame size, and quantization parameter (QP) value, where according to an embodiment of the present disclosure, high performance encoders and decoders serve to reduce the overall one-way latency between the cloud gaming server and the client.
The block diagram illustrates a device 1000 that may incorporate or be a personal computer, a server computer, a game console, a mobile device, or other digital device, each of which is suitable for implementing embodiments of the present invention. The device 1000 includes a central processing unit (CPU) 1002 for executing software applications and optionally an operating system. The CPU 1002 may be comprised of one or more homogeneous or heterogeneous processing cores.

According to various embodiments, the CPU 1002 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is specifically adapted for highly parallel, computationally intensive uses, such as applications configured for graphics processing during game execution, media and interactive entertainment applications, etc.

Memory 1004 stores applications and data for use by CPU 1002 and GPU 1016. Storage 1006 provides non-volatile storage and other computer readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices and signal transmission and storage media. User input devices 1008 communicate user input from one or more users to device 1000, and examples of device 1000 may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder/camera, and/or microphone.
The network interface 1009 enables the device 1000 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 1012 is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU 1002, memory 1004, and/or storage 1006. The components of the device 1000, including the CPU 1002, the graphics subsystem including the GPU 1016, the memory 1004, the data storage 1006, the user input devices 1008, the network interface 1009, and the audio processor 1012, are connected via one or more data buses 1022.

The graphics subsystem 1014 is further connected to the data bus 1022 and to the components of the device 1000. The graphics subsystem 1014 includes a graphics processing unit (GPU) 1016 and a graphics memory 1018. The graphics memory 1018 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 1018 may be integrated into the same device as the GPU 1016, connected to the GPU 1016 as a separate device, and/or implemented within the memory 1004.
The pixel data may be provided to the graphics memory 1018 directly from the CPU 1002. Alternatively, the CPU 1002 provides the GPU 1016 with data and/or instructions defining a desired output image, from which the GPU 1016 generates pixel data for one or more output images. The data and/or instructions defining the desired output image may be stored in the memory 1004 and/or the graphics memory 1018. In one embodiment, the GPU 1016 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU 1016 may further include one or more programmable execution units capable of executing shader programs.

The graphics subsystem 1014 periodically outputs pixel data from the graphics memory 1018 for an image to be displayed on the display device 1010 or projected by a projection system (not shown). The display device 1010 can be any device capable of displaying visual information in response to signals from the device 1000, including CRTs, LCDs, plasma displays, and OLED displays. The device 1000 can provide analog or digital signals to the display device 1010, for example.

Other embodiments for optimizing the graphics subsystem 1014 can include multi-tenancy GPU operation where a GPU instance is shared among multiple applications and the GPU supporting a single game is distributed. The graphics subsystem 1014 can be configured as one or more processing devices.

For example, the graphics subsystem 1014 may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem may implement the graphics and/or rendering pipeline for multiple games. That is, the graphics subsystem 1014 is shared between multiple games being executed.

In other embodiments, the graphics subsystem 1014 includes multiple GPU devices that are combined to perform graphics processing for a single application running on a corresponding CPU. For example, multiple GPUs can perform an alternative form of frame rendering, where GPU1 renders a first frame, GPU2 renders a second frame in successive frame periods, until the last GPU is reached, so that the first GPU renders the next video frame (e.g., if there are only two GPUs, GPU1 renders the third frame). That is, the GPUs cycle when rendering frames. Rendering operations can overlap, and GPU2 can begin rendering the second frame before GPU1 finishes rendering the first frame.
In another embodiment, multiple GPU devices can be assigned different shader operations in the rendering pipeline and/or graphics pipeline. A master GPU performs the main rendering and compositing. For example, in a group including three GPUs, master GPU1 can perform the main rendering (e.g., first shader operations) and compositing the output from slave GPU2 and slave GPU3, slave GPU2 can perform second shader operations (e.g., fluid effects such as rivers), slave GPU3 can perform third shader (particle smoke, etc.) operations, and master GPU1 composites the results from each of GPU1, GPU2, and GPU3. In this way, various GPUs can be assigned to perform different shader operations (flag waving, wind, smoke generation, fire, etc.) to render a video frame.
In yet another embodiment, each of the three GPUs can be assigned to different objects and/or portions of the scene corresponding to the video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (concurrently in parallel) or in different frame periods (sequentially in parallel).

Accordingly, the present disclosure describes methods and systems configured for streaming media content and/or receiving streamed media content, including providing encoder adjustments to improve the tradeoff between one-way latency and video quality in a cloud gaming system, where the encoder adjustments may be based on monitoring of client bandwidth, skipped frames, number of encoded I-frames, number of scene changes, and/or number of video frames exceeding a target frame size, where the adjusted parameters may include encoder bitrate, target frame size, maximum frame size, and quantization parameter (QP) value, where the high performance encoder and decoder serve to reduce the overall one-way latency between the cloud gaming server and the client.

It should be understood that the various embodiments defined herein may be combined or assembled into specific embodiments using the various features disclosed herein. Thus, the examples provided are only some possible examples and are not limited to the various embodiments in which more embodiments can be defined by combining various elements. In some instances, an embodiment may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

Embodiments of the present disclosure may be practiced in a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

With the above embodiments in mind, it should be understood that embodiments of the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings of the present disclosure, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

The present disclosure may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data, which can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed over network coupled computer systems, such that the computer readable code is stored and executed in a distributed fashion.

Although the operations of the method are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be coordinated to occur at slightly different times or distributed throughout the system, allowing processing operations to occur at various intervals relative to processing, so long as the processing of overlapping operations is performed in the desired manner.

Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative rather than limiting, and the embodiments of the present disclosure are not limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.

Claims (60)

1. A cloud gaming method, comprising:
generating a plurality of video frames when executing a video game on a cloud gaming server;
encoding the plurality of video frames at an encoder bit rate; and transmitting the plurality of compressed video frames from a streamer of the cloud gaming server to a client;
Measure the client's maximum receive bandwidth,
monitoring, at the streamer, the encoding of the plurality of video frames;
and dynamically adjusting parameters of the encoder based on the monitoring of the encoding. The dynamic adjustment of the parameters includes:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames exceeds the maximum receive bandwidth;
The method of claim 1 , further comprising increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being said QP parameter. The dynamic adjustment of the parameters includes:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames is within the maximum receive bandwidth;
determining that there is excess bandwidth when transmitting the group of video frames;
reducing a value of a QP parameter based on the excess bandwidth so that encoding is performed with greater precision, the parameter being the QP parameter;
The method of claim 1 , comprising: The dynamic adjustment of the parameters includes:
determining whether a number of video frames compressed as I-frames from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of I-frames;
increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being the QP parameter;
The method of claim 1 , comprising: The dynamic adjustment of the parameters includes:
determining that a number of video frames compressed as I-frames from a group of video frames from the plurality of compressed video frames is below a threshold number of I-frames;
The method of claim 1 , further comprising reducing a value of a QP parameter such that encoding is performed with greater precision, said parameter being said QP parameter. The dynamic adjustment of the parameters includes:
determining that a group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames meeting or exceeding a threshold, each of the number of video frames exceeding a target frame size;
The method of claim 1 , further comprising reducing at least one of the target frame size and maximum frame size due to the parameter. The method of claim 6, wherein the target frame size and the maximum frame size are equal. The dynamic adjustment of the parameters includes:
determining that a group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames being below a threshold, each of the number of video frames being within a target frame size;
The method of claim 1 , further comprising increasing at least one of the target frame size and maximum frame size as the parameter. The dynamic adjustment of the parameters includes:
determining whether a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of scene changes;
The method of claim 1 , further comprising increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being said QP parameter. The dynamic adjustment of the parameters includes:
determining that a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames is below a threshold number of scene changes;
The method of claim 1 , further comprising reducing a value of a QP parameter such that encoding is performed with greater precision, said parameter being said QP parameter. The method of claim 1, further comprising prioritizing smooth playback on the client by disabling skipping encoding of video frames. The method of claim 1, further comprising: at the encoder, dynamically adjusting an encoder bit rate speed based on the maximum receiving bandwidth of the client. A non-transitory computer-readable medium storing a cloud gaming computer program, comprising:
having program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server;
having program instructions for measuring a maximum receive bandwidth of a client;
and program instructions for encoding the plurality of video frames at an encoder bit rate, the plurality of compressed video frames being transmitted from a streamer of the cloud gaming server to a client;
program instructions for monitoring the encoding of the plurality of video frames at the streamer;
A non-transitory computer readable medium having program instructions for dynamically adjusting parameters of the encoder based on the monitoring of the encoding. The program instructions for dynamically adjusting the parameters include:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames exceeds the maximum receive bandwidth;
14. The non-transitory computer readable medium of claim 13, comprising program instructions for increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being the QP parameter. The program instructions for dynamically adjusting the parameters include:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames is within the maximum receive bandwidth;
determining that there is excess bandwidth when transmitting the group of video frames;
14. The non-transitory computer-readable medium of claim 13, comprising program instructions for reducing a value of a QP parameter based on the excess bandwidth such that encoding is performed with greater precision, the parameter being the QP parameter. The program instructions for dynamically adjusting the parameters include:
determining whether a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of scene changes;
14. The non-transitory computer readable medium of claim 13, comprising program instructions for increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being the QP parameter. The program instructions for dynamically adjusting the parameters include:
determining that a number of video frames compressed as I-frames from a group of video frames from the plurality of compressed video frames is below a threshold number of I-frames;
14. The non-transitory computer readable medium of claim 13, having programming instructions for reducing a value of a QP parameter such that encoding is performed with greater precision, the parameter being the QP parameter. The program instructions for dynamically adjusting the parameters include:
the group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames meeting or exceeding a threshold, each of the number of video frames exceeding a target frame size;
14. The non-transitory computer readable medium of claim 13 having program instructions for reducing at least one parameter of the target frame size and maximum frame size. The non-transitory computer-readable medium of claim 18, wherein in the computer program for cloud gaming, the target frame size and the maximum frame size are equal. The program instructions for dynamically adjusting the parameters include:
program instructions for determining that a group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames being below a threshold, each of the number of video frames being within a target frame size;
14. The non-transitory computer readable medium of claim 13 having program instructions for increasing at least one of the target frame size and maximum frame size as the parameter. The program instructions for dynamically adjusting the parameters include:
determining whether a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of scene changes;
14. The non-transitory computer readable medium of claim 13, comprising program instructions for increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being the QP parameter. The program instructions for dynamically adjusting the parameters include:
determining that a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames is below a threshold number of scene changes;
14. The non-transitory computer readable medium of claim 13, comprising program instructions for reducing a value of a QP parameter such that encoding is performed with greater precision, said parameter being the QP parameter. The non-transitory computer-readable medium of claim 13, further comprising program instructions for prioritizing smooth playback at the client by disabling skipping of encoding of video frames. The non-transitory computer-readable medium of claim 13, further comprising program instructions for dynamically adjusting an encoder bit rate rate at the encoder based on the maximum receiving bandwidth of the client. 1. A computer system comprising:
A processor;
and a memory coupled to the processor and having instructions stored therein, the instructions, when executed by the computer system, causing the computer system to perform a cloud gaming method, the cloud gaming method comprising:
generating a plurality of video frames when executing a video game on a cloud gaming server;
encoding the plurality of video frames at an encoder bit rate, and transmitting the compressed plurality of video frames from a streamer of the cloud gaming server to a client;
Measure the client's maximum receive bandwidth,
monitoring, at the streamer, the encoding of the plurality of video frames;
A computer system that dynamically adjusts parameters of the encoder based on the monitoring of the encoding. The method of dynamically adjusting the parameters includes:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames exceeds the maximum receive bandwidth;
26. The computer system of claim 25, further comprising increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being said QP parameter. The method of dynamically adjusting the parameters includes:
determining that the encoder bit rate used to encode a group of video frames from the plurality of video frames is within the maximum receive bandwidth;
determining that there is excess bandwidth when transmitting the group of video frames;
26. The computer system of claim 25, further comprising: reducing a value of a QP parameter based on the excess bandwidth so that encoding is performed with greater precision, the parameter being the QP parameter. The method of dynamically adjusting the parameters includes:
determining whether a number of video frames compressed as I-frames from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of I-frames;
26. The computer system of claim 25, further comprising increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being said QP parameter. The method of dynamically adjusting the parameters includes:
determining that a number of video frames compressed as I-frames from a group of video frames from the plurality of compressed video frames is below a threshold number of I-frames;
26. The computer system of claim 25, further comprising: reducing a value of a QP parameter such that encoding is performed with greater precision, said parameter being the QP parameter. The method of dynamically adjusting the parameters includes:
determining that a group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames meeting or exceeding a threshold, each of the number of video frames exceeding a target frame size;
26. The computer system of claim 25, wherein at least one of the target frame size and maximum frame size is reduced due to the parameters. The computer system of claim 30, wherein the target frame size and the maximum frame size are equal. The method of dynamically adjusting the parameters includes:
determining that a group of video frames from the plurality of video frames encoded and transmitted at the transmission rate includes a number of video frames, the number of video frames being below a threshold, each of the number of video frames being within a target frame size;
26. The computer system of claim 25, wherein at least one of the target frame size and maximum frame size is increased as the parameter. The method of dynamically adjusting the parameters includes:
determining whether a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames meets or exceeds a threshold number of scene changes;
26. The computer system of claim 25, further comprising increasing a value of a QP parameter such that encoding is performed with less precision, said parameter being said QP parameter. The method of dynamically adjusting the parameters includes:
determining that a number of video frames identified as having a scene change from a group of video frames from the plurality of compressed video frames is below a threshold number of scene changes;
26. The computer system of claim 25, further comprising: reducing a value of a QP parameter such that encoding is performed with greater precision, said parameter being said QP parameter. The method further comprises:
26. The computer system of claim 25, further comprising disabling skipping encoding of video frames to prioritize smooth playback at the client. The method further comprises:
26. The computer system of claim 25, wherein the encoder dynamically adjusts an encoder bit rate rate based on the maximum receiving bandwidth of the client. 1. A cloud gaming method, comprising:
generating a plurality of video frames when executing a video game on a cloud gaming server;
predicting a scene change for a first video frame for the video game, the scene change being predicted before the first video frame is generated;
generating a scene change hint that the first video frame is a scene change;
sending the scene change hint to the encoder;
delivering the first video frame to an encoder, the first video frame being encoded as an I-frame based on the scene change hint;
Measure the client's maximum receive bandwidth,
The method further comprising: determining whether to encode or not encode a second video frame received at the encoder based on the maximum receiving bandwidth of the client and a target resolution of a client display. further comprising: dynamically adjusting an encoder bitrate speed based on the maximum receiving bandwidth of the client;
40. The method of claim 37, further comprising transmitting the video frames to the client once the video frames are encoded. determining whether to encode or not encode the second video frame,
38. The method of claim 37, further comprising skipping encoding of the second video frame when the transmission rate to the client is low relative to the target resolution of the client display, such that the transmission rate to the client of a group of video frames from the compressed plurality of video frames exceeds the maximum receive bandwidth. determining whether to encode or not encode the second video frame,
38. The method of claim 37, further comprising: if a transmission rate to the client is high for the target resolution of the client display, encoding the second video frame normally such that the transmission rate to the client for a group of video frames from the plurality of compressed video frames is within the maximum receive bandwidth. determining whether to encode or not encode the second video frame,
41. The method of claim 40, further comprising encoding the second video frame at a lower precision if the transmission rate to the client is medium for the target resolution of the client display. Predicting a scene change for the first video frame includes:
executing, at the cloud gaming server, game logic built on a game engine of the video game to generate the plurality of video frames;
executing scene change logic to predict the scene change for the first video frame, the prediction being based on a game state collected during execution of the game logic;
generating the scene change hint using the scene change logic;
38. The method of claim 37, wherein the encoder sends the scene change hint before receiving the first video frame. The method of claim 42, wherein the scene change hints are delivered from the scene change logic to the encoder via an API. The method of claim 37, wherein the second video frame is compressed after the first video frame is compressed by the encoder. A non-transitory computer-readable medium storing a cloud gaming computer program, comprising:
having program instructions for generating a plurality of video frames when executing a video game on a cloud gaming server;
and program instructions for predicting a scene change for a first video frame for the video game, the scene change being predicted before the first video frame is generated;
generating a scene change hint that the first video frame is a scene change;
program instructions for transmitting the scene change hint to the encoder;
and program instructions for delivering the first video frame to an encoder, the first video frame being encoded as an I-frame based on the scene change hint;
having program instructions for measuring a maximum receive bandwidth of a client;
A non-transitory computer-readable medium having program instructions for determining whether to encode or not encode a second video frame received at the encoder based on the maximum receiving bandwidth of the client and a target resolution of a client display. further comprising program instructions for dynamically adjusting an encoder bit rate rate at the encoder based on the maximum receiving bandwidth of the client;
46. The non-transitory computer readable medium of claim 45 having program instructions for transmitting the video frames to the client once the video frames are encoded. The program instructions for determining whether to encode or not encode the second video frame include:
46. The non-transitory computer readable medium of claim 45, having program instructions for skipping encoding of the second video frame when the transmission rate to the client is low for the target resolution of the client display, such that the transmission rate to the client of a group of video frames from the plurality of compressed video frames exceeds the maximum receive bandwidth. The program instructions for determining whether to encode or not encode the second video frame include:
46. The non-transitory computer readable medium of claim 45, having program instructions for successfully encoding the second video frame such that, if a transmission rate to the client is high for the target resolution of the client display, the transmission rate to the client for a group of video frames from the plurality of compressed video frames is within the maximum receive bandwidth. The program instructions for determining whether to encode or not encode the second video frame include:
49. The non-transitory computer-readable medium of claim 48, having program instructions for encoding the second video frame at a lower precision if the transmission rate to the client is medium for the target resolution of the client display. The program instructions for predicting a scene change for the first video frame include:
program instructions for executing game logic built on a game engine of the video game at the cloud gaming server to generate the plurality of video frames;
and program instructions for executing scene change logic to predict the scene change for the first video frame, the prediction being based on a game state collected during execution of the game logic;
having program instructions for generating the scene change hint using the scene change logic;
46. The non-transitory computer-readable medium of claim 45, having program instructions for transmitting the scene change hint before the encoder receives the first video frame. The non-transitory computer-readable medium of claim 50, wherein in the computer program for cloud gaming, the scene change hint is delivered from the scene change logic to the encoder via an API. The non-transitory computer-readable medium of claim 45, wherein in the computer program for cloud gaming, the second video frame is compressed after the first video frame is compressed by the encoder. 1. A computer system comprising:
A processor;
and a memory coupled to the processor and having instructions stored therein, the instructions, when executed by the computer system, causing the computer system to perform a cloud gaming method, the cloud gaming method comprising:
generating a plurality of video frames when executing a video game on a cloud gaming server;
predicting a scene change for a first video frame for the video game, the scene change being predicted before the first video frame is generated;
generating a scene change hint, the first video frame being a scene change;
sending the scene change hint to the encoder;
delivering the first video frame to an encoder, the first video frame being encoded as an I-frame based on the scene change hint;
Measure the client's maximum receive bandwidth,
The computer system determines whether to encode or not encode the received second video frame at the encoder based on the maximum receiving bandwidth of the client and a target resolution of a client display. further comprising: dynamically adjusting, at the encoder, an encoder bit rate speed based on the maximum receiving bandwidth of the client;
54. The computer system of claim 53, further comprising: transmitting the video frames to the client once the video frames are encoded. determining whether to encode or not encode the second video frame,
54. The computer system of claim 53, further comprising: skipping encoding of the second video frame when the transmission rate to the client is low for the target resolution of the client display, such that the transmission rate to the client of a group of video frames from the compressed plurality of video frames exceeds the maximum receive bandwidth. The determining whether to encode or not encode the second video frame comprises:
54. The computer system of claim 53, further comprising: if a transmission rate to the client is high for the target resolution of the client display, then normally encoding the second video frame such that the transmission rate to the client for a group of video frames from the plurality of compressed video frames is within the maximum receive bandwidth. determining whether to encode or not encode the second video frame,
57. The computer system of claim 56, further comprising: encoding the second video frame at a lower precision if the transmission rate to the client is medium for the target resolution of the client display. Predicting a scene change for the first video frame includes:
executing, at the cloud gaming server, game logic built on a game engine of the video game to generate the plurality of video frames;
executing scene change logic to predict the scene change for the first video frame, the prediction being based on a game state collected during execution of the game logic;
generating the scene change hint using the scene change logic;
54. The computer system of claim 53, wherein the encoder transmits the scene change hint before receiving the first video frame. The computer system of claim 58, wherein the scene change hints are delivered from the scene change logic to the encoder via an API. The computer system of claim 53, wherein the second video frame is compressed after the first video frame is compressed by the encoder.