JP2022551587A - Synchronization and offset of VSYNC between cloud gaming server and client - Google Patents

Abstract

A method is disclosed that includes, at a server, setting a server VSYNC signal to a server VSYNC frequency. The server VSYNC signal corresponds to the generation of video frames during multiple frame periods of the server VSYNC frequency. In this method, at the client, the client VSYNC signal is set to the client VSYNC frequency. The method uses the server VSYNC signal to transmit compressed video frames from the server to the client over the network, the compressed video frames being based on generated video frames. The method decodes and displays the compressed video frames at the client, and analyzes the timing of one or more client operations as the client receives the plurality of compressed video frames to generate a server VSYNC signal and a client VSYNC signal. Adjust the relative timing between [Selection drawing] Fig. 4

Description

The present disclosure provides a streaming system configured to stream content over a network, and more specifically between a cloud gaming server and a client for reducing latency between the cloud gaming server and the client. of the vertical synchronization (VSYNC) signal.

In recent years, online services that enable streaming online or cloud games between a cloud game server and clients connected via a network have been continuously promoted. Streaming formats include the availability of game titles on demand, networking capabilities between players for multiplayer games, sharing of assets between players, instant experience sharing between players and/or spectators, friends playing video Being able to watch games, allowing friends to participate in their ongoing game play, etc. has become more popular. Unfortunately, demand is also driven by the capabilities of network connectivity and the limitations of processing performed on servers and clients that are responsive enough to render high-quality images delivered to clients.
For example, the results of all game activities performed on the server should be compressed and sent back to the client with low millisecond latency for the best user experience. Round-trip latency can be defined as the overall time from the user's controller input to the display of a video frame on the client. This includes the processing and transmission of control information from the controller to the client, the processing and transmission of control information from the client to the server, the use of that input in the server to generate video frames in response to that input, and the encoding unit to the encoding unit. processing and forwarding video frames (e.g., scanning out), encoding video frames, sending encoded video frames back to the client, receiving and decoding video frames, and either processing video frames prior to display or May include staging.
One-way latency is part of the round-trip latency consisting of the time from the start of the transfer (e.g., scanout) of a video frame to the encoding unit at the server to the start of display of the video frame at the client. can be defined as part Part of round-trip and one-way latency relates to the time it takes for a data stream to travel from a client to a server and from a server to a client over a communication network. Another part relates to processing on the client and server. Improvements in these operations, such as advanced strategies related to decoding and displaying frames, significantly reduce round-trips and one-way latencies between server and client, resulting in a better experience for users of cloud gaming services. provided.

It is against this background that the embodiments of the present disclosure have been made.

Embodiments of the present disclosure seek to reduce latency between streaming systems configured to stream content (e.g., games) over a network, and more specifically, cloud gaming servers, and clients. Aimed at synchronizing VSYNC signals between cloud gaming servers and clients. In the context of this patent application, "synchronization" should be taken to mean adjusting the signals so that they match in frequency, but may differ in phase. "Offset" should be interpreted to mean the time delay between signals. For example, the time from when one signal reaches its maximum to when the other signal reaches its maximum.

An embodiment of the present disclosure discloses a method. The method includes, at the server, setting a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during a plurality of frame periods at the server VSYNC frequency. The method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. The method includes decoding and displaying a plurality of compressed video frames at a client. The method includes analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives multiple compressed video frames. include.

An embodiment of the present disclosure discloses a method. The method includes generating multiple video frames at a server during multiple frame periods, where the frame periods are of approximately equal size. The method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The method includes transmitting from a server to a client a plurality of compressed video frames based on the plurality of video frames. The method includes decoding and displaying a plurality of compressed video frames at a client. The method analyzes the timing of one or more client operations to adjust the relative timing of the client VSYNC signal when the client receives the multiple compressed video frames, and performs the multiple compressed video frames at the server. Including generating.

Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing the method. The computer readable medium is program instructions for setting a server VSYNC signal to a server VSYNC frequency at a server, the server VSYNC signal for generating a plurality of video frames at the server during a plurality of frames at the server VSYNC frequency. Contains corresponding program instructions. A computer-readable medium includes program instructions for setting a client VSYNC signal to a client VSYNC frequency at a client. The computer readable medium includes program instructions for transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. A computer-readable medium includes program instructions for decoding and displaying a plurality of compressed video frames at a client. A computer readable medium analyzes the timing of one or more client operations and adjusts the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives multiple compressed video frames. Contains program instructions for

In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and memory coupled to the processor, the memory storing instructions that, when executed by the computer system, perform a method. Have a computer system do it. The method includes, at the server, setting a server VSYNC signal to a server VSYNC frequency, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during a plurality of frame periods at the server VSYNC frequency. The method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. The method includes decoding and displaying a plurality of compressed video frames at a client. The method includes analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives multiple compressed video frames. include.

Another embodiment of the present disclosure discloses another method. The method comprises, at the server, setting a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods. including. The method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. The method includes decoding and displaying a plurality of compressed video frames at a client. The method includes analyzing the timing of one or more client operations to set the amount of frame buffering used by the client as the client receives multiple compressed video frames.

Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing the method. The computer readable medium is program instructions at the server for setting the server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, wherein the server VSYNC signal corresponds to a plurality of video frames at the server during the plurality of frame periods. contains program instructions for setting, corresponding to the generation of A computer-readable medium includes program instructions for setting a client VSYNC signal to a client VSYNC frequency at a client. The computer readable medium includes program instructions for transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. A computer-readable medium includes program instructions for decoding and displaying a plurality of compressed video frames at a client. The computer readable medium provides program instructions for analyzing the timing of one or more client operations and setting the amount of frame buffering used by the client as the client receives multiple compressed video frames. include.

In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and a memory coupled to the processor, the memory storing instructions that, when executed by the computer system, perform a method. Have a computer system do it. The method comprises, at the server, setting a server VSYNC signal to a server VSYNC frequency defining a plurality of frame periods, the server VSYNC signal corresponding to generation of a plurality of video frames at the server during the plurality of frame periods. including. The method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The method includes transmitting a plurality of compressed video frames based on the plurality of video frames from a server to a client over a network using a server VSYNC signal. The method includes decoding and displaying a plurality of compressed video frames at a client. The method includes analyzing the timing of one or more client operations to set the amount of frame buffering used by the client as the client receives multiple compressed video frames.

Another embodiment of the present disclosure discloses another method. The method includes setting a plurality of VSYNC signals to a plurality of VSYNC frequencies in the plurality of devices, wherein corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies. The method includes transmitting multiple signals between multiple devices, which are analyzed and used to adjust relative timing between corresponding device VSYNC signals of at least two devices.

Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing the method. A computer readable medium is program instructions for setting a plurality of VSYNC signals to a plurality of VSYNC frequencies in a plurality of devices, wherein corresponding device VSYNC signals of corresponding devices are set to corresponding device VSYNC frequencies; Contains program instructions. A computer readable medium is program instructions for transmitting a plurality of signals between a plurality of devices analyzed and used to coordinate relative timing between corresponding device VSYNC signals of at least two devices. Contains program instructions.

In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and memory coupled to the processor, the memory storing instructions that, when executed by the computer system, perform a method. Have a computer system do it. The method includes setting a plurality of VSYNC signals to a plurality of VSYNC frequencies in the plurality of devices, wherein corresponding device VSYNC signals of corresponding devices are set to the corresponding device VSYNC frequencies. The method includes transmitting multiple signals between multiple devices, which are analyzed and used to adjust relative timing between corresponding device VSYNC signals of at least two devices.

Another embodiment of the present disclosure discloses another method. The method includes receiving encoded video frames at a client, and a server executing an application to produce rendered video frames, which are then encoded as encoded video frames at an encoder at the server. A encoded video frame includes one or more compressed encoded slices. The method includes decoding one or more encoded slices at a decoder of a client to generate one or more decoded slices. The method includes rendering one or more decoded slices for display at a client. The method includes initiating display of one or more decoded slices to be rendered prior to fully receiving the one or more encoded slices at the client.

Another embodiment of the present disclosure discloses a non-transitory computer-readable medium storing a computer program for performing the method. The computer readable medium is program instructions for receiving encoded video frames at a client and a server executing an application to produce rendered video frames, which are then encoded at an encoder at the server. The encoded video frame includes program instructions that have one or more encoded slices that have been compressed. A computer-readable medium includes program instructions for decoding one or more encoded slices at a decoder of a client to produce one or more decoded slices. A computer-readable medium includes program instructions for rendering one or more decoded slices for display at a client. The computer-readable medium includes program instructions for initiating display of one or more decoded slices rendered prior to complete reception of one or more encoded slices at a client.

In another embodiment of the present disclosure, a computer system is disclosed that includes a processor and memory coupled to the processor, the memory storing instructions that, when executed by the computer system, perform a method. Have a computer system do it. The method includes receiving encoded video frames at a client, and a server running an application to produce rendered video frames, which are then encoded as encoded video frames at an encoder at the server. A encoded video frame includes one or more compressed encoded slices. The method includes decoding one or more encoded slices at a decoder of a client to generate one or more decoded slices. The method includes rendering one or more decoded slices for display at a client. The method includes initiating display of one or more decoded slices to be rendered prior to fully receiving the one or more encoded slices at the client.

Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the disclosure.

The present disclosure can best be understood by reference to the following detailed description in conjunction with the accompanying drawings.

FIG. 1A is a diagram of the VSYNC signal at the beginning of a frame period, according to one embodiment of the present disclosure. FIG. 1B is a frequency diagram of a VSYNC signal, according to one embodiment of the present disclosure. 1 is a diagram of a system for providing games over a network between one or more cloud gaming servers and one or more client devices in various configurations, according to one embodiment of the present disclosure; FIG. Signals can be synchronized and offset to reduce one-way latency. FIG. 12 is a diagram for providing gaming between two or more peer devices, synchronizing and offsetting VSYNC signals for optimal timing of receipt of controller and other information between devices, according to an embodiment of the present disclosure; can be achieved. 4 illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, according to an embodiment of the present disclosure; 4 illustrates a multi-tenancy configuration between a cloud gaming server and multiple clients benefiting from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure; FIG. 11 illustrates one-way latency variation between a cloud gaming server and a client due to clock drift when streaming video frames generated from a video game running on the server, according to one embodiment of the present disclosure; FIG. 1 shows a network configuration including a cloud gaming server and a client when streaming video frames generated from a video game running on the server, according to an embodiment of the present disclosure, wherein the VSYNC signal between the server and the client is: Synchronized and offset to allow duplication of operations on the server and client, and to reduce one-way latency between the server and client. Possible variations in the timing of completion of decoding by the client with respect to the desired display time specified by the server, due to drift between the respective clocks at the cloud gaming server and the client, and the timing of the client and server, according to an embodiment of the present disclosure. FIG. 2 illustrates the variation of operation time and network latency; FIG. 12B includes a histogram showing the timing of completion of decoding by the client relative to the desired display time specified by the server, with subsequent measurements in the histogram due to drift between respective clocks at the cloud gaming server and client, according to an embodiment of the present disclosure. increase in decoding time. FIG. 12B includes a histogram showing the timing of completion of decoding by the client relative to the desired display time specified by the server, after compensating for the measured drift between the respective clocks at the cloud gaming server and the client, according to an embodiment of the present disclosure; FIG. , show consistent measurements of decoding time in subsequent histograms. [0014] Fig. 4 is a flow diagram illustrating a method for coordinating VSYNC signals between a cloud gaming server and a client for the purpose of reducing one-way latency, according to an embodiment of the present disclosure; A cloud gaming server for the purpose of reducing one-way latency, including constructing a histogram that provides the distribution of the timing of completion of decoding by the client relative to the desired display time specified by the server, according to one embodiment of the present disclosure. 4 is a flow diagram illustrating a method of aligning VSYNC signals when performing VSYNC signal adjustment between a server and a client, wherein a histogram is configured to determine the adjustment of the offset between the VSYNC signals at the server and the client, the histogram is also configured to determine the drift between the server VSYNC signal and the client VSYNC signal. 4 is a flow diagram illustrating another method for synchronizing VSYNC signals when performing VSYNC signal coordination between a cloud gaming server and a client for the purpose of reducing one-way latency, according to an embodiment of the present disclosure; FIG. is. FIG. 10 is a flow diagram illustrating a method for adjusting a client VSYNC signal in connection with generating compressed video frames at a server, where the video frames are generated during similarly sized frame durations, in accordance with an embodiment of the present disclosure; FIG. . [0019] Fig. 4 shows the construction of a histogram that provides the distribution of the timing of completion of decoding by the client with respect to the desired display time specified by the server, wherein the histogram is between VSYNC signals at the server and the client, in accordance with an embodiment of the present disclosure; is configured to determine the adjustment of the offset of the FIG. 4 is an illustration of a histogram providing the distribution of the timing of completion of decoding by a client with respect to the desired display time specified by the server, the histogram showing the offset between the VSYNC signals at the server and the client, according to an embodiment of the present disclosure; Configured to determine reconciliation. Fig. 10 is a flow diagram illustrating a method for constructing a histogram that provides a distribution of the timing of completion of decoding by a client with respect to a desired display time specified by a server, the histogram representing the required buffering of video frames decoded at the client; is configured to determine the amount of 4 is a flow diagram illustrating a method for adjusting relative timing between VSYNC signals between two or more devices, according to one embodiment of the present disclosure; 4 illustrates duplication of receiving, decoding, and rendering decompressed video frames for display at a client, according to one embodiment of the present disclosure; 1 is a flow diagram illustrating a method of cloud gaming in which encoded frames are received from a server at a client, decoded and rendered for display, decoding and display of video frames is May be overlapped for the purpose of reducing direction latency. 4 illustrates components of an exemplary device that can be used to carry out aspects of various embodiments of the present disclosure;

Although the following detailed description includes many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications to the following details are within the scope of the disclosure. . Accordingly, the aspects of the disclosure described below are presented without loss of generality to, and without imposing limitation on, the claims that follow this description.

In general, various embodiments of the present disclosure address latency and/or latency concerns between source and target devices when streaming media content (e.g., when streaming audio and video from a video game). Methods and systems configured to reduce qualitativeness are described. Specifically, in some embodiments of the present disclosure, VSYNC signals between cloud gaming servers and clients are synchronized and offset. Due to clock differences between the cloud game server and the client, the VSYNC signals of the cloud game server and client drift relative to each other. This drift leads to latency instability for the maximum frame period. For example, if the game is running at 60 Hz for video frame generation, there is an additional latency of 0-16.7 milliseconds that varies over time. By analyzing the worst-case or near-worst-case arrival times of compressed video frames at the client, the ideal VSYNC relationship between the cloud gaming server and the client can be determined. This ideal relationship can be established by adjusting the VSYNC frequency at either the cloud gaming server or the client, thus eliminating latency instability.
In other embodiments of the present disclosure, VSYNC signals between game consoles (e.g., game consoles) are synchronized and offset to provide ideal VSYNC relationships and minimal one-way latency between peer devices. . In particular, clock differences between peer devices (such as game consoles) in face-to-face games cause their VSYNC signals to drift relative to each other, resulting in erratic latency of up to a frame period. For example, if the game is running at 60 Hz for video frame generation, there is an additional latency of 0-16.7 milliseconds that varies over time. By exchanging timestamp information, the ideal VSYNC relationship between peer devices can be determined. This ideal relationship can be established by adjusting the VSYNC frequency on either of the peer devices, thus eliminating latency instability.
In still other embodiments of the present disclosure, dynamic client buffering and selective use of video frames received at the client from the cloud gaming server provide latency reduction and coordination. Knowledge of the server-side timing of video frame generation allows the client to determine the ideal display time for each frame. Frame buffering (single buffering, double buffering, triple buffering, etc.) can be dynamically adjusted based on the variability of compressed video frame arrival times at the client. Latency adjustments may also occur, such as choosing to skip display of latency arriving frames. In other embodiments of the present disclosure, one-way latency between the cloud gaming server and client may be reduced by duplicating the decoding of compressed video frames and their display. A cloud gaming client receives the compressed video frames from the cloud gaming server and decodes the compressed video frames. One-way latency can be reduced by starting display of a video frame before the frame is fully received or decoded at the client. Timing of transmission for display should predict the remaining time required to receive and decode a compressed video frame.

In particular the additional time required to generate complex frames on the server (such as scene changes), the increased time to encode/compress complex frames on the server, variable communication paths through the network, And latency instability can occur between the server and the client due to the increased time to decode complex frames at the client. Latency can also be erratic due to clock differences between the server and client. This causes the server and client VSYNC signals to drift. In one embodiment, this latency instability adjusts either the server VSYNC signal or the client VSYNC signal to bring the server VSYNC signal and the client VSYNC signal back into synchronized alignment (e.g., operate at the same frequency). ) can be removed by
In another embodiment, adjusting the timing offset between the server VSYNC signal and the client VSYNC signal is performed by considering near-worst-case latency requirements when receiving and displaying video frames at the client. Reduce one-way latency. In yet another embodiment, client-side dynamic buffering provides more display buffers at the client when latency increases and uses fewer display buffers when latency decreases. , to provide additional latency adjustments. In another embodiment, one-way latency can be further reduced by overlapping the decoding and display of video frames at the client.

With the above general understanding of the various embodiments, details of example embodiments will now be described with reference to various drawings.

Throughout this specification, references to "game", "video game", or "game application" or "application" are meant to describe any type of interactive application directed through the execution of input commands. By way of example only, interactive applications include applications for gaming, word processing, video processing, video game processing, and the like. Moreover, the terms introduced above are interchangeable.

Cloud gaming involves running a video game on a server to generate a game-rendered video frame, which is then sent to a client for display. The timing of operations on both the server and client may be associated with their respective vertical synchronization (VSYNC) parameters. Once the VSYNC signals are properly synchronized and/or offset between the server and/or client, operations performed on the server (e.g., generating and transmitting video frames over one or more frame periods) are performed on the client. (eg, displaying video frames on the display at a display frame or refresh rate corresponding to the frame period). In particular, a server VSYNC signal generated at the server and a client VSYNC signal generated at the client may be used to synchronize operations at the server and client. That is, when the server and client VSYNC signals are synchronized and/or offset, the server generates and transmits video frames in sync with how the client displays those video frames.

VSYNC signaling and vertical blanking interval (VBI) are incorporated to generate video frames and display those video frames when streaming media content between server and client. For example, the server will attempt to generate a game-rendered video frame with one or more frame periods defined by the corresponding server VSYNC signal (e.g., if the frame period is 16.7ms, then every frame period will be a video frame). Generating a frame results in 60 Hz operation, generating one video frame every two frame periods results in 30 Hz operation), which is then encoded and sent to the client. At the client, the received encoded video frames are decoded and displayed, and the client displays each video frame rendered for display beginning with the corresponding client VSYNC.

For illustrative purposes, FIG. 1A shows how VSYNC signal 111 may indicate the start of a frame period, where various operations may be performed at the server and/or client during the corresponding frame period. . When streaming media content, the server may use the server VSYNC signal to generate and encode video frames, and the client may use the client VSYNC signal to display the video frames. A VSYNC signal 111 is generated at a defined frequency corresponding to a defined frame period 110, as shown in FIG. 1B. In addition, VBI 105 defines the period from when the last raster line was drawn to the display during the previous frame period to when the first raster line (eg, top) is drawn to the display. As shown, after VBI 105, the video frames rendered for display are displayed via raster scan lines 106 (eg, raster line by raster line from left to right).

Additionally, various embodiments of the present disclosure reduce one-way latency and/or latency instability between source and target devices, such as when streaming media content (e.g., video game content). disclosed to do so. For illustrative purposes only, various embodiments for reducing one-way latency and/or latency instability are described within a server and client network configuration. However, the various techniques disclosed for reducing one-way latency and/or latency instability can be implemented in other network configurations and/or over peer-to-peer networks, as shown in FIGS. 2A-2D. It is understood that it can be implemented in For example, various embodiments disclosed for reducing one-way latency and/or latency instability can be implemented in various configurations (e.g., server and client, server and server, server and multiple clients, server and a plurality of servers, a client to a client, a client to a plurality of clients, etc.).

FIG. 2A illustrates gaming via network 250 between one or more cloud gaming networks 290 and/or servers 260 and one or more client devices 210 in various configurations, according to embodiments of the present disclosure. 200A is a diagram of a system 200A for providing the VSYNC signals of the server and client can be synchronized and offset and/or if dynamic buffering is performed at the client and/or decoding and display at the client. Operations can be duplicated to reduce one-way latency between server 260 and client 210 . In particular, the system 200A provides games over a cloud gaming network 290, and according to one embodiment of the present disclosure, the games are played by the corresponding user's client device 210 (e.g., thin client) playing the game. is running remotely from System 200A brings control of a game to one or more users playing one or more games over cloud gaming network 290 over network 250, either in single-player mode or multiplayer mode. can be done. In some embodiments, the cloud gaming network 290 may include multiple virtual machines (VMs) running on the host machine's hypervisor, one or more of which are available to the host's hypervisor. is configured to run a game processor module that utilizes hardware resources that are: Network 250 may include one or more communication technologies. In some embodiments, network 250 may include fifth generation (5G) network technology with advanced wireless communication systems.

In some embodiments, communications may be facilitated using wireless technology. Such technologies may include, for example, 5G wireless communication technologies. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network, where the service area covered by a provider is divided into smaller geographical areas called cells. Analog signals representing sound and images are digitized on the phone, converted by an analog-to-digital converter, and transmitted as a stream of bits. All 5G wireless devices within a cell connect to the cell's local antenna array and low-power automatic transceiver (transmitter and receiver) via frequency channels assigned by the transceiver from a pool of frequencies reused in other cells. ) and radio waves. Local antennas are connected to the telephone network and the Internet by high-bandwidth fiber optic or wireless backhaul connections. As with other cellular networks, mobile devices that move from one cell to another are automatically transferred to the new cell. It is understood that a 5G network is just one example type of communication network, and that embodiments of the present disclosure can utilize previous generations of wireless or wired communications, as well as later generations of wired or wireless technologies following 5G. want to be

As shown, game network 290 includes a game server 260 that provides access to multiple video games. Game server 260 may be any type of server computing device available in the cloud and may be configured as one or more virtual machines running on one or more hosts. For example, game server 260 may manage a virtual machine that supports a game processor that instantiates instances of a user's game. Thus, multiple game processors of game server 260 associated with multiple virtual machines are configured to execute multiple instances of one or more games associated with multiple users' gameplay. As such, the backend server support provides streaming of gameplay media (eg, video, audio, etc.) of multiple game applications to corresponding multiple users.
That is, game servers 260 are configured to stream data (eg, rendered images and/or frames of corresponding gameplay) back to corresponding client devices 210 over network 250 . As such, a computationally complex game application can continue to run on the backend server in response to controller input received and forwarded by the client device 210 . Each server can render images and/or frames, then encode (eg, compress) them and stream them to corresponding client devices for display.

For example, multiple users may access cloud gaming network 290 via communication network 250 using corresponding client devices 210 configured to receive streaming media. In one embodiment, client device 210 interfaces with a backend server (eg, game server 260 of cloud gaming network 290) configured to provide computing functionality (eg, including game title processing engine 211). It can be configured as a thin client that provides In another embodiment, the client device 210 may be configured with a game title processing engine and game logic for local processing of at least some of the video games generated by the video games running on the back end servers. It can be further utilized to receive streaming content or for other content provided by backend server support. For local processing, the game title processing engine includes basic processor-based functions for running the video game and services related to the video game. Game logic is stored on the local client device 210 and used to run the video game.

In particular, a corresponding user's (not shown) client device 210 may request access to a game over a communications network 250 such as the Internet and display images generated by a video game executed by a game server 260 . The images configured for rendering, and then encoded, are delivered to the client device 210 for display in association with the corresponding user. For example, a user may interact with an instance of a video game running on a game processor of game server 260 through client device 210 . More specifically, instances of video games are executed by game work processing engine 211 . Corresponding game logic (eg, executable code) 215 implementing the videogame is stored and accessible via a data store (not shown) and used to execute the videogame. The game title processing engine 211 can support multiple video games using multiple game logics, each selectable by the user.

For example, the client device 210 is configured to interact with the game title processing engine 211 associated with the corresponding user's gameplay, such as via input commands used to drive the gameplay. In particular, client device 210 may receive input from various types of input devices such as game controllers, tablet computers, keyboards, gestures captured by video cameras, mice, touch pads, and the like. Client device 210 can be any type of computing device having at least a memory and a processor module and can be connected to game server 260 via network 250 .
Backend game title processing engine 211 is configured to generate rendered images, which are delivered over network 250 for display on corresponding displays associated with client devices 210 . For example, via a cloud-based service, game-rendered images may be delivered by instances of corresponding games running on game execution engine 211 of game server 260 . That is, client device 210 receives encoded images (eg, encoded from game-rendered images generated through execution of a video game) and displays rendered images for display 11 . Configured. In one embodiment, display 11 includes an HMD (eg, displays VR content). In some embodiments, rendered images can be streamed wirelessly or wired to a smartphone or tablet directly from a cloud-based service or via a client device 210 (e.g., PlayStation Remote Play). can be done.

In one embodiment, game server 260 and/or game title processing engine 211 include basic processor-based functionality for performing services associated with games and game applications. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. is included. In addition, gaming application services include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, managing social friends, communicating with social network friends, communication channels, texting, instant messaging. , chat support, etc.

In one embodiment, cloud gaming network 290 is a distributed gaming server system and/or architecture. Specifically, distributed game engines that execute game logic are configured as corresponding instances of corresponding games. In general, a distributed game engine captures the functions of the game engine and distributes those functions for execution by multiple processing entities. Individual functions may also be distributed across one or more processing entities. A processing entity may be configured in various configurations, such as physical hardware, and/or virtual components or machines, and/or virtual containers. A container is different from a virtual machine because it virtualizes an instance of a game application running on a virtualized operating system.
The processing entity may utilize and/or rely on the servers and underlying hardware on one or more servers (computation nodes) of the cloud gaming network 290, which may be arranged in one or more racks. can be placed on. Coordination, allocation and management of the execution of their functions to the various processing entities is done by the distributed synchronization layer. As such, the execution of those functions is controlled by the distributed synchronization layer, enabling media (e.g., video frames, audio, etc.) to be generated for the game application in response to controller input by the player. . The distributed synchronization layer distributes critical game engine components/functions (e.g., via load balancing) across distributed processing entities so that they can be distributed and restructured for more efficient processing. It can be executed efficiently.

Game title processing engine 211 includes a central processing unit (CPU) and graphics processing unit (GPU) group configured to perform multi-tenancy GPU functionality. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on corresponding CPUs.

FIG. 2B is a diagram for providing a game between two or more peer devices, and according to one embodiment of the present disclosure, VSYNC signals are synchronized and offset to synchronize controller and other information between devices. Optimal timing of reception can be achieved. For example, face-to-face games may be performed using two or more peer devices directly connected via network 250 or via peer-to-peer communications (eg, Bluetooth®, local area networking, etc.). obtain.

As shown, the game is running locally on each of the corresponding user's client devices 210 (e.g., game consoles) playing the video game, which client devices 210 communicate via peer-to-peer networking. . For example, a video game instance is being executed by the game title processing engine 211 of the corresponding client device 210 . Game logic 215 (eg, executable code) implementing the video game is stored on the applicable client device 210 and used to execute the game. By way of example, game logic 215 may be delivered to the applicable client device 210 via portable media (eg, optical media) or via a network (eg, downloaded from a game provider via the Internet).

In one embodiment, the game title processing engine 211 of the corresponding client device 210 includes basic processor-based functionality for performing services associated with games and game applications. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformations, artificial intelligence, etc. is included. In addition, gaming application services include memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, managing social friends, communicating with social network friends, communication channels, texting, instant messaging. , chat support, etc.

Client device 210 can receive input from various types of input devices such as gestures captured by game controllers, tablet computers, keyboards, video cameras, mice, touch pads, and the like. Client device 210 can be any type of computing device having at least a memory and processor module that generates rendered images for execution by game title processing engine 211 and displays the rendered images on a display (e.g. , display 11, or head-mounted display—display 11 including HMD, etc.). For example, a rendered image may be an instance of a game running locally on the client device 210 to implement the corresponding user's gameplay, such as through input commands used to drive the gameplay. can be associated with Some examples of client devices 210 are personal computers (PCs), game consoles, home theater devices, general purpose computers, mobile computing devices, tablets, phones, or any other type capable of running an instance of a game. of computing devices.

FIG. 2C illustrates various network configurations that benefit from proper synchronization and offset of VSYNC signals between source and target devices, including the configurations shown in FIGS. 2A-2B, according to embodiments of the present disclosure. . In particular, various network configurations require proper alignment of the frequencies of the server and client VSYNC signals, and the server and client VSYNC signals, in order to reduce one-way latency and/or latency variations between the server and client. benefits from the timing offset of For example, one network device configuration includes a cloud gaming server (eg, source) to client (target) configuration. In one embodiment, the client may include a web RTC client configured to provide audio and video communication within a web browser. Another network configuration includes a client (eg, source) to server (target) configuration. Yet another network configuration includes a server (eg, source) to server (eg, target) configuration. Another network device configuration includes a client (eg, source) to client (target) configuration, where each client may be, for example, a game console for providing face-to-face gaming.

In particular, alignment of the VSYNC signals includes frequency synchronization of the server VSYNC signal and the client VSYNC signal for purposes of removing drift and/or to maintain an ideal relationship between the client VSYNC signal and the server VSYNC signal. It may also include adjusting the timing offset between the client VSYNC signal and the server VSYNC signal in order to reduce one-way latency and/or latency variation. To achieve proper alignment, in one embodiment, the server VSYNC signal may be adjusted to enforce proper alignment between the server 260 and client 210 pair.
In another embodiment, the client VSYNC signal can be adjusted to enforce proper alignment between the server 260 and client 210 pair. When the client and server VSYNC signals are aligned, the server VSYNC signal and the client VSYNC signal occur at substantially the same frequency and are offset from each other by a timing offset. The timing offset can be adjusted at any time.
In another embodiment, the alignment of the VSYNC signals can also synchronize the frequency of the VSYNCs of two clients, adjust the timing offset between their VSYNC signals for the purpose of removing drift, and/or Either VSYNC signal can be adjusted to achieve this alignment, which may include achieving optimal timing reception of controller and other information.
In yet another embodiment, the alignment can also synchronize the frequency of VSYNC of multiple servers, and synchronize the frequencies of the server VSYNC signal and the client VSYNC signal, for example for face-to-face cloud gaming. , adjusting the timing offset between the client VSYNC and server VSYNC signals. In server-to-client and client-to-client configurations, adjustments include synchronizing the frequency between the server VSYNC signal and the client VSYNC signal, and providing an appropriate timing offset between the server VSYNC signal and the client VSYNC signal. may include both. In a server-to-server configuration, alignment may include frequency synchronization between the server VSYNC signal and the client VSYNC signal without setting a timing offset.

FIG. 2D illustrates the communication between cloud gaming server 260 and one or more clients 210 benefiting from proper synchronization and offset of VSYNC signals between source and target devices, according to one embodiment of the present disclosure. Shows a multi-tenancy configuration. In a server-to-client configuration, coordination may include both synchronizing the frequency between the server and client VSYNC signals and providing an appropriate timing offset between the server and client VSYNC signals. be. In a multi-tenancy configuration, in one embodiment, the client VSYNC signal is coordinated at each client 210 to implement proper alignment between the server 260 and client 210 pair.

For example, the graphics subsystem may be configured to perform multi-tenancy GPU functions, and in one embodiment, one graphics subsystem provides graphics and/or rendering pipes for multiple games. line can be implemented. That is, the graphics subsystem is shared between multiple games running. In particular, the game title processing engine may also include CPU and GPU groups configured to perform multi-tenancy GPU functionality, and in one embodiment, one CPU and GPU group may provide graphics processing for multiple games. and/or a rendering pipeline. That is, the CPU and GPU groups are shared between multiple games being run. The CPU and GPU groups can be configured as one or more processing devices. In another embodiment, multiple GPU devices are combined to perform graphics processing for a single application running on corresponding CPUs.

FIG. 3 illustrates the general process of running a video game on a server to generate game-rendered video frames and sending those video frames to a client for display. Conventionally, some operations on game server 260 and client 210 are performed within frame periods defined by their respective VSYNC signals. For example, server 260 attempts to generate game-rendered video frames at 301 for one or more frame periods, as defined by corresponding server VSYNC signal 311 . The video frames are generated by the game in response to either control information delivered from the input device at operation 350 (eg, user input commands) or game logic not driven by control information. Transmission jitter 351 can be present when sending control information to server 260, and jitter 351 measures the variation in network latency from the client to the server (eg, when sending input commands).
As shown, the thick arrows indicate the current delay in sending control information to server 260, but due to jitter, the range of arrival times of control information at server 260 (e.g. range enclosed by arrows). At flip time 309 , the GPU reaches a flip command indicating that the corresponding video frame has been completely generated and placed in the frame buffer of server 260 . Server 260 then scans out/scans in (operation 302 , scan out may be aligned with VSYNC signal 311 ) for that video frame over subsequent frame periods defined by server VSYNC signal 311 . Execute (VBI omitted for clarity). The video frames are then encoded (operation 303) (eg, encoding begins after the occurrence of VSYNC signal 311 and the end of encoding may not be aligned with the VSYNC signal) and sent to client 210 ( Operation 304, the transmission may not be aligned with the VSYNC signal 311).
At client 210, encoded video frames are received (operation 305, reception may not be aligned with client VSYNC signal 312) and decoded (operation 306, decoding may not be aligned with client VSYNC signal 312). ), buffered and displayed (operation 307, start of display may be aligned with the client's VSYNC signal 312). In particular, client 210 displays each video frame rendered for display beginning with the corresponding occurrence of client VSYNC signal 312 .

One-way latency 315 can be defined as the latency from the start of transfer of a video frame to an encoding unit (eg, scanout 302 ) at the server to the start of display of the video frame at client 307 . That is, one-way latency is the time from server scanout to client display, accounting for client buffering. Each frame has latency from the start of scanout 302 to the completion of decoding 306, encoding 303 and transmission 304, network transmission between server 260 and client 210 with attendant jitter 352, and client reception. Due to the high degree of variability in the operation of servers such as 305, it may vary from frame to frame. As shown, the straight thick arrows indicate the current latency in sending the corresponding video frames to client 210, but due to jitter 352, the range of video frame arrival times at client 210 ( For example, the range bounded by the dashed arrow).
Buffering 320 is conventionally performed because one-way latency must be relatively stable (e.g., fairly consistent) for a good playing experience, resulting in , the low-latency display of individual frames (eg, from the start of scanout 302 to the completion of decoding 306) is delayed by a small frame period. This means that if the network is unstable, or if unpredictable encoding/decoding times occur, additional buffering is required to keep the one-way latency constant.

According to one embodiment of the present disclosure, one-way latency between a cloud gaming server and a client is due to clock drift when streaming video frames generated from a video game running on the server. , can vary. That is, the client VSYNC signal may drift with respect to frames arriving from server 260 due to the difference in frequency between server VSYNC signal 311 and client VSYNC signal 312 . Drift can be caused by minute differences in the crystal oscillators used by the server and client clocks. Further, embodiments of the present disclosure provide dynamic buffering for the client by performing one or more synchronizations and offsets of the VSYNC signal for alignment between the server and the client, and One-way latency is reduced by duplicating the decoding and presentation of video frames at the client.

FIG. 4 illustrates a highly optimized cloud gaming server 260 and a highly optimized client 210 when streaming video frames generated from a video game running on the server, according to an embodiment of the present disclosure. , overlaps server and client operations to reduce one-way latency, synchronizes and offsets the VSYNC signal between server and client, and eliminates one-way latency Not only is the time reduced, but the one-way latency variation between server and client is also reduced. In particular, FIG. 4 shows the desired alignment between the server and client VSYNC signals.
In one embodiment, adjustment of the server VSYNC signal 311 is performed to obtain proper alignment between the server and client VSYNC signals, such as in server and client network configurations. In another embodiment, adjustment of the client VSYNC signal 312 is performed to obtain proper alignment between the server and client VSYNC signals, such as from a multi-tenant server to a multi-client network configuration. For purposes of illustration, the adjustment of the server's VSYNC signal 311 is described in FIG. This is for the purpose of synchronizing the frequency of the server and client VSYNC signals and/or adjusting the timing offset between the corresponding client and server VSYNC signals, although it is understood that the client VSYNC signal 312 can also be used for adjustment. be.

As shown, FIG. 4 illustrates an improved process for running a video game on a server to generate rendered video frames and sending those video frames to a client for display in an embodiment of the present disclosure. indicate. This process is illustrated for the generation and display of a single video frame on the server and client. In particular, the server generates 401 game-rendered video frames. For example, server 260 includes a CPU (eg, game title processing engine 211) configured to execute games. The CPU generates one or more draw calls for a video frame, the draw calls containing commands placed in a command buffer for execution by the corresponding GPU of server 260 in the graphics pipeline. The graphics pipeline can include one or more shader programs at the vertices of objects in the scene to generate texture values that are rendered into video frames for display. In this case, the operations are executed in parallel via the GPU for efficiency. At flip time 409 , the GPU reaches a command buffer flip command indicating that the corresponding video frame has been completely generated and/or rendered and placed in the frame buffer of server 260 .

At 402, the server performs a scanout of the game-rendered video frames to the encoder. In particular, scan-out is performed scanline-by-scanline or in groups of consecutive scanlines, where a scanline refers to a single horizontal line of the display, eg, from screen edge to screen edge. These scanlines or groups of consecutive scanlines are sometimes referred to as slices and are referred to herein as screen slices. In particular, scanout 402 performs several processes that modify a game rendering frame, including overlaying it with another framebuffer or shrinking it to surround it with information from another framebuffer. can contain. During scanout 402, the modified video frames are then scanned into the encoder for compression. In one embodiment, scanout 402 is performed at occurrence 311 a of VSYNC signal 311 . In other embodiments, scanout 402 may be performed prior to generation of VSYNC signal 311 , such as at flip time 409 .

At 403, the game-rendered video frame (possibly modified) is encoded encoder slice by encoder slice at an encoder to generate one or more encoded slices. In this case, the coded slices are independent of scanlines or screen slices. As such, the encoder produces one or more encoded (eg, compressed) slices.
In one embodiment, the encoding process begins before the corresponding video frame scanout 402 process is completely finished. Additionally, the start and/or end of encoding 403 may or may not be aligned with server VSYNC signal 311 . A coded slice boundary is not limited to a single scanline, but may consist of a single scanline or multiple scanlines. Moreover, the end of an encoded slice and/or the start of the next encoder slice do not necessarily occur at the edges of the display screen (e.g., they can occur anywhere in the middle of the screen or in the middle of a scanline). ), the encoded slice need not traverse completely across the display screen. As shown, one or more encoded slices may be compressed and/or encoded, including compressed "encoded slice A" with hash marks.

At 404, the encoded video frames are transmitted from the server to the client, where the transmission may occur on an encoded slice by encoded slice basis, where each encoded slice is a compressed encoder slice. In one embodiment, the transmission process 404 begins before the encoding process 403 of the corresponding video frame is completely finished. Further, the start and/or end of transmission 404 may or may not be aligned with server VSYNC signal 311 . As shown, compressed encoded slice A is sent to the client independently of other compressed encoder slices of the rendered video frame. Encoder slices can be sent one at a time or in parallel.

At 405, the client receives the compressed video frames, re-encoded slice by slice. Further, the start and/or end of receive 405 may or may not be aligned with client VSYNC signal 312 . As shown, compressed encoded slice A is received by the client. Transmission jitter 452 can exist between server 260 and client 210 , jitter 452 measures variations in network latency from server 260 to client 210 . The lower the jitter value, the more stable the connection. As shown, the thick straight arrows indicate the current latency in sending the corresponding video frames to the client 210, but due to jitter, the range of arrival times of the video frames at the client 210 (e.g. , ranges delimited by dotted arrows). Variation in latency can also be due to one or more operations at the server, such as encoding 403 or transmitting 404, and network problems that introduce latency when transmitting video frames to client 210. FIG.

At 406, the client decodes the compressed video frame, again encoded slice by slice, producing decoded slice A (shown without hash marks), which is now ready for display. there is In one embodiment, decoding process 406 begins before receiving process 405 is completely completed for the corresponding video frame. Further, the start and/or end of decoding 406 may or may not be aligned with client VSYNC signal 312 . At 407, the client displays the decoded rendered video frames on the client's display. That is, the decoded video frames are placed, for example, in a display buffer that is streamed to the display device scanline by scanline.
In one embodiment, the display process 407 (i.e., streaming out to the display device) is performed after the decoding process 406 is completely finished for the corresponding video frame, i.e., the decoded video frame is fully in the display buffer. Start after resident. In another embodiment, display process 407 begins before decoding process 406 is completely finished for the corresponding video frame. That is, streaming out to the display device begins at an address in the display buffer at a time when only a portion of the decoded framebuffer resides in the display buffer. The display buffer is then updated or filled with the remaining portions of the corresponding video frame in time for display, so that updating of the display buffer is performed before those portions are streamed out to the display. Additionally, the start and/or end of display 407 is aligned with client VSYNC signal 312 .

In one embodiment, one-way latency 416 between server 260 and client 210 may be defined as the elapsed time from when scanout 402 is initiated to when display 407 is initiated. Embodiments of the present disclosure align VSYNC signals (e.g., synchronize frequencies and adjust offsets) between servers and clients to reduce one-way latency between servers and clients, and the one-way variation in latency between clients. For example, embodiments of the present disclosure can compute an optimal adjustment to the offset 430 between the server VSYNC signal 311 and the client VSYNC signal 312, resulting in server processing such as encoding 403 and transmission 404. When near worst case time required, near worst case network latency between server 260 and client 210, and near worst case client processing such as receive 405 and decode 406. However, the decoded rendered video frames are available in time for display process 407 . That is, there is no need to determine the absolute offset between server VSYNC and client VSYNC. It is sufficient to adjust the offsets so that the decoded rendered video frames are in time for the display process.

In particular, the frequencies of server VSYNC signal 311 and client VSYNC signal 312 may be aligned by synchronization. Synchronization is accomplished by coordinating the server's VSYNC signal 311 or the client's VSYNC signal 312 . For purposes of explanation, coordination is described in relation to the server's VSYNC signal 311, but it is understood that coordination may alternatively be performed on the client's VSYNC signal 312. FIG. For example, as shown in FIG. 4, a server frame period 410 (eg, the time between two occurrences 311c and 311d of server VSYNC signal 311) corresponds to a client frame period 415 (eg, two occurrences 312a of client VSYNC signal 312). and 312b). This indicates that the frequencies of the server's VSYNC signal 311 and the client's VSYNC signal 312 are also substantially equal.

The timing of the server's VSYNC signal 311 can be manipulated to maintain frequency synchronization of the server and client VSYNC signals. For example, the vertical blanking interval (VBI) of server VSYNC signal 311 may be increased or decreased over a period of time, eg, to account for drift between server VSYNC signal 311 and client VSYNC signal 312 . Manipulation of the vertical blanking (VBLANK) line in VBI provides for adjusting the number of scanlines used for VBLANK for one or more frame periods of the server VSYNC signal 311 . Reducing the number of VBLANK scanlines reduces the corresponding frame period (eg, time interval) between two occurrences of the server VSYNC signal 311 . Conversely, increasing the number of VBLANK scanlines increases the corresponding frame period (eg, time interval) between two occurrences of VSYNC signal 311 .
In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies between the client and server VSYNC signals 311 and 312 to be substantially the same frequency. Also, the offset between the server and client VSYNC signals can be adjusted by briefly increasing or decreasing the VBI before returning it to its original value. In one embodiment, the server VBI is adjusted.
In another embodiment, the client VBI is adjusted. In yet another embodiment, instead of two devices (server and client), there may be multiple connected devices, each of which may have its corresponding VBI adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (eg, without a server device). In another embodiment, the plurality of devices is one or more server devices arranged in one or more server/client architectures, multi-tenant server/client(s) architecture, or some combination thereof. and/or may include one or more client devices.

Alternatively, in one embodiment, the server's pixel clock (eg, located in the southbridge of the server's northbridge/southbridge core logic chipset) is manipulated to approximate the frequency of the server's VSYNC signal 311 over a period of time. Adjustments and/or fine-tuning may be performed to bring the frequency back into alignment between the server and client VSYNC signals 311 and 312 . Specifically, the server's southbridge pixel clock may be overclocked or underclocked to adjust the overall frequency of the server's VSYNC signal 311 . In this way, the frequency of the server VSYNC signal 311 is adjusted to align the frequencies between the client and server VSYNC signals 311 and 312 to be substantially the same frequency. The offset between VSYNC on the server and client can be adjusted by briefly increasing or decreasing the pixel clock of the client server before resetting the pixel clock of the client server to its original value.
In one embodiment, the server pixel clock is adjusted. In another embodiment, the client pixel clock is adjusted. In another embodiment, the client pixel clock is adjusted. In yet another embodiment, instead of two devices (a server and a client), there are multiple connected devices, each of which may have their corresponding pixel clocks adjusted. In one embodiment, each of the multiple connected devices may be an independent peer device (eg, without a server device). In another embodiment, the plurality of connected devices is one or more arranged in one or more server/client architectures, multi-tenant server/client(s) architecture, or some combination thereof. server device and one or more client devices.

FIG. 5A illustrates the drift 390 between the clocks of each of the cloud gaming servers 260 and the clients 210, as well as variations in time spent by server operations such as encoding 403 and transmitting 404, network wait, according to one embodiment of the present disclosure. Possible variations in the timing of completion of decoding 406 by client 210 when streaming video frames generated from a video game executed on server 260 due to time and client operations such as receiving 405 and decoding 406. It is a figure which shows.

The y-axis 501 indicates time in milliseconds. The x-axis 502 indicates time in minutes. In embodiments of the present disclosure, the server 260 may send the timestamp information to the client 210 along with the compressed video frames, or the server may send the timestamp information separately from the compressed video frames.
In one embodiment, this timestamp information may represent the time of occurrence of the server VSYNC signal just prior to scanout 402 of the corresponding video frame, derived from the server 260 pixel clock. That is, the timestamp indicates the desired display timing of the corresponding video frame, such as whether the video frame is to be displayed immediately.
In another embodiment, this timestamp information may represent a flip time, eg, the time of completion of rendering of the corresponding video frame, derived from the pixel clock of server 260 . In yet another embodiment, the normal frame period is used in place of the server VSYNC signal, and this timestamp information represents the start or end time of the corresponding frame period derived from the server 260 pixel clock. can be done.

Upon completion of decoding 406, client 206 records a time derived from the client's pixel clock and subtracts this time from the timestamp (delivered by the server) to obtain a "decryption timestamp" (i.e., the corresponding video time to complete the decoding, not the time it takes to decode the frame). The decode timestamps thus indicate the availability of the corresponding video frames for display at the client compared to the desired display times specified by the server (eg, indicated by the timestamps). As shown in FIG. 5A, in one embodiment, the decode timestamp of the first compressed video frame received by the client is assigned a value of zero (ie, normalized).
Also, all other decoding timestamps are calculated with reference to normalization (ie subtracted and normalization taken into account). Due to variability in server and client operation, and network latency, the decode timestamps measured for a series of compressed video frames may differ from the first compressed video frame 511 having a zero decode timestamp, as described above. It may be plotted as assigned distribution 510 . These decoding timestamps may be binned to create a histogram 515 as shown in FIG. 5B, which is described more fully below. Measurements 520, 530, and 540 show the distribution of decode timestamps computed for subsequent video frames received at the client. Decoding timestamps for measurements 520 , 530 and 540 are calculated with reference to the normalization defined by first compressed video frame 511 .

FIG. 5B includes a histogram showing the timing (e.g., corresponding timestamps) of completion of decoding by the client relative to the desired display times specified by the server, according to one embodiment of the present disclosure; The increase in decoding timestamps in subsequent histograms due to drift between respective clocks is shown. In particular, the decoding timestamps of each of the measurements 520, 530, and 540, determined later for subsequent video frames, may be binned to create a corresponding histogram. For example, the decoding timestamps plotted in measurements 520 may be binned to create histogram 525 . Similarly, the decoding timestamps plotted in measurement 530 may be binned to create histogram 535 and the decoding timestamps plotted in measurement 540 may be binned to create histogram 545. Each of histograms 515, 525, 535, and 545 may have unique characteristics. As shown, the width of the distribution of the decoding timestamps in each of the measurements 515, 525, 535, and 545 is approximately similar, but the later determined histogram was used to generate the VSYNC frequencies. show the decoding timestamps increasing as they are each shifted to the right, due to drift between the server and client clocks, etc.

In particular, decoding timestamp measurements 520, 530, and 540 (and their corresponding histograms 525, 535, and 545 shown in FIG. It shows the effect of the difference between the client VSYNC signal 312 (ie drift 390 in FIG. 3). A drift 390 between the server clock and the client clock can be shown as 590 in FIG. 5B as reflected in the corresponding VSYNC signal. For example, a 10 ppm (parts per million) difference in the server and client clocks due to the inaccuracy of the crystal oscillators used to generate the VSYNC frequencies on the server and client will result in a difference of 1 part per million between the server and client approximately every 30 minutes. This results in a frame period (16.7 ms) drift. Drift can take the form of a decrease or increase in decoding timestamps, depending on whether the server's VSYNC 311 is at a slightly higher or lower frequency than the client VSYNC signal 312 .

FIG. 5C includes a histogram showing the timing of completion of decoding by the client relative to the desired display time specified by the server, measured between respective clocks at the cloud gaming server and the client, according to one embodiment of the present disclosure. After compensating for drift, the subsequent histograms show consistent measurements of decoding time. In one embodiment, the amount of drift can be calculated by dynamically regenerating the histogram. Knowing the amount of drift allows corresponding adjustment of the VSYNC frequency and removal of the drift, resulting in histograms of 516, 526, 536, 546, etc. over time. As such, the plotted measurements 520, 530, and 540 do not show an increase in decoding timestamps (e.g., shift vertically upward along the x-axis 501 in FIG. 5A), but in FIG. , are plotted in more horizontal alignment with the measurements 510 (ie, no increase in one-way latency). This is reflected in FIG. 5C. In this figure, line 590' shows zero drift between the server clock and the client clock, reflected in the corresponding VSYNC signal (i.e., an increase in one-way latency in histograms 526, 536, and 546). not shown).

In another embodiment, the server transmits a compressed video frame every frame period (where the frame periods are of approximately equal length) and the corresponding server timestamp (e.g., used in the decoding timestamp calculation) is Instead of being sent from the server to the client, the corresponding server timestamp (for the corresponding video frame being received) is instead a previously calculated server timestamp (e.g. the previous received calculated by the client by adding the frame duration to the video frame). An initial server timestamp of an initial video frame and/or a timing signal may be delivered from the server to the client to initiate the timing process.

In yet another embodiment, the drift is determined, for example, by analyzing the variance between the timestamp information and the timing of receipt of the timestamp information at the client, separately or together with the compressed video frames, from the server to the client. Calculated using timestamp information sent to In yet another embodiment, instead of using the server VSYNC signal, the server can use frame periods of approximately equal size and calculate the drift of the server frame period relative to the client VSYNC (or several multiples thereof). . Therefore, the server frame period can be adjusted according to the drift calculation.

In other embodiments, instead of two devices (server and client), there are multiple connected devices, each of which measures their drift relative to one or more other devices. can be done. In one embodiment, each of the multiple connected devices may be an independent peer device (eg, without a server device). In another embodiment, the plurality of devices is one or more server devices arranged in one or more server/client architectures, multi-tenant server/client(s) architecture, or some combination thereof. and one or more client devices.

Therefore, using the measured drift between frequencies of the VSYNC signal between two devices (e.g., a server and a client, or any two devices in a plurality of network devices with server, client, and independent peers) can be used to condition the VSYNC signal to one or more devices. Adjustments may include removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period. Adjustments may also include overclocking or underclocking the corresponding clocks. The adjustment can be performed dynamically, with the corresponding adjustment of the VSYNC signal changing over time. In some embodiments, instead of using the VSYNC signal, the server may use frame periods of approximately equal size, and the server frame period may be adjusted in response to calculating drift.

Further, alignment of the server and client VSYNC signals as shown in FIG. 4 may include adjusting the server VSYNC signal 311 to provide an appropriate timing offset 430 between the corresponding client and server VSYNC signals. . In one embodiment, histogram data (eg, shown in at least FIGS. 5A-5C) is used to ensure that a predetermined number or threshold (eg, 99.99 percent) of received video frames arrive at the client in time. Additionally, a timing offset can be determined and displayed at the next appropriate occurrence of the client VSYNC signal 312 .
That is, the timing offset 430 (eg, shown in FIG. 4) is such that the video frame is ready for display (eg, received and decoded) in time for the minimum time to the next occurrence of the client VSYNC signal 312. and placed in the display buffer for streaming out at the client) to accommodate near-worst-case scenarios for video frames. As an example, a threshold of 99.99 percent provides 1 missing video frame out of 10,000 video frames generated at server 260 and displayed at client 210 .

In particular, to establish the appropriate offset 430, the frequency of server VSYNC signal 311 is manipulated over a period of time (eg, one or more frame periods) such that one or more of the corresponding server VSYNC signal 311 By moving the timing of occurrence so that both VSYNC signals synchronize their respective frequencies, the relative timing offset between the client and server VSYNC signals can be shifted or adjusted. The frequency of the client VSYNC signal 312 can be similarly manipulated to alternatively adjust the relative timing offset between the client and server VSYNC signals. Determining the appropriate offset can be performed dynamically, eg, iteratively over time, using corresponding dynamic manipulation of the VSYNC signal.
In other embodiments, rather than first removing the drift between the server VSYNC signal and the client VSYNC signal and then establishing a suitable offset between the VSYNC signals, instead the server VSYNC signal or the client VSYNC signal The offset is maintained by manipulating the frequency of more frequently to adjust the relative timing offset. In yet another embodiment, instead of using the server VSYNC signal, the server uses frame periods of approximately equal size and manipulates the server frame period or the client VSYNC signal such that for the client VSYNC (or multiples thereof): A relative timing offset of the server frame period can be adjusted.
In still other embodiments, instead of two devices (a server and a client), there are multiple connected devices, each of which manipulates its own VSYNC signal to cause one or more other devices to A relative timing offset of VSYNC can be adjusted with respect to VSYNC of . In one embodiment, each of the multiple connected devices may be an independent peer device (eg, without a server device). In another embodiment, the plurality of devices is one or more server devices arranged in one or more server/client architectures, multi-tenant server/client(s) architecture, or some combination thereof. and one or more client devices.

Flow diagrams 600A, 600B, and 600C of FIGS. 6A-6C, along with a detailed description of various client devices 210 and/or cloud gaming networks 290 (eg, within game server 260) of FIGS. 2A-2D, illustrate one implementation of the present disclosure. FIG. 10 illustrates a method for coordinating VSYNC signals between a cloud gaming server and a client for the purpose of reducing one-way latency, according to aspects. FIG.

In particular, FIG. 6A illustrates a method for adjusting relative timing between cloud gaming server and client VSYNC signals for the purpose of reducing one-way latency, according to one embodiment of the present disclosure. For example, flowchart 600A may be executed to adjust the timing between the corresponding client and server VSYNC signals shown in FIG.

At 601, the method includes setting a server VSYNC signal to a server VSYNC frequency at the server. As previously mentioned, the server VSYNC signal corresponds to the generation of video frames at the server during frame periods of the server VSYNC frequency. For example, a server runs a video game in streaming mode, and the server's CPU responds to input commands from a user to generate game-rendered video frames using the graphics pipeline enabled for streaming. A video game can be played in response.

At 603, the method includes, at the client, setting a client VSYNC signal to a client VSYNC frequency. The client VSYNC signal is used for rendering to the display associated with the client. That is, the timing of rendering and displaying video frames on the client can be based on the client VSYNC signal. For example, a video frame can be displayed beginning with the occurrence of the corresponding client VSYNC signal.

In one embodiment, the client VSYNC frequency is set approximately to the server VSYNC frequency. For example, the server can send a control signal to the client, which is used by the client to set the client's VSYNC signal to the apparent frequency of the server's VSYNC signal. That is, the control signal may include the apparent frequency at which the client VSYNC signal is set. That is, the client VSYNC frequency is set to the same apparent frequency as the server VSYNC frequency, but due to differences in the crystals used to clock the server and client, the actual server and client VSYNC frequencies may not match. There is

At 605, the method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over the network using the server VSYNC signal. In particular, game-rendered video frames generated in response to processing a video game by a server in streaming mode are delivered to an encoder configured to perform compression and generate a plurality of compressed video frames. (eg, while being scanned out 402). As previously mentioned, the start of encoding of the corresponding video frame may be aligned with the corresponding occurrence of the server VSYNC signal, or may occur before the corresponding occurrence, such as flip time. Compressed video frames are transmitted and/or delivered to the client for display during the game session. Transmission of a video frame need not start aligned with the server VSYNC signal, but can start as soon as a portion of the corresponding video frame or the complete video frame is encoded, as shown in FIG. can.

At 607, the method includes decoding and displaying a plurality of compressed video frames at the client. As mentioned above, the client receives multiple compressed video frames, which are then decoded by the client's decoder. For example, a client receives one or more encoded slices of corresponding compressed video frames. The compressed video frames are then decoded and placed in the display buffer. For example, a corresponding encoded slice of the compressed video frame is then decoded and the decoded video frame is placed in a display buffer. During decoding, the decoded slices can be rendered for display, rendering includes generating screen slices (e.g., scanlines) from the decoded slices of the corresponding video frames, which is It is then streamed to the client's display. In particular, the pixel data of the decoded slices of the corresponding video frames can be placed at appropriate addresses in the display buffer for streaming (eg, scanline by scanline) to the client's display.

At 610, the method analyzes the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives multiple compressed video frames. including doing. For example, to reduce one-way latency between server and client, and to reduce one-way latency variation, proper alignment (e.g., frequency synchronization and offset adjustment) between VSYNC signals of server and client Relative timing is adjusted for implementation. In one embodiment, proper timing between the server and client VSYNC signals is achieved by coordinating at least one of the server and client VSYNC signals. A more detailed discussion of adjusting the relative timing between the server and client VSYNC signals is provided in FIGS. 6B-6C below.

FIG. 6B is a flow diagram 600B illustrating a method for aligning VSYNC signals when performing VSYNC signal coordination between a cloud gaming server and a client, according to one embodiment of the present disclosure. Alignment includes, for example, synchronizing frequencies and/or adjusting offsets between server and client VSYNC signals, such as to reduce one-way latency. In particular, FIG. 6B provides additional detail regarding the relative timing adjustments between the server and client VSYNC signals outlined in operation 610 of FIG. 6A.

At 611, the method includes transmitting timestamp information associated with a plurality of video frames (eg, generated by the server as game-rendered video frames) from the server to the client. In one embodiment, the timestamp information is sent to the client along with the multiple compressed video frames. In another embodiment, the timestamp information is sent to the client separately from the multiple compressed video frames.

In particular, the timestamp information includes the corresponding timestamp of the corresponding video frame generated at the server as derived by the server's pixel clock. In one implementation, the timestamp of the corresponding video frame is used to scan the corresponding video frame into the encoder, such as during scanout (e.g., the occurrence of the server VSYNC signal just prior to scanout of the corresponding video frame). may occur on the corresponding occurrence of the server VSYNC signal to be executed. In another implementation, the corresponding video frame timestamps may occur at the corresponding flip times at the server. The timestamp indicates the desired display timing of the corresponding video frame determined by the video game, such as scanning or streaming to a local display, without transmission over the network.

When a client receives and decodes a compressed video frame, the timestamp information is processed to indicate the client's availability for display of the corresponding video frame compared to the desired display time (timestamp) specified by the server. A decryption timestamp is created. As previously explained, the client records the time derived from the client's pixel clock upon completion of decoding of the corresponding video frame. This decryption completion time is subtracted from the corresponding timestamp delivered by the server to produce the decryption timestamp. In addition, the first compressed video frame to be decoded uses a normalization factor that can be applied (e.g., added or subtracted) to all subsequently measured and/or calculated decoding times so that its decoding timestamp is It can be normalized to be adjusted to zero, after which the client can receive the compressed video frame.

At 613, the method includes constructing one or more histograms based on the measured and/or calculated decoding timestamps. In particular, a corresponding histogram is created by binning the decode timestamp information measured for compressed video frames received at the client over a period of time. As noted above, the decoded timestamps measured and normalized to the corresponding video frames compare when the decoded video frames are displayable on the client compared to the desired display time indicated by the server timestamps. indicates that there is In this way, the corresponding histogram provides the distribution of the timing of completion of decoding by the client relative to the desired display times specified by the server for multiple video frames. Therefore, in order to reduce the one-way latency and/or variation in one-way latency between the server and client, one or more generated histograms can be used to determine the relative timing can be adjusted.

At 615, the relative timing between the server and client VSYNC signals is adjusted to synchronize the server VSYNC frequency of the server VSYNC signal with the client VSYNC frequency of the client VSYNC signal. In one embodiment, drift is determined between the server VSYNC signal and the client VSYNC signal using corresponding histograms. For example, one or more histograms generated from video frames received by the client are continuously and dynamically updated. Analyze the histogram to determine the drift between the server VSYNC signal and the client VSYNC signal. For example, FIG. 5B shows how drift (eg, reflected by line 590) can be determined when plotting multiple histograms, as described above.

Alternative methods for determining the drift between the server and client VSYNC signals can be used for synchronization. For example, analysis of decoding timestamps over time can be performed to determine trends in increasing or decreasing decoding timing, which can be used to determine drift. In another example, drift may be calculated by analyzing the variance between server-generated timestamp information and client-based timing of receipt of the server-based timestamp information. Drift can also be measured between multiple connected devices, which can be independent peer devices, server and client devices, arranged in a peer-to-peer architecture, server/client architecture, or a combination thereof.

Further, based on timestamp information sent from the server to the client, at least one of the server VSYNC frequency and the client VSYNC frequency can be adjusted to compensate for the measured drift. For example, the frequency of the server VSYNC signal or the client VSYNC signal can be adjusted over a period of time so that the actual frequencies of the server and client VSYNC signals are approximately the same over that period. In this manner, the frequencies of the VSYNC signals of the server and client are synchronized.

As described above, the frequency of the corresponding server or client VSYNC signal can be adjusted by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, adjusting the VSYNC signal for a period of time. be able to. Adjustments may include overclocking or underclocking the corresponding pixel clocks of the server or client for a period of time. Further, adjustments can be performed continuously by dynamically adjusting the corresponding VSYNC signal appropriately over time.

At 617, the relative timing between the server and client VSYNC signals is adjusted by adjusting the relative offset between the server and client VSYNC signals based on the timestamp information. Similarly, the relative phase between the server and client VSYNC signals can be adjusted based on the timestamps. Once the frequencies of the server and client VSYNC signals are synchronized, relative phase or offset adjustments of the server and client VSYNC signals can be applied to the server or client VSYNC signals.

In particular, the adjustment of the offset between the server VSYNC signal and the client VSYNC signal is determined based on the near-worst-case decoding timestamps shown in the corresponding histograms. For example, the timing offset is such that a predetermined number or threshold (e.g., 99.99 percent) of received video frames arrive at the client within the time to be decoded and displayed at the next appropriate occurrence of the client VSYNC signal. can decide. Thus, even near-worst-case scenarios of one-way latency of video frames are taken into account when adjusting the timing offset between the server and client VSYNC signals, and near-worst-case scenario video frames are received; Decoded and paced in the display buffer for streaming out to the client display. Determining appropriate timing offsets is further described in connection with FIGS. 8A-8B.

As previously mentioned, adjusting the timing offset between the server and client VSYNC signals can be accomplished by adjusting the server VSYNC signal or the client VSYNC signal in one or more frame periods. For example, adjusting the timing offset can be performed by adjusting the frequency of the corresponding server or client VSYNC by one or more frame periods. In particular, the frequency of the corresponding server's or client's VSYNC signal can be adjusted by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the server's or client's corresponding pixel clock. Adjustable.

In some embodiments, rather than first removing the drift between the server and client VSYNC signals and then establishing a suitable offset between the VSYNC signals, instead of either the server VSYNC signal or the client VSYNC signal The timing offset is maintained by manipulating the frequency more frequently to adjust the relative timing offset. That is, the adjustment of the offset between the server VSYNC signal and the client VSYNC signal is continuously determined based on the near-worst-case decoding timestamps indicated by the corresponding histograms, which are used to determine frequent determinations of the timing offsets. It can be generated over a short period of time in operation.

FIG. 6C illustrates another method for aligning VSYNC signals when performing VSYNC signal coordination between a cloud gaming server and a client for the purpose of reducing one-way latency, according to an embodiment of the present disclosure. Figure 600C is a flow diagram illustrating a method; As previously mentioned, alignment includes frequency synchronization and/or adjustment of offsets between server and client VSYNC signals, such as to reduce one-way latency. In particular, FIG. 6C provides additional detail regarding the relative timing adjustments between the server and client VSYNC signals outlined in operation 610 of FIG. 6A.

Specifically, an alternative method for determining drift between the server and client VSYNC signals is used for synchronization, as outlined in operation 620, including operations 621 and 623. FIG. Specifically, at 621, the method includes periodically transmitting timing information from the server to the client. For example, the timing information determined from the server pixel clock includes the start of the corresponding frame period, the length of time of the corresponding frame period, the scanout timing of the video frame to the encoder, the flip time of the corresponding video frame, etc. obtain.

Also, at 623, the timing information is analyzed to determine the drift between the server VSYNC signal and the client VSYNC signal. For example, drift may be calculated by analyzing the variance between server-generated timing information and client-based timing of receipt of the server-based timing information. In other embodiments, the drift may be measured for the server frame period used to generate the video frames, where the server frame period drift is measured with reference to the client VSYNC signal, or some multiple thereof. be done. Drift can also be measured between multiple connected devices, which can be independent peer devices, server and client devices, arranged in a peer-to-peer architecture, server/client architecture, or a combination thereof.

Once the drift is determined, the frequency of the server VSYNC signal or the client VSYNC signal can be adjusted to compensate for the drift, and the adjustments can be applied over a period of time. As described above, the measured drift between the frequencies of the server and client VSYNC signals can be used to adjust the VSYNC signal at the server or client over time. Adjusting the VSYNC signal may include removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period, or overclocking or underclocking the corresponding pixel clocks of the server or client. Sometimes.

After compensating for drift between frequencies of the server and client VSYNC signals, the relative phase or offset between the server and client VSYNC signals can be adjusted based on the timestamp information, the adjustment being either the server VSYNC signal or the client VSYNC signal. applicable to signals. In particular, relative phase or offset adjustments are performed based on timestamp information associated with server-generated video frames.

Specifically, at 611, timestamp information is sent from the server to the client. In embodiments, the timestamp information is transmitted with the plurality of video frames or transmitted separately from the plurality of video frames. Operation 611 of flowchart 600C was previously described in conjunction with flowchart 600B of FIG. 6B. For example, the timestamp information determined by the server pixel clock may indicate the timing of the corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame into the encoder (eg, during scanout). In another implementation, the timestamp information may indicate the timing of the corresponding flip times of the corresponding video frames.

As mentioned above, the timestamp information is used by the client to create decoding timestamps, each of which is compared to the desired display time (timestamp) specified by the server for the client's corresponding video frame. Indicate the availability of the display in . The decoding timestamp can be derived by subtracting the time at the client indicating completion of decoding of the corresponding video frame from the corresponding server-based timestamp information. Normalization can also be applied when generating decoding timestamps.

Drift can be performed without using timestamp information, so for example to generate multiple histograms, one histogram is generated at a particular time and the used. That is, the histogram can be updated by extending the decoding timestamps contained in the histogram, but there is no need to generate multiple histograms over different time periods. Specifically, at 613-A, a histogram is created based on the decoding timestamps. Operation 613-A of flow diagram 600C is similar to operation 613 described above with respect to flow diagram 600B of FIG. 6B. For example, the decoding timestamp information is binned against the video frames received and decoded by the client over time, timing the completion of decoding by the client relative to the desired display time specified by the server (such as the server timestamp information). provides the distribution of

At 617, the relative phase or offset between the server and client VSYNC signals may be adjusted based on the timestamp information, and the adjustment may be applied to the server VSYNC signal or the client VSYNC signal. Operation 617 of flowchart 600C was previously described in conjunction with flowchart 600B of FIG. 6B. In particular, offset adjustments are determined based on near-worst-case decoding timestamps as indicated by continuously updated histograms. For example, the timing offset is such that a predetermined number of threshold (e.g., 99.99 percent) of received video frames arrive at the client within the time to be decoded and displayed at the next appropriate occurrence of the client VSYNC signal. can decide. Determining appropriate timing offsets is further described in connection with FIGS. 8A-8B.

Adjustment of the timing offset between the server and client VSYNC signals is performed by adjusting the server or client VSYNC signals by one or more frame periods. For example, adjustments may be made to one or more frames by removing or adding raster scanlines in the vertical blanking interval of the corresponding frame period, or by overclocking or underclocking the corresponding pixel clocks of the server or client. This can be done by adjusting the frequency of the server or client VSYNC signal for a period of time.

In some embodiments, drift operation 620 of FIG. 6C is not performed when establishing an appropriate offset between VSYNC signals. Instead, the timing offset is maintained by manipulating the frequency of the server VSYNC signal or the client VSYNC signal more frequently to adjust the relative timing offset. That is, adjustments to the offset between the server VSYNC signal and the client VSYNC signal are continuously determined based on the near-worst-case decoding timestamps indicated by the corresponding histograms, and adjustments to the offset are shortened. can be performed at intervals of time.

Along with a detailed description of the various client devices 210 and/or cloud gaming network 290 (eg, within game server 260) of FIGS. 2A-2D, flowchart 700 of FIG. Figure 3 shows an alternative method for reducing one-way latency to and from the client; In particular, flowchart 700 illustrates a method for adjusting a client VSYNC signal in connection with generating compressed video frames at a server, where the video frames are generated during similarly sized frame periods, according to one embodiment of the present disclosure. generated.

At 710, the method includes generating multiple video frames at the server during multiple frame periods, where the frame periods are of approximately equal size. In one embodiment, the cloud gaming server may turn off or not implement the server VSYNC signal as it does not need to be displayed on the server when the video frames are streamed to the client. Alternatively, the server may utilize regular (eg, periodic) or nearly regular signals used for timing during generation of game-rendered video frames when processing a video game. For example, instead of using the server's VSYNC signal, the server may use frame periods of approximately equal size. The generation of the multiple video frames occurs within multiple frame periods such that the game-rendered video frames are generated within the corresponding frame periods. The server runs the video game in streaming mode, and the server's CPU runs the video game in response to input commands from the user to generate game-rendered video frames using the graphics pipeline. be able to.

At 720, the method includes setting the client VSYNC signal to the client VSYNC frequency at the client. The client VSYNC signal is used for rendering to the display associated with the client. The timing of rendering and displaying video frames at the client can be referenced to the client VSYNC signal, and the corresponding video frames can be displayed starting with the occurrence of the corresponding client VSYNC signal.

At 730, the method includes transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client. In particular, game-rendered video frames are delivered to the server's encoder (eg, during scanout 402), and the encoder is configured to perform compression on the game-rendered video frames. Multiple compressed video frames are sent (eg, streamed) to a client for display, such as during a game session. The transmission of a compressed video frame need not coincide with the frame period, so that transmission can occur as soon as it is part of the corresponding video frame, or when a complete video frame has been encoded. may be done.

At 740, the method includes decoding and displaying the plurality of compressed video frames at the client. As described above, the client receives and decodes multiple compressed video frames. For example, a client can receive one or more encoded slices of a corresponding compressed video frame and decode it. The decoded video frames are placed in a display buffer. During decoding, the decoded slices can be rendered for display, rendering involves generating screen slices (e.g., scanlines) from the corresponding decoded slices of video, which are then , streamed to the client's display. For example, the pixel data of the decoded slices of the corresponding video frames may be placed at appropriate addresses in the display buffer for streaming (eg, scanline by scanline) to the display.

At 750, the method includes transmitting timing information associated with multiple frame periods from the server to the client. Timing information may indicate when each frame period begins at the server. Since the frame periods are approximately equal, the timing of one frame period delivered to the client (e.g. server's timestamp) allows the client to track the timing of each frame period (e.g. periodically to the last computed timestamp). ).
In this manner, the client can correlate timing information received from the server or calculated at the client with corresponding video frames received at the client. Client-determined timing information is the desired display timing of the corresponding video frame as determined by the video game running on the client, such as when the video frame was generated and theoretically scanned or streamed to a local display. can be given without transmission over the network.

At 760, the method analyzes the timing of one or more client operations to adjust the relative timing of the client VSYNC signals and reproduces the multiple compressed video frames at the server when the client receives the multiple compressed video frames. Including generating frames. In particular, the server frame period (eg, duration) or the client VSYNC signal can be manipulated to adjust the relative timing offset of the server frame period with respect to the client VSYNC signal (or multiples thereof).

For example, the drift between the client-calculated server frame period and the client VSYNC signal can be determined. Drift compensation can be applied to the server frame period or the client VSYNC signal for synchronization. For example, at the client, the frequency of the client VSYNC signal can be adjusted by removing or adding raster scan lines in the vertical balance interval of the corresponding frame period, and the VSYNC signal can be adjusted for a period of time. Adjustments may also include overclocking or underclocking the client's pixel clock. In addition, one or more histograms can be created on the client to adjust the relative offsets, o determining how much the decoded video frame will look like on the client compared to when the video frame was generated on the server. Indicate when available (e.g. required display time). Histograms can be constructed using the same technique as described above, with minor modifications such as using client-determined frame periods to indicate when video frames are generated and displayed.

FIG. 8A is a diagram 800A illustrating the construction and use of a histogram 850 that provides the distribution of the decoding timestamps of video frames and client display of video frames for desired display times specified by the server, as described above. A histogram is constructed to determine the adjustment of the offset between the VSYNC signals at the server and client, according to one embodiment of the present disclosure. As shown, video frames are generated at the server (operation 401) and scanout is performed on the game-rendered video frames to the encoder (operation 402) for compression (operation 403) and sent to the client. (operation 404). The client receives the encoded video frames (operation 405), decompresses the encoded video frames (operation 406), and renders the video frames for display (e.g., the decoded video frames in operation 407). convert video frames to scanlines).

As noted above, server-based timestamp information is delivered to the client in association with the compressed/encoded video frames for the purpose of creating one or more histograms used to determine offsets. The timestamp information provides the desired display time specified by the server. A server may not send a compressed video frame every frame period. In particular, the histogram may include decoded timestamps that indicate the availability of video frames for display at the client compared to desired display times specified by the server (eg, server-based timestamp information). Since the server and client timestamps may be defined by individual unsynchronized clocks, it may be useful to normalize the decoding timestamps. For example, normalization may include subtracting the value of the first decoded timestamp from all decoded timestamps. This causes the first decoding timestamp to be zero and all subsequent timestamps to be associated with it.

As shown in FIG. 8A, the constructed histogram 850 can be used to determine the appropriate offset between the server VSYNC signal and the client VSYNC signal. As mentioned above, the histogram provides the distribution of decoding timestamps. This indicates the availability of the display for the desired display time. The VSYNC offset 430 between server and client is indicated when a predetermined number or threshold (eg, 99.99 percent) of received video frames have arrived at the client and the next appropriate occurrence of the client VSYNC signal. exist to be decoded in time. In one embodiment, the remaining number (eg, 0.01 percent) is too late to be displayed and may be missed. In other words, the VSYNC offset 430 corresponds to near-worst-case latency when receiving, decoding, and displaying a rendered video frame.
Encoding 403, transmission 404, reception 405, and decoding 406 of FIG. 8A illustrate near-worst frames (eg, the 99.99th percentile). With the proper VSYNC offset 430 established, there is no margin between decoding this frame and displaying it. One or more client buffers 820 may have a low decode timestamp (e.g., less than the 25th percentile) such that encode 403, send 404, receive 405, and decode 406 are binned early in the histogram (e.g., below the 25th percentile). (which exhibits the lowest latency in one direction). In this particular example, four buffers are required, three for the frames being decoded (three buffers 820) and one for the currently displayed frame (not shown).

A series of theoretical timing diagrams 850A through 850D are provided for client VSYNC signal 312, and timing diagram 850C (and accompanying display 407) shows ideal client VSYNC 312C. Since there is no direct clock or timestamp synchronization (e.g., via a third party timing mechanism such as a universal clock), the offset 430 is not set directly, instead the current client VSYNC 312 uses the near worst case of the histogram , resulting in ideal client VSYNC timing 312C as described above. Alternatively, the server VSYNC can be adjusted to create the appropriate offsets, as described above.

Server timestamp information is collected and/or received by the client. As mentioned above, the timestamp information may include the time the corresponding video frame was generated (e.g., flip time, when scanout occurred, the time of the server VSYNC signal when scanout occurred). occurrence, etc.). Additional information may be collected at the server and/or client and used to create or interpret histograms, and is referred to as "histogram information", as described in more detail below.

On the server side, the additional histogram information includes the number of scene changes, encoding time average and/or standard deviation of I-frame encoding times, and average and/or standard deviation of P-frame encoding times. May contain statistics. Encoding time statistics may be delivered as periodic messages from the server to the client. Additionally, the histogram information may include the time to prepare the encoder slices by the encoder. This may be delivered as a periodic message from the server to the client. The histogram information may also include actual server-side VSYNC timing and target VSYNC timing, which may be added to the packet header. Additionally, the histogram information may include the average number of slices per I-frame versus P-frame.

At the server, the histogram information may include round-trip time (RTT) measurements to derive one-way network latency for transmitting encoded slices (e.g., compressed encoder slices). be. The RTT measurement can be used to determine the transmission time required to send packets to the client (eg, without further processing performed by the client such as decoding and rendering). For example, the RTT can be determined by sending heartbeat packets from the server to the client. This packet contains a unique identifier. The client sends a heartbeat response back to the server with a unique identifier so that the server can calculate the RTT. One-way network latency is about half the RTT. When used to create a histogram, periodic measurements of RTT can be used to analyze and/or determine network or transmission jitter (eg, spikes in RTT). For example, a one-way network latency measurement measured via RTT can be used as the transmission time of all video frames received until the next RTT measurement.

At the client, additional histogram information may include the decoding time for each received encoded video frame. Further, the histogram information may include render preparation time for each decoded video frame, and render preparation may include converting decoded video frame slices into scanlines or screen slices.

Additionally, for servers, additional histogram information may include a maximum transmission rate that defines the total network throughput (eg, bandwidth) that the server considers available to the client. This can be used to determine the maximum rate at which an encoder slice of an encoded video frame can be sent. The maximum rate fluctuates based on the stability of the network connection to the client, and the offset can be dynamically adjusted to accommodate fluctuations. Furthermore, the maximum transmission rate can be adjusted independently of the encoder parameters, so slices can be transmitted more quickly if the encoder is configured not to produce slices at the maximum transmission rate.

For example, maximum bandwidth or maximum transmission rate may be determined by a feedback mechanism from the client. One way to do this is to have the client return the number of packets received in a range of incremental sequence IDs (identifiers) or frames. For example, a client may report that it has received 145 out of 150 frames for sequence IDs 100-250. The server can calculate packet loss, know the amount of bandwidth sent in that series of packets, and determine the client's maximum bandwidth. The amount of bandwidth transmitted is constantly fluctuating due to variable bitrates, scene complexity, etc., so the client cannot make this decision. That is, the client does not know whether the server is sending the maximum bandwidth it can handle at any given moment. For example, the maximum bandwidth is 15 Mbps (megabits per second), but the scene complexity can be low because the user is displaying menus (static video frames are low complexity, frame-to-frame variation no). As a result, only 2 Mbps is transmitted. So if the client reports 0% packet loss, the server will not be told if the client can still handle 15 Mbps. Therefore, the true maximum bandwidth can only be determined if the server is sending the maximum bandwidth.

FIG. 8B shows a histogram 850 showing the distribution of decoding timestamps. In this case, in one embodiment, normalization is performed such that the numerically smallest decode timestamp is assigned a value of zero (eg, the smallest decode timestamp is subtracted from all decode timestamps). In particular, the x-axis indicates the time of the corresponding decoding timestamp in milliseconds, such as from 0 to over 60 milliseconds (ms). The y-axis shows the number of video frames received by the client for the corresponding decoding timestamp.

For purely illustrative purposes, the timestamp of decoding can vary in the range of about 60 milliseconds (milliseconds), and the desired display time specified by the server, compared to the desired display time on the client for a video frame. Indicates 60 ms variation in display availability. That is, some frames may be displayed approximately 60ms earlier or later than others. Variations in the availability of a particular frame for display can be due to variations in server and client processing, scene complexity, network path variations, packet delay variations, and other factors.
By analyzing the worst or near-worst decoding timestamps, the ideal relationship between the server and client VSYNC signals can be determined. That is, as described above, determine the ideal relative offset between the timing of the client VSYNC signal and the server VSYNC signal to maximize the number of received and decompressed video frames that can be displayed with the appropriate client VSYNC signal. be able to. As such, diagram 800B shows the width of the distribution of decoding timestamps 755 (eg, approximately 57 milliseconds) within which 99.99 percent of the video frames received by the client arrive and the client It is decoded in time for display at the next appropriate occurrence of the VSYNC signal.

Using this width of the distribution of decoding timestamps 755 (including all decoding timestamps up to the near-worst case, but excluding those beyond), the overall buffering required for the decoded video frame can be determined. If width 755 is less than the frame period, two buffers are required. One is needed for the frame when decoding and the other is needed for display. For example, if the width is greater than one frame period and less than two frame periods, three buffers are required. In the particular example of a width of 57ms, if the frame period is 16.67ms, then 5 frame buffers are required. The decode timestamp indicates the availability of the decoded frame for the desired display time, thus video frames with lower decode timestamps are kept in the buffer for a longer period of time before display, and video frames with higher decode timestamps are not displayed. It is held in the previous shorter time buffer.

In one embodiment, the histogram is dynamically regenerated. In another embodiment, the amount of frame buffering is set dynamically by the client over time. In yet another embodiment, frames that arrive and are too late to be decoded to be displayed at the desired display time are skipped (ie, not displayed).

Along with a detailed description of the various client devices 210 and/or cloud gaming networks 290 (eg, within game servers 260) of FIGS. 2A-2D, flowchart 900 of FIG. Shows how to construct a histogram that provides the distribution of the elapsed timings of video frames from being generated on the game server to arriving and/or being ready to be displayed to the client, the histogram being a measure of buffer size on the client. Configured for decisions. As mentioned above, histograms are also constructed to determine the appropriate offset between the VSYNC signals at the server and client.

Operations 601, 603, 605, 611, and 613 were previously described in conjunction with flowchart 600A of FIG. 6A and flowchart 600B of FIG. 6B, and disclose adjusting the relative timing between the server and client VSYNC signals. (e.g. synchronizing frequencies and adjusting timing offsets or phases). In summary, at 601, the method includes setting, at the server, a server VSYNC signal to a frequency corresponding to generation of video frames at the server during frame periods of the server VSYNC signal. At 603, the method includes setting a client VSYNC signal corresponding to a frequency at the client to be used for rendering to a display associated with the client. At 605, the method includes transmitting compressed video frames based on the generated video frames from the server to the client over the network using the server VSYNC signal.

At 611, the method includes sending timestamp information associated with the compressed video frames to the client. For example, the timestamp information may be sent with or separately from the compressed video frames, and the timestamp information is theoretically transmitted over the network when determined by the video game, such as when to scan or stream to a local display. gives an indication of the desired display timing of the corresponding video frame without transmission over. When a client receives and decodes a compressed video frame, the timestamp information is processed and compared to the desired display time specified by the server (e.g., server timestamp) to determine the availability of the corresponding video frame for display at the client. A decryption timestamp is created that indicates In one embodiment, the decoding timestamps can be normalized because the server and client timing can be defined by corresponding individual clocks that are not synchronized. A full discussion of timestamp information is provided in connection with FIGS. 6B-6C and 8A-8B and is equally applicable in connection with FIG.

At 613, the method includes constructing a histogram based on the measured and/or calculated decoding timestamps at the client. For example, a corresponding histogram may be created by binning decode timestamp information associated with compressed video frames received and decoded at the client over a period of time. The histogram shows that the decode timestamp indicates the availability of the video frame for display on the client compared to the desired display time specified by the server (e.g., the server timestamp), so the histogram shows the server (server time It also provides a distribution of the timing of completion of decoding of video frames received by the client relative to the desired display times specified by stamp information, etc.). A full discussion of timestamp information is provided in connection with FIGS. 6B-6C and 8A-8B and is equally applicable in connection with FIG.

At 910, the method includes measuring the width of the histogram at particular points in time. For example, the width of the distribution of decoding timestamps in the histogram is such that a predetermined number or threshold (e.g., 99.99 percent) of received video frames is within the time period indicated at the next appropriate occurrence of client VSYNC signal 312. (for clarity, the remaining 0.01% of the received video frame is not included when measuring width).
In particular, the width of the histogram can be used to set the amount of frame buffering required by the client at any given time. Thus, at 920, the method dynamically sets a number of display buffers on the client based on the width of the histogram and the frame duration of the synchronized server and client VSYNC signals, and the histogram 750 is a specific is generated at the point of As noted above, if the width is less than a frame period, two frame buffers are required, for example. Thus, video frames with lower decode timestamps are retained in the buffer for a longer period of time, while video frames with higher decode timestamps are retained in the buffer for a shorter period of time.

Along with detailed descriptions of the various client devices 210 and/or cloud gaming networks 290 (eg, within game servers 260) of FIGS. 2A-2D, flowchart 1000 of FIG. Figure 4 shows a method for adjusting the relative timing between VSYNC signals between devices. In particular, flowchart 1000 may be used to compensate for drift and/or adjust offsets or phases between two or more VSYNC signals of corresponding devices.

At 1010, the method includes setting a plurality of VSYNC signals to a plurality of VSYNC frequencies at the plurality of devices, wherein the corresponding device VSYNC signals of the corresponding devices are set to the VSYNC frequencies of the corresponding devices. That is, each device uses its corresponding pixel clock to set its corresponding VSYNC signal. Further, the frequencies may be similar, such as set to the same apparent frequency, but the actual frequencies may differ due to differences between the various pixel clocks. These VSYNC signals can be used for video frame generation (eg, server in server/client architecture) and/or display of video frames (eg, client in server/client architecture). Also, these VSYNC signals can be used both for generating video frames and for displaying video frames. For example, in peer-to-peer architecture devices, where each device is running a video game locally, the timing can coordinate the execution and display of video frames.

At 1020, the method includes transmitting a plurality of signals among a plurality of devices, which is analyzed and used to adjust relative timing between VSYNC signals of corresponding devices of at least two devices. be done. Relative timing can be coordinated between devices configured in a server/client architecture or between devices configured in a peer-to-peer architecture. For example, the signal may include server timestamp information or server timing information that gives an indication of when the corresponding video frame is intended to be displayed by the server, as described above. As such, the VSYNC signals of multiple devices may be synchronized (eg, synchronizing the frequency of the VSYNC signals) by determining the drift between at least two VSYNC signals. Also, timing offsets and/or timing phases can be adjusted between at least two VSYNC signals.

Specifically, in one embodiment, at least two of the devices may be configured in a server/client architecture. In another embodiment, devices are arranged in a multi-tenant configuration (eg, one server for multiple client devices). For example, the first device may be a server device whose server VSYNC signal is set to the server VSYNC frequency. The server VSYNC signal corresponds to the generation of video frames during execution of an application on the server device during frames on the server VSYNC frequency. A plurality of compressed video frames are transmitted over the network from the server device to each of the remaining devices (such as client devices) in the plurality of devices based on the server VSYNC signal. For example, the server VSYNC signal provides timing for the generation and encoding of video frames at the server. Compressed video frames are based on video frames being generated by the server device. Each receiving device (such as the rest) decodes and displays the received compressed video frames. The display of decoded video frames can be synchronized between each receiving device.

In particular, relative timing may be adjusted between devices to compensate for drift and/or adjust timing offsets or phases between VSYNC signals of the devices. Drift and timing offsets or phase adjustments can be determined using the techniques described above in connection with FIGS. 6A-6C, 7, and 8A-8B. The adjustment of the relative timing between the VSYNC signals of two devices can occur in either device, and the raster scan lines of the vertical blanking interval of the corresponding frame period of the corresponding device VSYNC signal of the corresponding device. , or overclocking or underclocking the corresponding clock of the corresponding device.

In particular, in one embodiment at least two of the devices may be configured in a peer-to-peer architecture. For example, each device may be an independent peer device. That is, none of the devices are server devices. As such, the device may be configured for peer-to-peer gaming. Each device generates multiple video frames by processing the same video game. An independent peer device may be operating in multiplayer mode for a particular video game with backend server support for controlling multiplayer game sessions.
A backend server can manage state data for each user of a multiplayer game session, thereby enabling state sharing between devices. The state data may include game state data that defines the state of gameplay (of the game application) for that user at a particular point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, and the like. In this manner, objects and characters can be inserted into each game environment of users participating in a multiplayer game session, allowing each user's gameplay to be customized to each user through shared state.
Also, each user's gameplay can be synchronized based on state sharing. That is, the video frames being displayed on each device can be synchronized to reflect synchronized gameplay. As such, one user may not be able to benefit from continuously receiving and displaying video frames on a corresponding device ahead of other users' gameplay video frames. Alternatively, no backend server is involved, in which case the VSYNC relationship between peers is between receiving control or state information from other peers and displaying video frames using information received from other peers. Optimized to minimize latency.

FIG. 11A illustrates duplication of receiving, decoding, and rendering decompressed video frames for display at client 210, according to one embodiment of the present disclosure. In particular, one-way latency between a server (not shown) and client 210 in a cloud gaming application can be reduced by redundant operations of receiving, decoding, and displaying a particular video frame.

For example, a cloud gaming application client receives and decodes video frames. In particular, a client that receives encoded video frames 1105 in receive operation 405 expects the server to run the video game to produce game-rendered video frames, which are then encoded in the server's encoder and then encoded. delivered to the client as a video frame 1105. Encoded video frame 1105 includes one or more encoded slices that are compressed by the server's encoder. The client includes a decoder configured to decode one or more encoded slices within an encoded video frame at decoding operation 406 . In one embodiment, the decoding process begins before the corresponding video frame is completely received at the client. A decoded video frame 1106 includes one or more encoder slices because the decoder performs decoding for each encoded slice. The decoded video frame 1106 is then prepared for display, such as by rendering the information in the decoded video frame 1106 into scanlines or screen slices. The client-rendered video frame 1107 is then ready for display.

One-way latency between the server and the client can be reduced by having the client 210 begin displaying the video frame at operation 407 before the video frame is fully decoded at operation 406 . In particular, one or more decoded slices of a video frame may be prepared for rendering on a display before the video frame is fully decoded. That is, the display operation at 407 overlaps the decoding operation at 406 . In particular, the first encoded slice (such as slice A) must arrive and be decoded before the client's scanout can start displaying. Additionally, all subsequent encoded slices must arrive and be decoded before their respective decompressed data can be rendered and scanned out for display.

Furthermore, in addition to duplicating the receiving and decoding operations at the client, even before the encoded video frames transmitted by the server are completely received at the client, one is then rendered in preparation for display. Or multiple decoded slice representations may occur. That is, one or more of the receive, decode, and display operations at the client may overlap for corresponding video frames. Additionally, if you are overlapping multiple operations on both the server and the client, one of the rendered video frames that are then rendered in preparation for display may not be displayed even before the scanout operation on the server is fully completed. One or more decoded slices can be displayed on the client, where in one embodiment the scanout delivers the game-rendered video frames to the encoder on the server.

The overlap of displaying at operation 407 and decoding at operation 406 can be performed for each encoder slice. In this way, an encoded slice can be displayed before one or more subsequent encoded slices are received. To do this, forward error correction (FEC) data must be interleaved between coded slices of corresponding video frames. Specifically, an encoded slice may be divided into one or more network packets. A FEC packet can be used to modify one or more packets associated with a slice. As such, FEC packets may be interleaved between packets of multiple slices. In this way, forward error correction can be used early to detect missing slices and/or corrupted packets without waiting for the entire set of packets (e.g., data and FEC) for a frame to be received by the client. can be corrected. As a result, the decryption operation and the display operation on the client can be duplicated.

In one embodiment, a decryption timestamp can be created for each slice to indicate the slice's availability for display at the client compared to the desired display time specified by the server. The decode timestamp takes the completion time of the slice decoding 406 at the client, minus the timestamp received from the server that indicates the ideal display time of the frame, which decompressed slice is used in the display process 407. Add time (i.e., add 0 ms if you need the decompressed slice data immediately, add 8.33 ms if you need the slice data in the middle of the 16.67 ms frame period, for example). can be calculated by It may be useful to normalize the decryption timestamps in some way, such as subtracting the first decryption timestamp from all other timestamps.

The decode timestamps can be arranged in histograms similar to those shown in FIGS. 5A-5B and 8A-8B. Use the worst-case or near-worst-case (e.g., 99.999%) decoding timestamps determined by the histogram to adjust relative server and client VSYNC timings, thereby reducing one-way latency can be reduced. A very high threshold, such as 99.999%, is desirable because if a slice arrives and is decoded late, the existing contents of the display buffer will be used for display, causing visible corruption or "tearing"; It defines that one frame is missing out of 100,000 video frames generated by the server 260 and displayed by the client 210 .

2A-2D, along with a detailed description of the various client devices 210 and/or cloud gaming networks 290 (eg, within game server 260) of FIGS. shows a cloud gaming method in which frames are received at a client from a server, decoded and rendered for display, and the decoding and display of video frames can be overlapped for the purpose of reducing one-way latency. . The ability to overlap one or more operations on the client is achieved by managing one-way latency between the server and the client, as previously described in FIGS. 4-10. For example, the relative timing between server and client VSYNC signals may be adjusted to reduce and/or minimize one-way latency variations between server and client.

At 1110, the method includes receiving encoded video frames at a client, and a server executing an application to produce rendered video frames, which are then encoded at an encoder at the server. A coded video frame, which is coded as a frame, includes one or more coded slices that have been compressed. For example, the server uses the server VSYNC signal to generate multiple video frames. Each video frame may be sent to an encoder for compression, and each video frame may be encoded into one or more coded slices. As mentioned above, the start of encoding of the corresponding video frame can be aligned with the server VSYNC signal. The compressed video frame is then sent to the client, and transmission does not have to coincide with the server VSYNC signal, and begins as soon as an encoder slice or complete video frame is encoded. Compressed video frames are received by a client.

At 1120, the method includes decoding one or more encoded slices at a decoder of a client to generate one or more decoded slices. In one embodiment, decoding of one or more encoded slices may begin prior to receiving a complete encoded video frame at the client. For example, a client receives one or more encoded slices of corresponding video frames. Each of the encoded slices is then decoded and placed in a display buffer, and the decoded video frames are placed in the display buffer.

At 1130, the method includes rendering one or more decoded slices for display at the client. In particular, during the decoding process, decoded slices can be rendered for display, which can include generating screen slices (e.g., scanlines) from the decoded slices of corresponding video frames. included, which is then streamed to the client's display.

At 1140, the method, in one embodiment, causes the client to begin displaying one or more decoded slices to be rendered before fully receiving the one or more encoded slices. include. In particular, decoded slices placed in the display buffer may be immediately streamed to the client's display. Therefore, client operations for receiving and displaying may overlap.

In another embodiment, the method includes beginning display of the one or more decoded slices rendered on the display prior to fully decoding the one or more encoded slices. . In particular, decoded slices placed in the display buffer may be immediately streamed to the client's display. Therefore, client operations for decryption and display may overlap.

FIG. 12 illustrates components of an exemplary device 1200 that can be used to carry out aspects of various embodiments of the present disclosure. For example, FIG. 12 illustrates streaming and/or streaming of media content, such as adjusting the VSYNC signal of a server or client to synchronize and/or adjust the offset of the VSYNC signal between the server and client, according to embodiments of the present disclosure. 1 illustrates an exemplary hardware system suitable for receiving media content, suitable for providing dynamic buffering at a client, and suitable for overlapping decoding and display of video frames at the client.
This block diagram may incorporate or be a personal computer, server computer, game console, mobile device, or other digital device, each suitable for implementing embodiments of the present invention. A device 1200 is shown. Device 1200 includes a central processing unit (CPU) 1202 that runs software applications and, optionally, an operating system. CPU 1202 may be comprised of one or more homogeneous or heterogeneous processing cores.

According to various embodiments, CPU 1202 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments have one or more microprocessor architectures specifically adapted for highly parallel and computationally intensive applications, such as media and interactive entertainment applications, applications configured for graphics processing during game execution. It can be implemented using a CPU.

Memory 1204 stores applications and data used by CPU 1202 and GPU 1216 . Storage 1206 provides non-volatile storage and other computer-readable media for applications and data and includes fixed disk drives, removable disk drives, flash memory devices and CD-ROM, DVD-ROM, Blu-ray, etc. -ray®, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 1208 communicate user input from one or more users to device 1200 and include keyboards, mice, joysticks, touchpads, touchscreens, still or video recorders/cameras, and / Or there may be a microphone.
Network interface 1209 allows device 1200 to communicate with other computer systems over electronic communications networks, and may include wired or wireless communications over local area networks and wide area networks such as the Internet. Audio processor 1212 is adapted to generate analog or digital audio output from instructions and/or data provided by CPU 1202 , memory 1204 and/or storage 1206 . The components of device 1200, including CPU 1202, graphics subsystem 1214, e.g. 1222.

Graphics subsystem 1214 is also connected to data bus 1222 and the components of device 1200 . Graphics subsystem 1214 includes a graphics processing unit (GPU) 1216 and graphics memory 1218 . Graphics memory 1218 includes display memory (eg, a frame buffer) used to store pixel data for each pixel of the output image. Graphics memory 1218 may be integrated into the same device as GPU 1216 , connected as a separate device from GPU 1216 , and/or implemented within memory 1204 . Pixel data may be provided directly from CPU 1202 to graphics memory 1218 . Alternatively, CPU 1202 provides data and/or instructions defining the desired output image to GPU 1216, which causes GPU 1216 to generate pixel data for one or more output images. Data and/or instructions defining a desired output image may be stored in memory 1204 and/or graphics memory 1218 . In embodiments, GPU 1216 includes 3D rendering functionality that generates output image pixel data from instructions and data that define scene geometry, lighting, shadowing, texture, motion, and/or camera parameters. GPU 1216 may further include one or more programmable execution units capable of executing shader programs.

Graphics subsystem 1214 periodically outputs image pixel data from graphics memory 1218 for display on display device 1210 or projection by a projection system (not shown). Display device 1210 can be any device capable of displaying visual information in response to signals from device 1200, including CRT, LCD, plasma, and OLED displays. Device 1200 can provide display device 1210 with analog or digital signals, for example.

Other embodiments for optimizing the graphics subsystem 1214 may include multi-tenancy GPU operation, where GPU instances are shared among multiple applications, and distributed GPUs supporting a single game. Graphics subsystem 1214 may be configured as one or more processing devices.

For example, graphics subsystem 1214 may be configured to perform multi-tenancy GPU functionality, and in one embodiment, one graphics subsystem provides graphics and/or rendering for multiple games. Pipelines can be implemented. That is, the graphics subsystem 1214 is shared between multiple games that are running.

In other embodiments, graphics subsystem 1214 includes multiple GPU devices combined to perform graphics processing for a single application running on the corresponding CPU. For example, multiple GPUs can perform an alternative form of frame rendering, where for example GPU1 renders the first frame, GPU2 renders the second frame in consecutive frame periods, and when the last GPU is reached , the first GPU renders the next video frame (eg, GPU1 renders the third frame if there are only two GPUs). That is, the GPU rotates when rendering a frame. Rendering operations can overlap, in which GPU2 can start rendering a second frame before GPU1 finishes rendering the first frame.
In another embodiment, multiple GPU devices may be assigned different shader operations in the rendering and/or graphics pipeline. The master GPU is doing the main rendering and compositing. For example, in a group containing three GPUs, master GPU1 may perform main rendering (e.g., first shader operation) and perform compositing of output from slave GPU2 and slave GPU3, while slave GPU2 may perform second shader (e.g., , rivers, etc.) operations, slave GPU3 can perform a third shader (eg, particle smoke) operation, and master GPU1 combines the results from each of GPU1, GPU2, and GPU3. In this way, different GPUs can be assigned to perform different shader operations (flag waving, wind, smoke generation, fire, etc.) to render video frames. In yet another embodiment, each of the three GPUs can be assigned to different objects and/or portions of the scene corresponding to the video frame. In the above embodiments and implementations, these operations can be performed in the same frame period (concurrently in parallel) or in different frame periods (sequentially in parallel).

Accordingly, the present disclosure provides a method for streaming media content and/or streaming media content, including adjusting the VSYNC signal of a server or client to synchronize and/or adjust the offset of the VSYNC signal between the server and client. , provide dynamic buffering at the client, and overlap decoding and display of video frames at the client.

It should be understood that various embodiments defined herein can be combined or incorporated into specific implementations using various features disclosed herein. Accordingly, the examples provided are only a few possible examples, and are not limited to various implementations in which various elements can be combined to define more implementations. In some instances, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

Embodiments of the present disclosure may be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments of the disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wire-based or wireless networks.

With the above embodiments in mind, it should be understood that embodiments of the disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulations of physical quantities. Any of the operations described herein that form part of embodiments of the present disclosure are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The device may be specially constructed for the required purpose. Alternatively, the apparatus may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may prove more convenient to construct more specialized apparatus to perform the required operations. Sometimes there is.

The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer-readable medium is any data storage device that can store data that can later be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data. Includes storage devices. The computer-readable medium can include computer-readable tangible medium distributed over network-connected computer systems so that the computer-readable code is stored and executed in a distributed fashion.

Although the method operations have been described in a particular order, other housekeeping operations may be performed between the operations, or the operations may occur at slightly different times, so long as the processing of the overlay operations is performed in the desired manner. or may be distributed within the system to allow processing operations to occur at various intervals associated with the processing.

Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the embodiments are to be considered illustrative rather than limiting, and the embodiments of the present disclosure are not limited to the details provided herein, but within the scope and equivalents of the appended claims. May be changed within the object.

Claims (94)

at a server, setting a server VSYNC signal to the server's VSYNC frequency, said server VSYNC signal corresponding to generation of video frames at said server during frame periods of said server VSYNC frequency;
At the client, set the client VSYNC signal to the client VSYNC frequency,
transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal;
at the client, decode and display the plurality of compressed video frames; and
analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames; how to. 2. The method of claim 1, further comprising transmitting timestamp information associated with the plurality of generated video frames from the server to the client. 3. The method of claim 2, wherein the timestamp information is transmitted with the plurality of compressed video frames or the timestamp information is transmitted separately from the plurality of compressed video frames. the timestamp information includes timestamps corresponding to generated corresponding video frames;
3. The method of claim 2, wherein the corresponding timestamps occur at corresponding flip times at the server or at corresponding occurrences of the server VSYNC signal used to scan the corresponding video frames to an encoder. Method. 3. The method of claim 2, further comprising adjusting at least one of the server VSYNC frequency and the client VSYNC frequency to be similar based on the timestamp information sent from the server to the client. 3. The method of claim 2, further comprising adjusting the relative offset between the server VSYNC frequency and the client VSYNC signal based on the timestamp information sent from the server to the client. 2. The method of claim 1, further comprising adjusting at least one of the server VSYNC signal and the client VSYNC signal to adjust the relative timing. continuously analyzing the timing of client operations; and
2. The method of claim 1, further comprising dynamically adjusting the relative timing by adjusting at least one of the server VSYNC signal and the client VSYNC signal based on the analysis. Further, at the server or client, determine a drift between the server VSYNC frequency and the client VSYNC frequency; and
8. The method of claim 7, wherein the server or client adjusts the corresponding server VSYNC frequency or the client VSYNC frequency in one or more frame periods to compensate for the drift. Adjusting at least one of the server VSYNC signal or the client VSYNC signal includes:
8. The method of claim 7, removing or adding a raster scanline in the vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal. The adjustment of at least one of the server VSYNC signal or the client VSYNC signal includes:
8. The method of claim 7, overclocking or underclocking the corresponding clocks of the server or the client. The server includes an encoder for generating the plurality of compressed video frames, the plurality of compressed video frames generated in response to processing of an application by the server in streaming mode and delivered to the client during a session. 2. The method of claim 1, wherein: 13. The method of Claim 12, wherein the client is configured to receive and decompress the plurality of compressed video frames for rendering on a display associated with the client. 2. The method of claim 1, wherein timing of said decoding by said client of said plurality of compressed video frames is analyzed and used to adjust said relative timing between said server VSYNC signal and said client VSYNC signal. the method of. Further, analyzing the timing of the decoding by the client of multiple encoded slices within the multiple compressed video frames and adjusting the relative timing between the server VSYNC signal and the client VSYNC signal. 15. The method of claim 14. 15. The method of claim 14, wherein near-worst-case decoding times of the plurality of compressed video frames are analyzed to adjust the relative timing between the server VSYNC signal and the client VSYNC signal. 17. The method of claim 16, wherein the near worst case decoding time is determined by measuring a distribution of decoding times of the plurality of compressed video frames received at the client. providing a corresponding timestamp at the server for each of the plurality of video frames produced in an encoder at a corresponding scanout of the corresponding video frame;
deliver the corresponding timestamps of the plurality of video frames to the client;
constructing a histogram giving the timing of each of the plurality of video frames, the video frames generated and generated for the client at corresponding decoding times at the client associated with the corresponding timestamps provided at the server; is to be delivered,
the server VSYNC signal such that a threshold percentage of the plurality of compressed video frames arrive at the client and are decoded in time for display at the corresponding occurrence of the client VSYNC signal indicated by the corresponding timestamp. 2. The method of claim 1, further comprising offsetting the client VSYNC signal from . generating a plurality of video frames by the server during a plurality of frame periods, where the frame periods are of approximately equal size;
At the client, set the client VSYNC signal to the client VSYNC frequency,
transmitting from the server to the client a plurality of compressed video frames based on the plurality of video frames;
at the client, decode and display the plurality of compressed video frames; and
analyzing the timing of one or more client operations to adjust the relative timing of the client VSYNC signal; A method for generating compressed video frames. 20. The method of claim 19, further comprising periodically transmitting timestamp information associated with said plurality of frame periods from said server to said client. Further, at the server or client, determine a drift between the server VSYNC frequency and the client VSYNC frequency; and
20. The method of claim 19, wherein the server or client adjusts the corresponding server VSYNC frequency or the client VSYNC frequency in one or more frame periods to compensate for the drift. In adjusting the corresponding server VSYNC frequency or the client VSYNC frequency,
22. The method of claim 21, further comprising removing or adding a raster scan line in a vertical blanking interval of a frame period of the server VSYNC signal or client VSYNC signal. said adjustment of said corresponding server VSYNC frequency or said client VSYNC frequency comprising:
22. The method of claim 21, overclocking or underclocking corresponding clocks of the server or the client. A non-transitory computer readable medium storing a computer program for performing the method,
program instructions for setting a server VSYNC signal at a server to a server VSYNC frequency, said server VSYNC signal for generating a plurality of video frames at said server during a plurality of frames at said server VSYNC frequency. Correspondingly,
having program instructions for setting a client VSYNC signal at the client to a client VSYNC frequency;
having program instructions for transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal;
said client having program instructions for decoding and displaying said plurality of compressed video frames; and
analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames; A non-transitory computer-readable medium having program instructions for further comprising program instructions for transmitting timestamp information associated with the plurality of generated video frames from the server to the client;
the timestamp information includes timestamps corresponding to generated corresponding video frames;
25. The method of claim 24, wherein the corresponding timestamps occur at corresponding flip times at the server or at corresponding occurrences of the server VSYNC signal used to scan the corresponding video frames to an encoder. Non-Transitory Computer-Readable Medium. 25. The non-transitory computer of claim 24, further adjusting at least one of the server VSYNC frequency and the client VSYNC frequency to be similar based on the timestamp information sent from the server to the client. readable medium. 25. Further comprising program instructions for adjusting the relative phase or offset between the server VSYNC frequency and the client VSYNC signal based on the timestamp information transmitted from the server to the client. The non-transitory computer-readable medium as described in . 25. The non-transitory computer-readable medium of claim 24, further comprising program instructions for adjusting at least one of said server VSYNC signal and said client VSYNC signal to adjust said relative timing. program instructions for determining, at the server or client, a drift between the server VSYNC frequency and the client VSYNC frequency; and
29. The non-transient device of claim 28, further comprising program instructions for adjusting, at the server or client, the corresponding server VSYNC frequency or the client VSYNC frequency in one or more frame periods to compensate for the drift. computer-readable medium. The program instructions for adjusting at least one of the server VSYNC signal or the client VSYNC signal comprise:
29. The non-transitory computer-readable medium of claim 28, having program instructions for removing or adding raster scan lines in the vertical blanking interval of a frame period of a corresponding server VSYNC signal or client VSYNC signal. Program instructions for adjusting at least one of the server VSYNC signal or the client VSYNC signal comprise:
29. The non-transitory computer-readable medium of claim 28, having program instructions for overclocking or underclocking corresponding clocks of the server or the client. setting a server VSYNC signal at a server to a server VSYNC frequency defining a plurality of frame periods, said server VSYNC signal corresponding to generating a plurality of video frames at said server during said plurality of frame periods. and
At the client, set the client VSYNC signal to the client VSYNC frequency,
transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal;
at the client, decode and display the plurality of compressed video frames; and
and analyzing the timing of one or more client operations to set the amount of frame buffering used by the client as the client receives the plurality of compressed video frames. 33. The method of claim 32, wherein the amount of frame buffering is dynamically set by the client over time. 33. The method of claim 32, wherein the amount of frame buffering defines the number of frames to buffer at the client. 33. The method of claim 32, further comprising adjusting relative timing between the server VSYNC signal and the client VSYNC signal based on the timing of the one or more client operations. transmitting timestamp information associated with the plurality of video frames;
constructing a histogram giving the timing of each video frame, the video frame being generated and delivered to the client at a corresponding decoding time at the client associated with the corresponding timestamp information;
measure the width of the histogram at a given moment; and
33. The method of claim 32, dynamically setting a number of display buffers at the client based on the width of the histogram and the frame duration. The corresponding timestamp information of the corresponding generated video frame occurs at the corresponding flip time at the server or at the server VSYNC signal used to scan the corresponding video frame to the encoder. 37. The method of claim 36, occurring upon corresponding occurrence. The amount of frame buffering used by the client is an input for latency adjustment, which delays display of frames at the client if the frames arrive later than a predetermined time. 33. The method of claim 32, comprising deciding to skip. 33. The method of claim 32, wherein the decoded compressed video frames are displayed using the client VSYNC signal. A non-transitory computer-readable medium storing a computer program for performing the method,
program instructions for setting a VSYNC signal at a server to a VSYNC frequency of the server defining a plurality of frame periods, wherein the server VSYNC signal corresponds to a plurality of video frames at the server during the plurality of frame periods. corresponds to the generation of
having program instructions for setting a client VSYNC signal at the client to a client VSYNC frequency;
having program instructions for transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal;
having program instructions for decoding and displaying, at the client, the plurality of compressed video frames; and
program instructions for analyzing the timing of one or more client operations and setting the amount of frame buffering used by the client when the client receives the plurality of compressed video frames; A non-transitory computer-readable medium, comprising: 41. The non-transitory computer-readable medium of claim 40, wherein in the method the amount of frame buffering is dynamically set by the client over time. 41. The non-transitory computer-readable medium of claim 40, wherein in the method the amount of frame buffering defines the number of frames to buffer at the client. 41. The method of claim 40, further comprising program instructions for adjusting the relative timing between the server VSYNC signal and the client VSYNC signal based on the timing of the one or more client operations. Temporary computer-readable medium. having program instructions for transmitting timestamp information associated with the plurality of video frames;
program instructions for building a histogram giving said timing of each video frame, said video frame being generated and generated for said client at a corresponding decoding time at said client associated with said corresponding timestamp information; having program instructions for measuring the width of said histogram at an instant in time; and
41. The non-transitory computer-readable medium of claim 40, further comprising program instructions for dynamically setting a number of display buffers at the client based on the width of the histogram and the frame duration. In the method, the corresponding time stamp information of the corresponding video frame generated is VSYNC at the corresponding flip time at the server or at the server VSYNC used to scan the corresponding video frame to an encoder. 45. The non-transitory computer-readable medium of claim 44, occurring upon corresponding occurrence of a signal. In the method, the amount of frame buffering used by the client is an input for latency adjustment, wherein the latency adjustment is a 41. The non-transitory computer-readable medium of claim 40, determining to skip display of frames. 41. The non-transitory computer-readable medium of claim 40, wherein in the method the decoded compressed video frames are displayed using the client VSYNC signal. a computer system,
a processor;
a memory coupled to the processor and storing instructions that, when executed by a computer system, cause the computer system to perform a method, the method comprising:
setting a server VSYNC signal at a server to a server VSYNC frequency defining a plurality of frame periods, said server VSYNC signal corresponding to generation of a plurality of video frames at said server during said plurality of frame periods. and
At the client, set the client VSYNC signal to the client VSYNC frequency,
transmitting a plurality of compressed video frames based on the plurality of video frames from the server to the client over a network using the server VSYNC signal;
at the client, decode and display the plurality of compressed video frames; and
A computer system that analyzes the timing of one or more client operations to set the amount of frame buffering used by the client as the client receives the compressed video frames. 49. The computer system of claim 48, wherein in the method the amount of frame buffering is dynamically set by the client over time. 49. The computer system of claim 48, wherein in the method the amount of frame buffering defines the number of frames to buffer at the client. The method further comprises:
transmitting timestamp information associated with the plurality of video frames;
constructing a histogram giving the timing of each video frame, the video frame being generated and delivered to the client at a corresponding decoding time at the client associated with the corresponding timestamp information;
measure the width of the histogram at a given moment; and
49. The computer system of claim 48, dynamically setting the number of display buffers at the client based on the width of the histogram and the frame duration. a method,
setting the plurality of VSYNC signals to the plurality of VSYNC frequencies in the plurality of devices, the corresponding device VSYNC signals of the corresponding devices being set to the VSYNC frequencies of the corresponding devices;
A method of transmitting a plurality of signals between said plurality of devices, said signals being analyzed and used to adjust relative timing between corresponding device VSYNC signals of at least two devices. Further, the first device is a server device, for setting a server VSYNC signal to a server VSYNC frequency for said server device, said server VSYNC signal transmitting an application at said server device during said plurality of said server VSYNC frequencies. Supports generation of video frames for multiple frame periods during execution,
using the server VSYNC signal to transmit a plurality of compressed video frames based on the plurality of video frames from the server device to each of the remaining ones of the plurality of devices over a network; and
each of the remaining ones of the plurality of devices decoding and displaying the plurality of compressed video frames;
53. The method of Claim 52, wherein the display of the plurality of compressed video frames on each of the remaining ones of the plurality of devices is synchronized. 54. The method of claim 53, wherein each of said remaining devices of said plurality of devices is a client device configured in a client architecture comprising a server and said server device. 55. The method of claim 54, wherein said server device is configured in a multi-tenant configuration with said remaining ones of said plurality of devices. further generating a corresponding plurality of video frames when processing an application on a corresponding device of the plurality of devices; and
the corresponding device of each of the plurality of devices displaying the corresponding plurality of video frames;
53. The method of claim 52, wherein display of corresponding video frames on the multiple devices is synchronized. each of the plurality of devices is an independent peer device;
53. The method of claim 52, wherein the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming. 53. The method of claim 52, wherein at least one of said plurality of signals has timestamp information. 53. The method of claim 52, wherein the plurality of signals are analyzed to synchronize device VSYNC signals of the plurality of devices. the relative timing between corresponding device VSYNC signals of at least two devices comprising:
60. The method of claim 59 adjusted by removing or adding raster scan lines in the vertical blanking interval of the corresponding frame period of the VSYNC signal of the corresponding device of the corresponding device. the relative timing between corresponding device VSYNC signals of at least two devices comprising:
60. The method of claim 59 adjusted by overclocking or underclocking corresponding clocks of corresponding devices. further determining the drift between the VSYNC frequency of the first device of the first device and the VSYNC frequency of the second device of the second device;
53. The method of claim 52, wherein the first device or the second device adjusts the VSYNC frequency of the corresponding device in one or more frame periods to compensate for the drift. A non-transitory computer-readable medium storing a computer program for performing the method,
having program instructions for setting the plurality of VSYNC signals to the plurality of VSYNC frequencies in the plurality of devices, wherein the corresponding device VSYNC signals of the corresponding devices are to set the VSYNC frequencies of the corresponding devices; ,
comprising program instructions for transmitting a plurality of signals between the plurality of devices, the plurality of signals being analyzed and adjusted to adjust the relative timing between corresponding device VSYNC signals of at least two devices; A non-transitory computer-readable medium used. program instructions for setting a server VSYNC signal at a server to a server VSYNC frequency, wherein the server VSYNC signal is for a plurality of video frames of an application at the server during a plurality of frames at the server VSYNC frequency; corresponds to the generation of
having program instructions for transmitting a plurality of compressed video frames based on the video frames from the server to the plurality of devices over a network using the server VSYNC signal; and
each of the plurality of devices further having program instructions for decoding and displaying the plurality of compressed video frames;
64. The non-transitory computer-readable medium of claim 63, wherein said display of said plurality of compressed video frames on said plurality of devices is synchronized. In the method, each of the plurality of devices is a client device;
65. The non-transitory computer-readable medium of claim 64, wherein in said method each of said plurality of devices is configured in a server and client architecture. program instructions for generating a corresponding plurality of video frames of an application on a corresponding device of the plurality of devices; and
a corresponding device of each of the plurality of devices further comprising program instructions for displaying the corresponding plurality of video frames;
64. The non-transitory computer-readable medium of claim 63, wherein display of corresponding multiple video frames on the multiple devices are synchronized. wherein each of the plurality of devices is an independent peer device;
67. The non-transitory computer-readable medium of claim 66, wherein in the method the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming. 64. The non-transitory computer-readable medium of claim 63, wherein the method analyzes the plurality of signals to synchronize device VSYNC signals of a plurality of devices. a computer system,
a processor;
a memory coupled to the processor and storing instructions which, when executed by the computer system, cause the computer system to perform a method, the method comprising:
setting the plurality of VSYNC signals to the plurality of VSYNC frequencies in the plurality of devices, setting the corresponding device VSYNC signals of the corresponding devices to the VSYNC frequencies of the corresponding devices;
A computer system for transmitting a plurality of signals between said plurality of devices, said plurality of signals being analyzed and used to adjust relative timing between VSYNC signals of corresponding devices of at least two devices. The method further comprises:
setting a server VSYNC signal at a server to a server VSYNC frequency, said server VSYNC signal corresponding to generation of video frames of an application at said server during frames at said server VSYNC frequency;
using the server VSYNC signal to transmit a plurality of compressed video frames based on the video frames from the server to the plurality of devices over a network; and
each of the multiple devices decoding and displaying multiple compressed video frames;
70. The computer system of Claim 69, wherein said display of said plurality of compressed video frames on said plurality of devices is synchronized. In the method, each of the plurality of devices is a client device;
71. The computer system of claim 70, wherein in said method each of said plurality of devices is configured in a server and client architecture. further generating a corresponding plurality of video frames of an application on a corresponding device of the plurality of devices; and
each corresponding device of the plurality of the plurality of devices displaying the corresponding plurality of video frames;
70. The computer system of claim 69, wherein display of corresponding video frames on said multiple devices is synchronized. wherein each of the plurality of devices is an independent peer device;
73. The computer system of claim 72, wherein in the method the plurality of devices are arranged in a peer-to-peer configuration for peer-to-peer gaming. 70. The computer system of claim 69, wherein the method analyzes the plurality of signals to synchronize device VSYNC signals of the plurality of devices. A method of cloud gaming, comprising:
A client receives an encoded video frame and a server runs an application to produce a rendered video frame, which is then encoded as an encoded video frame in an encoder of said server, said encoding a coded video frame having one or more coded slices that have been compressed;
decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices;
render the one or more decoded slices for display at the client; and
starting display of the one or more decoded slices to be rendered prior to complete reception of the one or more encoded slices at the client. 76. The method of claim 75, further comprising initiating decoding of the one or more encoded slices prior to receiving the encoded video frame completely at the client. At the start of display of the one or more decoded slices to be rendered,
76. The method of claim 75, beginning display of the one or more decoded slices rendered prior to fully decoding the one or more encoded slices. Further, at the server, setting a server VSYNC signal to a VSYNC frequency of the server, the server VSYNC signal corresponding to generating a plurality of rendered video frames at the server during a plurality of frames at the server VSYNC frequency. death,
at the client, setting a client VSYNC signal to the client VSYNC frequency;
transmitting compressed video frames based on the rendered video frames from the server to the client over a network using the server VSYNC signal;
at the client, decode and display the plurality of compressed video frames; and
analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames; 76. The method of claim 75, wherein an adjustment of the relative timing between the server VSYNC signal and the client VSYNC signal is made, the adjustment comprising:
Transmit timestamp information associated with the plurality of generated video frames from the server to the client, wherein corresponding timestamps of corresponding rendered generated video frames correspond to corresponding timestamps at the server. occurring at flip time or at the corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to the encoder;
adjusting at least one of the server VSYNC frequency and the client VSYNC frequency to be similar based on the timestamp information sent from the server to the client; and
79. The method of claim 78, performed by adjusting the relative offset between the server VSYNC frequency and the client VSYNC signal based on the timestamp information sent from the server to the client. 80. The method of Claim 79, wherein the timestamp information is transmitted with the plurality of compressed video frames or the timestamp information is transmitted separately from the plurality of compressed video frames. The relative timing between the server VSYNC signal and the client VSYNC signal is
79. The method of claim 78, adjusted by removing or adding raster scan lines in the vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal. A non-transitory computer-readable medium storing a computer program for performing the cloud gaming method,
having program instructions for receiving encoded video frames at a client, a server executing an application to generate rendered video frames, which are then encoded at an encoder at the server; encoded as a frame, said encoded video frame comprising one or more encoded slices that have been compressed;
having program instructions for decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices;
having program instructions for rendering the one or more decoded slices for display at the client; and
A non-transitory computer having program instructions for initiating display of the one or more decoded slices rendered prior to fully receiving the one or more encoded slices at the client. readable medium. 83. The non-transitory computer of Claim 82, further comprising program instructions for starting decoding of said one or more encoded slices prior to receiving a complete encoded video frame at said client. readable medium. said program instructions for initiating display of said one or more decoded slices to be rendered;
83. The method of claim 82, comprising program instructions for initiating display of the one or more decoded slices rendered prior to fully decoding the one or more encoded slices. Temporary computer-readable medium. At the server, having program instructions for setting a server VSYNC signal to a VSYNC frequency of the server, the server VSYNC signal being a plurality of rendered video at the server during a plurality of frame periods of the server VSYNC frequency. corresponds to frame generation,
at the client, comprising program instructions for setting a client VSYNC signal to a client VSYNC frequency;
having program instructions for transmitting a plurality of compressed video frames based on the plurality of rendered video frames from the server to the client over a network using the server VSYNC signal;
having program instructions for decoding and displaying the plurality of compressed video frames at the client; and
analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames; 83. The non-transitory computer-readable medium of Claim 82, further comprising program instructions for: In said program instructions, an adjustment of said relative timing between said server VSYNC signal and said client VSYNC signal is made, said adjustment comprising:
Program instructions for transmitting timestamp information associated with the plurality of generated rendered video frames from the server to the client, wherein the corresponding timestamps of corresponding rendered generated video frames are: , at a corresponding flip time at the server or at a corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to an encoder;
program instructions for adjusting at least one of the server VSYNC frequency and the client VSYNC frequency to be similar based on the timestamp information sent from the server to the client;
and program instructions for adjusting the relative phase or offset between the server VSYNC frequency and the client VSYNC signal based on the timestamp information transmitted from the server to the client. 86. The non-transitory computer-readable medium of Clause 85. 87. The non-transitory computer of Claim 86, wherein in said method said timestamp information is transmitted with said plurality of compressed video frames, or said timestamp information is transmitted separately from said plurality of compressed video frames. readable medium. In the method, the relative timing between the server VSYNC signal and the client VSYNC signal comprises:
86. The non-transitory computer-readable medium of claim 85 adjusted by removing or adding raster scan lines in vertical blanking intervals of frame intervals of the server VSYNC signal or the client VSYNC signal. a computer system,
a processor;
a memory coupled to the processor and storing instructions which, when executed by the computer system, cause the computer system to perform a method of cloud gaming, the method comprising:
A client receives an encoded video frame and a server runs an application to produce a rendered video frame, which is then encoded as an encoded video frame in an encoder of said server, said encoding a encoded video frame having one or more encoded slices that have been compressed;
decoding the one or more encoded slices at a decoder of the client to generate one or more decoded slices;
render the one or more decoded slices for display at the client; and
A computer system that initiates display of the one or more decoded slices rendered prior to complete reception of the one or more encoded slices at the client. In said method,
90. The computer system of Claim 89, further comprising initiating decoding of the one or more encoded slices prior to receiving a complete encoded video frame at the client. In the method, at the start of display of the one or more decoded slices to be rendered,
90. The computer system of claim 89, beginning displaying the one or more decoded slices rendered on the display before fully decoding the one or more encoded slices. The method further comprises:
at the server, setting a server VSYNC signal to the server's VSYNC frequency, the server VSYNC signal corresponding to generation of a plurality of rendered video frames at the server during a plurality of frame periods of the server VSYNC frequency. and
at the client, setting a client VSYNC signal to the client VSYNC frequency;
transmitting compressed video frames based on the rendered video frames from the server to the client over a network using the server VSYNC signal;
at the client, decode and display the plurality of compressed video frames; and
analyzing the timing of one or more client operations to adjust the relative timing between the server VSYNC signal and the client VSYNC signal as the client receives the plurality of compressed video frames; 90. The computer system of claim 89, wherein: In the method, an adjustment of the relative timing between the server VSYNC signal and the client VSYNC signal is made, the adjustment comprising:
Transmit timestamp information associated with the plurality of generated video frames from the server to the client, wherein corresponding timestamps of corresponding rendered generated video frames correspond to corresponding timestamps at the server. occurring at flip time or at the corresponding occurrence of the server VSYNC signal used to scan the corresponding video frame to the encoder;
adjusting at least one of the server VSYNC frequency and the client VSYNC frequency to be similar based on the time stamp information sent from the server to the client; and the time sent from the server to the client. 93. The computer system of claim 92, by adjusting the relative offset between the server VSYNC frequency and the client VSYNC signal based on stamp information. In the method, the relative timing between the server VSYNC signal and the client VSYNC signal comprises:
93. The computer system of claim 92 adjusted by removing or adding raster scan lines in the vertical blanking interval of a frame period of the server VSYNC signal or the client VSYNC signal.

Images