TWI615803B - System and method for improving the graphics performance of hosted applications - Google Patents

Abstract

The present invention discloses a system and method for efficiently processing a video stream using limited hardware and / or software resources. For example, one embodiment of a computer-implemented method for efficiently processing a video stream using a processor pipeline having a plurality of pipeline stages includes identifying a bottleneck stage within the processor pipeline, the bottleneck stage processing the video Frame of the stream; receiving a feedback signal from the bottleneck stage at one or more upstream stages, the feedback signal providing an indication of the speed at which the bottleneck stage processes the frames of the video stream; and adjusting the response in response The speed at which one or more upstream stages process the frames of the video stream to estimate the speed at which the bottleneck stage processes the frames of the video stream.

Description

System and method for improving image performance of hosted application

The present invention relates generally to the field of data processing systems and, more particularly, to a system and method for improving the performance of graphics for hosted applications.

This application claims the right of U.S. Provisional Application No. 61 / 210,888, filed on March 23, 2009, and filed on August 7, 2009 with the title of "System and Method for Accelerated Machine Switching" Partial continuation application No. 12 / 538,077, and a partial continuation application (CIP) application No. 10 / 315,460 entitled “Apparatus and Method for Wireless Video Gaming”, filed on December 10, 2002 To the assignee of this CIP application.

For low-latency applications such as video games, it's important to perform image operations as efficiently as possible. However, attempts to speed up the image rendering process can lead to unintended visual distortions, such as "image tearing", that is, displaying information from two or more different frames on a display device with a single screen drawing. The embodiments of the present invention described below provide various techniques for improving the efficiency of image rendering while reducing these unintended visual distortions.

101‧‧‧control system

102‧‧‧Server

103‧‧‧Local Storage Network

104‧‧‧Low-latency video compression

105‧‧‧Disk Array

106‧‧‧Control signal

106a‧‧‧Control signal

106b‧‧‧Control signal

110‧‧‧Internet

112‧‧‧Low-latency video decompression

113‧‧‧Control signal logic

115‧‧‧Home / office client

121‧‧‧ input device

122‧‧‧Monitor / Standard Resolution TV / High Resolution TV

202‧‧‧Server

204‧‧‧Video Compressor

206‧‧‧Internet

209‧‧‧route

210‧‧‧Hosting Services

211‧‧‧user venue

215‧‧‧Home & Office Client / Platform / User Client

221‧‧‧External input device

222‧‧‧Monitor / Standard Resolution TV / High Resolution TV

241‧‧‧Central office, front-end system, residential tower

241‧‧‧WAN interface

242‧‧‧WAN interface

243‧‧‧Firewall / Router / Antenna

251‧‧‧Control signal

252‧‧‧User Site Routing

253‧‧‧ User Internet Provider

254‧‧‧Internet

255‧‧‧Server-centric routing

256‧‧‧Frame calculation

257‧‧‧Video Compression

258‧‧‧Video decompression

301‧‧‧ inbound internet traffic

302‧‧‧ Inbound Routing / Inbound Routing Network

311‧‧‧disk array

312‧‧‧disk array

315‧‧‧ Delay Buffer / Disk Array

321‧‧‧Application / Game Server

322‧‧‧Application / Game Server

325‧‧‧Application / Game Server

329‧‧‧ Uncompressed Video / Audio

330‧‧‧Common Video Compression / Shared Pool / Common Hardware Compression

339‧‧‧compressed video / audio / outbound internet traffic

340‧‧‧outbound routing / outbound routing network

350‧‧‧ Video / Audio and / or Grouped View of Delay Buffer

399‧‧‧ Outbound Internet Traffic

401‧‧‧Central Processing Unit (CPU)

402‧‧‧Image Processing Unit (GPU)

403‧‧‧Memory

405‧‧‧Back buffer

406‧‧‧Pre-buffer

408‧‧‧video game code / video game output / uncompressed video output

410‧‧‧Image Information

430‧‧‧Image Engine

1301‧‧‧actual camera position

1302‧‧‧ Predict camera position

1303‧‧‧actual background

1304‧‧‧Acting background

P1‧‧‧Central Processing Unit

P2‧‧‧Image Processing Unit

P3‧‧‧Monitor

P4‧‧‧ Bottleneck stage

Q12‧‧‧ queue

Q23‧‧‧ queue

Q34‧‧‧ queue

FIG. 1 illustrates a system architecture for executing an online video game according to an embodiment of the present invention.

FIG. 2 illustrates an online video game on which an online video game can be played according to one embodiment of the present invention Different communication channels.

FIG. 3 illustrates one embodiment of a system architecture for compressing audio / video generated by a video game.

Figure 4 illustrates a system architecture according to one embodiment of the invention.

5 to 12 illustrate data flow and feedback between various system components used according to an embodiment of the present invention.

FIG. 13 illustrates the difference between a predicted camera position and an actual camera position.

The invention will be more fully understood from the subsequent embodiments and accompanying drawings, however this should not be seen as limiting the disclosed subject matter to the specific embodiments shown, but merely for purposes of explanation and understanding.

Specific details (such as device type, system configuration, communication method, etc.) are set forth in the following description to provide a complete understanding of the invention. However, those of ordinary skill in the relevant arts will understand that implementation of the described embodiments does not necessarily require these specific details.

The assignee of this application has developed an online video game and application host system. For example, certain embodiments of this system are described in U.S. Patent Application No. 12 / 538,077, filed August 7, 2009, entitled "System and Method for Accelerated Machine Switching" , This application claims the right of U.S. Provisional Application No. 61 / 210,888, filed on March 23, 2009, and is filed on December 10, 2002 under the title of "Apparatus and Method for Wireless Video Gaming", No. 10 / 315,460 Part of the continuation (CIP) application, it is assigned to the assignee of this CIP application. These applications are sometimes referred to as "co-applications" and are incorporated herein by reference. Some related aspects of the online video game and application host system described in the equivalent application will now be briefly described, followed by a detailed description of a visualization and encryption system and method for hosting applications .

An exemplary online video game and application host system

FIG. 1 illustrates one embodiment of a video game / application hosting service 210 described in the same application. The hosting service 210 hosts applications running on the server 102, and the servers 102 receive input from an input device 121 via a home or office client 115 and send to the hosting service 210 via the Internet 110. The servers 102 respond to the input and therefore update their video and audio output compressed by the low-latency video compression 104. The compressed video is then streamed through the Internet 110 for the home or office client 115 to decompress, and then displayed on a monitor or SD / HDTV 122. This system is a low-latency streaming interactive video system as described more fully in the aforementioned "Applications in the Same Application".

As shown in Figure 2, a variety of network technologies with varying degrees of reliability, such as wired or fiber optic technologies that are generally more reliable Technology) Implement a network connection between the hosting service 210 and the home and office client 215. Any of these client devices may have their own input devices (e.g. keyboard, buttons, touch screen, trackpad or inertial recognition pen, video capture camera and / or motion tracking camera, etc.) or An external input device 221 (eg, a keyboard, a mouse, a game controller, an inertial sensing pen, a video capture camera, and / or a motion tracking camera, etc.) may be used with a connection wire or wireless connection. As described in more detail below, the hosting service 210 includes servers of various performance levels, including servers with high-performance CPU / GPU processing capabilities. While playing a game or using an application on the hosting service 210, a home or office client device 215 receives keyboard and / or controller input from the user, and then transmits the controller input to the server via the Internet 206 Hosting service 210 that executes game code in response and generates a continuous video output frame (a series of video images) to the game or application software (for example, if the user presses a button (this will guide the screen To the right), the game program will then generate a series of video images showing the character to the right). This series of video images is then compressed using a low-latency video compressor, and the hosting service 210 then uses the The Internet 206 transmits a low-latency video stream. The home or office client device then decodes the compressed video stream and presents the decompressed video image on a monitor or TV. Therefore, the computing and graphics hardware requirements of the client device 215 are greatly reduced. The client 215 only needs to have the processing capability to pass keyboard / controller input to the Internet 206 and decode and decompress a decompressed video stream received from the Internet 206. In fact, any personal computer today can use The software on its CPU does the above (for example, an Intel Corporation Core Duo CPU operating at close to 2GHz can decompress a 720p HDTV encoded using a compressor such as H.264 and Windows Media VC9). And in the case of any client device, a dedicated chip can also perform video decompression in real time for these standards at far lower cost and power consumption than a general-purpose CPU such as a modern PC requires. Note that in order to perform the functions of transmitting controller input and decompressing video, the home client device 215 does not need any special image processing unit (GPU), optical disk drive or hard disk drive.

As games and application software become more complex and more realistic, they will require higher-performance CPUs, GPUs, more RAM, and larger and faster disk drives. The computing power may continue to increase, but the end user will not be required to update the home or office client platform 215, because its processing requirements will use a given video decompression algorithm to maintain a constant display resolution and frame rate . Therefore, the illustrated system does not have the hardware limitations and compatibility issues encountered today.

Furthermore, because the game and application software is only executed on the server in the hosting service 210, the user ’s home or office (unless otherwise described, as used herein, the “Office” shall include any non-residential environment, Include, for example, a copy of the game or application software (in the form of optical media, or as download software) that does not exist in the school dormitory. This obviously alleviates the possibility of illegally copying (piracy) a game or application software, and alleviates the possibility of piracy, misappropriation or other damage to a valuable database that can be used by a game or application software. In fact, if you are playing a game or application software, you need a special server that you ca n’t actually use in your home or office (for example, Expensive, extremely large, or extremely noisy devices), even if a pirated copy of the game or application software is obtained, it will not work at home or in the office.

FIG. 3 illustrates one embodiment of the components of a server center for a hosting service 210 described below. Regarding the hosting service 210 illustrated in FIGS. 1 to 2, unless otherwise described, a hosting service 210 control system 101 controls and cooperates with components of this server center.

Inbound Internet traffic 301 from the user client 215 is directed to an inbound route 302. Generally, inbound Internet traffic 301 will enter the server center via a high-speed fiber optic connection to the Internet, but any network connection with sufficient bandwidth, reliability, and low latency will be sufficient. Inbound routing 302 is one of the networks (the network can be implemented as an Ethernet, Fibre Channel network, or the network can be implemented by any other transmission method) switches and one of the routing servers that support these switches System that retrieves the arriving packets and routes each packet to the appropriate application / game ("app / game") servers 321-325. In one embodiment, a packet passed to a particular app / game server is representatively received from the client and / or can be forwarded by other components (e.g., network components such as gateways and routers) in the data center / Change one of the data subsets. In some cases, for example, if a game or application is running on multiple servers simultaneously, packets will be routed to more than one server 321 to 325 at a time. The RAID arrays 311 to 312 are connected to the inbound routing network 302, so that the app / game servers 321 to 325 can read and write to the RAID arrays 311 to 312. In addition, a RAID array 315 (which can be implemented as multiple RAID arrays) is also connected to the inbound route 302 and can read data from the RAID array 315 from the app / game server 321 to 325. The inbound routing 302 can be implemented in a variety of prior art network architectures, including a tree structure of switches, the inbound Internet traffic 301 at its root; implemented in a mesh structure interconnecting all kinds of devices Medium; or implemented as a series of interconnected sub-networks, where centralized traffic in intercommunication devices is isolated from centralized traffic in other devices. A network configuration is a SAN. Although it is usually used for storage devices, it can also be used for devices General high-speed data transfer between. In addition, the app / game servers 321 to 325 may each have multiple network connections to the inbound route 302. For example, a server 321 to 325 may have a network connection to a subnet attached to one of the RAID arrays 311 to 312 and another network connection to a subnet attached to another device.

As mentioned previously, the app / game servers 321 to 325 may be configured to be all the same, some different, or all different. In one embodiment, each user generally uses at least one app / game server 321 to 325 when using a hosting service. For ease of explanation, it will be assumed that a given user is using the app / game server 321, but a user can use multiple servers, and multiple users can share a single app / game server 321 to 325. As previously described, the control input sent by the user from the client 215 is received as inbound Internet traffic 301 and routed to the app / game server 321 via the input route 302. The app / game server 321 uses the user's control input as the control input for a game or application running on the server, and calculates the next video frame and the audio associated with it. The app / game server 321 then outputs the uncompressed video / audio 329 to the shared video compression 330. The app / game server can output the uncompressed video via any method (including one or more Gigabit Ethernet connections), but in one embodiment, the video is output via a DVI connection, and The audio and other compression and communication channel status information is output via a universal serial bus (USB) connection.

The shared video compression 330 compresses uncompressed video and audio from the app / game servers 321 to 325. The compression can be implemented entirely in hardware or in software running on hardware. A dedicated compressor may exist for each app / game server 321 to 325, or if the compressors are fast enough, a given compressor may be used to compress video / audio from more than one app / game server 321 to 325 . For example, at 60fps, the video frame time is 16.67ms. If a compressor can compress a frame in 1ms, the compressor can be used to compress video / audio from up to 16 app / game servers 321 to 325 by sequentially capturing input from the server, of which The compressor retains each video / audio The state of the video compression process, and the content of the processing is switched as the compressor cycles through the video / audio streams from these servers. This results in significant cost savings in compression hardware. Because different servers will complete the frames at different times, in one embodiment, the compressor resources are in a common set with the common storage components (e.g., RAM, flash memory) used to store the state of each compression process. Area 330, and when a frame of servers 321 to 325 is completed and ready to be compressed, a control component determines the compression resources available at this time, and provides the compression resource of the server's compression program status and uncompressed to the compression resources. Video / audio frame for compression.

Note that the part of the compression process status of each server contains information about the compression itself, such as the decompressed frame buffer data of previous frames that can be used as a reference to one of the P-tiles, the resolution of the video output; compression quality; Tile structure; bits allocated to each microbrick; compression quality, audio format (for example, stereo, surround sound, Dolby® AC-3). But the compression program status also contains information about the communication channel status information about the following: peak data rate and whether a previous frame is currently output (and therefore the current frame should be ignored), and whether there are potentially channel characteristics that should be considered in compression Excessive packet loss such as influencing decisions on compression (e.g., in terms of the frequency of I-tiles, etc.). As the peak data rate or other channel characteristics change over time, as determined by app / game server 321 to 325 which supports one of the user monitoring data sent from the client 215, the app / game server 321 to 325 Send relevant information to the shared hardware compression 330. These and other features of the hosting service 210 are described in detail in the same application as the application.

The shared hardware compression 330 also encapsulates the compressed video / audio using a method such as described above, and if appropriate, applies FEC code, copies some data, or takes other steps to fully ensure that the video is received by the client 215 / Audio and decompression ability to make quality and reliability as high as possible.

Some applications, such as those described below, require the video / audio output of a given app / game server 321 to 325 to be available simultaneously at multiple resolutions (or in other multiple formats). If the app / game server 321 to 325 thus informs the shared hardware compression 330 resource, the uncompressed video / audio 329 of the app / game server 321 to 325 will be in different formats, different resolutions and / or at the same time Compressed in different packet / error correction structures. In some cases, certain compression resources are shared among multiple compression programs that compress the same video / audio (for example, in many compression algorithms, there is a step of scaling the image to multiple sizes before applying compression. If needed Output images of different sizes, you can use this step to serve multiple compression steps at the same time). In other cases, different compression resources will be required for each format. In any case, the compressed video / audio 339 for all the various resolutions and formats required for a given app / game server 321 to 325 (regardless of one or more) will be output to the outbound route 340 simultaneously. In one embodiment, the output of the compressed video / audio 339 is in UDP format, so it is a unidirectional packet stream.

The outbound routing network 340 includes a series of routing servers and switches that compress compressed video / audio streams through an outbound Internet traffic 339 interface (which will typically be connected to a fiber optic interface to the Internet). Guide to desired user or other destination and / or return to delay buffer 315 (implemented as a RAID array in an implementation) and / or return to the inbound route 302 and / or through a private network (not Display) for video distribution. Note (as described below) that the outbound route 340 can simultaneously output a given video / audio stream to multiple destinations. In one embodiment, this is implemented using Internet Protocol (IP) multicasting in which a given UDP stream intended to be streamed to one of multiple destinations is broadcast simultaneously, and the broadcast is routed through the outbound route 340 Forwarded by routing server and switch. The multiple destinations of the broadcast may be multiple users / clients via the Internet, multiple app / game servers 321-325 via inbound routing 302, and / or one or more delay buffers 315. Therefore, the output of a given server 321 to 325 is compressed into one or more formats, and each compressed stream is directed to one or more destinations.

Further, in another embodiment, if a user simultaneously uses multiple app / game servers 321 to 325 (for example, using a parallel processing configuration to generate a complex 3D output of various scenes) and each server generates a part of the obtained image, the video output of multiple servers 321 to 325 can be combined into a combined frame by the shared hardware compression 330, and it will be like the video from now on The output from a single app / game server 321 to 325 generally handles the video output as described above.

Note that in one embodiment, one copy of all the videos generated by the app / game server 321 to 325 (presenting at least the resolution of the video viewed by the user or higher) is at least a certain number of minutes (in one embodiment (15 minutes, 15 minutes) in the delay buffer 315. This allows each user to "rewind" to view videos from sessions to view previous work or achievements (in the case of a game). Therefore, in one embodiment, each compressed video / audio output 339 stream routed to a user client 215 is also multicast to a delay buffer 315. When video / audio is stored in a delay buffer 315, a directory on the delay buffer 315 is located on the network address of the app / game server 321 to 325 via the source of the delayed video / audio and the delay buffer A cross-reference can be found between the delayed video / audio locations on 315.

Image processing in one embodiment of an online gaming system

For low-latency applications such as video games, it's important to perform the graphics operations as efficiently as possible. However, attempts to speed up the image rendering process can lead to unintended visual distortions, such as "image tearing", that is, displaying information from two or more different frames on a display device with a single screen drawing. The embodiments of the present invention described below provide various techniques for improving the efficiency of image rendering while reducing these unintended visual distortions.

As illustrated in FIG. 4, in one embodiment, each application / game server 321 is equipped with a central processing unit (CPU) 401 for executing a video game code 408 stored in the memory 403 and for executing One of the image commands of the video game output 408 is a graphics processing unit (GPU). The architecture of the CPU and GPU is well known, and therefore this article will not describe these units and the instructions / commands executed by these units in detail. In short, the GPU is capable of processing applications such as one or more images through one or more of Open GL or Direct 3D. Specify a graphics command library using a programmatic interface (API). The code for implementing these image APIs is shown as the image engine 430 in FIG. 4. When the CPU processes the video game code 408, it passes the image commands specified by the API to the GPU that executes the commands and generates the video output 408. It should be noted, however, that the underlying principles of the invention are not limited to any particular image standard.

In one embodiment, the CPU and GPU are pipeline processors, which means that a set of data processing stages are connected in series within the CPU and GPU, so that the output of one stage is the input of the next stage. For example, the CPU pipeline usually includes an instruction fetching phase, an instruction decoding phase, an execution phase, and a retirement phase, each of which may have multiple sub-phases. A GPU pipeline can have more stages, including (for example and without limitation) transformation, vertex lighting, scene transformation, primitive generation, projection transformation, clipping, viewshed transformation, rasterization, texturing, fragment shading, and display. These pipeline stages are well understood by the average skilled person and will not be described in detail herein. The elements of a pipeline are usually executed in parallel or in a time-segmented manner and a certain amount of queue storage is usually required between the stages of the pipeline.

The above stages and the required queues between these stages each add a certain amount of delay to the execution of the image command. The following embodiments of the present invention provide techniques for minimizing this delay. Reducing latency is important because it expands the market for a device. In addition, the manufacturer of a device may have no control over the source of significant delays. For example, a user can attach a high-latency TV to a video game console, or can use a multimedia device (such as an online video game, a medical device or a military device controlled via the Internet, and Targets on the front line are engaged while the operator remains safely behind.)

As illustrated in FIG. 4, an embodiment of the present invention includes a back buffer 405 and a front buffer 406 to store a video game video frame generated by the image engine 430 when a user plays a video game. Each "frame" consists of a set of pixel data representing a screen image of a video game. In operation, each frame is generated in the back buffer as image commands are executed using image data. When a frame is completed in the back buffer, the The frame is sent to a line-by-line scan output buffer 406 before the frame to generate the uncompressed video output 408. The scan output procedure can be performed at a predetermined standard frequency (for example, 60 Hz or 120 Hz as implemented in a standard CRT or LCD monitor). The uncompressed video output 408 may then be compressed using various advanced low-latency video compression techniques described in the same application. Of course, there is no need to scan out the frame buffer from the video card (for example, via a digital video interface (DVI)) as suggested above. The frame buffer can be sent directly to the compression hardware via, for example, an internal bus of the application server (for example, a PCI Express bus). The frame buffer can be copied in memory by one of the CPU or GPU. The compression hardware may be (for example and without limitation) a CPU, a GPU, hardware installed in a server, and / or hardware on a GPU card.

Figure 5 shows an asynchronous pipeline, in which the queues (Q12, Q23, Q34) between each processing stage (P1, P2, P3, P4) save the data generated from the previous stage until it is consumed by the next stage. In one embodiment of the invention, the various stages described herein are stages within the GPU 402. The delay of this pipeline is the sum of the time (Tp1, Tp2, Tp3) consumed in transforming the data in each stage plus the time (Tq1, Tq2, Tq3) consumed by the data in each queue.

The obvious first step in minimizing the delay is to minimize queues or even discard them completely. A common way to minimize the delay is to synchronize the pipeline stages according to FIG. 6. Each stage operates on different sets of data at the same time. When all stages are ready, they all pass their data to the next stage in the pipeline. Queuing becomes unimportant and will not be shown in the drawing. The delay of a synchronization pipeline is the number of stages multiplied by the time to complete the slowest stage.

This slowest stage in the pipeline is the bottleneck P4 in all diagrams. This stage is usually a fixed feature of a device that the designer cannot control. Figure 7 shows the data flow downstream of the bottleneck stage. Note that there is no need to queue or synchronize. Delay is the sum of the time it takes to complete each phase. Delays cannot be slower than this sum.

This inspired a method for minimizing the delay of the pipeline stage upstream of the bottleneck according to Figure 8. method. If the first pipeline stage knows exactly how much time each pipeline stage will consume and when the bottleneck stage will request new data, it can be predicted when new data will begin to be produced that will be just ready for the bottleneck stage. Therefore, in an embodiment, the first pipeline stage may reduce its clock to slow down data processing based on when the bottleneck stage needs the new data. This technique can be called a phase-locked pipeline. The total delay is the sum of the time of each pipeline stage.

Another embodiment is illustrated in FIG. 9, where the bottleneck stage is manually moved to the first pipeline stage by slowing down the first pipeline stage to be slightly slower than the actual bottleneck stage. The box labeled 5 in P1 starts after box 3 in P4. Box 4 in P1 should also be slightly slower than the top of box 2 in P4. This is a common case in video games where the bottleneck is a physical connection between the computer and the monitor. There is a disadvantage in Figure 9: there must be some queues (not shown) that cause delays between stages P3 and P4. Another disadvantage is that the delay experienced by the user can drift over time, decrease steadily and then suddenly increase but start to decrease again. It can also result in discarded frames. Developers usually minimize the drop frame by driving the first stage at a rate as close to one of the bottleneck stages as possible. However, this rate is usually not known. If the first stage is driven to be even slightly faster than the bottleneck rate, the queues in the system will fill up and delay the upstream stage. Ironically, trying to use this method to minimize latency will run the risk of maximizing latency.

In one embodiment of the invention (shown in FIG. 10), the first stage is limited to the same rate as the bottleneck stage. The separation distance at the top of each number box in P1 should be the same as the separation distance at the top of each box in P4. The rate at which P1 generates frames just matches the rate at which P4 consumes frames. Feedback must be provided from the bottleneck stage to the first stage to ensure that the rates just match. Each stage provides feedback that includes (but is not limited to) the time required to run the data and the time spent in the queue. The phase-locked component maintains statistical information at each stage, and can accurately predict with a predetermined degree of confidence that when the bottleneck stage requires data, the data will be prepared and queued to a minimum. Note that a universal clock is not required in this embodiment. Phase-locked components require only relative time. Therefore, each pipeline stage can use a different clock. In fact, such clocks can potentially separate thousands of miles In different physical devices. In summary, in this embodiment of the invention, a bottleneck phase is identified based on timing constraints. Feedback is then provided from the bottleneck phase to the upstream phase to allow the upstream phase to precisely match the bottleneck phase. The phase of the upstream phase is adjusted to minimize wasted time in the queue.

The preceding icon illustrates a lightweight application. These applications are inefficient because the hardware is idle for most of the time. As illustrated in FIG. 11, one embodiment of the present invention that forms an inexpensive design is a design that assigns minimum hardware resources to each stage but still guarantees that each stage is faster than one of the bottleneck stages. In this case, the phase-locked method only gains less than the fully synchronized pipeline according to one of FIG. 6. Another example is a computer game that uses higher resolution textures to render more polygons, uses more anti-aliasing, special effects, until the frame rate starts to decrease.

This embodiment directly leads to another embodiment of the present invention, in which advanced image processing is performed using a minimum of hardware but with low latency. In this embodiment, the video stream is subdivided into two logical parts that can be processed independently: (a) a part with few resources and severe delay, and (b) a part with many resources and delay tolerance. These two parts can be combined in a hybrid system as illustrated in FIG. A particular instance (of many possible examples) would be a computer game known as a "first-person shooter" in which a user moves around from the perspective of a game character in a 3-dimensional world. In this type of game, the background and non-player characters consume a lot of resources and tolerate delays ("b" in Figure 12 represents "background"), and the resources that consume the player's images consume little and cannot Tolerate latency ("a" stands for "avatar" in Figure 12) (ie, because any performance performance over extremely low latency performance will result in an unexpected user experience). When the user pulls the trigger, he expects to see his weapon fire immediately. In the illustrated specific embodiment, the game is implemented using a personal computer, with a central processing unit (CPU) as stage P1 and an image processing unit (GPU) as stage P2. The monitor labeled P3 is the bottleneck stage. In this case, "monitor" means any device that consumes an uncompressed video stream. It may be compression hardware.

In this embodiment, the CPU completes its work on the background image labeled 3b before completing its work on the avatar image labeled 2a. However, in order to reduce the delay associated with the avatar, the GPU processes 2a before processing 3b, thereby rendering the avatar 2a on a previously rendered background 2b (to perform the avatar's motion as effectively as possible) and outputs the frame , And then immediately begin to show the background of the next frame (labeled 3b). The GPU can be idle for a short time to wait for data from the CPU to complete the next frame. In this embodiment, the CPU is idle waiting for the phase lock to signal that it is time to make a list of one of the drawing commands for the user's avatar and pass the list to the GPU. The CPU then immediately draws the background of the new frame, but the new frame cannot be the next frame, because the GPU will start to draw the next frame. The CPU is never ready for the next frame in time. Therefore, the CPU must start drawing the background for the frame after the next frame. This situation is similar to the operation of a synchronization pipeline as illustrated in FIG. 6.

The phase difference of such a frame between the avatar and the background is acceptable to the user in most cases. However, where the highest possible quality is desired, the following additional techniques can be used. High-latency path predicts input to produce data. In the first-person shooter example, the camera position is predicted in advance. When combining the output of the high-latency path with the output of the low-latency path, modify the output (eg, background) of the high-latency path to more closely match the output produced using the actual input rather than the predicted input. In the first-person shooter example, the background will be translated, scaled, and / or rotated to match the actual camera position. Note that this means that the high-latency path will have to be rendered slightly larger than one of the areas actually viewed by the player as illustrated in Figure 13. Figure 13 shows an actual camera position 1301, a predicted camera position 1302, and an actual background 1303 and a show background 1304. Therefore, if a user is playing a game (where a character is running towards a tree), the tree is slightly closer in each frame, which means larger. The user fired a shot and hit the tree. In a mixed scene, the tree is one frame behind the shot. So for a frame, things may appear to be "wrong" (ie, the shot does not seem to hit). To compensate, the described embodiments of the invention An embodiment enlarges the tree to approximate what it should appear in a firing frame.

As another example, when a user is playing a first-person shooter video game and pressing the fire button, the user wants to immediately see the sparks emitted from the gun. Therefore, in an embodiment, the program draws a fired gun on a previously rendered background and the game measures the drawing time so that the game is completed to be retrieved by the next stage in the pipeline (the next stage is the dvi output (vsync) or encoder input or some other bottleneck). The game then draws its best guess as to how the background of the next frame should be. If the speculation is not good, an embodiment modifies the background to more closely match the background that it should present when rendering from the correct camera position. Therefore, the technique shown in FIG. 13 is a simple affine warp. More complex techniques used in other embodiments use the z-buffer to achieve better results.

In an embodiment, the various functional modules and related steps illustrated herein may be implemented by specific hardware components containing hard-wired logic (such as an application-specific integrated circuit ("ASIC")) for performing the steps. Or by any combination of program computer components and custom hardware components.

In one embodiment, the modules may be implemented on a programmable digital signal processor ("DSP"), such as a TMS320x architecture from Texas Instruments (eg, a TMS320C6000, TMS320C5000, ..., etc.). A variety of different DSPs can be used while still meeting these fundamental principles.

Embodiments may include various steps as stated above. The steps may be embodied in machine-executable instructions that cause a general-purpose or special-purpose processor to perform certain steps. In some or all drawings, various components (such as computer memory, hard disk drive, input device) that are not related to these fundamental principles are omitted to avoid confusing related aspects.

The disclosed subject matter elements may also be provided as a machine-readable medium for storing such machine-executable instructions. The machine-readable medium may include, but is not limited to, flash memory, optical discs, CD-ROMs, DVD RPM, RAM, RPEMO, EEPROM, magnetic or optical cards, propagation media, or other types of machines suitable for storing electronic instructions Readable media. For example, the present invention can be downloaded as a computer program, which can be implemented from a remote computer via a communication link (for example, a modem or a network connection) by a data signal embodied in a carrier wave or other propagation medium. (E.g., a server) to a requesting computer (e.g., a client).

It should also be understood that the disclosed subject matter component may also be provided as a computer program product, which may include one of the instructions stored thereon that can be used to program a computer (e.g., a processor or other electronic device) to perform a series of operations Machine-readable media. Alternatively, these operations may be performed by a combination of hardware and software. The machine-readable medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs and magneto-optical disks, ROM, RAM, RPEMO, EEPROM, magnetic or optical cards, propagation media, or other types suitable for storing electronic instructions Media / machine-readable media. For example, the disclosed subject matter component can also be downloaded as a computer program product, where the program can be transmitted via a communication link (e.g., a modem or network) by a data signal embodied in a carrier wave or other propagation medium. Connection) from a remote computer or electronic device to a requesting program.

In addition, although the disclosed subject matter has been described in connection with specific embodiments, several modifications and changes are also fully within the scope of the present invention. Therefore, this description and the drawings should be viewed in an interpretive and not a restrictive sense.

Claims (4)

A computer-implemented method for efficiently processing a video stream with a processor pipeline having a plurality of pipeline stages, comprising: identifying a bottleneck stage in the processor pipeline, the bottleneck stage having a first clock and Processing the frame of the video stream; receiving a feedback signal from the bottleneck stage at one or more upstream stages, at least one of the upstream stages having a second clock, the feedback signal containing information about the operation of the bottleneck stage Information on the time required for the data and information on the time consumed by the data queue; and adjust the speed of the frame of the video stream in the one or more upstream stages in response to estimate the processing of the video stream in the bottleneck stage. Waiting for the speed of the frame, wherein the speed is at least partially adjusted by modifying a frequency of the second clock, wherein the video stream is generated by a user playing a code of a video game, and wherein a host servo The video game is executed on a server, wherein the user plays the video game from a client computer, and wherein the pipeline stages are stages within the host server. The method of claim 1, wherein the processor pipeline includes one or more stages in a central processing unit (CPU) and one or more stages in an image processing unit (GPU). A method for processing a video stream includes the following operations: identifying a bottleneck stage in a processor pipeline, the processor pipeline having a plurality of pipeline stages, wherein the bottleneck stage has a first clock and processes the video stream Frame; at one or more upstream stages, receiving a feedback signal from the bottleneck stage, wherein at least one of the upstream stages has a second clock, and wherein the feedback signal Contains information about the time required for the operational data of the bottleneck stage and information about the time consumed by the data queue; and adjusts the speed of the frame of the video stream in the one or more upstream stages in response to estimate the bottleneck stage Processing the speed of the frames of the video stream, wherein the speed is at least partially adjusted by modifying a frequency of the second clock, and wherein each of the operations is performed by hardware including hard-wired logic Component execution, wherein the video stream is generated by a user playing a video game code, and wherein the video game is executed on a host server, wherein the user plays the video game from a client computer, And the pipeline stages are the stages in the host server. The method of claim 3, wherein the processor pipeline includes one or more stages in a central processing unit (CPU) and one or more stages in an image processing unit (GPU).