TW200951728A - Mothod for reporting recorded video preceding system failures - Google Patents

Abstract

A system and method are described for intelligently allocating client requests to server centers provide real-time streaming interactive video. For example, one embodiment of a computer-implemented method comprises: strategically positioning a plurality of application server centers at different geographical locations; receiving a request from a client to execute an online application; determining the latency requirements based on the type of application requested by the client; and forwarding the client request to a particular application server center within the plurality based at least on the latency requirements of the requested application.

Description

200951728 VI. Description of the Invention: [Technical Field of the Invention] The present disclosure is generally in the field of a data processing system that improves the ability of a user to manipulate and access audio and video media. This application is entitled "Apparatus and Method f〇r wireless" for the application of the wireless video game on December 10, 2002.

Video Gaming) No. 10/315, 460 part of the continuation (CIp) application, which has been given to the assignee of this CIP application. φ [Prior Art] Since the days of Thomas Edison, recorded audio and film media have become one aspect of society. In the early 2nd century, there were widespread distributions of recorded audio media (magnetic columns and records) and movie media (coin-operated jukeboxes and movies), but the two technologies are still in their infancy. In the late 1920s, 'combining movies and audio on the basis of best-selling' followed by the combination of color film and audio. Radio broadcasts gradually evolved into broadcast-selling audio media that largely support advertising. When the television (TV) broadcasting standard was established in the mid-1940s, television and radio were joined in the form of broadcast-selling media to bring previously recorded or live-lived movies to the home. By the middle of the 20th century, most American households had a phonograph record player for playing recorded audio media, a radio for receiving live broadcasts, and for live broadcasts. A television that broadcasts audio/video (A/V) media. These three "media players" (record player, radio and τν) group 139958.doc 200951728 are often combined into a common public speaker cabinet to become the "media center" of the family. Although media choices are limited for consumers, the media “ecosystem” is very stable. Most of the users know how to use the "Media Broadcasting Machine" and enjoy the full range of their capabilities. At the same time, publishers of the media (mostly film and television studios, and music companies) are able to distribute their media to both cinemas and families without extensive piracy or "secondary sales" (ie, already used) Resale of the media). Often, publishers do not receive revenue from secondary sales' and therefore secondary sales reduce the publisher's revenue for new sales of purchasers who would otherwise be able to use the media themselves. Although there are indeed used records sold during the mid-20th century, such sales will not have a major impact on record publishers because they are different from movies or video (which are usually viewed once or only several times by adults). ), music tracks can be listened to hundreds or even thousands of times. Therefore, the music media is far more durable than the film/viewer (that is, it has a lasting value for adult consumers). Once the purchase of ρβ y has been made, the right consumer likes the music, and the levy may keep it for a long time. From the middle of the century to the present, the media ecosystem has undergone a series of fundamental changes in the interests and losses of consumers and publishers. In the audio » recorded! § (especially in the case of high quality wins), there is indeed a degree of materiality; the lack of introduction of ^ ^, the degree of flat fee is convenient, but it also marks the beginning of the use of 2 for consumers purely for convenience. Many records, but increasing (four) 4 tapes to record their own collection of slaps of the film) will be pirated. Also, consumers will record music played on the radio by 139958.doc 200951728 instead of buying a record or tape from the publisher. The emergence of consumer VCRs has led to more consumer convenience, as VCRs can now be set to record TV shows, which can be viewed at a later time, and vcr has also led to the establishment of a video rental industry where movies and τν programming can be " Access on an as-needed basis. The rapid development of popular home media devices since the mid-1980s has led to unprecedented choice and convenience for consumers, and has led to a rapid expansion of the media publishing market. Today, consumers face multimedia choices and multimedia devices, many of which are tied to specific forms of media or to specific publishers. A keen media consumer may connect a bunch of devices to the τν and the computer in each room, resulting in a "mouse-writing" cable to one or more TVs and/or personal computers (PCs) and a group of remote controls. In the context of this application, the term "personal computer" or "PC" refers to any type of computer suitable for use in a home or office, including desktop computers, Macint〇sh8 or other non-Windows computers, and Windows. Compatible devices, Unix variants, laptops, etc.). Such devices may include video game consoles, VCRs, DVD players, audio surround sound processors/amplifiers, satellite set-top boxes, and electric pro-TV set-top boxes. In addition, for enthusiastic consumers, there may be multiple devices with similar functions due to compatibility issues. For example, 'consumers may own both HD-DVD and Blu-ray DVD players, or Microsoft Xbox. Both with the Sony Playstation® video game system. In fact, due to the incompatibility of some games across several versions of the game console, consumers may have both a later version (such as the 'Xbox 360®'). Frequently, consumers are confused about which video 139958.doc 200951728 input and which remote end to use. Even after placing the disc in the correct player (eg 'DVD, HD-DVD' Blu-ray, xb〇x or piaystati〇n), selecting the video and audio inputs for the device and finding the correct remote control' Consumers still face technical challenges. For example, in the case of a wide-screen DVD, the user may need to first determine the correct aspect ratio (eg, 4:3, full, zoom in, wide zoom, cinema wide, etc.) and then on their TV or monitor the screen The upper aspect ratio is determined by the upper s. Similarly, the user may need to first determine the correct audio surround sound system format (e.g., AC-3, Dolby Digital, DTS, etc.) and then set the correct audio surround sound system format. Often, 'consumers are unaware that they may not have access to the media content of their television or audio system's full capabilities (for example, watching movies that are squeezed in the wrong aspect ratio, or listening to stereo audio rather than surround sound) . Increasingly, Internet-based media devices have been added to the stack of devices. Audio devices like the Sonos® digital music system stream audio directly from the Internet. Similarly, devices like the 81-ton 13〇\1' entertainment player record video and stream it over the home network or Streaming through the Internet 'where the video can be viewed remotely on the PC. And Internet Protocol Television (IPTV) services provide cable TV-like services via digital subscriber line (DSL) or other home Internet connections. There have also been recent efforts to integrate a single media feature into a single device, such as the Moxi® Media Center and the PC running the Windows XP Media Center version. While each of these devices provides a little convenience for the functions they perform, each lacks universal and simple access to most media. In addition, often due to the need for the storage of the 139958.doc 200951728 or the local storage, these devices often cost a few 7G to make a difference. These modern consumer electronic devices usually consume a lot of power and even consume a lot of power when idle. This means that it is expensive and time-consuming, jg, and Jia energy. For example, if a consumer forgets to cut a device or switch it to a different video input, the device may continue to operate. In addition, because none of these devices are complete solutions, they must be integrated with other devices in the home, leaving the user with a mouse-line and many remote controls.

In addition, when many newer Internet-based devices work properly, it often provides a more general form of media (compared to the form that might otherwise be available). For example, devices that stream video over the Internet often only make the video material string A 'incapable of being accompanied by a DVD interactive "extra billing item" stream, such as video "production", game or director commentary. . This is due to the fact that interactive materials are often produced in a specific format intended for the specific device that handles interactivity at the local end. For example, each of DVD, HD-DVD, and Blu-ray Disc has its own specific interactive format. Any home media device or local computer (which may be developed to support all popular formats) will require some degree of sophistication and flexibility that would be too expensive and complicated for consumer operations. This problem is exacerbated. If a new format is introduced in the future, the local device may not have the hardware capability to support the new format, which means that the consumer will have to purchase the upgraded local media device. For example, if a higher resolution video or stereo video is introduced later (for example, each eye 139958.doc 200951728 stream), the local device may not have the computing power to decode the video. Or, or may not have hardware for outputting the video in a new format (eg, false-twisting to achieve stereoscopic vision by 120 fps video synchronized with shuttered glasses), where 6 prints 8 are delivered to each One eye, if the consumer's video hardware can only support 6 〇 fps video, this option will not be available in the absence of upgraded hardware purchases). The problem of media device obsolescence and complexity is a serious problem when it comes to inflammation and cutting-edge interactive media (especially video games).

Modern video game applications are basically divided into four main non-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2, and PS3); M1Cr0S0ft Xb〇x® and Xbox 36〇8; & Nintend 〇8 and

WiiTM; and PC-based games. Each of these platforms is different from the others, so that games written to be executed on one platform are typically not executed on another platform. There may also be compatibility issues between a generation of devices and next-generation devices. Even if a big-playing software game developer builds a software game designed independently of a particular platform, the implementation of a specific game on a particular platform requires a proprietary software layer (often referred to as the "game development engine") to adapt the game. Used on the platform. Each platform is sold to the consumer or itself as a "console" (i.e., a separate box attached to the TV or monitor/speaker). Typically, video games are sold on optical media such as Blu-ray, DVD_ROM or CD_R〇M, which contains video games embodied in sophisticated real-time software applications. As home broadband speeds increase, video axes are becoming increasingly available for download. 139958.doc 200951728 Due to the immediacy and high computing requirements of the video game, the special requirements for achieving platform compatibility with §fl game software are extremely demanding. For example, we may expect full game compatibility from a generation of video games to next-generation video games (for example, from XBox to XBox 360, or from Playstati〇n 2 ("ps2") to Playstation 3 ("PS3")). Sex, as there is a general compatibility of productivity applications (eg, Microsoft Word) from one pc to another pc with a faster processing unit or core. However, this is not the case for video games. Because when a generation of video games is released, the game makers usually seek the highest possible performance for the (four) grid points, so the system is often dynamically architected so that many games written for the previous generation system are slightly The next generation system does not work. For example, XBox is based on the x86 family of processors, while the χΒ〇χ 36 is based on the PowerPC family. Techniques can be used to simulate the previous architecture', but assuming that the video game is an instant application, it is often impractical to achieve the exact same behavior in the simulation. ❿ & is known to consumers, video game console manufacturers and video game software publishers. For the consumer, it means keeping the old generation of the game console and the new generation of video game consoles connected to the TV, so that it is enough to play all the games. For console manufacturers, the cost associated with slower simulations and slower adoption of the new console. And + in the publisher, and g, which means that multiple versions of the new game may have to be released to cover all potential consumers _ not only released for each version of the video game (phantom, XBox, piaystati〇n) And often releases versions for each version of a given trademark (eg, PS2 and PS3). Example 139958.doc 200951728 And, develop a separate version of the electronic art "Crazy Rugby 8" for XBox XBox 360, PS2, PS3, Gamecube, Wii and PC platforms and other platforms. Portable devices such as cellular phones and portable media players also challenge game developers. Increasingly, these devices are connected to a wireless data network and are capable of downloading video games. However, there are a variety of cellular phones and media devices in the market that have many different display resolutions and computing capabilities. Also, because such devices typically have power consumption, cost, and weight constraints, they typically lack high-level graphics-accelerating hardware like graphics processing units ("GPUs") (such as NVIDIA's 2 devices from Santa Clara, CA). ). Therefore, game software developers typically develop a given game title that is used for many different types of portable devices. The user can find that the given game title is not available for its particular cellular phone or portable media player. In the case of a home game console, hardware platform manufacturers typically ask software game developers for a version of the tax on their ability to publish games on their platform. Honeycomb telephony wireless communication companies often also ask game publishers for royalties to download games to cellular phones. In the case of pc games there is no royalties paid for publishing games, but game developers often face high costs due to the high consumer service burden used to support multiple PC configurations and possible installation issues. In other words, pcs often have fewer barriers to piracy of game software because they can be easily reprogrammed by technically savvy users and games can be more easily pirated and more easily distributed (eg, via the Internet). . Therefore, for the software game developer I39958.doc 200951728, there are costs and disadvantages found in the game console, cellular phone and pc. For game publishers of consoles and PC software, the cost is not limited to this. In order to distribute the game via the retail channel, the publisher asks the retailer for a wholesale price that is lower than the sale price to give the retailer a profit margin. Publishers must also pay for the cost of manufacturing and distributing physical media for the game. Retailers often also ask publishers for "price protection fees" to cover possible unforeseen expenses (such as games not being sold out) or the price of the game is reduced, or the retailer must refund some or all of the wholesale price and/or recover from the purchaser. game). In addition, retailers often ask publishers for help in selling games in advertising flyers. In addition, retailers are increasingly purchasing back games from users who have completed the game, and then the games are sold as used games, and the revenue of the used games is typically not shared with the game publisher. The following facts add to the cost burden imposed on game publishers: games are often pirated and distributed via the Internet for users to download and make free copies. As the Internet broadband speed increases and broadband connectivity becomes more widespread in the United States and around the world (specifically, to the home and to the "Internet cafes" that lease PCs connected to the Internet), games are increasingly being Assigned to a PC or console via download. In addition, broadband connections are increasingly used to play multiplayer and large multiplayer online games (the two are acronym "MMOG" | allegations). These changes mitigate some of the costs and issues associated with retail distribution. Downloading online games solves some of the disadvantages of game publishers because distribution costs are usually small and there are fewer or no 139958.doc •12- 200951728 There is a cost of unsold media. But the downloaded games are still subject to piracy, and because of the Big Eight, which often has many billions of bytes, it can take a very long time to download. In addition, a plurality of games can be filled with small magnetic machines such as those sold together with a portable computer or together with a video game console. However, insofar as the game or MM〇G requires an online connection to make the game playable, the piracy problem is alleviated because the user is usually required to have a valid user account. Unlike linear media (such as 'video and music') that can be captured by the camera to capture video on the display screen or by microphone to record audio from the speaker, each video game experience is unique and cannot be used for simple video/audio Record to copy. As a result, MMOGs can be protected from piracy and thus support commerce even in areas where copyright law is not enforced and pirated. For example, Vivendi SA's World of Warcraft MM〇G has been successfully deployed and has not been pirated throughout the world. And many online or MM〇G games (such as Unden

Lab's "Second Life" MM(J) generates the revenue of the game operator via the economic model built into the game, where assets can be brought, sold and even built using online tools. Therefore, mechanisms other than the purchase or subscription of conventional game software can be used to pay for the use of online games. Although piracy is often mitigated due to the nature of the online or MMOG, online game operators still face the remaining challenges. Many games require a large amount of local (ie, in-home) processing resources for online or proper operation. The right user has a low-performance local computer (for example, a computer without a GPU, such as Low-end laptop), then it may not be able to play the game. In addition, the game console is aging, which is far behind the current technical form I39958.doc •13- 200951728 L and can be processed more months and months. High-level navigation. Even if the user's native pci is sufficient to handle the game's computational requirements, 'installation complexity often exists. There may be drive incompatibility (for example, if you download a new game, you may install a new version of the graphics driver) It causes the previously installed game that relies on the old version of the graphics driver to be inoperable. As more games are downloaded, the control port can be used to reproduce the local disk. ##Discover the defect and fix it or if the game is modified (For example, if the game developer finds that the level of the game is too difficult to play or too easy to play) 'Complex games usually receive the download patch from the beta age 1 vendor over time (pate Hp These patches require new downloads, but sometimes not all users complete all patch downloads. At other times, the downloaded patch introduces additional compatibility or disk space consumption issues. Large data download may be required to provide graphical or behavioral information to the local PC or console. For example, if the user enters the room in the MMOG and encounters a graphic material or has a user Scenes or characters that are not available on the machine must download the material of the scene or character. If the internet connection is not fast enough, this can cause substantial delay during the game. In addition, if the scene or person encountered If you need storage or computing power that exceeds the storage space or computing power of the local PC or console, it can create a situation where the user cannot continue in the game or must continue with a reduced quality graphic. Therefore, online or MMOG games often limit their storage and/or computational complexity requirements. In addition, they often limit the data transfer during the game. Online or MMOG games can also narrow the market for users of playable games. 139958.doc -14 - 200951728 In addition, technically knowledgeable users are increasingly reversing the original copy of the game and modifying the game to make It can be cheating. Cheating can be as simple as repeating a button press that is faster than manpower (for example, to shoot very quickly). In a game that supports game_asset trading, cheating can lead to fraudulent transactions involving The degree of deception of assets with actual economic value. When online or MM〇G economic models are based on the trading of such assets, this can lead to substantial harmful consequences for game operators. The cost of developing new games can be achieved with PCs and consoles. Produce more sophisticated games (for example, with more realistic graphics (such as instant ray tracing), and more realistic behaviors (such as 'instant physics simulations)). In the early days of the video game industry, video game development was very similar to application software development; that is, most of the development costs were in the development of software (as opposed to the development of graphics, audio and behavioral elements or "assets"). ), such as their software development that can be developed for use in movies with a wide range of special effects. Today's cutting-edge video game development efforts are more like software development than special effects. For example, many video games offer simulations of the 3-D world' and produce characters, props, and environments that are increasingly real (i.e., computer graphics that appear to be realistic with live action images taken by photography). One of the most challenging aspects of photo-realistic game development is to create a human face that can't be distinguished from a living person's face. Facial capture techniques (such as 'ContourTM Authenticity Capture System developed by Mova of San Francisco, CA) capture the precise geometry of the performer's face and track the precise geometry of the performer's face with high resolution while the performer is in motion shape. This technology allows the reproduction of a 139958.doc •15·200951728 3D face on a PC or game console that is virtually indistinguishable from the captured live action face. Accurately capturing and reproducing "photo-realistic" faces is useful in several ways. First, 'highly recognizable celebrities or athletes are often used in video games (often employed at high cost), and imperfections may be obvious to the user', making the viewing experience distracting or unpleasant. Often 'requires a high degree of detail to achieve a high degree of photorealistic sensibility - potentially the need to reproduce a large number of polygons and high-resolution textures (in the case where polygons and/or textures change with the movement of the face on the basis of the frame) . When a high polygon count scene with detailed texture changes quickly, the game-enabled pc or game console may not have enough RAM to store enough polygons and textures for the required number of moving frames produced in the game clip. data. In addition, a single or single drive that is typically used on a PC or game console is typically much slower than RAM and typically cannot keep up with the maximum data rate that the GPU can accept in rendering polygons and textures. When a game usually loads most polygons and textures into RAM, this means that the complexity and duration of a given scene is largely limited by the capacity of the ram. In the case of, for example, facial animation, this may limit the PC or game console to a low-resolution face that is not realistic, or limited to being paused only during the game and loaded for more frames. Polygons and textures (and other materials) were previously made into animated realistic faces in a limited number of frames. The field PC or console displays a message similar to the "Loading..." watch. The progress bar moves slowly across the screen to the user Aw of today's complex video game. The next scene is a disk (unless there is another 139958.doc 200951728 参 parameter, otherwise the "disk" of this article refers to non-volatile optical media or magnetic media, and non-magnetic such as semiconductor "flash" memory. Disc media) The delay in loading can take a few seconds or even a few minutes. This wastes time and can make gamers quite frustrated. As discussed previously, a large or all delay may be due to the loading time of polygons, textures, or other material from the disk, but may also be the case when the processor and/or GPU in the console is ready for the scene. When it comes to the data, it takes a part of the loading time. For example, a soccer video game allows players to choose among a large number of players, groups, sports fields, and weather conditions. Therefore, depending on the choice of the Het group σ, the different polygons, textures, and other materials (collectively referred to as "objects") for the scene are required for the month b (for example, different groups have different colors and patterns on their uniforms). It is possible to enumerate many or all of the various arrangements and pre-calculate many or all of the objects in advance and store the objects on a disk for storing the game. However, if the number of permutations is large, the storage required for all objects may be too large to be mounted on the disk (or too unrealistically impossible to download. Therefore, the existing and console systems are usually in the complexity of the scene and the duration of the playback. Both are constrained and suffer long loading times for complex scenes. Another significant limitation of prior art video game systems and application software systems is that they increasingly use large databases such as 'polygons' for 3D objects. And textures), these large databases need to be loaded into the pc or game console for processing. As mentioned above, when the databases are on the disk at the local end, the databases can be spent Load for a long time. 'If the database is stored in a remote location and stored via the Internet 139958.doc -17· 200951728 take 'the load time is usually much more serious. In this case, download large times The library may take a few minutes, hours, or even days. In addition, the library often generates a lot of fees (for example, for detailed in games, movies, or historical footage). There is a buckle model for sailing ships) and it is intended to be sold to the local end user. However, once the database is downloaded to the local user, there is a risk of being pirated. In many cases, the user only Estimate the database to see if it is appropriate for the user (for example, when making

It is desirable to download the database for the purpose of whether the user's 30 costumes have a satisfactory appearance or appearance when performing a specific movement. Long load times can be a hindrance to users who estimate the 3D database before deciding to make a purchase. Similar problems arise in MMOGs (specifically, games that allow users to take advantage of increasingly customized characters). For pcs or game consoles that display characters, they need to be able to access data with 3D geometry (polygons, textures, etc.) and the behavior of the characters (for example, if the character has a shield, then the shield is strong enough to deflect the spear) Library. Usually, when MM〇G* is played for the first time, a large database of characters is available under the initial copy of the game, and the initial copy of the game is available on the game disc or downloaded to the disk at the local end. However, as the game progresses, if the user encounters a person or object that the database is not available at the local end (for example, if another user has created a custom character), the user or the object must be downloaded before the person or object can be displayed. database. This can result in a substantial delay in the game. Given the sophistication and complexity of video games, another challenge for video game developers and publishers in the case of prior art video game consoles is at 139958.doc -18- 200951728 at: Developing video games often takes 2 years In 3 years, the cost is tens of millions of US dollars. Assume that the new video game console platform is introduced at a rate almost every five years, then game developers need to start their games in the years before the new game console is released. Development work to make video games available at the same time when a new platform is released. Several consoles from competing manufacturers are sometimes released at about the same time (for example, within - or two years of each other), but it remains to be seen that each console is popular (eg, which console will produce the maximum video) Game software sales). For example, in the recent console cycle ®, MlCr〇S〇ft ΧΒ〇χ 360, S〇ny motion 3 and Nintendo Wii are scheduled to be introduced in approximately the same general time frame. But in the years leading up to these introductions, game developers must essentially “press” which console platforms will be more successful than others and invest their development resources accordingly. Film production companies must also share their limited production resources for a long time before the film is released based on the film that it estimates may be successful. Game production is increasingly similar to film production given the growth of the investment required for video games, and ❹ game production companies routinely invest their production resources based on their estimates of the future success of a particular video game. But unlike film companies, this bet is not based solely on the success of the production itself; the truth is that it is based on the success of the game console on which the game is intended to execute. Simultaneously launching games on multiple consoles mitigates the risk, but this extra effort increases costs and often delays the actual release of the game. The application software and user environment on the PC are becoming more computationally intensive, dynamic and interactive, making it more visually appealing to the user and making it more useful and intuitive. For example, the new Windows VistaTM operating system 139958.doc _ 19· 200951728 and the successive versions of the Macintosh® operating system have visual effects. High-end graphics tools, such as MayaTM from Autodesk, offer very cutting-edge 3D rendering and animation capabilities that push the limits of current state of the art CPU and GPU. However, the calculation requirements for these new tools create many practical problems for users and software developers of such products. Because the visual display of the operating system (OS) must work on a variety of computers (including those that are no longer sold but can still be upgraded with the new OS), the graphics requirements are largely influenced by the computer that OS intends to use. The minimum commonality limit (which typically includes computers that do not include GPUs). This severely limits the graphics capabilities of 〇8. In addition, battery powered portable computers (e.g., laptops) limit visual display capabilities because high computing activity in the CPU or GPU typically results in higher power consumption and shorter battery life. Portable computers typically include software that automatically reduces processor activity to reduce power consumption when the processor is not being utilized. In some computer models, the user can manually reduce the processor activity. For example, the S〇nyiVGN_SZ28〇p laptop contains rStamina on one side (for low performance, longer battery life) On the other side, the switch labeled rSpeed" (for high performance, short battery life)" must be able to execute on a portable computer even if the computer is performing at its peak performance capability H rate. It can also work in place. As a result, OS graphics performance often remains available at much lower computing power than current state of the art. Frequently selling high-end, computationally intensive applications (such as Maya), expecting that these applications will be used for high performance, usually yielding much higher performance, and more expensive and less portable, The minimum commonality requirement. As a result, these applications have a much lower target audience than generic os (or generic productivity applications, similar 嶋e) and are typically sold in much lower quantities than generic OS software or general purpose application software. Potential audiences are subject to restrictions' because it is often difficult for prospective producers to try out these computationally intensive applications in advance. For example, suppose students want to know how to use Maya or already know the programs.

I hope to try Maya before investing (this may involve executing a high-end computer from Maya). When a student or potential purchaser can download a demo version or receive a physical media copy of the Maya demo version, it will not be able to perform a computer that can perform Maya to its full potential (for example, to handle complex 3 scenes) Comprehensive evaluation of the product. This essentially limits the audience of these South End applications. It also makes the sale price higher, as development costs are usually paid off through purchases that are much smaller than the number of purchases by the generic application. High-priced apps also generate more incentives for individuals and businesses that use pirated copies of the application software. As a result, high-end application software suffers from piracy, although publishers of the software have made significant efforts to mitigate the piracy through various technologies. However, even when using pirated high-end applications, it is impossible for users to exclude the need to invest in expensive current state of the art PCs to perform pirated copies. Therefore, although the user can obtain the use of the software application at a fraction of the actual retail price of the software application, the user of the pirated software still needs to purchase or obtain an expensive PC in order to fully utilize the application. 139958.doc -21 200951728 This is also true for users of high-performance pirated video games. Although pirates can get the game at a fraction of the actual price of the game, they still need to purchase the expensive computing hardware needed to play the game properly (for example, Gpu-enhanced PC' or a high-end video game console like XBox 360. The false cost of the game is usually for the entertainment of consumers. The extra cost for high-end video game systems can be too expensive. The average annual income of current workers is quite low (relative to the current average annual income of workers in the United States). The country (such as 'China) is even worse, a much smaller percentage of people have high-end video game systems or high-end PCs. In these countries, users can pay for computers connected to the Internet. "Internet cafes" are quite common. Often, these Internet cafes have older models or low-end PCs that do not have high-performance features (such as Gpus that would otherwise allow players to play computationally intensive video games). Key factors for the success of games executed on low-end PCs (such as Vivendi's "World of Warcraft", which is highly successful in China and is usually played in Chinese Internet cafes. In contrast, computationally intensive games (such as Second Life) are less likely to be played on pCs installed in Chinese Internet cafes. These games do not actually achieve the low performance of only accessing Internet cafes. Users of PCs. There are also obstacles to users who are considering purchasing a video game and are first willing to try out a demo version of the game by downloading the demo to their home via the Internet. The video game demo is often the all-around version of the game. Some features are disabled, or restrictions are imposed on the amount of game play. This may involve a long process of downloading billions of bytes of data before the game can be installed on a PC or console and executed on a PC or console ( It may take a few hours.) In the case of PC 139958.doc -22· 200951728, it may also involve calculating which special drivers the game needs (such as 'DirectX or OpenGL drivers), downloading the correct version, installing the correct version' and then Determine if the PC can play the game. This latter step may involve determining if the PC has sufficient processing (CPU and GPU) capabilities, sufficient RAM, and compatibility. OS (for example, some games are executed on wind〇ws XP and not on Vista). Therefore, after trying to perform the long process of video game presentation, the user may find that the video game demo is impossible to play (given the user's PC) Configuration. Worse, once the user has downloaded a new drive to try the demo, these drive versions may be incompatible with other games or applications that the user is accustomed to using on the PC, so the demo installation can be The inoperable game or application is inoperable. These obstacles not only frustrate users, but also create obstacles for video game software publishers and video game developers to sell their games. Another problem that is not economically beneficial is as follows Facts related: A given pc or game console is usually designed to meet the performance requirements of the application and/or game. For example, some PCs have more or less RAM, a slower or faster CPU, and a slower or faster pu (if they have a GPU). Some games or applications take advantage of the full computing power of a given pc or console, while some games or applications do not take advantage of the full computing power of a given pc or console. If the user's game or application selection does not meet the peak performance capabilities of the local PC or console, then the user may be wasting money on the PC or console due to unused features. In the case of a console, the console manufacturer may pay more than the cost of funding the console. 139958.doc •23- 200951728 Another issue that exists in the sale and enjoyment of video games involves allowing the user to watch others play the game before the user implements the purchase of the game. There are several prior art methods for recording video games for playback at a later time. For example, U.S. Patent No. 5,558,339 teaches that game state information (including game controller actions) is recorded during a "game play" on a video game client computer (owned by the same or different users). Status information

It can be used at a later time to replay some or all of the game actions on a video game client computer (e.g., a PC or console). A significant disadvantage of this method is that for a user who views a recorded game, the user must have a video game client computer capable of playing the game and must have a video game application executing on the computer to enable playback. The game play is equivalent when the recorded game state is recorded. In addition, the video game application must be written in such a way that there is no possible difference in execution between the recorded game and the played back game. For example, the game graphics system is calculated based on the frame of the frame. For many games, depending on whether the scene is particularly complex or if there are other delays that slow down execution (eg 'in PCJi, another process may be taking place so that the cpu loop is taken from the game application), the game logic can sometimes target A graph that is shorter than a frame time or longer than a frame time to calculate the image displayed as the next frame. In this type of game, the "Prevention - "" frame calculated at a slightly less time than the frame time (for example, a few CPU clock cycles) will eventually appear. When using the same game state information for the same scene, it may be easier to spend a few more CPU cycles than a frame time (for example, if the internal hangout is slightly external to 139958.doc -24- 200951728 The dram bus is out of phase, and even if there is no large delay from another process that takes a few milliseconds of CPU time from the game processing, it introduces a delay of several cycles. Therefore, when playing back the game, the frame becomes two Frame time calculations are not calculated as a single frame time. Some behaviors are based on the frequency at which the game calculates new frames (for example, when the game samples input from the game controller). When playing the game, for different behaviors This deviation in the time reference does not affect the game play, but it can cause the game being played back to produce different results. For example, if the basketball track is calculated at a stable 60 β fPs rate, the game controller input is based on When the rate of the calculated frame is sampled, the calculated frame rate may be 53 fps when the game is recorded, and is calculated when the game is played back. The frame rate may be 52 fps'. This can cause basketball to be blocked from entering the basket, which can lead to different results. Therefore, using game state recording video games requires very careful game software design to ensure the same game state information is used. Replay produces exactly the same result. Another prior art method for recording video games is to record only the video output of a PC or video game system (eg, to a VCR, a DVD recorder, or to a video capture board on a PC). The video can then be rewinded and replayed, or alternatively 'recorded video uploaded to the Internet (usually after video compression). The disadvantage of this method is that when playing back a 3D game sequence' The user is limited to viewing the sequence only by the self-test viewpoint (the sequence is recorded from it). In other words, the user cannot change the view point of the scene. In addition, when playing on the home PC or game console via the Internet, the record is recorded. The compressed video of the game sequence is available to other users 139958.doc •25· 200951728, even if the video is instant It is also impossible to upload the video of (4) to the Internet in real time. The reason is because many families connected to the Internet in the world have a highly asymmetric broadband connection (for example, fox and electric data modems usually A high-resolution video sequence with a much higher downlink frequency than the upload bandwidth often has a wider bandwidth than the network's upload bandwidth, making it impossible to upload instantly. Therefore, after playing the game sequence (possibly A few minutes or even hours), before another time on the Internet can view the game, there is a significant delay. Even though this delay is © in a specific situation (for example, watching the outcome of a game player that appeared at a previous time) Tolerable, but it eliminates the ability to watch a live game (eg, a basketball tournament played by a winning player) or "live replay" when playing a game live. Another prior art method allows viewers with television receivers to watch video game live broadcasts, but only under the control of the television producer. Some television channels in the United States and other countries provide video game video channels, where a television viewer can view a particular video game user (e.g., a top player in a tournament) on a video game channel. This is accomplished by feeding the video output of the video game system (pc and/or console) to a video distribution and processing device for the television channel. This is the case when a TV channel broadcasts a live basketball game, several of which provide live feeds from different angles around the basketball court. The television channel can then utilize its video/audio processing and effects devices to manipulate the output from various video game systems. For example, a TV channel can overlay text indicating the status of different players on top of video from a video game (as it can be overwritten during the live broadcast of the basket 139958.doc -26 - 200951728 ball game), and the TV The channel can add audio from commentators who can discuss the actions that occur during the game. In addition, the video game output can be combined with a camera that records the video of the actual player of the game (e.g., exhibiting the player's emotional response to the game). One of the problems with this approach is that the live video footage must be instantly available as a video distribution and processing device for the television channel to make it stimulating for live broadcasts. However, as previously discussed, when the video game system is executed from the home (especially when the broadcast of a package © the video of a live broadcast of a camera that is capturing the real world video of the game player), this is often not the case. may. In addition, in the case of tournaments, the concern is that the player in the family can modify the game and cheat as previously described. For these reasons, such video game broadcasts on television channels are often equipped with players and video game systems that are gathered at a common location (for example, at a television studio or in the sports%), where the television production equipment can be connected.视频 Video feed from multiple video game systems and potential live broadcast cameras. In spite of these prior art video game television channels, it is possible to provide a very stimulating decision for the television viewer (which is similar to the live broadcast of the sports event (for example, the same as the video game player presented by the "athlete"), not only based on Its actions in the video game world, and according to its actions in the real world) 'but these video game systems are often limited to situations where players are physically close to each other. In addition, because the television channels are broadcast, each broadcast channel can only display a video stream selected by the producer of the television channel. Due to these limitations and the high cost of broadcast time, production equipment and production staff 139958.doc •27* 200951728, these TV channels usually only show top players participating in top tournaments. In addition, a given television channel that broadcasts a full screen image of a video game to all television viewers displays only one video game at a time. This severely limits the choice of TV viewers. For example, a TV viewer may not be interested in a game displayed at a given time. Another viewer may only be interested in watching a particular player's game play that is not shown by the video channel at a given time. In other situations, the viewer may only be interested in seeing how the player in the game handles a particular level of the game. Other viewers may wish to control the view point (from which the video game is viewed), which is different from the view point selected by the production team or the like. In short, TV viewers may have countless preferences in watching video games (even if several different TV channels are available, the particular broadcast of the TV network should not be suitable for all of the above reasons, making the prior art video game TV channel There are significant limitations in presenting video games to television viewers. • Another disadvantage of prior art video game systems and application software systems is that they are complex and often suffer from errors, crashes, and/or unwanted and unwanted behaviors ( Collectively referred to as "defects" Although games and applications typically undergo debugging and tuning processes (often referred to as "software quality assurance" or SQA) before release, 'almost unchanged: once a game or application is released into the field The vast majority of the audience, the defects will suddenly appear. Unfortunately, 'software developers are difficult to identify and track many defects after the release. Software developers may be difficult to recognize the flaws. Even when they understand a defect, 'may exist only Can be used to identify what is causing the defect 139958.doc -28 - 20095 For example, the user can call the game developer's consumer service hotline and leave a message 'The message states: When playing the game, the screen starts to flash, then changes to solid blue and the PC freezes. It provides the SQA team with very little information that is useful in tracking defects. Some games or applications that are connected online can sometimes provide more information under certain conditions. For example, sometimes a "watchdog" can be used. The process of monitoring whether a game or application "crashes." The watchdog process collects information about the state of the game or application process when the game or application crashes (for example, the state of use of the memory stack, the extent to which the game or application progresses) Statistics, and then uploading the information to the SQA team via the Internet. But in a complex game or application, the information can take a very long time to decrypt, in order to accurately determine that the user is What to do. However, it is impossible to determine what sequence of events caused the crash. Associated with the PC and the game console One problem is that it suffers from service problems that are extremely inconvenient for consumers. Service issues also affect 1> (or the manufacturer of the game console, as it usually needs to send a special box to safely ship a broken PC or console, And thus the cost of repairs (if the pc or console is under warranty). The game or software publisher can also be affected by the loss of sales (or online service usage) caused by the PC and/or console in the repair state. Figure 1 illustrates a prior art video game system such as Sony Playstati0_3, Micr〇s〇ft xb〇x 360®, Nintend0 WiiTM, Wind〇ws-based PC or APple Macintosh. Central processing unit (10)u) for executing code (usually 139958.doc -29· 200951728

A graphics processing unit (GPU) for performing high-level graphics operations, and multiple forms of input/output (1/〇) for communicating with external devices and users. For the sake of simplicity, these components are shown as being combined into a single unit 100. The prior art video game system of FIG. 1 is also shown to include an optical media drive 104 (eg, 'DVD-R〇M drive); a hard drive 103 for storing video game code and data; for playing multiplayer games, A network connection for downloading games, patches, demos, or other media. 105. A random access memory (RAM) 101 for storing code currently being executed by the CPU/GPU 100; for receiving during game play Game controller 106 from the user's input command; and display device 1〇2 (eg, sdtv/HDTV or computer monitor The prior art system shown in Figure 1 suffers from several limitations. First, with RAM 1〇1 Compared with the speed, the optical disk drive 1〇4 and the hard disk drive ι〇3 tend to have a much slower access speed. When working directly through the RAM 1〇1, since the RAM HH usually has a much higher bandwidth and does not The fact that the CPU/Gpu 1〇〇 can handle the number of polygons per second possible when reading code and data directly from the hard disk drive 103 or the optical drive 1〇4 can be dealt with by the relatively long search delay of the optical disk mechanism. Much more per second The number of polygons. However, only a limited amount of RAM is provided in such prior art systems (for example, 256 512 dead bytes). Therefore, the sequence of "Loading..." is often required, where rami〇i is periodically filled. Data for a scene under the video game. - Some systems attempt to overlap code loading and game play at the same time, but this can only be done when there is a known sequence event (for example, if driving along a road , the geometry of the 139958.doc -30- 200951728 that can be approached by the roadside while driving the car) ^ For complex and/or fast scene changes, this type of overlap usually does not work. For example In the case where the user is in the middle of the campaign and the RAM 101 is completely filled with the information indicating the object in the view of the moment, 'If the user quickly moves the view to the left to view that the RAM is not currently loaded in the 101 The object in it will cause a discontinuity in the action, because there is not enough time to load the new object from the hard disk drive i 〇 3 or the optical disk drive 1 〇 4 into the RAM 101. The other system of Figure 1 The problem is due to hard The limitation of the storage capacity of the machine 1〇3 and the optical media丨〇4. Although the disk storage device can be manufactured to have a relatively large storage capacity (for example, 500 million bytes or 500 million bytes) Above) 'but it still does not provide sufficient storage capacity for the particular situation encountered in current video games. For example, as previously stated, a soccer video game may allow users to be in many groups around the world, Choose between players and sports fields. For each group, each player and each sports field, a large number of texture mappings and environment mappings are needed to characterize the 3D surface of the world (for example, each group has a unique sweatshirt, each of which needs A unique texture map). One of the techniques used to solve this latter problem is that for the game, the texture and environment map are pre-computed once the user has selected the texture and environment map. This can be "and many processes that are intensively' including decompressing images, 3d mapping, adding shadows, organizing data structures, and the like. Therefore, when the video game performs such a tens of calculations, there may be a delay for the user. One way to reduce this delay is in principle. The most (four) game execution performs all such calculations, including small, player rosters and sports fields. Each permutation. The release version of the game thus 139958.doc • 31 - 200951728 will include all such pre-processed material stored on optical media 104 or on one or more servers on the Internet, when the user makes a selection At this time, selected pre-processed materials for a given group, player roster, and sports field selection are downloaded to the hard disk drive 103 only via the Internet. However, as a practical matter, the pre-arrangement of each permutation possible in the game play Loaded-shell material may easily be a terabyte of data that far exceeds the capacity of today's optical media devices. In addition, the information used for a given group, player roster, and sports field selection may be easily Hundreds of millions of bytes of data or hundreds of millions of bytes of data. In the case of home network connections (for example, 10 Mbps), via the Internet Downloading this data will take longer than calculating the data at the local end. Therefore, the prior art game architecture shown in Figure 1 allows the user to experience significant delays between large scene transitions of complex games. Another problem with prior art methods of prior art methods shown in this is that video games tend to become higher order and require more CPU/GPU processing power over the years. Therefore, even with an unlimited amount of RAM, video games are hard. The physical requirements also exceed the peak level of processing power available in these systems. Therefore, users need to upgrade their game hardware every few years to keep them in sync (or play newer games at a lower quality level). One consequence of the trend of video games is that machines for video games for home use are generally not economical' because their cost is usually determined by the requirements of the highest-performance games they can support. For example, XBox may be used. 360 to play a game like "Gears of War", which requires high-performance CPU, GPU and hundreds of millions of bytes of ram, 139958.doc •32- 200951728 It is possible to use XBox 360 to play "pac Man", a game from the 1970s that requires only a few thousand bytes of RAM and a very low-performance CPU. . In fact, the XBox 360 has enough computing power to host many of the same “Pacman” games at the same time. In most hours of the week, the video game console is usually cut off. According to a July 2006 Nielsen Entertainment study of active gamers aged 13 and over, on average, active gamers spend 14 hours a week or only 12% of the total time of the week to play console video games. . This means that the flat beta video game console is idle 88% of the time. This is an inefficient use of expensive resources. It is particularly meaningful to assume that the video game console is often funded by the manufacturer to reduce the purchase price (which is expected to be earned back by royalties from future video game software purchases). Video game consoles also incur costs associated with virtually any consumer electronics device. For example, the electronics and mechanisms of the system need to be housed in a housing. Manufacturers need to provide service guarantees. Retailers selling the system are required to receive sales for the system and/or for the sale of video game software. All of these factors add to the cost of the video game console, which must be funded by the manufacturer, delivered to the consumer, or funded by both the manufacturer and the consumer. In addition, piracy is a big problem in the video game industry. In fact, the security agencies used on each of the larger video game systems have been "broken" over the years, resulting in unauthorized copying of video games. For example, the xb〇x 360 security system ruptured in July and the user now (4) downloaded an illegal copy. Downloadable games (for example, games for pc.Mac) Special 139958.doc •33- 200951728 Don't be vulnerable to piracy. In the world's top squad (where piracy is not strong), there is virtually no viable market for independent video game software, because users can easily purchase pirated copies at a cost-small rate with legitimate copies. copy. λ, in many parts of the world, the cost of the game console is a high percentage of revenue, so that even if piracy is controlled, few people can afford the current state of the art game system. ❹ ❹ In addition, the market for used games has reduced the revenue of the video game industry. When the user becomes bored with the game, they can sell the game to a store that resells the game to other users. This unauthorized but widespread practice significantly reduces the revenue of game publishers. Similarly, when there is a platform change every few years, typically about 50% of sales are reduced. This is because when the user knows that a newer version of the platform is about to be released, the user stops buying the game for the older platform (for example, when the piaystati〇n 3 is about to be released, the user stops purchasing the Playstation 2 game) 〇 combination As a result, the loss of sales and the increased development costs associated with the new platform can have a significant adverse effect on the profitability of game developers. The new game console is also very expensive. The Xbox 36, Nintend〇 Wii and Sony Playstation 3 are all retailed for hundreds of dollars. High-capacity PC gaming systems can cost up to $8000. This represents a significant investment by the user, in particular, considering the fact that the hardware became obsolete in a few years and many systems were purchased for the child. One of the above problems is an online game in which the game code and data host are hosted on a server and delivered to the client machine as required, and the compressed video and audio are streamed via a digital broadband network. Some of the 139958.doc -34- 200951728 divisions (such as F-land's G-Cluster, which is now a subsidiary of SOFTBANK Broadmedia in Japan) currently offer these services online.

Business. Similar gaming services become available in local networks such as hotels and their networks provided by DSL and TV providers. The larger drawback of these systems is the problem of latency', i.e., the time it takes for the signal to travel to the game server and travel from the game server, which is typically located at the "head end" of the operator. Fast motion video games (also known as "twitch" video games) require very low latency between when the user performs the action by the game controller and when the display screen is updated to show the result of the user action. A low latency is required to make the user feel the game respond "immediately". The user is satisfied with different delay intervals depending on the type of visual game and the proficiency of the user. For example, for slow informal games (like backgammon) or slow action role-playing games, a 100 millisecond delay may be tolerable, but in fast action games, over 70 milliseconds or 80 milliseconds. The delay can cause the user to perform worse in the game and is therefore unacceptable. For example, in a game that requires fast response time, there is a sharp drop in accuracy when the delay is increased from 50 milliseconds to 1 millisecond. When the game or application server is installed in a nearby controlled network environment or the network path to the user can predict and/or can tolerate the peak bandwidth of the network environment, the control delay is easy in terms of maximum delay and consistency of delay Waiter (for example, the user observes the steady motion from the digital video stream via the network). This level of control can be achieved either between the cable TV network headend and the home of the cable user, or from the DSL central office 139958.doc -35- 200951728 to the DSL user's home, or from the server Or in the user's commercial office area network (LAN) environment. It is also possible to obtain a point-to-point private connection with a specific hierarchy of guaranteed bandwidth and delay between businesses. But in the game or application system that hosts the game console in a server center connected to the universal Internet and then streams the compressed video to the user via a broadband connection, many factors cause delays, resulting in previous Serious limitations in the deployment of technical systems. In a typical broadband-connected home, users can have a DSL or cable modem for Broadband® service. These broadband services typically incur up to 25 milliseconds of travel delay (and sometimes more) between the user's home and the universal Internet. In addition, there is a route to the server center via the Internet. And the delay in trips caused by the trip. The delay through the Internet changes based on the delay in the route of the given data and the data being routed. In addition to routing delays, tripping delays are also incurred due to the speed of light passing through the fibers interconnecting most of the Internet. For example, the dip is about -1 GG0 mile. A round trip delay of about 22 milliseconds is incurred due to the speed of light passing through the fiber and other consuming. The extra delay can be incurred due to the data rate of the data streamed over the Internet. For example, if a user has a DSL service sold under "6 Mbps DSL service", in practice, the user will most likely get at least 5 Mbps under the line throughput, and will likely see periodically due to various factors. Connection degradation caused by (such as congestion at the Digital Subscriber Line Access Multiplexer (DSLAM) during peak load time). A similar problem can arise if there is congestion in the network of the 139958.doc •36-200951728 in the network of the congested or electrical data system system via the neighboring loop, so that it will be used to The data rate of the connected cable modem sold by the "6 Mbps Cable Data Server Service" is reduced to much less than the data rate. If the data packet at a stable rate of 4 Mbps is streamed unidirectionally from the server center via the connection in the User Datagram Protocol (UDp) format, if everything works properly, the data packet will pass without Incurs additional delays, but if there is congestion (or other obstruction) and only 3.5 Mbps can be used to stream data to the user, then in a typical situation, the packet will be discarded, resulting in lost data, or the packet will be at a point of congestion The queue is queued until it can be sent, thereby introducing additional delays. Different congestion points have different capacity for storing the delayed packets, so in some cases, the packets that cannot be successfully resolved are discarded immediately. In other cases, millions of bits of data are queued and eventually sent. However, in almost all cases, the queue at the point of congestion has a capacity limit, and once it exceeds these limits, the queue will overflow and the packet will be discarded. Therefore, in order to avoid incurring additional delays (or worse, the loss of packets), it is necessary to avoid exceeding the data rate capacity from the game or application server to the user. Delays are also incurred due to the time required to compress the video in the server and decompress the video in the client device. When the video game executed on the server is calculating the next frame to be displayed, further delay is incurred. Currently available video compression algorithms suffer from high data rates or high latency. For example, 'motion JPEG is a compression algorithm with only in-frame loss, which is characterized by low delay. Each frame of the video is compressed independently of each other frame of the video. When the client device receives a frame of compressed motion JPEG video, its i39958.doc -37· 200951728 can immediately decompress the frame and display the frame, resulting in very low latency. However, because each frame is compressed separately, the algorithm cannot take advantage of the similarity between successive frames' and thus only the in-frame video compression algorithm suffers from very high data rates. For example, 6〇 fps (frames per second) 64〇χ480 motion JPEG video may require 40 Mbps (megabits per second) or 40 Mbps (megabits per second) or more. These high data rates for these low resolution video windows will be too expensive in many broadband applications (and for most consumers' Internet-based applications). In addition, because each frame is independently compressed, artifacts in the frame that may be generated due to lossy compression may appear at different locations in successive frames. This can result in a visual artifact that appears to the viewer as moving when the video is decompressed. Other compression algorithms (such as MPEG2, H.264, or VC9 from Microsoft Corporation) can achieve high compression ratios when used in prior art configurations, but at the expense of high latency. These algorithms utilize inter-frame compression and in-frame compression. Periodically, the algorithms perform only in-frame compression of the frame.此种 This type of frame is called a key frame (commonly called an "I" frame). These algorithms then typically compare the I-frame to the previous frame and the subsequent frame. Instead of compressing the previous frame and the successive frames independently, the algorithm determines how the image changes from the I frame to the previous frame and the subsequent frame, and then stores the changes as: "B" frame (for The change before the I frame) and the "P" frame (for the changes after the I frame). This results in a much lower data rate than just in-frame compression. However, it usually comes at the expense of higher latency. The I frame is usually much larger (often 10 times larger) than the B or P frame, and therefore, 139958.doc • 38· 200951728 takes a prolonged time proportionally at a given data rate. Consider, for example, a situation where the frame is 10 times the size of the B frame and the p frame, and for each single j frame, there are 29 3 frames, 10 frames, 59 frames. A total of 60 frames, or for each "Frame Group" (GOP). Therefore, at 60 fPs, there is one 60 frame OOP per second. Assume that the transmission channel has a maximum data rate of 2 Mbps. In order to achieve the highest quality video in the channel, the compression algorithm will generate a 2 Mbps stream, and given the above ratio, this will result in 2 million bits per frame (Mb) / (59 + l 〇) = 3 〇 , 394 ❹ one place 〇 and 3 935 3,935 bits per 1 frame. When receiving a compressed video stream by a decompression algorithm, in order to stably play video, it is necessary to decompress and display each frame at regular intervals (for example, 60 £>. To achieve this result, if any picture The frame is subject to transmission delay, and all frames need to be delayed by at least one delay, so the worst case frame delay will define the delay for each view frame. Because the frame is the largest, the frame introduces the longest transmission. Delay, and the entire I frame will have to be received before the decompressible and display frame (or between any boxes depending on the I frame). Assuming the channel data rate is 2 Mbps 'the transfer frame will cost 3 〇 3,935/2 Mb=i45 milliseconds. An inter-frame video compression system (as described above) that uses a large percentage of the bandwidth of the transmission channel will experience long delays due to the large size of the I frame relative to the average size of the frame. In other words, when the prior art interframe compression algorithm achieves an average lower data rate per frame than the in-frame compression algorithm (eg, 2 Mbps vs. 40 Mbps), it still suffers high due to the large! frame. Peak data rate per frame (eg, 3〇3 935*6〇=182 Mbps). But keep in mind that the above analysis assumes that both the p-frame and the B-frame are much smaller. 139958.doc -39- 200951728 Although this is generally true, this does not hold for frames with high image complexity that are not related to previous frames, high motion, or scene changes. In these cases, the P or B frame can become j The frame is generally large (if the p-frame or b-frame becomes larger than the I-frame, the tip compression algorithm usually "forces" the frame and replaces the P-frame or B-frame with the I-frame). , j frame size data rate peak can appear in the digital video stream at any time. Therefore, for compressed video, when the average video data rate is close to the data rate capacity of the transmission channel (often for this situation, given for The high data rate requirement of the video), the high peak data rate from the I frame or the large p-frame or B-frame results in a high frame delay. Of course, the above discussion only characterizes the large b-picture from G〇p. The compression algorithm generated by the frame, p-frame or I-frame is delayed. If B is used In the frame, the delay will be more south. The reason is because all b frames and j frames after the B frame must be received before the B frame can be displayed. Therefore, in the picture group (GOP) sequence such as BBBBBIPPPPPBBBBBIPPPPP There are 5 B frames in front of each I frame, and the first B frame can be displayed by the video decompressor only after receiving the subsequent B frame and I frame. Therefore, if the video is made 60 fps (ie, 16.67 ms/frame) stream, before the first B frame can be decompressed, regardless of the fast channel bandwidth, receiving five B frames and I frames will cost ι6.67* 6 = 1 〇〇 milliseconds, and this is only the case of $ b frames. A compressed video sequence with 3 frames of B is quite common. In addition, at low channel bandwidths such as 2 Mbps, the effects of delay due to the size of the I frame are largely added to the delay caused by waiting for the B frame to arrive. Therefore, on the 2 Mbps channel, in the case of a large number of b 139958.doc • 40-200951728 frames, it is quite easy to use a prior art video compression technique with a delay of more than 5 milliseconds or more than 500 milliseconds. If the b-frame is not used (at the expense of a lower compression ratio for a given quality level), then the B-frame delay is not incurred' but still causes the delay due to the peak frame size described above. The problem is exacerbated by the nature of many video games. The video compression algorithm utilizing the GOP structure described above is largely optimized for use with live video or movie material that is intended for passive viewing. Usually, the camera (real camera, or the virtual camera in the state of the computer generated by the computer) and the scene are relatively stable, just because if the camera or scene moves too far and bumpy, the video or movie material (a) usually looks good. Unpleasant, and (b) if it is being viewed, when the camera suddenly bumps back and forth, 'the viewer is usually unable to follow the action closely (for example, if the camera is disturbed while shooting the child who blows out the butterfly on the birthday cake and Suddenly bumping back and forth between the cakes, the viewers usually focus on the children and the cake, regardless of the brief interruption of the camera when it suddenly moves. In the case of video conferencing or video conferencing, the camera can be held in a fixed position and not moved at all, resulting in very little data spikes. But 3D high-motion video games are characterized by constant motion (for example, considering 3d competitions, where the entire frame is in fast motion during the competition's performance time, or considering the first-person shooter game, where the virtual camera is strange Moving back and forth bumpyly) These visual games can produce frame sequences with large and frequent peaks, where the user may need to clearly see what happened during their sudden movements. Therefore, in 3D high motion video games, compression artifacts 139958.doc •41- 200951728 are far intolerable. Therefore, the video output of many video games (due to their nature) produces a compressed video stream with very high and frequent peaks. Assuming that the user of the fast motion video game has a small tolerance for high latency, and given all of the above delay reasons, there have been limitations to video games hosted by server hosts that stream video over the Internet. In addition, applications that require interactive applications are hosted on a common Internet and video streams are subject to similar restrictions for users of such applications. These services require network configuration, where the colocation server is placed directly at the headend (in the case of cable broadband) or in the central office (in the case of digital subscriber line (DSL)) or in a commercial context [αν (or on a specially graded private connection) to control the route and distance from the client device to the server to minimize latency and adapt to peaks without incurring delays. LANs (typically rated at 1 Mbps to 1 Gbps) and leased lines with sufficient bandwidth typically support peak bandwidth requirements (for example, 18 Mbps peak bandwidth is one of the 100 Mbps LAN capacity fractions). If special adaptation is made, the peak bandwidth requirement can also be accommodated by the residential broadband infrastructure. For example, on a cable TV system, a dedicated bandwidth can be given for digital video traffic that can handle peaks such as large I frames. In addition, on DSL systems, higher speed DSL modems (considering high peaks) can be supplied, or specially graded connections that can handle higher data rates can be supplied. However, conventional cable modems and DSL infrastructure attached to the universal Internet are far from tolerant for peak bandwidth requirements for compressed video. Therefore, online services (hosting a video game or application host in a server center that is a long distance from the client device, and 139958.doc • 42- 200951728 is then made via the Internet via a conventional residential broadband connection. The compressed video output stream) suffers from significant latency and peak bandwidth requirements - especially for games and applications that require very low latency (eg, first-person shooters and other multi-use, interactive action games, or An application that needs a quick response time). The present disclosure will be more fully understood from the following description of the embodiments and the accompanying drawings, which are not to be construed as limited. . Specific details (such as device type, system configuration, communication method, etc.) are set forth in the following description in order to provide a thorough understanding of the disclosure. However, it will be appreciated by those skilled in the art that the specific embodiments described in the practice may not require such specific details. 2a to 2b provide a high-level architecture of two embodiments, in which a video game and a software application are hosted by a colocation service 21 and hosted by a user site 2 under a subscription service, "user location It means that wherever the user is located, if the mobile device is used, it includes the outdoor device. The A-side device 205 is stored via the Internet 2〇6 (or other public network or private network). take. Client device 2() 5 can be a general purpose computer with internal or external display device 222 having a wired or wireless connection to the Internet (such as Microsoft WindWws or [like a % based or APple company) Macintosh computer, or it can be used to output video and audio to the monitor (II) terminal device (such as a wired or wireless connection to the Saki network) for monitoring II or TV 222, or it can be speculated to be 139958. Doc •43· 200951728 Comes with mobile devices with wireless connectivity to the Internet. Any of these devices may have its own user input device (eg, keyboard, button 'touch screen, track pad' or inertial-sensing wand, video capture camera and/or Or a motion tracking camera, etc.) or it may use an external input device 221 that is connected by wire or wirelessly (eg, keyboard, mouse, game controller, inertial sensing stick, video capture camera, and/or motion tracking camera) Wait). As described in more detail below, the "hosting service 21" includes servers of various performance levels (including those with high-capacity CPU/GPU processing capabilities). During playback of the game or use of the application on the colocation service 210, the home or office-to-client device 205 receives keyboard and/or controller inputs from the user, and then it inputs the controller via the Internet 2〇 6 Transfer to the colocation service 210' The colocation service 21, in response to this, execute the game code and generate successive frames for the video output (video sequence) of the game or application software (eg, if the user Pressing the button that will direct the person on the screen to the right, the game program will then produce a sequence of video images showing the person moving to the right. The video image sequence is then compressed using a low latency video compressor, and the colocation service 21 then transmits the low latency video stream over the internet 206. The home or office client device then decodes the compressed video stream and reproduces the decompressed video image on a monitor or TV. Therefore, the computational and graphics hardware requirements of the client device 205 are significantly reduced. The client 205 only needs to have the processing capability for forwarding the keyboard/controller input to the Internet 206 and decoding and decompressing the compressed video stream received from the Internet 2〇6, in fact, today I39958 .doc • 44· 200951728 Any personal computer can do this in software on its CPU (for example, Intel & Dual Core cPU running at approximately 2 GHz can decompress compressors using H.264 and Windows Media VC9 Coded 72〇ρ Μτν). Moreover, in the case of any of the client devices, the dedicated chip can also be executed on-line for much lower cost than the general purpose CPU and much less power consumption than the general purpose CPU, such as is required for modern PCs. Standard video decompression. Notably, in order to perform the functions of transferring controller input and decompressing video, the home client device 205 does not require any specialized graphics © processing unit (GPU), optical disk drive or hard disk drive (such as shown in FIG. 1). Demonstrated prior art video game system). As games and application software become more complex and more photo-realistic, they will require higher performance CPUs, Gpus, more RAM, and larger and faster drives, and will enable The computing power at the management service 210 is continually upgraded, but the end user will not need to upgrade the home or office client platform 205 because the video or office to the client will be enabled by a given video decompression algorithm. The processing requirements of platform 2〇5 remain constant for display resolution and frame rate. Therefore, there are no hardware limitations and compatibility issues seen today in the systems illustrated in Figures 2a through 2b. In addition, because the game and application software is only executed in the server in the host hosting service 21, in the user's home or office (unless otherwise conditional), the "office" as used herein will include any non- There is never a copy of the game or application software (in the form of optical media, or downloaded software) in seven backgrounds, including, for example, classrooms. This significantly reduces the possibility of illegal copying (piracy) of games or application software and the possibility of pirating valuable libraries that can be used by games or application software. In fact, if you need a specialized server (for example, a very expensive, large or noisy device) to play games or application software that is not feasible for home or office use, even if you get a game or application software. A copy of the pirated copy will not be operated in the home or office. In one embodiment, the colocation service 210 is directed to a game or application software developer (which generally refers to a software development company, a game play or movie studio, or a game or application software publisher) 220 that designs a video game. A software development tool is provided to enable it to design a game that can be executed on the colocation service 21〇. Tools such as HI allow developers to take advantage of the features of colocation services (the features are usually not available in a standalone PC or game console) (for example, very fast access to very large databases of complex geometries (unless otherwise conditional) Otherwise, "geometry" will be used herein to refer to the polygons, textures, rigging, lighting, behavior, and other components and parameters that define the 3D dataset). Under this structure, different business models are possible. Under a model, the host agency service 210 charges the subscription fee from the end user and pays royalties to the developer 22, as shown in Figure 2a. In an alternate implementation (shown in Figure 2b), the developer 220 charges the subscription fee directly from the user and pays the colocation service 210 for the cost of hosting the game or application content. These basic principles are not limited to any particular business model used to host an online game or application host. Video Compression of Warm Compression 139958.doc -46- 200951728 As discussed earlier, one of the significant problems with online video game services or application software services is latency. The 70 ms _ 80 ms delay (from the moment the input device is actuated by the user to the time the response is displayed on the display device) is the upper limit for games and applications that require fast response times. However, this is very difficult to achieve in the case of the architecture shown in Figures 2a and a few due to a large number of actual and physical constraints. As indicated in Figure 3, when a user subscribes to an internet service, the connection is typically rated at a nominal maximum data rate 301 to the user's home or office. Depending on the provider's policy and routing device capabilities, the maximum data rate may be more or less strictly enforced, but the actual available data rate is typically lower due to one of many different reasons. For example, there may be too many network traffic at the DSL central office or on the local cable modem loop, or there may be noise on the cable, causing the dropped packets, or the provider may establish per user per The maximum number of digits in a month. Currently, the maximum downlink data rate for cable and DSL services is typically in the range of hundreds of kilobits per second (Kbps) to 30 Mbps. Honeycomb services are typically limited to hundreds of Kbps of data. However, the speed of broadband services and the number of users subscribed to broadband services will increase dramatically over time. Currently, some analysts estimate that 33% of US broadband users have data rates below 2 Mbps or above 2 Mbps. For example, some analysts predict that by 2010, more than 85% of US broadband users will have data rates of 2 Mbps or more. As indicated in Figure 3, the actual available maximum data rate 302 can fluctuate over time. Therefore, under low latency, online gaming or application software, 139958.doc •47· 200951728, it is sometimes difficult to predict the actual data rate available for a particular video stream. For a given amount of scene complexity and motion at a given number of frames per second (fps) with a given resolution (eg, 64〇χ48〇@6〇 maintains the data rate required for the quality level) The 3〇3 rise is higher than the actual maximum data rate 302 (as indicated by the peak in Figure 3), and several problems can arise. For example, some Internet services will only discard packets, resulting in users. Lost data and distorted/lost images on the video screen. Other services will temporarily buffer (ie, queue) extra packets and provide them to the client at the available data rate, resulting in a delay Increase - unacceptable results for many video games and applications. Finally, some Internet service providers see the increase in data rate as a malicious attack (such as denial of service attacks (by computer hackers to make the network A well-known technique for connection deactivation)), and the user's internet connection will be disconnected during a specific time period. Therefore, the embodiments described herein seek to ensure that it is used for viewing The required data rate of the game will not exceed the maximum available data rate. ® Hosting Service Architecture Figure 4a illustrates the architecture of the Hosting Service 21〇 according to an embodiment. The Hosting Service 210 can be located in a single server center. Medium, or may be spread across multiple server centers (providing a low-latency connection to users with a lower latency to a particular server-centric path than others to provide load balancing between users, and in one or Redundancy is provided in the event of multiple server center failures.> The colocation service 21〇 can eventually include tens of thousands or even millions of servers 4〇2, so the servo is very large to use 139958. Doc -48- 200951728 user base. The colocation service control system 4.1 provides overall control over the colocation service 210 and directs routers, servers, video compression systems, § ten fees and accounting systems. Etc. In an embodiment, the colocation service control system 401 is implemented on a Linux-based decentralized processing system. The processing system is bound to be used for storage for storage. Those with information, RAIr database of information and statistics for the server system) arrays. In the above description, the various actions implemented by the colocation service 21 are initiated and controlled by the colocation service control system 401, unless attributed to other specific systems. The colocation service 210 includes a number of servers 402, such as those currently available from Intel, IBM, and Hewlett Packard, among others. Alternatively, the server 402 can be assembled into a custom component configuration, or the server 402 can ultimately be integrated to implement the entire server as a single wafer. Although this figure shows a small number of servers 402 for purposes of illustration, in actual deployments there may be as few as one server 402 or as many as millions or millions of servers 402. Server 402 can be configured in the same ® mode (as an example of some configuration parameters, with the same CPU type and performance; with or without GPU, and with GPU, with the same GPU type and performance; with the same number of CPUs And GPU; RAM with the same amount and the same type/speed; and with the same RAM configuration), or various subsets of the word processor 402 can have the same configuration (for example, 25% of the servers can be configured in a specific way) 50% of the servers are configured in a different way, and 25% of the servers are configured in yet another way, or each server 4〇2 can be different. 139958.doc -49- 200951728 In one embodiment, the server 402 is free of magnetic disks, that is, does not have its own native mass storage device (which is an optical or magnetic storage device, or a semiconductor-based storage device). A large-capacity storage component such as a flash memory or a servo-like function, each server accessing a shared mass storage via a fast backplane or network connection. In one embodiment, the quick connect is a storage area network (SAN) 403 connected to a Series of Independent Redundant Disk Array (RAID) 4〇5 series, with connections between devices implemented using Ultra High Speed Ethernet. As is known to those skilled in the art, SAN 403 can be used to group many RAID arrays 405 together, resulting in extremely high bandwidth - close to or possibly exceeding the amount of RAM that can be used in current game consoles and PCs. bandwidth. Moreover, while RAID arrays based on rotating media such as magnetic media often have significant seek time access latency, RAID arrays based on semiconductor memories can be implemented with much lower access latency. In another configuration, some or all of the servers 4〇2 provide some or all of their own mass storage at the local end. For example, the server 402 can store frequently accessed information (such as a copy of its operating system and video game or application) on a low-latency local flash memory-based storage. A SAN is used to access a RAID array 405 with a higher seek latency based on rotating media to access a large database of geometry or game state information less frequently. In addition, in one embodiment, the colocation service 210 uses the low latency video compression logic 404 described in detail below. Video compression logic 404 can be implemented in software, hardware, or any combination thereof (described in its specific embodiments below). Video compression logic 404 includes logic for compressing audio and visual material 139958.doc -50-200951728. In operation, when a video game is played via a keyboard, mouse, game controller or other input device 421 or an application at the user location 211 is used, the control signal logic 413 on the client 415 will indicate The control button 4〇6a·b (usually in the form of a UDP packet) pressed by the button (and other types of user input) is transmitted to the colocation service 21〇. In the future, the control signals from a given user are routed to the appropriate server (or to multiple servers if multiple servos respond to the user's input device) 402. Control signal 406a can be routed to server 402 via the SAN as illustrated in Figure 4a. Alternatively or additionally, control signals 4〇6b may be routed directly to server 402 via a hosted service network (e.g., an Ethernet-based local area network). Regardless of how the control signals 4〇6a-b are transmitted, the or the servers execute the game or application software in response to the control signals 4〇6a_b. Although not illustrated in Figure 4a, various network connectivity components (such as firewalls and/or gateways) can handle the hosted services 21 (e.g., colocation services 21 and the Internet 41). Incoming and outgoing traffic between the edge of the user premises 211 (between the Internet 41 and the home or office client 415). The graphics and audio output of the executed game or application Soft Zhao (i.e., the new video image sequence) is provided to the low latency video compression logic 404, and the low latency video compression logic 4 is based on low latency video compression techniques (such as, The techniques described herein compress the video image sequence and compress the video via the Internet 41 (or, as described below, via the best high-speed network service bypassing the universal Internet) The stream (usually with compressed or uncompressed audio) is transmitted back to 139958.doc 51 200951728 to the client 415. Next, the low latency video decompression logic 412 on the client 415 decompresses the video and audio streams and reproduces the decompressed video stream and typically plays the decompressed audio stream on the display device 422, or may be in a display device 422 separate speakers play audio or no audio at all. Note that although input device 421 and display device 422 are shown as stand-alone devices in Figures 2a and 2b, they may be integrated into a client device such as a portable electrical touch or mobile device. The home or office client 41 5 (described previously as home or office client 205 in Figures 2a and 2b) can be a very low cost and low power device with very limited computational or graphical performance and possibly very limited The local mass storage device does not have a local large-capacity storage. In contrast, each server 402 coupled to the SAN 403 and the plurality of RAIDs 405 can be a computing system that is exceptionally efficient, and in fact, if multiple servers are cooperatively used in a side-by-side configuration, almost There are no restrictions on the amount of computation and graphics processing that can be tolerated. In addition, the computational power of the server 402 is provided to the user due to the low latency video compression 4〇4 and the low latency video decompression 412 (perceived by the user). When the user presses the button on the input device ,, the image on display 422 is updated in response to the button press (with no meaningful delay in perception), as if the game or application software was executing at the local end. For &, for a very low-performance computer or a home or office-to-client 415 that is only a low-cost chip that implements low-latency video decompression and control signal logic 413, effectively appears to be at the remote location available at the local end Provide users with any computing power. This is (4) for the ability to play the highest-end 'processor-intensive (usually new) video games and applications with the highest performance 139958.doc •52· 200951728. Figure 4c shows a very basic and inexpensive family Or office client device 465. This device is an embodiment of the home or office client 415 from Figures 4a and 4b. It is about 2 inches long. It has an Ethernet jack 462 that interfaces with an Ethernet cable (PoE), which receives its power from the Ethernet jack 462 and its connection to the Internet. Sex. The device is capable of performing NAT in a network that supports Network Address Translation (NAT). In an office environment, many new Ethernet switches have ❿ PoE and bring PoE directly to the Internet in the office. Jack. In this case, what is required is an Ethernet cable from the wall jack to the customer terminal 465. If the available Ethernet connection does not carry power (for example, in a home with a DSL or SEM data machine but no PoE), there is an inexpensive wall-type "brick" (ie, power supply) available. It will accept an Ethernet-free electrical meter without power and output an Ethernet network with PoE. The client 465 contains control signal logic 413 (Fig. 4a) coupled to the Bluetooth wireless interface, and the Bluetooth wireless interface with a Bluetooth input device such as a keyboard, mouse, game control device and/or microphone and/or earphone. 479 interface. Moreover, one embodiment of the client 465 can output video at 120 fps with the display device 468. The display device 468 can support 12 〇 fps video and signal a pair of opaque glasses 466 (typically via infrared) for Each successive frame shades one eye and then covers the other eye. The effect that the user feels is to "jump out" to display the stereoscopic 3D image of the honor screen. One of the operations is supported. The display device 468 is a Samsung HL-T5076S. Since the video stream for each eye is separate, in one embodiment where the two independent video streams are compressed by the host 139958.doc • 53· 200951728, the frames are interleaved in time and the graph The box is decompressed in client 465 in two separate decompression processes. The client 465 also includes low-latency video decompression logic 412, which decompresses incoming video and audio and outputs 'HDMI (High Definition Multimedia Interface), connector 463 via HDMI (High Definition Multimedia Interface), connector 463 It is inserted in SDTV (Standard Definition Television) or HDTV (High Definition Television) 468 to provide video and audio to the TV, or to be inserted into the monitor 468 supporting HDMI. If the user's monitor 468 does not support ❹ HDMI ’, HDMI to DVI (Digital Visual Interface) can be used, but the audio will be lost. The 'display capability (eg, supported resolution, frame rate) 464 is conveyed from the display device 468 under the HDMI standard, and then this information is transmitted back to the colocation service 21 via the internet connection 462, thus The colocation service 210 can compress the compressed video in a format suitable for the display device.

Figure 4d shows a home or office client device 475, which is identical to the Home® or Office Client Device 465 shown in Figure 4C, except that the client device 475 has more external interfaces. Again, the client 475 can accept P〇E to power, or it can occupy an external power adapter (not shown) that is plugged into the wall. The video camera 477 provides the compressed video to the client 475 using the client 475 USB input. The compressed video is uploaded by the client 475 to the colocation service 210 for use as described below. The low latency compressor will be built into camera 477 using the compression techniques described below. In addition to having an Ethernet connector for its internet connection, the user 475 also has an 802.11 g wireless interface to the Internet. Both interfaces 139958.doc -54- 200951728 are capable of using NAT within a network that supports NAT. In addition to the HDMI connector for outputting video and audio, the client 475 also has a dual-link DVI-I connector, and the dual-link DVI-I connector includes an analog output (and has a standard that will provide VGA output). Adapter cable). It also has an analog output for composite video and S video. For audio, the client 475 has a left/right analog stereo RCA jack and has a TOSLINK output for digital audio output. In addition to the Bluetooth wireless interface to the input device 479, it also has a USB jack for interfacing to the input device. Figure 4e shows an embodiment of the internal architecture of the client 465. All or some of the devices shown in this figure can be implemented in a field programmable logic array, in a custom ASIC, or in several discrete devices (custom design or off-the-shelf). Ethernet 497 with PoE is attached to the Ethernet interface 481. Power 499 is derived from Ethernet 497 with PoE and is connected to the remaining devices in the client 465. Bus 480 is a common stream for the communication between devices. The control CPU 483 that executes the small client control application from flash memory 476 (almost any small CPU is suitable, such as the MIPS R4000 series CPU at 100 MHz with embedded RAM) is implemented for the network (ie , the Ethernet interface) is stacked and also communicates with the colocation service 2 组态 and configures all devices in the client 465. It also processes the interface with input device 469 and sends the packet (along with the user controller data subject to forward error correction protection) back to the colocation service 21〇. 139958.doc •55· 200951728 In turn, the control CPU 483 monitors the packet traffic (for example, whether the packet is lost or delayed) and the timestamp of its arrival. This information is sent back to the colocation service 210' so that it can constantly monitor the network connection and adjust the content it sends accordingly. The flash memory 476 is initially loaded with a control program for controlling the CPU 483 and a serial number unique to the particular client 465 unit at the time of manufacture. This sequence number allows the colocation service 21 to uniquely identify the user 465 unit. The Bluetooth interface 484 wirelessly communicates to the input device 469 via its antenna (within the user terminal 465). Video Decompressor 4 8 6 is a low latency video decompressor configured to implement the video decompression described herein. A large number of video decompression devices exist, either as off-the-shelf products or as intellectual property (ιρ) with a design that can be integrated into FPGa or custom ASICs. One company that provides IP for the η.264 decoder is Ocean Logic of Manly of NSW Australia. The advantage of using IP is that the compression technique used in this article does not match the compression standard. Some standard decompressors are flexible enough to be configured to accommodate the compression techniques in this article' but some standard decompressors may not. However, there is complete flexibility in redesigning the decompressor as needed in the case of IP. The output of the video decompressor is coupled to a video output subsystem 487 which couples the video to the video output of the HDMI interface 490. The audio decompression subsystem 488 is either implemented using an available standard audio decompressor' or it may be implemented as an IP, or may be implemented, for example, in I39958.doc -56 - 200951728

The audio decompression is implemented in the control processor 483 of the Vorbis audio decompressor. The device that implements the audio decompression is coupled to an audio output subsystem 489 that couples the audio to the audio output of the HDMI interface 490. Figure 4f shows an embodiment of the internal architecture of the client 475. As can be seen, in addition to the additional interface and optional external DC power from the power adapter plugged into the wall (and if so, the optional external DC power replacement will be from the Ethernet PoE 497), The architecture is the same as the architecture Φ of the client 465. The functionality common to the client 465 will not be repeated below, but the additional functionality will be described below. The CPU 483 communicates with additional devices and configures additional devices.

The WiFi subsystem 482 provides wireless internet access via its antenna as an alternative to Ethernet 497. The WiFi subsystem is available from a number of manufacturers, including Atheros Communications of Santa Clara, CA. USB subsystem 485 provides an alternative to Bluetooth communication for wired USB input device 479. The USB subsystem is fairly standard and can be easily used in FPGA ® and ASICs and is often built into existing devices that perform other functions such as video decompression. The video output subsystem 487 produces a wider range of video outputs than the video output in the client 465. In addition to providing HDMI 490 video output, it provides DVI-I 491, S-Video 492 and Composite Video 493. Also, when the DVI-I 491 interface is used for digital video, the display capability 464 is passed back from the display device to the control CPU 483 so that it can notify the host escrow service 210 of the capabilities of the display device 478. The 139958.doc •57- 200951728 interface provided by the video output subsystem 487 is a fairly standard interface and is readily available in many forms. The audio output subsystem 489 digitally outputs audio via the digital interface 494 (S/PDIF and/or Toslink) and rotates the analog form of the audio via the stereo analog interface 495. Round trip delay analysis ❹ Of course, in order to realize the benefits of the previous paragraph, the user's action between the input device 421 and the effect of seeing the action on the display device 42〇 should be no more than 7 milliseconds _80 milliseconds. This delay must account for all factors in the path from the input device 421 in the user premises 211 to the colocation service 21A and back to the user premises 211 to the display device 422 again. Figure 4b illustrates the various components and networks through which signals must travel, and above these components and the network is a timetable that lists exemplary delays that can be expected in actual implementation. Note that Figure 4b is simplified to show only the important path routing. Other routes for the other features of the system are described below. A double-headed arrow (e.g., 'arrow 453') indicates a round-trip delay and a single-headed arrow (e.g., arrow 457) indicates a one-way delay, and "~" indicates an approximate measurement. It should be noted that there will be real-world situations in which the listed delays are not achievable, but in a large number of situations, in the United States, DSL and electricity (4) connections to the user's premises are used, and such materials can be made in the situation. Also, as described in the right to voice, the Internet's cellular wireless nesting US does not work in the system's but most current US honeycomb-style beigu systems (such as EVD〇) incur very high latency and are displayed in the middle. Delay. However, these basic principles can be implemented in the future cellular technology of the ‘(4) delay of this level. 139958.doc -58- 200951728 Beginning with input device 421 at user premises 211, a user control signal is sent to user terminal 415 once the user actuates input device 421 ' (which may be a stand-alone device such as a set top box, or It can be a software or hardware that is executed in another device such as or a mobile device, and it is packetized (in one embodiment, the #UDP format) and gives the destination address to the packet to reach the colocation service. 21 〇. The packet will also contain information indicating which user the control signal is from. The control signal packets are then forwarded to the WAN interface 442 via a firewall/router/NAT (Network Address Translation) device 443. The WAN interface 442 is an interface device provided to the user premises 211 by the user's ISP (Internet Service Provider). The WAN interface 442 can be a cable or DSL modem, a WiMax transceiver, a fiber optic transceiver, a cellular data interface, an Internet Protocol interface (Internet Pr〇t〇c_1_over-powerline interface), or to the Internet. Any of a number of other interfaces. Additionally, firewall/router/NAT device 443 (and (possibly) WAN interface 442) can be integrated into client 415. An example of this would be a mobile phone that includes software for implementing the functionality of the home or office client 415, and for wirelessly routing and connecting to the Internet via a certain standard (eg, 802.11g). member. The WAN interface 442 then routes the control signals to a "presence point" 441 for the user's Internet Service Provider (ISP), which is a WAN transporter that provides connectivity to the user premises 211. A facility with an interface between a universal Internet or a private network. The nature of the point of existence will vary depending on the nature of the Internet service provided. For DSL, it will typically be the central office of the telephone company to which the DSLAM is located. For the 139958.doc -59- 200951728 electric data modem, it will usually be the forefront of the cable multi-system operator (MSO). The cellular system, which will typically be the nature of the control '...w) associated with the cellular tower, will then route the control signal packets to the universal Internet 41. The control signal packets are then routed to the side interface 444 to the host escrow service 210 via the interface that is most likely to be the fiber optic transceiver interface. WAN 444 will then route the control signal packets to, logically (which can be implemented in many different ways, including Ethernet switches and routing feeders), routing logic to fully estimate the user's address and control signals Route to the correct server for a given user. The server 402 then treats the control signals as inputs to the game or application software executing on the feeder 4〇2 and uses the control signals to process the next frame of the game or application. Once the next frame is generated, the video and audio are output from the server 402 to the video compressor 4〇4. Video and audio can be output from the server 402 to the compressor 404 via various components. The compressor 404 can be initially placed in the server 402 so that compression can be performed at the local end within the server ❹ 402. Alternatively, it may be connected via a network to a network (either a private network between server 402 and video compressor 404, or a network via a shared network such as SAN 403) (such as Ethernet) Network connection) Outputs video and/or audio in a packetized form. Alternatively, the video feed device 402 can be output' via a video output connector (such as a 'DVI or VGA connector) and then captured by the video compressor 404. Alternatively, the audio can be output from the server 402 as digital audio (e.g., via a TOSLINK or S/PDIF connector) or analog audio, and the analog audio is digitized and encoded by the audio compression logic within the video compressor 4〇4. 139958.doc -60- 200951728 Once the video 404 has captured the video frame from the server 402 and the audio generated during the frame time, the video compressor will compress the video using the techniques described below. Audio. Once the video and audio are compressed, it is packetized by an address to send it back to the user's client 415 and routed to the WAN interface 444. The WAN interface 444 is then routed through the universal Internet 410. The video and audio packets, the universal Internet 410 then routes the video and audio packets to the user's Isp presence point 1. The presence point 441 routes the video and audio packets to the WAN interface 442 at the user's premises, WAN interface 442 The video and audio packets are routed to the firewall/router/NAT device 443, and the firewall/router/NAT device 443 then routes the video and audio packets to the client 415. The client 415 decompresses the video and audio, and then displays the video on the display device 422 (or the built-in display device of the user terminal) and sends the audio to the display device 422 or to a separate amplifier/speaker or to the user terminal. Amplifier/speaker. In order for the user to feel that the entire process just described is perceptually free of lag, the round trip delay needs to be less than 70 milliseconds or 80 milliseconds. Some of the delay delays described in the return trip path are controlled by the colocation service 210 and/or the user, while other delay delays are not controlled by the colocation service 210 and/or the user. Nonetheless, based on the analysis and testing of a large number of real-world situations, the following are approximate measurements. The one-way transmission time 451 for transmitting the control signal is typically less than 丨 milliseconds, and the round-trip routing 452 via the user's premises is typically performed using an easy-to-access consumer-grade firewall/router/NAT switch on the Ethernet. 139958.doc 61 200951728 Completed in seconds. The user ISP extensively changes its round trip delay 453, which is typically seen between milliseconds and 25 milliseconds for DSL and cable modem providers. The back-and-forth delay of the Universal Internet 41 is how the routing is routed and whether there are any faults on the route (and these issues are discussed below), but the general Internet is usually quite optimal. Routing and delay are largely determined by the speed at which light passes through the fiber (the distance given to the destination). As discussed further below, 1000 miles have been determined as the approximate farthest distance that is desired to place the colocation service 21 away from the user premises 211. At 1 mile (2,000 miles back and forth), the actual transmission time for signals via the Internet is approximately 22 milliseconds. The WAN interface 444 to the colocation service 21 is typically a commercial grade fiber optic high speed interface with negligible latency. Therefore, the universal internet latency 454 is typically between 1 millisecond and 10 milliseconds. The one-way routing 455 delay via the colocation service 210 can be achieved in less than one millisecond. Server 4〇2 will typically calculate a new perimeter box for the game or application at less than one frame time (which is 16.7 milliseconds at 60 fps), so 16 milliseconds is the reasonable maximum one to be used. Delay 456. In the best hardware implementation of the video compression and audio compression algorithms described in this article, compression 45? can be completed in 1 millisecond. In the next best version, compression can take up to 6 milliseconds (of course, a less desirable version can take longer, but these implementations will affect the total latency of the round trip and will require other delays to be shorter (eg, subtractable) Small via the allowable distance of the universal Internet) to maintain a 7 〇 _ 8 〇 millisecond delay target). The round trip delay of Internet 454, user ISP 453 and user location route 452 has been considered, so the remaining video decompression 458 extension 139958.doc -62· 200951728, video decompression 458 delay depends on video decompression 458 is implemented in a dedicated hardware or in a software implemented on a client device 415 (such as a PC or mobile device), the size of the visual display and the performance of the decompressing CPU. Typically, decompression 458 takes between 1 millisecond and 8 milliseconds. Thus, the worst-case back-and-forth delay that the user of the system shown in Figure 4a can expect to experience is determined by adding together all the worst-case delays seen in practice. It is: 1 + 1+25+22+1 + 16+6+8=80 milliseconds. In addition, 'actually' is in practice (with the description of the misconceptions discussed below) 'This is roughly the prototype version of the system shown in Figure 4a (using an off-the-shelf Windows PC as a client device and home DSL in the US) And the electric gauge data machine connection) see the back and forth travel delay. Of course, situations that are better than the worst can lead to much shorter delays, but you can't rely on them to develop widely used business services. In order to achieve the delays listed in Figure 4b via the universal internet, the video compressor 404 and the video decompressor 412 (from Figure 4a) in the client 415 are required to generate a packet stream having very specific characteristics such that The sequence of packets generated from the entire path of the colocation service 210 to the display device 422 is not subject to delay or excessive packet loss, and in detail 'continuously falls on the Internet connection via the user ((4) WAN interface 442 and firewall / Router / NAT 433) can be used within the constraints of the user's bandwidth. In addition, the video compressor must generate a sufficiently robust packet stream so that it can tolerate normal Internet and network transmissions; see the inevitable packet loss and packet reordering. Low-Temperature Video Compression 139958.doc -63· 200951728 To accomplish the above objectives, an embodiment employs a new video compression method that reduces the delay and peak bandwidth requirements for video transmission. Prior to describing such embodiments, an analysis of current video compression techniques will be provided with respect to Figures 5 and 6a through 6b. Of course, if the user has sufficient bandwidth to handle the data rate required by such techniques, then such techniques can be used in accordance with the underlying principles. Note that this does not address audio compression, but rather states that audio compression is implemented concurrently and synchronously with video compression. Prior art audio compression techniques that meet the requirements for this system exist. Figure 5 illustrates a particular prior art for compressing video in which each individual video frame 501-503 is compressed by compression logic 520 using a particular compression algorithm to produce a series of compressed frames 511-513. One embodiment of this technique is "Motion JPEG" in which each frame is compressed based on a discrete cosine transform (DCT) according to a joint image expert group (jpeg) compression algorithm. Various different types of compression algorithms can be used, however, these basic principles are still adhered to (for example, wavelet-based compression algorithms such as jpEG·2000). One of the problems with this type of compression is that it reduces each The data rate of the frame 'but it does not take advantage of the similarity between successive frames to reduce the data rate of the total video stream. For example, as illustrated in Figure 5, assume that the frame rate of 640x480x24 bits/pixel = 640*480*24/8/1024=900 kilobytes/frame (KB/frame) is given For a quality image, Motion JPEG may only compress the stream by 1/10 ' resulting in a stream of 90 KB/frame. In 6 frames/sec, 'this will require 90 KB*8 bits*60 frames/sec = 42.2 Mbps channel bandwidth' for almost all of the US home Internet today with 139958.doc 200951728 s will be extremely high bandwidth and too high bandwidth for many office internet connections. In fact, assuming that it requires a constant stream of data at such high bandwidths, and it will only serve one user, it will consume 1% of the 131^ Ethernet LAN even in an office LAN environment. A large percentage of the bandwidth and a heavy Ethernet switch that supports the LAN. Therefore, compression for motion cameras is inefficient when compared to other compression techniques, such as those described below. In addition, a single frame compression algorithm using a lossy compression algorithm (eg, 15 £(} and ;]?£(}_2〇〇〇) produces compression artifacts that may be unobtrusive in still images (eg False shadows in dense foliage in the scene may not appear as artifacts, because the eye does not know exactly how dense leaves should be presented. However, once the scene is in motion, artifacts may protrude because eye detection An artifact that changes from the frame to the frame, although the artifact is in the area of the scene (in which the artifact may not be noticeable in still images). This leads to a red scene in the sequence of frames. The perception of "noise" is similar to the "snow" noise seen during TV reception. Of course, this type of compression can still be used in the specific embodiments described in this article, but generally In order to avoid background noise in the scene, a high feed rate (ie, low compression ratio) is required for a given perceived quality. Other types of compression (such as H.264, or Windows Media VC9, MPEG2 and MPEG4) in compressed view Both are more efficient in the stream because they take advantage of the similarity between successive frames. These techniques all rely on the same general technique for compressing video. Therefore, although the H.264 standard will be described, the same general principles apply. For a variety of other compression algorithms. A large number of 264 264 pressure 139958.doc -65- 200951728 reducer and decompressor available, including x264 open source software library for compression Η.264 and FFmpeg open for decompressing H.264 Source Software Library Figures 6a and 6b illustrate an exemplary prior art compression technique in which a series of uncompressed video frames 501-503, 559-561 are compressed by compression logic 620 into a series of "I frames" 611, 671. "P-frame" 612-613; and "B-frame" 070. The vertical axis in the figure generally represents the resulting size of each of the encoded frames (although the frames are not drawn to scale) As described above, those skilled in the art have a good understanding of the video writing code using the I frame, the B frame and the P frame. In short, the I frame 611 is the DCT of the completely uncompressed frame 501. Based compression (similar to compressed as described above) JPEG image). The size of the p-frames 612-613 is typically significantly smaller than the size of the frame 611 because it utilizes the data from the previous I-frame or P-frame; that is, it contains the indication of the previous I-frame or P-picture. The changed material between the boxes. In addition to the B-frame using the frame of the subsequent reference frame and (possibly) the reference frame in the previous frame, the B-frame 670 is similar to the P-frame. 'It will be assumed that the desired frame rate is 6 frames/sec. Each I frame is about 160 Kb, and the average P frame and B frame are 16 Kb and a new I frame is generated every second. Under this set of parameters, the average data rate will be: 160 Kb + 16 Kb * 59 = ll Mbps. This data rate falls appropriately within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid background noise problems from intra-frame coding only, because P-frames and B-frames track the difference between frames, so compression artifacts tend to appear and disappear from the frame to the frame. To reduce the background noise problem described above. 139958.doc -66- 200951728 One of the problems with compression of the above type is that although the average data rate is relatively low (for example, 1.1 Mbps), a single I frame may take several frame times to transmit. For example, using prior art, a 2.2 Mbps network connection (eg, a DSL or cable modem with a 2.2 Mbps peak from the maximum available data rate 302 of Figure 3a) would typically be sufficient to stream video at 1.1 Mbps 'per 60 frames a 160 Kbps I frame. This will be done by having the decompressor put a 1 second video into the queue before decompressing the video. Within 1 second, 1.1 Mb of data will be transmitted 'it will be easily accommodated by the maximum available data rate of 2.2 Mbps, even if it is assumed that the available data rate may periodically drop by as much as 50%." Unfortunately, this prior art The method will delay the video by 1 second due to the 1 second video buffer at the receiver. This delay is sufficient for many prior art applications (e.g., 'playback of linear video), but it is extremely long latency for fast motion video games that cannot tolerate delays greater than 7 milliseconds _8 milliseconds. If an attempt is made to eliminate the 1 second video buffer, it will still not cause sufficient delay reduction for the fast motion video game. For example, as previously described, the use of the frame will require receipt of all the frames before the frame.图 frame and [frame. If it is assumed that approximately 59 non-1 frames are split between the frame and the frame, then there will be at least 29 frames and the I frame received before any frame can be displayed. Therefore, regardless of the available bandwidth of the channel, a delay of 1/6 sec duration or a delay of milliseconds for each of the 29 1 30 frames is required. Obviously, his time is extremely long. Therefore, the other method will be to eliminate the frame and only use the frame and ρ map C (this is followed by . For a given quality level, the data rate will increase by 139958.doc •67· 200951728 plus, but for this example For consistency, we continue to assume that each frame is 160 Kb and the average P frame size is 16 Kb, and therefore the data rate is still 1.1 Mbpsh. This method eliminates the inevitable introduction of the B frame. The delay, because the decoding of each p-frame depends only on the previously received frame. The problem with this method is that the frame is much larger than the average p-frame, so that it is on the low-frequency wide channel (such as most In the home and in many offices, the transmission of the I-frame adds a substantial delay. This is illustrated in Figure 6b. The video stream data rate 624 is lower than the maximum data rate 621 available (except for the I-frame), where The peak data rate 623 required for the frame far exceeds the maximum data rate 622 available (and even exceeds the rated maximum data rate of 621). The data rate required for the P frame is less than the maximum data rate available. Even at 2.2 Mbps. The rate peak is steadily maintained at its 2 2 Mbps peak rate' and will also cost 160 Kb/2.2 Mb = 71 ms to transmit the I frame, and if the maximum data rate 622 is reduced by 50% (1.1 Mbps), it will take 142 milliseconds. To transmit the I frame. Therefore, the delay in the transmission I frame will fall somewhere between 71 milliseconds and 142 milliseconds. This delay is added to the delay identified in Figure 4b (the delays are in the worst case) A total of 7 milliseconds), therefore, this will result in a total round trip delay of 141-222 milliseconds from the moment the user actuates the input device 421 until the image is presented on the display device 422, which is extremely high. The data rate drops below 2.2 Mbps, and the delay will increase further. Also note that ISP "blocking" often exceeds the peak data rate 623 of the available data rate 622. There are often serious consequences. Devices in different ISPs will behave differently, but When the data rate is much higher than the available data rate 622 139958.doc -68- 200951728 ^ the following behavior in the DSL and the electric data machine isp when " 曰 曰 :: (4) by the packet into the Column Packet delay (introduction delay), (8) discard some or all packets '(4) disable connection for a period of time (most likely because 1SP is concerned about malicious attacks, such as "denial of service" attack, therefore" to transmit packets at full data rate Streams (having characteristics such as those not shown in Figure 6b) are not a viable option. The value of the 620 can be exceeded in the colocation service 2H) (four) ^ below the maximum available data rate

The data rate of the rate is transmitted 'and thus introduces the unacceptable delay described in the previous paragraph. In addition, the video stream data rate sequence shown in Figure 6b appears to be a very "tame" H1 data rate sequence and will be expected to result from the compression of video from the video sequence. The video sequence does not change very much and has very little motion (eg, as would be common in an sfL conference call, in a video conference call, the camera is in a fixed position with very little motion, and the scene ( For example, objects in the seated person § inflammatory words show less exercise). The video stream data rate sequence 634 shown in Figure 6c is a typical sequence that is expected to be visible from video with much more motion, such as may be produced in a movie or video game or in an application software. Note that there is also a P-frame peak (such as 63 5 and 636) that is quite large and in many cases exceeds the maximum available data rate, except for the peak 633 of the work frame, although the peaks of these P-frames are not as good as the I-picture. The box peak is generally quite large, but it is still so large that it cannot be carried by the ramp at full data rate, and as the I-frame peak-like peak of the P-frame must be transmitted slowly (by increasing the delay). 139958.doc -69- 200951728 On a high-bandwidth channel (for example, 100 Mbps LAN, or high-bandwidth ι〇〇Mbps private connection), the network will be able to tolerate such things as a frame peak ο] or a p-frame peak 636 The big peak, but in principle, can maintain low latency. However, such networks are often shared among many users (for example, in an office environment), and the "peak" data will affect the performance of the LAN, especially if the network traffic is routed to a private shared connection (eg In the case of remote data center to office). First, remember that this example is an example of a relatively low resolution video stream of 640M80 pixels at 6〇 fps. 6〇 fps © l92〇x 1080 HDTV streams are easily handled by modern computers and monitors, and 2560x1440 resolution displays at 60 fps are increasingly available (for example, Apple's 30" displays). Using H 264 compression, a high motion video sequence of 1920x1080 as follows may require 4 5 Mbps to achieve a reasonable quality level. If the I-frame peak is assumed to be 1〇 of the nominal data rate, it will produce a 45 Mbps peak and a small but still quite large p-frame peak. :3⁄4•Several users are receiving video streams on the same Mbps network (for example, the private network connection between the office and the data center), so it is easy to see how the peak of the video stream from several users can happen. Alignment, thereby drowning the bandwidth of the network, and possibly flooding the bandwidth of the backplane of the switch supporting the user on the network. Even in the case of ultra-high-speed Ethernet, if enough users have enough peaks for simultaneous alignment, they can flood the network or network switch. In addition, once 256〇xl44〇 resolution video becomes more common, the average video stream data rate may be 9.5 Mbps, which may result in a 95 Mbps peak data rate. Needless to say, (4) The 1 Mbps connection between the center and the office (which is now an extraordinarily fast connection) will be completely defeated by the peak traffic of 139958.doc •70-200951728 from a single user. Therefore, even if LAN and private network connections are more tolerant for peak-streaming video, streaming video with high peaks is not required and may require special planning and adaptation by the office. Of course, for standard linear video applications, these issues are not a problem 'because the data rate is "smoothed" at the transmission point and used for each frame - the material is below the maximum available data rate 622 and is decompressed] Before the frame, p frame and B frame sequence, the 'buffer in the client side stores the I frame, P © ® box and B frame sequence. Therefore, the f rate on the network remains close to the average data rate of the video stream. Unfortunately, this introduction delay, even if the B-frame is not used, is unacceptable for low-latency applications such as video games and applications that require fast response times. A prior art solution for mitigating video streams with high peaks is to use techniques commonly referred to as "blame positioning meta-rate" (CBR) encoding. Although the term CBR appears to imply that all graphs (4) are shrunk to have the same bit material (ie, A is small)' but it is often referred to as a dust-shrinking paradigm, in the compression paradigm 'allows a certain number of frames to be crossed (In our case, the maximum bit rate of the "fixed frame." For example, if the CBR constraint is applied to the code under the conditions of the figure (which limits the bit rate to, for example, a nominal maximum data rate of 62K70. /.), then the compression algorithm will limit the compression of each of the frames so that any frame that would normally be compressed using more than 70% of the maximum f-rate 621 will be in fewer bits. compression. The result of this is that it will normally require more bits to maintain a given quality level. The frame is "extremely lacking" and the image quality of these frames will be less than 139958.doc 71 . 200951728 The image quality of other frames rated at more than 70% of the maximum data rate of 62% is poor. This method can produce acceptable results for a particular type of compressed video (where (a) less motion or scene change is expected and (b) the user can accept periodic quality degradation. A good example of a suitable CBR application is a video conference call because there are fewer peaks and there is a temporary degradation in quality (for example, if the camera is swept, resulting in significant scene motion and large peaks, it may not exist during the glance) Sufficient bits are used for high quality image compression, which will result in degraded image quality), which most users can accept. Unfortunately, CBR is not well suited for many other applications with high complexity scenarios or large amounts of motion and/or requiring a reasonably constant quality level. 0 The low latency compression logic used in one embodiment uses a number of different Technology to solve the many problems of streaming low latency compressed video while maintaining high quality. First, the low-latency compression logic 4〇4 only generates the j-frame and the 1> frame, thereby alleviating the need to wait for a number of frame times to decode each of the 6 frames, as illustrated in the figure, in- In an embodiment, the low latency algorithm 404 divides each uncompressed frame 7〇1·76〇 into a series of “tiles” and individually encodes each image block into a frame or ρ frame. In this paper, the group of compressed frames and frames are referred to as "r frames" 711-770. In the particular example shown in Figure 7a, each of the uncompressed frames is subdivided into 16 image blocks of a 4x4 matrix. However, these basic principles are not limited to any particular sub-dividing mechanism. In one embodiment, the low latency compression logic 4〇4 divides the video frame into a plurality of image blocks and encodes the image block from each frame (ie, 139958.doc • 72· 200951728 is compressed) into i. Frame (ie, compressing the image block as if it were 1/16 of the size of the full image) and compressing the "mini" frame to I frame compression) and leaving The image block is encoded as a p-frame (ie, the compression for each "mini" 1/16 frame is p-frame compression). The image blocks compressed into I frames and the image blocks compressed into P frames will be referred to as "I image blocks" and "P image blocks", respectively. The image block to be encoded as an I image block is changed with each successive video frame. Therefore, in a given frame time, only one of the image blocks in the video frame is an image block, and the remaining ones of the image blocks are P image blocks. For example, in Figure 7a, the image block of uncompressed frame 701 is encoded as image block 1 and the remaining 1-150 image blocks are encoded as p-image blocks (匕 to 匕5). An R frame 711 is generated. In the next uncompressed video frame 702, the uncompressed picture busy block 101 is encoded as an I picture block I and the remaining picture blocks 2 and 2 to 15 are encoded as P picture blocks (P 〇 'and P2 to P 〗 5) to generate R frame 712. Therefore, the I image block and the P image block used for the image block are gradually interleaved in time on successive frames. The process continues until an R image block 77 is generated (the last image block in the matrix is encoded as an I image block (ie, so far. The process then restarts, resulting in, for example, frame 711 (ie, for image block 0) Another R frame, such as an I picture block, etc. Although not illustrated in Fig. 73, in an embodiment, the first R frame of the video sequence of the R frame contains only the image block (i.e., Subsequent P-frames have reference image data (from which motion is calculated. Or, in one embodiment, the startup sequence uses the same I-image block type as normal, but does not include encoding for the image block that has not been encoded with the image block. The p-image block of the image block. In other words, the specific image block is not encoded together with any data before the arrival of the first frame image 139958.doc • 73· 200951728 block, thereby avoiding the video stream data rate 934 in FIG. 9a. The peak is activated, which is described in further detail below. In addition, as described below, various sizes and shapes can be used for the image blocks while still complying with these basic principles. The video solution is performed on the client 415. The compression logic 412 decompresses each image block as if it were a separate video sequence of the small I frame and the P frame, and then reproduces each image block to the frame buffer of the drive display device 422. For example, use The image block 视 of the video image is decompressed and reproduced from the frames 711 to 770 of the R frames 711 to 770. Similarly, the image block 1 is reconstructed from the R frames 7 to 77, and the like. The decompression of the j-frame and the p-frame is well known in the art, and the decompression of the J image block and the Ρ image block can be performed by the plurality of execution entities of the video decompressor at the user terminal 415. Execution is done. Although the multiplication process seems to increase the burden on the client 415, it does not actually increase the computational burden on the client 415 because the image block itself is proportionally smaller (relative to the extra In terms of the number of processing), therefore the number of pixels displayed is the same 'as if there is a processing and using the full size of the conventional ones] [frame and ρ frame. This R frame technique significantly reduces the usually associated with the I frame Peak bandwidth (Figure 6b and Figure 6c) As explained, because any given frame is mainly composed of P frames that are usually smaller than the I frame. For example, suppose that the typical frame is 160 Kb, then the frame illustrated in Figure 7a Each of the 1 image blocks will be approximately 1/16 or 1〇Kb for this amount. Similarly, assuming that the field p frame is 16 Kb, it is used for each of the image blocks illustrated in Figure 7a. The p-frame can be approximately 1 Kb. The final result is approximately 1 〇Kb+15M Kb=25 Kb2R Figure 139958.doc -74- 200951728 box. Therefore 'every 60 frame sequence will be 25 Kb*60=l. 5 Mbps. So 'at 60 frames per second, this would require a channel capable of maintaining a bandwidth of 15 Mbps' but with a much lower peak due to the dispersion of the I image block throughout the 60 frame spacing. Note that in the previous example, the average data rate was M Mbps with the same assumed data rate for both the frame and the p frame. This is because in the previous example, only a new I frame is introduced every 60 frame time, and in this example, the 16 image blocks constituting the I frame are cycled in 16 frame times, and therefore, An equalization of a j-frame is introduced every 16 frame times, resulting in a slightly higher average data rate. Despite this, in practice, the introduction of more frequent I frames does not linearly increase the data rate. This is due to the fact that the P frame (or P image block) mainly encodes the difference from the previous frame to the next frame. Therefore, if the previous frame is quite similar to the next frame, then the P frame will be very small. If the previous frame is quite different from the next frame, the P frame will be very large. However, because the P-frame is largely derived from the previous frame, rather than being derived from the actual frame, the resulting coded frame may contain more errors than a frame with a sufficient number of bits (for example, Visual artifacts). In addition, when a P frame follows another P frame, an error accumulation can occur (when there is a long P frame sequence, it gets worse). The current video compressor will detect the fact that the quality of the image is degraded after a sequence of P frames, and if necessary, it assigns more bits to the subsequent 1) frame to improve quality, or if it is the most For an effective action process, replace the frame with a frame. Therefore, when using a long ρ frame sequence (for example, 59 frames, as in the previous example above), it is usually the case that the scene has a large number of complex 139958.doc •75· 200951728 noise and/or motion, usually For the P frame, more bits are needed (because it becomes farther away from the I frame). Or 'from the relative view point to see the p-frame, the p-frame that closely follows the J-frame tends to require fewer bits than the P-frame farther from the I-frame. Thus, in the example shown in Figure 7a, the p-free frame is farther apart from the frame (before the 1 frame) and in the previous example the 'P frame can be from I The frame is separated by 59 frames. Therefore, in the case of more frequent I frames, the P frame is smaller. Of course, the exact relative size will vary based on the nature of the video stream. However, in the example of Figure 7 & if the I image block is 10 Kb, the size of the P image block can be only 0.75 kb ', resulting in 10 Kb+15. *0.75 Kb=21.25 Kb, or at 60 frames per second, the data rate will be 21.25 Kb*60=l.3 Mbps, or have an i frame at 1_1 Mbps followed by 59 p frames The data rate of the stream is about 16 high. /(^ Again, the relative result between these two methods for video compression will vary depending on the video sequence, but in general, we have found by experience that 'for a given quality level, using the r frame is better than using I The /p frame sequence requires about 20% more bits. However, of course, the r frame sharply reduces the peak'. This makes the video sequence available at a much slower delay than the I/p frame sequence. The nature of the video sequence, The R frame is configured in a number of different ways for the reliability of the channel and the available data rate. In an alternative embodiment, a number of image blocks other than 16 are used in the 4χ4 configuration. For example, it can be 2χ1 or 1 Two image blocks are used in the x2 configuration, and four image blocks can be used in a 2χ2, 4X1 or 1χ4 configuration. You can use 6 image blocks in a 3x2, 2x3, 6x1 or 1x6 configuration or can be used in 4χ2 8 image blocks are used in the 2x4, 8x1 or 1 χ8 configuration (as shown in Figure 7b). Note that the 'image block does not need to be square, the video map 139958.doc •76· 200951728 does not have to be square, or even Rectangular ❶ breaks down the image block to best fit the video stream used and Regardless of the shape of the program, in another embodiment, the loop of the I image block and the P image block is not locked to the number of image blocks. For example, in the 8 image block 4x2 configuration, it can still be as shown in Fig. 7b. The 16-cycle sequence is used for the description. The sequential uncompressed frames 721, 722, and 723 are each divided into 8 image blocks 〇_7, and each image block is individually compressed. From R frame 731, only the image block The image is compressed into 1 image block, and the remaining image blocks are compressed into P image blocks. For the subsequent R frame 732, all 8 image blocks are compressed into p image blocks, and then for the subsequent R frame 733, the image Block 1 is compressed into the image block and the other image blocks are compressed into P image blocks. In addition, the 16 frames continue to be sorted, and only one I image block is generated every other frame, so in the 15th picture During the frame time period (not shown in FIG. 7b) and during the 16th frame time period, the last I image block for the image block 7 is generated (using all p image block compression R frame 78〇). Then, the sequence Again, image block 0 is compressed into j image block and other image blocks are compressed into P The image block begins. As in the previous embodiment, the first frame of the entire video sequence will typically be an I image block to provide a reference β image block and a p image block for the P image block forward from the other point. The loop is not even an even multiple of the number of shirts. For example, in the case of 8 image blocks, before using another frame, there is a block of images that can be followed by each frame. Both are two frames of the image block. In yet another embodiment, if, for example, a particular area of the screen is known to have more motion (requiring more frequent I image blocks) and other areas are more static (eg , showing the score of the game) (requiring less frequent I image blocks), compared with other image blocks 139958.doc -77- 200951728, can more often sort a specific image block together with the 丨 image block. Furthermore, although each frame has a single work image block as illustrated in Figures 7a-7b, multiple j image blocks can be encoded in a single frame (depending on the bandwidth of the transmission channel). Conversely, a particular frame or frame sequence can be transmitted without a block (i.e., only a P block). The reason why the method of the previous paragraph works properly is that although it does not have to be scattered across each single frame; the image block seems to cause a large peak, but the behavior of the system is not so simple. Because each image block is compressed separately from other image blocks, when the image block becomes smaller, the stone of each image block can become less effective because the compressor of a given image block cannot utilize other images. Similar image features of the block and similar motion. Therefore, dividing the screen into 16 image blocks will generally result in less efficient coding than dividing the screen into 8 image blocks. However, if the screen is divided into 8 image blocks and it causes the data of a full I frame to be introduced every 8 frames instead of every 16 frames, it results in a much higher data rate overall. . Therefore, by introducing a full frame every 16 frames (rather than every 8 frames), the total data rate is reduced. Moreover, by using eight larger image blocks (rather than 16 smaller image blocks), the total data rate is reduced, which also reduces the peak of the data caused by the larger image blocks to some extent. In another embodiment, the low latency video compression logic 404 in FIGS. 7a and 7b is preconfigured or based on image quality in each image block by virtue of known characteristics of the video sequence to be compressed. The ongoing analysis automatically controls the allocation of bits to each of the image blocks in the R circle. For example, in some competitive video games, the player's car (which is scene J39958.doc -78· 200951728: for no motion) occupies most of the lower half of the screen, while the upper half is completely It is filled with roads, buildings and scenery that are approaching, and two are almost always in motion. If the compression logic class assigns an equal number of bits to each image block, the image block (image block 4_7) on the lower half of the screen in the uncompressed frame 721 in circle 7b will generally be compared. The image block in the upper half of the screen in the uncompressed frame 721 (the image block is compressed by the same product f. If this particular game or game is known to have such characteristics The operator of the colocation service 210 can set up the compression logic 404 to allocate more bits to the image block at the top of the screen (as compared to the bits allocated to the image block at the bottom of the screen). Alternatively, _ logic 404 may estimate the compression quality of the image block after compressing the frame (using one or more of a number of compressed print metrics, such as a beacon signal to noise ratio (PSNR)) 'and if determined at a particular time window In the case where a particular image block consistently produces better quality results, it gradually assigns more bits to the image block that produces lower quality results until the various image blocks reach a similar level of quality φ. In an example, compressor logic 404 allocates bits to Achieving higher quality in a particular image block or image block group. For example, it provides a better overall perceived appearance to have a higher quality at the center of the screen than at the edges. In one embodiment, to improve video quality The resolution of a particular region of the stream, video compression logic 404 uses a smaller image block to encode the video stream with relatively more scene complexity and/or motion (with relatively less scene complexity and video streaming) / or the area of motion), for example, as illustrated in Figure 8, in an R frame 811 (may be followed by a series of R frames with the same image block size 139958.doc -79· 200951728 J ( A small image block is used around the moving person 805 in one of the areas (not shown). Then, when the character 8〇5 moves to a new area of the image, 'around the new area in the other-R frame 812. Using smaller image blocks, as explained, as described above, various sizes and shapes can be used as "image blocks" while still following these basic principles. Although the cyclic I/P image blocks described above are substantially reduced Video streaming speed The peak in the rate, but it does not completely eliminate the peak, especially in the case of fast-changing or highly complex video images (such as in movies, video games, and some application software). For example, in a sudden scene During the transition, a complex frame may be followed by another complex frame that is completely different. Even if several I image blocks can lead the scene transition only a few frame times, it is not helpful in this case because of the new The material of the frame is independent of the previous I image block. In this case (and in other cases where a large number of images change even if not everything changes), the video compressor 4〇4 will determine many, if not all, of the The Ρ image block is more efficiently coded as an I image block and results in a very large peak in the data rate of the frame. As previously discussed, it is only for most consumer-grade Internet connections (and many office connections), which can only be "blocked" beyond the maximum data rate available and the maximum data rate shown in Figure 6c as 622. 621 information. Note that the rated maximum data rate 621 (e.g., "6 Mbps DSL") is essentially a sales figure for a user considering purchasing an internet connection, but generally it does not guarantee a performance level. For the purposes of this application, it is irrelevant because we only focus on the maximum data rate 622 available when streaming video over a connection. Thus, in Figures 9a and 9C, when the solution to the peak problem is described in 139958.doc • 80 - 200951728, the nominal maximum data rate is omitted from the graph and only the maximum available data rate 922 is shown. The video streaming data rate must not exceed the maximum available data rate of 922. To solve this problem, the first thing the video compressor 404 does is to determine the peak data rate 941, which is the data rate at which the channel can be processed steadily. This rate can be determined by a number of techniques. A data rate test stream that is increasingly subject to this technology is gradually sent from the colocation service 210 to the user 415 (Figs. 4a and 4b) and the user is enabled. The end provides feedback on the level of packet loss and delay to the colocation service. After the packet loss and/or delay begins to show a sharp increase, after the private message reaches the maximum available data rate, the colocation service 210 can gradually reduce the test stream data rate until the client 415 reports at a reasonable time. The test stream has been received in the cycle (packet loss and acceptable acceptable level of delay are close to minimum). This determines the peak maximum data rate 94 which will then be used as the peak data rate for streaming video. Over time, the peak data rate 9 network 41 will fluctuate (e.g., if another user in the family begins to use severely = network connection), and the client 415 will need to monitor the peak data ambiguously == Check if the packet loss or delay increases (indicating the maximum available and: 2 drops below the previously determined peak data rate 941), and right so, then the peak data rate 9 goes 1 household (10) is now sealed = ground, money time If π β , and the delay remains at the optimal level, then, ask the video compressor to slowly increase the data rate (for example, if the family rate is available to view the most available Internet connection) Serious use - the user has stopped again. · People wait until the packet is lost 139958.doc 200951728 Loss and/or the higher delay indication has exceeded the maximum data rate 922 available, and can be reconsidered the lower peak data 941 Level, but this lower level may be souther than the level before testing the increased data rate. Therefore, the peak data rate 941 can be found by using this technique (and other techniques like this). And periodically adjust as needed. "Peak data rate 9" is determined by the video pressure, the reducer 4〇4 is used to make the maximum data rate. The logic used to determine the peak data rate can be == at location 211 and/or It is implemented on the colocation service 21. At the use site 21!, the client device 415 performs calculations to determine the peak data rate and transmits this information back to the colocation service 21; in the colocation service At 210, the server 4主机2 of the colocation service performs a calculation to determine the peak data rate based on statistics received from the client 415 (eg, peak loss, delay, maximum data rate, etc.). An example scene complexity and/or motion instance video stream data rate 934 is generated using a cyclic ί/P image block compression technique as previously described and illustrated in Figures, Figures, and 8. Video Compressor The 404 is configured to output compressed video at an average data rate of 94 低于 below the peak data rate, and note that most of the time, the video stream data rate remains below the peak. A comparison of the material rate 941»data rate 934 with the video stream data rate 634 shown in Figure 6c (which is generated using an I/P/B or I/p frame) shows that the cyclic P/P image block compression is smoothed. Multiple data rates. However, at frame 2 times peak 952 (which is close to 2 times peak data rate 942) and frame 4 times peak 954 (which is close to 4 times peak data rate 944), the data rate still exceeds the peak data rate 941. It is unacceptable. In practice, even for high-motion video from fast-changing video games, the peak value exceeding the peak data rate 941 appears in the frame of less than 2%, exceeding The peak of the 2x peak data rate 942 is rare, and the ridge value of more than 3 times the peak data rate of 943 is rare. However, when it does occur (e.g., during a scene transition), the data rate required must produce a good quality video image. One way to solve this problem is to simply configure the video compressor 404 to have its maximum data rate output as the peak data rate 941. In the will, the resulting video output quality during the peak frame is poor because the compression algorithm is "extremely lacking". What is caused is that compression artifacts occur when there is a sudden transition or rapid motion, and in time, the user begins to realize that artifacts always appear suddenly when there is a sudden change or rapid motion, and it can become quite annoying. Although the human visual system is quite sensitive to visual artifacts that occur during sudden changes or rapid motion, it is not very sensitive to the reduction in frame rate detected in such situations. In fact, when these sudden changes occur, it seems that the human visual system is focused on tracking these changes, and if the frame rate temporarily drops from 60 fps to 30 fps and then immediately returns to 60 fps, then human vision The system will not notice. In addition, in the case of very sharp transitions (such as sudden scene changes), if the frame rate drops to 2 〇 fps or even 15 fps and then immediately returns to 60 fps, the human visual system will not notice. As long as the frame rate is reduced, it only happens occasionally. For human observers, it seems that the video system is constantly being executed at 60 fps. This feature of the human visual system is utilized by the technique illustrated in Figure 9b. 139958.doc -83- 200951728. Server 402 (from Figures 4a and 4b) produces an uncompressed video output stream at a stable frame rate (in one embodiment, at 60 fps). The timetable shows the output of each frame 961-970 for each 1/60 second. Starting from frame 961, each uncompressed video frame is output to the low latency video compressor 4〇4, and the low latency video compressor 404 compresses the frame for less than one frame time, resulting in The compressed frame of the first frame 丨 981. The data produced for the compressed frame 1 981 may be larger or smaller depending on many of the factors previously described. If the data is small enough to be transmitted to the client 415 at a round frame time (1/6 sec) or less than one frame time, then the transmission time (xmit time) 991 (indicating the transmission time) It is transmitted during the length of the arrow of duration. In the next frame time, the feeder 402 produces an uncompressed frame 2, which is compressed into a compressed frame 2 982, and at a peak data rate 941 during a transmission time 992 less than one frame time. It is transmitted to the client 415. Next, in the next frame time, the server 4〇2 produces an uncompressed frame 3 963. When the uncompressed frame 3 963 is compressed by the video compressor 4〇4, the resulting compressed frame 3 983 is more data than can be transmitted at the peak data rate 941 in the frame time. Therefore, it is transmitted during the transmission time (2 times peak) 993, which occupies all the frame time and the time of the next frame time. Now, during the next frame time, the server 402 generates another uncompressed frame 4 and outputs it to the video compressor 404, but the data is ignored and illustrated by 974. This is because video compressor 404 is configured to ignore other uncompressed video frames that arrive when they are still transmitting previously dusted frames. Of course, the video decompressor of the client 139958.doc • 84 - 200951728 4 1 5 will not receive the frame 4, but it simply displays the frame 3 over the display device 422 for 2 frames time ( That is, temporarily reduce the frame rate from 60 fps to 30 fps). The next frame 5' server 402 outputs uncompressed frame 5 965, which is compressed into compressed frame 5 985 and transmitted in frame 1 during transmission time 995. The video decompressor of client 415 decompresses frame 5 and displays it on display device 422. Next, the server 402 outputs the uncompressed frame 6 966, and the video compressor 4〇4 compresses it into a compressed frame 6 986, but the data obtained at this time is very large. The compressed frame is transmitted at the peak data rate 941 during the transmission time (4 times peak) 996, but it takes almost 4 frame times to transmit the frame. During the next three frame times, the video compressor 404 ignores the three frames from the server 4〇2, and the decompressor of the client 415 stably holds the frame 6 on the display device 422 for a period of time 4 Frame time (ie, temporarily reduce the frame rate from 6〇fps to 15 fps). Finally, the server 4〇2 outputs the frame 1〇97〇, and the video compressor 404 compresses it into the compressed frame 1〇987, and transmits it during the transmission time 997, and the solution of the client 415 The compressor decompresses the frame 10 and displays it on the display device 422 and restarts the video again at 6 〇 fps. /Thinking, although the video compressor 404 discards the video frame from the video stream generated by the server 4〇2, it does not discard the audio material (regardless of the form of the audio system), and discards the video map. The frame-time video compressor 4〇4 continues to compress the audio data and transmits it to the client 415. The client 415 continues to decompress the audio data and provides the audio to the user for playback of the sound 139958.doc -85- 200951728 No matter what device. Therefore, during the period in which the frame is discarded, the audio continues without being attenuated. Compressed audio consumes a relatively small percentage of the bandwidth compared to compressed video and therefore does not have a significant impact on the overall data rate. Although not illustrated in any of the > material rate maps, there is always a data rate capacity reserved for the compressed audio stream within the peak data rate 941. The example just described in Figure 9b is chosen to illustrate how the frame material is under % during the peak data rate, but not stated when using the cyclic P/P image block technique described previously & There are very few peaks and consequent discards, and R also makes this a high scene complexity/high dynamics column (such as those found in video games, movies, and an application software). Therefore, the reduced frame rate is rare and temporary, and the human visual system does not measure them. If the frame rate reduction mechanism just described is applied to the video stream data rate shown in Figure 9 & then the resulting video stream data is illustrated in Figure 9c.

rate. In this example, the '2x peak 952 has been reduced to a flattened 2x peak 953' S4 times the bee 955 has been reduced to a flattened 4x peak 955, and the entire video stream data rate 9 3 4 remains at or Below the peak data rate of 94 Ιώ J | JFC. ” 丨畑 筏 , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , In an office environment on an Ethernet or 8 (2) wireless network (for example, a (10) s connection between a data center and an office), it can transmit high without peaks. 139958.doc •86- 200951728 Motion video streaming so that multiple users (for example, 1920x1080 at 60 fps at 4.5 Mbps) can use LAN or shared private data connections without flooding the network or network with overlapping peaks Path Switch Base. Data Rate Adjustment In one embodiment, the colocation service 210 initially evaluates the channel's available maximum data rate 622 and delay to determine the appropriate data rate for the video stream and then dynamically adjusts the data in response thereto. rate In order to adjust the data rate, the colocation service 210 can, for example, modify the resolution of the video stream to be sent to the client 415® and/or the number of frames per second. Also, the colocation service can adjust the compressed video. The quality level. When changing the resolution of the video stream (e.g., from 1280x720 resolution to 640x360), the video decompression logic 412 on the client 41 5 can scale the image to maintain the same image size on the display screen. In one embodiment, in the event that the channel is completely logged out, the colocation service 210 suspends the game. In the case of a multiplayer game, the colocation service reports to other users that the user has quit the game and/or will Game pause ❹ 卩 for other users. Discard or late packet In one embodiment, if the data is lost due to packet loss between video compressor 404 and client 415 in Figure 4a or 4b, or The video decompression logic 4丨2 can be mitigated because the packets that arrive too late to decompress and satisfy the delay requirements of the decompressed frame are lost in an unordered manner. False shadow. In a streaming I/P frame implementation, if there is a lost/delayed packet, the entire screen is affected, which may cause the screen to freeze completely. 13995S.doc -87 - 200951728 - Time period or other screen width Visual artifacts. For example, if a lost/delayed packet causes a loss of 1 frame, the decompressor will lack a reference for all p-frames to follow before receiving a new cap. If the p-frame is lost 'Then it will affect the p-frames that are used for the entire screen. Depending on how long before the frame appears, this will have a longer or shorter visual impact. Use the interlacing shown in Figure 7a and Figure 71) I/p blocks, missing/delayed blocks are less likely to affect the entire screen because they only affect the image blocks contained in the affected package. If the data of each image block is sent in individual packets, (4) if the packet is lost, it only affects the image block. Of course, the duration of the visual artifact will depend on how much the image block is lost and how many frames will be spent before the image block appears in the case of the loss of the p-image block, but the different images on the screen are assumed. Blocks are updated very frequently (perhaps with each frame) by 1 frame, and even if the image block is affected on the screen, other image blocks may not be affected. In addition, if an event causes several packets to be lost at the same time (for example, 'the spike of the temporary interrupt e data stream in the power adjacent to the DSL line), some image blocks will be more affected than other image blocks, but because some images The block will be updated quickly by the new image block, so it is only temporarily affected. Moreover, in the case of the implementation of the stream P/P frame, not only the 1 frame is the most critical frame, but the I frame is extremely large, so if there is an event causing the packet to be discarded/delayed, then it is much smaller. The I image block has a higher probability of being affected than the I frame (that is, if any part of the I frame is lost, it is impossible to decompress the I frame). Since all of these reasons are compared to the case of the Ι/P frame, the use of the Ϊ/Ρ image block results in a much smaller visual artifact when the packet is discarded/delayed. 139958.doc -88 - 200951728 An embodiment seeks to reduce the effects of lost packets by intelligently encapsulating the collapsed video blocks into TCP (Transmission Control Protocol) packets or UDP (User Datagram Protocol) packets. For example, in an embodiment, the image block is aligned with the packet boundary whenever possible. Figure 10 & illustrates how the image block can be encapsulated in a series of packets i _ _ 1005 without implementing this feature. Specifically, in Figure i〇a, the image block crosses the packet boundary and is inefficiently packaged so that the loss of a single packet causes multiple frames to be lost.

Lost. For example, if the packet 1〇〇3 or 1〇〇4 is lost, three image blocks are lost, resulting in visual artifacts. In contrast, Figure 10b illustrates image block packaging logic 1010 for intelligently encapsulating image blocks within a packet to reduce the effects of packet loss. First, image block encapsulation logic 1010 aligns the image block with the packet boundary. Therefore, the image blocks ΤΙ, Τ3, Τ4, Τ7, and Τ2 are aligned with the boundaries of the packets 1〇01_1〇〇5, respectively. Image block encapsulation logic also attempts to combine image blocks into packets in a potentially more efficient manner without crossing the packet boundaries. Based on the size of each of the image blocks, the image block and the redundancy are combined in one packet 1〇〇1; the Τ3 and Τ5 are combined in one packet 10〇2; the image blocks Τ4 and Τ8 are combined in one In L 1003, the image block D8 is added to the packet 1〇〇4; and the image block D2 is added to the packet 1005. Therefore, under this scheme, a single packet loss will result in the loss of no more than 2 image blocks (rather than the 3 image blocks as illustrated in Figure 〇a). An additional benefit of the embodiment shown in Figure 1b is that the image blocks are transmitted in a different order in which they are displayed within the scene/image. If the neighboring packet is lost due to the same event that interferes with the transmission (which would affect the areas on the screen that are not close to each other 139958.doc -89 - 200951728 domain), this method produces a less noticeable artifact on the display. An embodiment uses forward error correction (FEC) techniques to protect a particular portion of the video stream from channel errors. As is known in the art, FEC techniques such as Reed-Solomon and Viterbi generate error correction information and attach it to the material transmitted via the communication channel. If an error occurs in the basic data (for example, I frame), FEC can be used to correct the error. The FEC code increases the data rate of the transmission, so ideally it is only used when it is most needed. If the material is being sent and it will not result in a very noticeable visual artifact, then the FEC code may preferably not be used to protect the material. For example, a P-Picture block immediately before a lost I-picture block will only produce a 1/60 second visual artifact on the screen (ie, the image block on the 'screen will not be updated). This visual artifact can hardly be detected by the human eye. As the p-image block is further backwards from the I-image block, the loss of the P-image block becomes more and more noticeable. For example, if the image block cycle pattern is such that one I image block is followed by 15 P image blocks before the I image block is available again, if the P image block is immediately after the I image block ❹ 丢失, then It causes the image block to display an incorrect image for 15 frame times (at 60 fps, he will be 250 milliseconds). The human eye will easily detect a stream interruption of 250 milliseconds. Therefore, the more the p image block is from the new I image block (that is, the closer the P image block follows the I image block), the more noticeable the artifact is. As discussed earlier, in spite of this, in general, the closer a p-picture block follows an I-picture block, the smaller the data used for the P-picture block. Therefore, the P-image block following the I image block is not only more secure for protection from loss, but also smaller. In addition, in general, the smaller the data, the smaller the FEC code needed to protect it. Thus 'as illustrated in Ua', in one embodiment, due to! The importance of image blocks in the video stream, only! The image block has a fecma. Therefore, the coffee 1101 contains 1 image block! (10) The error correction code and fec "(10) contain an error correction code for the I picture block 11G3. In this embodiment, no FEC is generated for the picture block.

In the embodiment illustrated in Figure m, a code is also generated for a p-image block that is most likely to cause visual artifacts when lost. In this embodiment, b FEC 1105 provides an error correction code for the first 3 p-image blocks but not for the following p-picture blocks. In another embodiment, the FEC code is generated for the p-image block with the smallest data size (which would tend to be the P-block that was originally generated after the j-image block, which is most critical for protection). In another embodiment, instead of transmitting the FEC code along with the image block, the image block is transmitted twice, each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment shown in FIG. lie, 1^(: codes 1111 and 1113 are generated for the audio packets 111〇 and 1112 respectively transmitted from the colocation service at the same time as the video. The integrity of the audio in the video stream is maintained. Sex is especially important because distorted audio (eg, clicks or clicks) will result in a particularly undesirable user experience. FEC Ma helps ensure that the audio content is on the user side; brain 415 is reproduced without distortion. In an embodiment, instead of transmitting the FEC code together with the audio data, the audio data is transmitted twice, each time in a different packet. 139958.doc -91 · 200951728 If one packet is lost/delayed, then another In addition, in one embodiment illustrated in FIG. 9D, FEC codes 1121 and 1123 are used for user input commands (eg, button presses) U 2 for uplink transmission from the client terminal 415 to the colocation service 21, respectively. 〇 and 1122. This is important because 'missing button presses or mouse movements in video games or applications may result in an undesirable user experience. In another embodiment 'not the FEC code Instead of the user inputting the command data, the user inputs the command data twice, each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment, the host generation The management service 210 evaluates the quality of the wanted channel with the client 415 to determine whether to use the FEC, and if so, determines which portion of the video, audio and user commands should be applied for FEC. The "quality" of the evaluation channel may include The functions such as estimating packet loss, delay, etc. If the channel is particularly unreliable, the colocation service 2 can apply FEC to all I video blocks, p video blocks, audio and user commands. If the channel is reliable, the colocation service 21 can apply FEC only for audio and user commands, or FEC can be applied to audio or video, or FEC can be used at all. Various other arrangements of FEC applications can be used, The basic principles are still adhered to. In an embodiment, the colocation service 210 constantly monitors the status of the channel and changes the FEc policy accordingly. Referring to Figures 4a and 4b, when the packet is lost/delayed, resulting in loss of image block data, or if the packet may be lost due to a particularly bad packet loss, the FEC cannot correct the lost image block data. 139958.doc -92 · 200951728 The client 415 evaluates how many frames remain before receiving a new 1 image block and compares it to the round trip delay from the + client 415 to the colocation service 21 (). The right round trip delay is less than the new The (9) image block should reach the number of previous frames. Then the client 415 sends a message to the colocation service 21 () to request a new I image block. This message is routed to the video (4) device 4G4, and it is not generated. For the p-image block of the image block that has been lost, but the image block is generated. It is assumed that the system shown in Figures 4a and 4b is designed to provide a round trip delay of typically less than 80 milliseconds, which results in the image block being corrected. Within 8 〇❹ milliseconds (at 60 fPs, the frame has a duration of 16.67 milliseconds, so in full frame time, the 80 millisecond delay will result in a corrected image block of 83 to 33 milliseconds, 83.33 milliseconds is 5 Frame time, which is a noticeable disruption apos but far less (e.g.) for the frame 25〇 Shu 5 millisecond interrupt introduction Note). When the compressor 404 is out of its normal cyclic order to produce such an image block, 'if the I image block will cause the bandwidth of the frame to exceed the available bandwidth, the compressor 404 will delay the loop of the other image blocks so that Other image blocks receive P-blocks during the frame time (even if one image® block will normally be an I-picture block during the frame), and then the usual loop will continue from the next frame' and will usually The image block that has received the image block in the previous frame will receive the I image block. Although this action temporarily delays the phase of the R frame cycle, it will usually be visually unobtrusive. Video and Audio Retractor/Decompression Device Implementation Figure 12 illustrates a particular embodiment in which multiple cores and/or multiple processors 1200 are used to compress eight image blocks in parallel. In one embodiment, a dual core processor, quad core xeon CPU computer 139958.doc -93. 200951728 system implemented at 2.66 GHz or higher is used, and each core implements an open source Χ264 Η.264 compressor as a separate process. However, various other hardware/software configurations can be used while still adhering to these basic principles. For example, each of the CPU cores can be replaced by an H.264 compressor implemented in an FPGA. In the example shown in Figure 12, the cores 1201-1208 are used to process the I and Q blocks simultaneously as eight separate threads. As is well known in the art, current multi-core and evening processor computer systems are integrated with Microsoft's Windows XP Professional (64-bit or 32-bit version) and Linux multi-threaded processing systems. It is inherently capable of multi-thread processing. In the embodiment illustrated in Figure 12, because each of the § cores is only responsible for one image block' so it operates largely independently of the other cores, each performing a separate instantiation of x264. Use PCI Express xl-based DVI capture cards (such as the Sendron video imaging ip development board from Microtronix of Oosterhout in the Netherlands) to capture uncompressed video at 640x480, 800x600 or 1280x720 resolutions and FPGA on the card Direct memory access (DMA) is used to transfer the captured video to the system ram via the DVI bus. The image blocks are configured in a 4x2 configuration 1205 (although this is illustrated as a square image block, in this embodiment it has a 160x240 resolution). Each instantiation of χ264 is configured to compress one of the eight 160x240 image blocks, and is synchronized such that each core enters a loop after initial I image block compression, each frame and another The frames are out of phase to compress an I image block followed by seven Ρ image blocks, as illustrated in Figure 12, at each frame time, using the previously described technique to obtain the resulting pressure I39958.doc -94 - 200951728 The reduced image blocks are combined into a packet stream, and the compressed image block is then transmitted to the destination client 415. Although not illustrated in Figure 12, 'if the data rate of the combined 8 image blocks exceeds the specified peak data rate 941', then all 8 χ264 processes will temporarily suspend the necessary frame time' until the 8 for the combination has been transmitted. The data of the image block. In one embodiment, client 415 is implemented as a software executing on 8 instantiated PCs of FFmpeg. The receiving process receives eight image blocks and routes each of the image blocks to FFmpeg for instantiation, and FFmpeg instantiates the decompressed image blocks and reproduces them to the appropriate image block locations on display device 422. The client 415 receives the keyboard, mouse or game controller input from the input device driver of the PC and transmits it to the server 402. The server 402 then receives the input device data received by the application and applies it to the game or application executed on the server 4〇2, and the server 402 is a PC that executes Windows using the Intel 2.16 GHz dual-core CPU. Server 402 then generates a new frame and exports the new frame to the motherboard based graphics system via its DVI output or via the DVI output of the NVIDIA 8800GTX PCI Express card. At the same time, the server 402 outputs audio generated by the game or application via its digital audio output (eg, S/PDIF), which is coupled to a dual quad-core Xeon-based PC that implements video compression. Digital audio input. The Vorbis open source audio compressor is used to compress audio simultaneously with video, regardless of the core that can be used to process the thread. In one embodiment, the core of the compressed image block is first compressed to first perform audio pressure 139958.doc -95· 200951728, and then the audio signal is transmitted together with the compressed video, and the Vorbis audio is used on the client terminal 41 5 . Decompressor to decompress the compressed audio. The colocation service server center branches the light traveling through the glass (such as an optical fiber) at a fraction of the speed of the light in the vacuum' and thus the exact propagation speed of the light in the fiber can be determined. However, in practice, considering the routing delay, transmission inefficiency, and other elapsed time, we have observed that the optimal delay on the Internet reflects a transmission speed that is closer to 50% of the speed of light. Therefore, the optimal 1 mile round trip delay is about 22 milliseconds, and the best 3 mile round trip delay is about 64 milliseconds. Therefore, a single server on the US coast will be too far apart to serve the client on the other coast (which may be up to 3 miles away) with the desired delay. However, as illustrated in Figure 13a, if the colocation service 210 is located at the center of the United States (e.g., Kansas, Nebraska, etc.), the distance to any point in the continental United States is about 1500 miles or 1,500 miles. Below, the round-trip internet latency can be as low as 32 milliseconds. Referring to Figure 4b, note that although the worst case delay allowed by the user ISP 453 is 25 milliseconds, it is generally observed that in the case of DSL and cable modem systems, a delay of approximately 1 〇 -1 5 milliseconds is observed. Again, Figure 4b assumes that the maximum distance from the user premises 211 to the colocation center 210 is 1000 miles. Thus, in the case of the typical 15 millisecond user ISP round trip delay used and the maximum internet distance of 1500 miles for a 32 millisecond round trip delay, the time from when the user actuates the input device 421 The total round trip delay for seeing a response on display device 422 is 139958.doc • 96- 200951728 1 + 1 + 15+32+1 + 16+6+8=80 milliseconds. As a result, an 80-millisecond response time is typically achieved over an Internet distance of 1,500 miles. This will allow any user location in the US mainland with a sufficiently short user ISP delay of 453 to be accessed in a centrally located single server center. In another embodiment illustrated in Figure 13b, the colocation service 21" server centers HS1-HS6 are strategically located around the United States (or other geographic area) 'a larger hosted service server center Positioned close to high population centers (eg, HS2 and HS5). In one embodiment, server centers HS1-HS6 exchange information via network 1301. Network 13〇1 may be an internet or private network or a combination of both. In the case of multiple server centers, users with high user ISP delays 453 can be served at lower latency. Although the distance on the Internet is indeed a factor in the latency of travel back and forth via the Internet, other factors that are largely unrelated to latency also work. Sometimes the packet flow is routed to a remote location via the Internet and returned again, resulting in a delay from the long loop. Sometimes there are routing devices that are not operating properly on the path, resulting in delays in transmission. There are sometimes traffic that overloads the path, which introduces delays. In addition, sometimes, there is a failure to prevent the user's ISp from being routed to a given destination. Therefore, although the universal Internet usually provides connections from one point to another with fairly reliable and optimal routing and latency, this fairly reliable and optimal route and delay is largely determined by distance (especially In the case of a long-distance connection that leads to the outside of the local area of the user, but the reliability and delay are not guaranteed and often cannot be used by the 139958.doc 97· 200951728 user's place to the general Achieved by a given destination on the Internet.

In an embodiment, when the user client 41 5 is initially connected to the colocation service 210 to play a video game or use an application, the client is in startup with the available colocation server server Hs 1 _HS6 Each of them communicates (eg, using the techniques described above). If the delay is low enough for a particular connection, use the connection. In one embodiment, the client communicates with all of the colocation service server centers or a subset of the colocation service server centers, selecting the colocation service server center with the lowest latency connection. The client can select the service center with the lowest latency connection, or the server center can identify the server center with the lowest latency connection and provide this information (for example, in the form of an internet address) to the client. The host is overloaded and/or the user's tour is less loaded. The hosted service can be used if the specific hosted service server or the application can tolerate the delay to the other server center. Directed to another host agent service server. In this case, the game or application being executed by the user will be paused on the word server 4〇2 at the user's overloaded server center. The game or application status data is transferred to the other host hosting service server center. Then (4) restart the game or program. In the implementation, the colocation service 210 will wait until the game or application reaches the natural pause point (for example, between levels in the game, or when the user starts "save" in the application towel to advance: In yet another embodiment, the colocation service is 21°, the user activity is stopped for a specified time period (eg, α 139958.doc -98 - 200951728) and then the transmission will begin at that time. As described above, in an embodiment, the colocation service 210 subscribes to the Internet bypass service 440 of Figure 14 in an attempt to provide a guaranteed delay to its client. As used herein, the Internet wrap The service provides a service from a point in the Internet to another point with a guaranteed private network route (eg, delay, -bee rate, etc.). For example, if the colocation service 210 is positive Using the AT&T DSL service provided by San Francisc〇 to receive a large amount of traffic from the user (rather than routing to the central office based in AT&T San Francisco), the colocation service 210 will Hiring a high-capacity private service from a service provider (possibly AT&T itself or another provider) between one or more of the central office based in San Francisco and the server center for the colocation service 210 Data connection. Then, from all the hosted service server center HS1-HS6 via the universal Internet to San Francisc〇, the route using the user of Ατ & τ DSL leads to excessive delay, then the private data can be used instead. Connection. Although private data connections are usually more expensive than routes via the universal Internet, as long as they maintain a small percentage of the colocation service 21 to the user, the total cost impact is low and the user will experience A more consistent service experience. In the event of a power failure, the ship center often has two spare power layers. The first layer is usually from the battery (or from an alternate ready-to-use energy source, such as keep running and attached to the hair The standby power of the flywheel of the motor, which provides power immediately when the mains fails, and keeps the servo center running. If the power failure is temporary And the power mains returns quickly (eg 139958.doc -99· 200951728 in /7 hours), then the battery needs to keep the center of the feeder running. But if the power failure lasts for a long period of time, it usually starts Generators (such as diesel powered) replace the battery and the generators can operate as long as they have fuel. These generators are extremely expensive because they must be capable of generating as much power as the servo center typically draws from the mains. Each of the colocation services HS1 HS5i|j shares user data with each other so that when there is a power failure in a server center, the game and application in progress can be suspended, and then the game or ^ The application state f is transferred from each server 4G2 to the word server 402' at the other server center and then each user's client 415 is notified to direct it to the new server 4〇2 . Assuming that such occasions occur occasionally, the user is transferred to a hosted service server center that is unable to provide the best latency (ie, the user will only have to tolerate the duration of the higher latency duration power failure). Accepted, it will allow for the option of transferring a much wider range of users. For example, given the time difference across the US, users on the East Coast may be sleeping at n:3〇pM, while users on the West Coast are starting to peak in video games at 8:3〇PM. . If there is a power failure in the hosted service server center on the West Coast, there may not be enough West Coast servers 4〇2 for handling all users at the other hosting server center. In such a case, some users may be transferred to a colocation service server center with an available server 402 on the East Coast, and the only consequence for the user will be a higher latency. Once the user has transferred from the server center where the power has been lost, the server center can then start its server and set the orderly cut off for the battery (or other immediate power backup) 139958.doc -100· 200951728 Turn off all equipment before exhausting. In this way, the cost of the generator for the center of the feeder can be avoided. In one embodiment, during the time of severe loading of the colocation service 210 (either due to peak user loading or because one or more server centers fail) 'based on the game or application being used by the user The program delay requires the user to be transferred to another server center. Therefore, users of games or applications that require low latency will be given the preference for available low latency server connections with limited © supply. Master Managed Service Features Figure 15 illustrates an embodiment of the components of the server center for the hosted services 210 utilized in the following description of the features. As with the colocation service 210 illustrated in Figure 2a, the components of this server center are controlled and coordinated by the colocation service 210 control system 4〇1 unless otherwise conditional. The incoming Internet traffic 15〇1 from the user client 415 is directed to the incoming route 1 502. Usually, the Internet traffic server will enter the server center via high-speed fiber connection to the Internet, but any network connection component with sufficient bandwidth, reliability and low latency will be sufficient. . Inbound routing 1502 is a network (which can be implemented as an Ethernet, fiber channel network, or via any other transport component) switch and a system that supports the routing servers of the switches, which arrives Packets are routed and routed to the appropriate application/game server 1521-1525. In one embodiment, the 'delivery to a particular application/game server packet indicates a subset of the data received by the client 139958.doc -101 - 200951728 and/or may be other components within the data center (eg, Road connection components, such as gateways and routers, are used to translate/change. In some cases, the packet is routed to more than one server 1521-1525 each time, for example, if the game or application is executing concurrently on multiple servers. The RAID arrays 1511-1512 are connected to the incoming routing network 1502 so that the application/game server ι521_1525 can read the raid

Array 1511-1512 and write to RAID array 1511-1512. In addition, RAID array 1515 (which can be implemented as multiple RAID arrays) is also coupled to incoming routing 1502, and data from RAID array 1515 can be read from application/game servers 1521-1525. Incoming routing 15〇2 can be implemented in a variety of prior art network architectures (including tree-structured switches, inbound Internet traffic 丨5〇1 at its root); interconnecting all of the various devices' mesh structures Implemented in the middle; or as a sub-network sequence of interconnects (concentrated traffic among interworking devices and centralized traffic isolation among other devices). One type of network is configured as S AN 'which, although commonly used for storage devices, can also be used for general high speed data transfer between devices. Also, the application/game servers 1521-1525 can each have multiple network connections to the incoming route 15〇2. For example, the servers 1521-1525 may have a network connection to a subnet attached to the RAID arrays 1511-1512 and another network connection to a subnetwork attached to other devices. The application/game servers 1521-1525 can be configured identically, somewhat differently, or all differently, as previously described with respect to server 402 in the embodiment illustrated in Figure 4a. In one embodiment, each user typically has at least one application/game servo 139958.doc -102 - 200951728 1521-1525 when using the colocation service. For simplicity of explanation, it will be assumed that a given user is using the application/game server 1521 'but multiple servers can be used by one user, and multiple users can share a single application/game server 1521-1525. The user's control input sent from the client 415 (as previously described) is received as an incoming Internet service and routed to the application/game server 1521 via the incoming route 1502. The application/game server 1521 uses the user's control input as a control input to the game or application executing on the server and calculates the video and the next frame of the audio associated therewith. The application/game server 1521 then outputs the uncompressed video/audio 1529 to the shared video compression 15 3 0. The application/game server can output uncompressed video via any component (including one or more super-fast Ethernet connections), but in the embodiment the 'video system is connected via D VI and the audio is output. And other compression and communication channel status information is output via a universal serial bus (USB) connection. Shared Video Compression 1530 compresses uncompressed video and audio from the application/game server ι521_ 1525. This compression can be implemented entirely in hardware or in hardware that executes the software. There may be a dedicated compressor for each application/game server 1521-1525, or if the compressor is fast enough, a given compressor may be used to compress video from more than one application/game server 1521-1525 / Audio. For example, at 6 3 3, the video frame time is 16.67 milliseconds. If a compressor can compress 1 frame in i milliseconds, then the compressor can be used to compress from up to 16 applications/game servers 139958.doc • 103 by taking input from one server after another. · 200951728 1521-1525 video/audio, which stores the state of each video/audio compression process and switches the background as it cycles through the video/audio stream from the server. This results in substantial cost savings for the compressed hardware. Since different servers will complete the frame at different times, in one embodiment, the compressor resources are in a shared state with shared storage components (eg, RAM, flash memory) for storing the state of each compression process. In the cluster 153, and when the server 152 1-1525 is complete and ready to be compressed, the control component determines which compression resource is available at the time, provides the state of the compression process of the server and the uncompressed Compressed video/audio frame. Note that 'one of the states of the compression process for each server includes information about the compression itself, such as the decompressed frame buffer data of the previous frame (which can be used as a reference for the P image block) , resolution of video output; quality of compression; image block structure; allocation of bits per image block; compression quality, audio format (eg stereo, surround sound, Dolby® AC-3). However, the compression process state also includes information about the following communication channel status: peak data rate 941, and whether the previous frame (as illustrated in Figure 9b) is currently being output (and therefore the current frame should be ignored), and potentially There are channel characteristics that should be considered in compression (such as excessive packet loss) that affect compression decisions (eg, in terms of the frequency of video blocks, etc.). Since the peak data rate 941 or other channel characteristics change over time 'as determined by the application/game server M21-1525 that supports each user to monitor the data sent from the client 415, the application/game The server 1521-1 525 sends the relevant information to the shared hardware compression 139958.doc -104. 200951728 1530. The shared hardware compression 1530 also encapsulates the compressed video/audio using components such as those previously described, and, where appropriate, applies a fec code, copies specific material, or takes other steps to adequately ensure video. The /sfl data stream is received by the client 415 and is decompressed with viable high quality and reliability. Some applications, such as those described below, require that the video/audio output of a given application/game server 1521·1525 be available simultaneously at multiple resolutions (or in multiple other formats). If the application/game server 1521-1525 thus notifies the shared hardware compression 153 resource, the uncompressed video audio 1529 of the application/game server 1521_1525 will be in a different format, with different resolutions and/or Simultaneous compression in different packet/error correction structures. In some cases, some compression resources may be shared among multiple compression processes that compress the same video/audio (eg, in many compression algorithms, there are steps by which the image is scaled to multiple sizes before applying compression. Need to output images of different sizes, this step can be used to simultaneously servo several compression processes. Under other conditions, separate compression resources will be required for each format. In any case, it will be used for a given application/game server. The compressed video/audio 1539 of all the various resolutions and formats required by 1521-1525 (one or more) is simultaneously output to the outgoing route 154. In one embodiment, the output of the compressed video/audio 1539 is output. It is in UDP format, so it is a one-way packet flow. The outgoing routing network 1540 contains a series of routing servers and switches, and the I39958.doc -105· 200951728 series routing server and switch will each compressed video. /The audio stream is directed to the desired network via the Internet Protocol 1599 interface, which is typically connected to the optical interface of the Internet. The user or other destination and/or return to the delay buffer 1515, and/or return to the incoming route 15〇2, and/or output via a private network (not shown) for video distribution. (As described below): The outgoing route 1540 can simultaneously output a given video/audio stream to multiple destinations. In one embodiment, this is implemented using Internet Protocol (IP) multicast, where the broadcast is intended to be simultaneously The given stream is streamed to multiple destinations and the broadcast is repeated by the routers and switches in the outgoing route 1540. The multiple destinations broadcasted can be used over the Internet to multiple uses. The client 415, via the incoming route 15〇2, to the plurality of application/game servers (10), and, or to the one or more delay buffers 1515», therefore, compresses the output of the given server 1521_1522 In one or more formats, and directing each compressed stream to one or more destinations. Additionally, in another embodiment, if multiple application/game servers 1521 1525 are simultaneously used by a user (eg, , used to produce complex In the parallel processing configuration of the 3D output of the hybrid scene) and each server generates a part of the obtained image, the video output of the plurality of servers 1521-1525 can be combined into a combined frame by the shared hardware compression 丨53〇, and The composite frame is processed as described above from the other side as if it were from a single application/game server 1521_1525. Note that in one embodiment, a copy of all video generated by the application/game server 1521_ 1525 will be generated. (At least in the resolution or more resolution of the video viewed by the user) is recorded in the delay buffer 1515 for up to 139958.doc • 106 - 200951728 a certain number of minutes (15 minutes in one embodiment) . This allows each user to "rewind" the video from each session to check previous work or performance (in the case of the game). Thus, in one embodiment, each compressed video/audio output 1539 stream routed to user client 415 is also multicast to delay buffer 1515. When the video/audio is stored on the delay buffer 1515, the directory on the delay buffer 1515 provides the network address and delay buffer of the application/game server 1521-1525 (which is the source of the delayed video/audio). A reference between the location of the delayed video/audio® can be found on the device 1515. The live, ready-to-play, instantly playable game application/game server 1521-1525 can be used not only to execute a user's application or video game, but also to establish support for host generation via The user interface application of the hosted service 210 of the navigation and other features of the service 210. A screen shot of the user interface application is shown in Figure 16 ("Game Viewfinder" screen). This particular user interface screen allows the user to view the ® (or delayed) 15 games played by other users on the spot. Each of the "thumbnail" video windows (such as '1600) is a live video window in motion that displays a video from a user's game. The view shown in the thumbnail may be the same view that the user is looking at or it may be a delayed view (eg, if the user is playing a fighting game, the user may not want other users to see where they are hidden and It can choose to delay any view of its game play for a period of time (eg '10 minutes)). The view can also be a camera view of the game that is different from the view of any user. The user can select the game to be viewed simultaneously based on a variety of criteria via the 139958.doc 107. 200951728 menu selection (not shown in this description). As a small sample of exemplary choices, the user may select a random selection of games (such as those shown in circle 16), all categories of games (both played by different players), only top players of the game, games A player of a given level, or a lower level player (for example, if the player is learning the basics), a player who is "scoring" (or a competitor), a game with the largest number of viewers, and the like. Note 'Usually, each user will decide whether the video from their game or application can be viewed by others, and if so, decide which other people can view and when they can be viewed by others, and decide whether the video can only be delayed. In the case of inspection. The application/game server 152 1-1525 that produces the user interface screen shown in FIG. 16 is requesting a game from the user to the user (the application/game server 1 521-1 525) The application/game server 1521-1525 sends a message to obtain the 视5 video/audio feeds. The message is sent via the incoming router 丨5〇2 or another network. This message will include the size and format of the video/audio that was requested and will identify the user viewing the user interface screen. A given user may choose to select the "Piracy" mode and not allow any other user to view the video/audio of their game (from their view point or from another view point), or as described in the previous paragraph, the user may choose Allow viewing of video/audio from its games, but delay the video/audio being viewed. The user application/game server 1521_ 1525 (which receives and accepts requests for its video/audio to be viewed) will therefore acknowledge to the requesting server and will also notify the shared hardware compression 153 that 139958.doc -108 is required. · 200951728 Generates an additional compressed video stream of the requested format or screen size (assuming the format and screen size are different from the format and screen size already generated), and it will also indicate the destination of the compressed video (ie, the request server) ). If the requested video/audio is only delayed, the requesting application/game server 1521-1525 will be notified as such, and will find the location of the video/audio in the directory on the delay buffer 1515 as a delay. The video/audio source of the application/game server 1521_1525 originates from the delay buffer 1515 to obtain delayed video/audio. Once all such requests have been generated and processed, up to 15 live broadcast thumbnail-sized video streams are routed from the outgoing route 1540 to the incoming route 15〇2, to the application/game server that generates the user interface screen. The device 1521·1525 will be decompressed and displayed by the server. The delayed video/audio stream may be oversized, and if so, the application/game server 1521_1525 will decompress the streams and scale the video stream down to the thumbnail size. In one embodiment, the request for audio/visual is sent to a central "management" service (not shown in Figure i5) similar to the hosted service control system of Figure h (and managed by a central "management" service The central "management" service then redirects the requests to the appropriate application/game server 1521-1525. Moreover, in one embodiment, a request may not be required because the thumbnails are "pushed" to the user of the user who is allowed to. All of the simultaneous audio from 15 games can produce harsh sounds. The user can choose to mix all the sounds together in this way (perhaps to get the "beep" feeling from all the actions being viewed), or the user can choose to listen to the audio from one game at a time. Single-Travel 139958.doc -109- 200951728 The choice of play is done by moving the yellow selection box 1601 to the given game (the yellow frame can be moved by using the arrow keys on the keyboard, by moving the mouse, by Move the joystick either by pushing a direction button on another device such as a mobile phone). Once a single game is selected, it is only played from the audio of the game. Also, game information 1602 is displayed. In the context of this game, for example, the Publisher Logo ("EA") and the game logo "Need for Speed Carbow Canyon" and the orange bar indicate the person who plays the game or views the game at a particular time under relative conditions. The number (in this case, a lot, so the game is "hot"). In addition, a "status" is provided indicating that there are 8 different instantiations in which i 45 players are actively playing Need for Speed games (ie, the game can be played by individual player games or multiplayer games), and there are 680 Viewer (this user is one of them). Note that such statistics (and other statistics) are collected by the hosted service control system 401 and stored on the RAID arrays 1511-1512 for maintaining the host hosting service 21 〇 operation and for appropriate Billing the user and paying the fee to the publisher who provided the content" - some statistics are recorded as a result of the actions performed by the service control system 4.1 and some statistics are provided by the individual application/game server 1521- 1525 is reported to the service control system 4〇1. For example, when the game is being viewed (and when the game is stopped viewing), the application/game server 1521-1525 executing the game viewfinder application sends a message to the colocation service control system 401 to The colocation service control system 401 can update how many games are in the view of the statistics. Some statistics can be used by user interface applications such as this game viewfinder application. 139958.doc -110· 200951728 If the user clicks the start button on their input device, they will see the thumbnail video in the yellow box and the thumbnail video will be live broadcast to full screen size. This effect is shown in the process of Figure 17. Note that the size of the video window 1700 is increased. In order to implement this effect, the application/game server 1521-1525 requests the full screen size of the game routed to it from the application/game server 1521-1525 executing the selected game (analyzed by the user's display device 422) A copy of the video stream Q application executing game/game server 1521-1525 notifies the shared hardware compressor 153 that a copy of the thumbnail size of the game is no longer needed (unless another application/game server 1521- 1525 requires such a thumbnail), and then it directs the shared hardware compressor 1530 to send a copy of the full screen size of the video to the application/game server 1521-1525 that amplifies the video. The user playing the game may or may not have a display device 422 having the same resolution as the display device of the user who zooms in on the game. In addition, other viewers of the game may or may not have a display 422 that has the same resolution as the user who zoomed in on the game (and may have different audio playback components, such as stereo or surround sound). Therefore, the shared hardware compressor 1530 determines whether a suitable compressed video/audio stream that satisfies the requirements of the user requesting the video/audio stream has been generated and if the appropriate compressed video/audio stream does exist, the shared hardware The compressor 1530 notifies the routing router 54 to route the replica of the stream to the application/game server 1521-1525 that amplifies the video, and if the appropriate compressed video/audio stream does not exist, the compressed video is suitable for Another copy of the user and directs the outgoing route to send the stream back to the incoming route 1502 and the application/game server 1521-1525 that amplifies the video. 139958.doc -111 - 200951728 This server that now receives the full screen version of the selected video will decompress the full-fledged version and gradually scale it up to full size. Figure 18 illustrates how the screen looks after the game is fully enlarged to full screen and the game is displayed at full resolution of the user's display device 422 (as indicated by the image pointed by arrow 1 800). The application/game server 1521-1 525 executing the game viewfinder application sends a message to other application/game servers 1521-1 525 that provide thumbnails to indicate that the thumbnails are no longer needed and hosted to the host. The control server 4〇1 sends a message to indicate that the other games are no longer viewed. At this time, the only display generated is the overlay 1801 at the top of the screen, which provides information and menu control to the user. Note that as the game progresses, the viewer grows to 2, 5, 3 viewers. In the case of so many viewers, there must be many viewers with display devices 422 that have the same or close Resolution (per application/game server 1 521 -1 525 has the ability to scale video for adjustment of fit). Since the game shown is a multiplayer game, the user can decide to join the game at some point. The colocation service 21 may or may not allow users to join the game for a variety of reasons. For example, the user may have to pay for the game and choose not to pay, the user may not have enough levels to join the particular game (eg, for other players, it will not be competitive), or use The Internet connection may not have a sufficiently low latency to allow the user to play (for example, there is no latency constraint for viewing the game, so the view can be viewed remotely without delay (actually, Play on another continent, but for games played 139958.doc 112 200951728, the delay must be low enough for the user to (a) enjoy the game' and (b) be at a lower latency The other players connected are equal). If the user is not allowed to play, the application/game server 1521-1 525 that provides the user interface of the game viewfinder user will request the host hosting service control server 401 to initiate (ie, locate and start) via The application/game server 1521_ 1525 is suitably configured to play a particular game to load the game from the RAID arrays 1511-1512, and then the colocation service control server 401 will direct the incoming route 1 502 to come from The user's control signal is sent to the application/game server of the current hosted game and now the hosted game application/game server will guide the shared hardware compression 1530 self-compression from the hosted game viewfinder application. The video/audio of the application/game server switches to compress the video/audio from the application/game server of the current hosted game. The vertical synchronization of the game viewfinder application/game service with the new application/game server of the colocation game is not synchronized, and thus there may be a time lag between the two synchronizations. Because the shared video compression hardware 1530 will begin to compress the video after the application/game server 1521-1525 completes the video frame, the first frame from the new server can be completed than the full frame time of the old server. Early, the first frame from the new server may be before the transmission is completed by the compressed frame (for example, consider the transmission time 992 of Figure 9: if the uncompressed frame 3 963 is completed earlier) One and a half of the box time will then impact the transmission time 992). In this case, the shared video compression hardware 1530 will ignore the first frame from the new server (eg 'is ignored (974) frame 4 964), and the client 415 will come from the old service 139958.doc - The last frame of the 113-200951728 device maintains an extra frame time, and the shared video compression hardware 1530 will begin compressing the next frame time video from the new application/game server of the colocation game. For the user, the transition from one application/game server to another application/game server will be visually seamless. The colocation service control server 4 then switches the application/game server 1521_1525, which informs the host escrow game viewfinder, to the idle state' until it is needed again. The user can then play the game. In addition, the exception game will be played instantly on the sensation (because the game has been loaded from the Raid array 1511-512 to the app/game server i52i_i525 at a rate of 1 billion bits per second) and will be The ideal drive, scratchpad configuration (in wind〇ws2 condition) loads the game along with the operating system that is configured for the game to the server that is exactly for the game, and is not possible with Other applications that compete for the operation of the game are executed on the server. Also, as the user progresses through the game, each of the segments of the game will be loaded at a rate of one billion bits per second (ie, 8 seconds per billion bytes) from the RAID array 1511-1512. Loaded into the server, and because of the huge storage capacity of the RAID array 1511·1512 (because it is a shared resource for many users, it can be very large, but still cost effective), allowing pre-computation of geometry settings or other The game clips are set up and stored on the RAID arrays 1511-1512 and loaded very quickly. In addition, because the hardware configuration and computational power of each application/cruise server 1521_1525 is known, pixel and vertex shading can be pre-computed. Therefore, the game can be launched almost instantly 'it will be executed in an ideal environment, 139958.doc -114- 200951728 and the subsequent clips will be loaded almost instantly. But 'in addition to these advantages' the user will be able to view other people playing the game (via the previously described game viewfinder, and other components), and both decide whether the game is fun' and if so, then watch others Learning skills. In addition, the user will be able to demonstrate the game instantly without having to wait for a large download and/or installation, and the user will be able to play the game instantly (perhaps on a less expensive trial basis ± or on a longer term basis) . In addition, the user will be able to play the game on a Windows® PC, on a television set, at home, while on the move, and even on a mobile phone, with a wireless connection of sufficiently low latency. In addition, this can be done without having a copy of the game. As previously stated, the user may decide not to allow his game play to be viewed by others, allowing his game to view 'allow its game to be viewable by selected users' after delay or allow its game to be viewable by all users. In any case, in the embodiment, the video/audio is stored in the delay buffer φ 1515 for 15 minutes' and the user will be able to "rewind" and view the first.

The game is just played, and the game is paused, the game is played back slowly, the game is decided, etc., as it is done while watching a digital video recorder (dvr). Although in this example, the user is playing the game, but the right user is using the application, the same "DVR" capability is available. This may be useful in verifying previous work and in other applications as detailed below. In addition, if the game is designed to have the ability to fall back based on game state information so that the camera view can be changed, the 3D DVR will also be supported, but it will need to be designed to support "3D 139958.doc" -115- 200951728 DVR" ability. The "dvr" capability of using the delay buffer 1515 will work in conjunction with any game or application (of course, limited to video generated when using a game or application), but in the case of a game with 3D 0¥11 capability. The user can control the 3D r through of the previously played clip, and cause the delay buffer 15 15 to record the video and record the game state of the game segment. Therefore, a particular "crossing" is recorded as compressed video, but since the game state will also be recorded, different crossings may be at a later date in the same segment of the game. The users on the colocation service 21 will each have a user page in which the user can post information about himself and other materials. One of the things that the user will be able to announce is a video clip from a game that has been saved. For example, if the user has overcome the particular difficulty of the game, the user can just "rewind" to the location where he or she gets the big result in the game, and then guide the colocation service 21 to a certain Video clips of duration (eg, 3 seconds) are saved on the user's user page for viewing by other users. The only thing to do to implement this application/game server 1521_1525 that the user is using is to play back the video stored in the delay buffer 1515 to the RAID arrays 1511 - 1512 and then index the video segments to the user. On the user page. The right game has the ability to have a 3D DVR, and as described above, the user can also record the game state information required by the 3D DVR and make it available to the user's user page. The game is designed to have "bystanders" in addition to active players. 139958.doc • 116· 200951728 (ie, 'users who can travel in the 3D world without observing and observe movements) The game viewfinder app will enable the user to join the game as a bystander and player. From the point of view of the self-examination, there is no difference between the user being a bystander and not the active player for the colocation system 210. Loading the game onto the application/game server 1521-1525 and the user will control the game (eg, controlling the virtual camera p of the world to see the only difference will be the user's gaming experience. Multiple User Collaboration® Hosting Another feature of Service 210 is the ability of multiple users to collaborate while viewing live video (even if they are viewed using different devices). This is useful when playing games and when using applications. Many PCs and mobile phones are equipped with video cameras and have the ability to perform instant video compression (especially when the image is small). Also, a small camera is available that can be attached to the TV' and is used in software or in many hardware for compressing video. It is not difficult to implement instant compression in one of the compression devices. In addition, many PCs and all mobile phones have microphones, and the headphones are available with a microphone. The combination has local video/audio compression capability (specifically, use this article) The cameras and/or microphones of the low latency video compression technology described herein will enable the user to view video and/or The message is transmitted from the user premises 211 to the colocation service 210 along with the input device control data. When using the techniques, the capability described in 囷 19 can be achieved: the user can have his video and audio 1900 appear in another On the screen of a user's game or application. This example is a multiplayer game where teammates collaborate in the car. The video/audio using 139958.doc -117- 200951728 can only be viewed selectively by their teammates. /Hearing. Also, because there is no delay in effectively using the techniques described above, players will be able to talk to each other or exercise for an uninterestable delay. This video/audio integration is made by the user. The compressed video and/or audio of the camera/microphone is completed as the incoming Internet service 丨5 〇1 arrives. Then the incoming route 1502 routes the video and/or audio to the permitted viewing/hearing of the video and / or audio application/game server 1521-1525. Then 'users who choose to use video and/or audio for each application/game server 1521-1525 decompress Shrink video and/or audio and integrate it as needed to appear in a game or application, such as illustrated by 1900. The example of Figure 19 shows how to use the collaboration in a game, but the collaboration can be used for An extremely powerful tool for the application. Consider a situation where a large building is being designed by a Chicago-based architect for a New York-based real estate developer for the group, but the decision involves moving on and happening to be in Miami. Financial investors in the airport and need to make decisions about specific design elements of the building (how they relate to the buildings in the vicinity) to meet both investors and real estate developers. Assume that the construction company has an attachment in Chicago A high-resolution monitor for PC cameras, real estate developers have laptops with cameras in New York, and investors have mobile phones with cameras in Miami. The construction company can use the colocation service 210 to host a powerful architectural design application that can be highly realistic 3D reproduction' and it can take advantage of the large database of buildings in New York City and the database of buildings that are not being counted. . The architectural design application will execute on one of the applications / 139958.doc 200951728 game server 1521-1525 (or on several if it requires a lot of computing power). Each of the three users at the disparate location will be connected to the colocation service 21〇, and each will have a simultaneous view of the video output of the architectural design application, but it will be targeted for each The user has a given device and network connection characteristics that are appropriately sized by the shared hardware compression 1530 (eg, the construction company can pass the 2 quotient

Using the Internet connection to see 256〇xl44〇6〇fps, New York real estate developers can see 1280x720 60 fps images via their 6 Mbps connection on their laptops, and investors can use their mobile phones 250 KbPs honeycomb data connection see 320x180 60 fps image). Each party will hear the other party's voice (the conference call will be handled by any of the many widely available conference call suite software in the application/games ΐ5225) and via the user input device With the actuation of the button, the user will be able to use his local camera to make the video appear. As the meeting progresses, 'architects will be able to show through the photo-realistic feeling that the building looks like what it is when it rotates the building and makes it adjoin another building in the area, and all The average video will be visible under the resolution of the display devices of all parties. The local device used by any party will be able to handle the paste without any problem, not to mention downloading or even storing. Reproduce the huge database needed for the surrounding buildings in New York City. From the point of view of the fragrant of each of the users, although the distance is very long, and despite the disparate local device, it will be recognized as one and will be in an unbelievable sense of reality. Sewing experience. When the good news conveys its emotional state, in addition, when one wants its face to be seen, it can do so. In addition, if the real estate 139958.doc • 119· 200951728 developers or investors want to control the construction process (which is the keyboard, the crater, the input mouse, keypad or touch screen of the DX Dan), the job can be,

It can respond without a perceived delay (assuming its network connection: reasonable delay). For example, in the case of a mobile phone, if it acts: a WiFi network connected to the airport, it will have a very low latency. 2 It uses the cellular data network available in the United States today, which is likely to be noticeably lagging behind. However, for most of the purposes of the conference (where investors are watching architects controlling building crossings or talking about video conference calls), even honeycomb delays should be acceptable. Finally, at the end of the cooperative meeting call, the real estate developer and investment side will comment on it and stop the colocation service, the construction company will be able to "rewind" the video of the meeting recorded on the delay buffer 1515 and Review comments, facial expressions and/or actions applied to the 3D model of the building during the meeting. If there are specific segments that it wishes to save, then the segments of the video/audio can be moved from the delay buffer 1515 to the RAID arrays 1511-1512 for archival storage and later playback. And 'from a cost perspective', if an architect only needs to use New York City's computing power and a large database for a 15-minute conference call, then it only needs to pay for the time when the resources are used, without having to have high capabilities. Workbench does not have to purchase expensive copies of large databases. Video Enriched Society S Services Hosting Services 210 is an unprecedented opportunity for building video-rich community services on the Internet. Figure 20 shows an illustrative user page for a game player on the colocation service 21〇. Like a game viewfinder app

139958.doc -12EN 200951728 The same program as the 'user page' is an application executed on one of the application/game servers 1521_ 1525. All thumbnails and video windows on this page show a constantly moving video (if the clip is short, it loops). By using a video camera or by uploading video, the user (the user name is "KILLHAZARD j" can publish his own video 2 (other users can view the video). The video is stored on the RAID array 1511-512. Also, when other users come to the user page of KILLHAZARD, if KILLHAZARD is using the colocation service 210 at this time, Beck will display the live video of whatever it is doing 2001 (assuming it is allowed to view its use) The user of the page views it. This will request the KILLHAZARD to be active from the service control system 401 by the application/game server 1521-1525 hosted by the host page application (and if so, request that it is being used) The application/game server 1521·1525) is completed. Then, using the same method used by the game viewfinder application, the compressed video stream of the appropriate resolution and format is sent to the application executing the user page application. / Game server 1521-1525 and display it. If the user selects a window with a live broadcast of KILLHAZARD And then properly click on its input device, the window will zoom in (again using the same method as the game viewfinder application), and the live video will be viewed by the user's display device 422 (suitable for Viewing the characteristics of the user's Internet connection) Filling the screen. This is a key advantage over the prior art method: the user who views the user's page can see the game played live by the user, and does not have the ability to Play the game's local computer or game console. It is 139958.doc • 121- 200951728 Users provide an excellent opportunity to see the users on the user page displayed as "active", and they are able to view the game. The opportunity for a user to wish to try or be good at the game. The video clips recorded or uploaded from the camera of KILLHAZARD's partner 2002 are also displayed on the user page, and the bottom of each video clip refers to the text that should not be played online by the partner (for example, six_sh〇t is playing the game) Dragon Knight (Eragon) and MrSnuggles99 offline, etc.). By clicking on the menu item (not shown), the slotted video clip is switched from displaying the recorded or uploaded video to the partner of the game currently playing the colocation service 21〇 at the moment in the game. Live video of the content. Therefore, it becomes a game viewfinder that is a group of partners. If a partner game is selected and the user clicks on the game, the game will be zoomed in to the full screen and the user will be able to watch the game played live on the full screen. Again, the user of the game viewing the partner does not own a copy of the game and does not have a native computing/game console resource for playing the game. The game view is effective instantaneous. As described earlier, when a user plays a game on the colocation service 21, the user can "rewind" the game and find the video clip he wishes to save' and then save the video clip to his user. page. These are called "Brag Clips". Video clips 2〇〇3 are self-reward clips 2〇〇3 saved by KILLHAZARD from previous games played by them. The number 2004 shows how many times the self-reviewed clip has been viewed, and when the self-reward clip is viewed, the user has an opportunity to rate it, and the number of icons 2005 of the orange keyhole shape indicates how high the rating is. When the user views the 139958.doc 122-200951728 user page, the self-review clip 2003 is continuously cycled along with the rest of the video on the page. If the user selects and clicks on one of the self-review clips 2〇〇3, it is zoomed in to present the self-reward clip 2003, and DVR control that allows playback, pause, rewind, fast forward, step, and the like. The self-reward clip 2003 playback is loaded by the application/game server 1521_ 1525 by the user-recorded self-review clips stored on the RAID array 1511_1512 and decompressed and played back to implement 0. The self-reward clip 2003 can also be a 3D DVR" video clip from a 3D DVR capable game (i.e., a sequence of game states from a game that can be played back and allows the user to change the camera view point). In this case, game state information is also stored in addition to the particular "crossing" compressed video recordings that the user performs while recording the game segment. When the user page is being viewed and all of the thumbnails and video windows are constantly looping, the 3D dvr self-reward clip 2003 will cause the user to record the self-reward clip of the compressed video when the user records the "travel" of the game segment. 〇3 is constantly circulated. However, when the user selects the 3D DVR self-review clip 2〇〇3 and clicks the 3D DVR self-review clip 2〇〇3, the user will be able to click in addition to the DVR control that allows the compressed video clip to be played. Give it a button for the 3D ability of the game clip. It will be able to independently control the camera "crossing" during the game segment, and if it wishes (and the user of the user page so allows), it will be able to record the alternate self-reward clip in the form of compressed video. The alternative self-reward clip "crossing" will then be available to other viewers of the user page (immediately, or after using the ♦ page owner 139958.doc • 123- 200951728 has the opportunity to verify the clip) . This 3D DVR Self-Cue Clip 2003 capability is enabled by launching a game that will replay the recorded game state information on another application/game server 152 1-1 525. Because the game can be launched almost instantaneously (as previously described), so it is activated (its playback is limited to the game state recorded by the self-reward clip) and the user is allowed to record the compressed video to the delay buffer 1515. It is not difficult to use the camera to "cross" at the same time. Once the user completes the "crossing", the game is deactivated. From the user's point of view, launching the "crossing" with the 3D DVR self-reward clip 2003 is no more difficult than controlling the DVR control of the linear self-reward clip 2003. It does not know the game or even know how to play the game. It is only a virtual camera operator staring at the 3D world during the game segment recorded by another operator. The user will also be able to add their own audio to the self-reward clip (or record or upload from the microphone in this way, using the self-reward clip to create custom animations using the characters and actions from the game. This animation technique Often referred to as "machinima." As the user progresses through the game, they will reach different skill levels. The played game will report the results to the service control system 4G1, _0_ these skill levels will also be displayed On the user page. Interactive animated online advertising has changed from text to still images, video, and now to interactive clips 'usually using Adobe Flask animation streamlined to implement the use of the thin client The reason is: user 139958.doc -124· 200951728 is usually expected to push the lock to the heart of the heart. The privilege of being delayed is less resistant than the wide two = on a very low-performance PC, and therefore: = Two Hearts: Interactive advertising will work properly. The will of the Adobe Flash animation-simple client is at the level of interactivity and experience (to reduce the download Inter) restricted on the duration.

The S1 interactive advertisement 'where the user will be in the showroom in the showroom ^ (4) _ car exterior and internal color' simultaneous ray tracing show how the car does not look like. The user then selects the avatar to drive the car, and then the user can use the car for driving on a race track or through a country other than Monaco. The user can select a larger engine or a better tire, and then see how the changed configuration affects the ability of the car to accelerate or maintain stability. Of course, the advertisement is effectively a cutting-edge 3D video game. But for such an ad that can be played on a PC or video game console, it will require 100 MB downloads, and in the case of a PC, it may need to install a special drive' and may lack sufficient CPU or GPU computing power on the PC. It is not executed at all. Therefore, such advertisements are impractical in prior art configurations. In the colocation service 210, the advertisements are placed almost immediately, and are preferably executed regardless of the capabilities of the user's client 415. As a result, it delivers more quickly than a streamlined, user-friendly ad, with a richer experience and a high degree of reliability. Streaming geometry RAID arrays 1511-512 and inbound routing 1502 during instant animation provide such fast data rates with such low latency that it is possible to design depend on t39958.doc -125- 200951728 RAID array 1511-15 12 And routing 15〇2 to reliably deliver geometric video games and applications directly in the middle of the game play or in the application during live play (eg, traversal with complex databases). In prior art systems, such as the video game system shown in the figure, the large number of storage devices available (especially in practical home devices) is so slow that it is not possible to stream geometry during game play (except for the required The geometry is slightly predictable outside the case). For example, in a driving game where a given road exists, the geometry of the building used to enter the field of view can be reasonably properly predicted and the mass storage device can search for the location of the upcoming geometry in advance. But in complex scenes with unpredictable changes (for example, in a battle scene with complex characters around), the RAM on the gPC or video game system is completely filled with the geometry of the object currently in view, and Then, if the user suddenly turns his character to view his character, why if the geometry is not preloaded into the RAM, there may be a delay before the geometry can be displayed. In the colocation service 210, the RAID arrays 1511-1512 can exceed the speed data of the ultra-fast Ethernet speed, and under the SAN network, it is possible to achieve better than 10 billion bits of Ethernet. The network is 10 billion bits per second faster than other network technologies. 10 billion bits per second will carry billions of bytes of data in less than one second. Approximately 170 million bits (21 MB) of data can be loaded at 60 fps frame time (16 67 ms). Of course, even in a RAID configuration, the rotating media will still incur a delay greater than one frame time', but the flash memory based on the flash § 己 体 最终 will eventually rotate with 139958.doc -126- 200951728 Media RAID The array is generally large and will not incur this high latency. In one embodiment, a cache with a large amount of RAM writes is used to provide very low latency access. Therefore, at a sufficiently high network speed and a large amount of memory with low latency, the geometry can be streamed to the application/game server 1521-1525 generally faster than the CPU and/or GPU can process the 3D data. . Thus, in the previously given example, where the user suddenly turns his character and looks backwards, the Φ geometry of all the characters behind him can be loaded before the character completes the rotation, and thus, for the user, It would seem that it is in a world that is as realistic as a real-life photo with live action. As discussed earlier, one of the last boundaries in a photo-realistic computer animation is the human face, and due to the sensitivity of the human eye to incompleteness, the slightest error from a photo-realistic face can result from viewing. The negative reaction of the person. Figure 22 shows the use of ContourTM Authenticity Capture Technology (the following is the subject of the application: Application No. 10/942,609, "Apparatus and method for capturing the ❹ motion of a performer"; "Apparatus and method for improving the identification of a performer", No. 10/942, 413, "Apparatus and method for improving marker identification within a motion capture", February 25, 2005 "Apparatus and method for performing motion capture using shutter synchronization", application No. 11/077,628, filed on March 10, 2005; App 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 127-200951728 method for performing motion capture using a random pattern on capture surfaces"; "System and method for performing motion capture using phosphor application techniques"; June 7, 2006, pp. 11/449,131, filed on June 7, 2006 Application No. 1 1/449,043 "System and method for "Performance motion capture by strobing a fluorescent lamp"; "System and method for three dimensional capture of stop-motion O animated characters", filed on June 7, 2006, each of these applications The live broadcast performance captured by the assignee of this CIP application has resulted in a very smooth capture surface that achieves a high polygon count tracking surface (i.e., the polygon motion accurately follows the movement of the face). Finally, photo-realistic results are produced when the live broadcast of the live video is mapped onto the tracking surface to create a textured surface. Although current GPU technology is capable of rendering many polygons in the tracking surface and texture and illuminating the surface in real time, if each of the polygons and textures changes between m (which will produce the most photorealistic result), then All available RAM for modern PC or video game consoles will be quickly consumed. Using the stream geometry technique described above, the geometry is continuously fed into the application/game server 1521-1 525 so that it can continually create a photo-realistic face to allow for almost indistinguishable The video game on the facial face of the live action became practical. Integration of Linear Content and Interactive Features 139958.doc -128- 200951728 Movies, TV shows and audio materials (collectively "Linear Content") are widely available in many forms for home and office users. Linear content is available on physical media such as CD, DVD, HD-DVD and Blu-ray media. It can also be recorded by DVRs from satellite and cable TV broadcasts. In addition, it can be downloaded via satellite and cable τν (PPV) content and video on demand (VOD) on cable τν. Increasing linear content can be downloaded via the Internet and serial & content Available. Today, there is really no place where you can experience all the features associated with linear media. For example, DVDs and other video optical media typically have interactive features that are not available at other locations (such as director reviews, "trivia" shorts, etc. Online music sites have cover art and songs that are not normally available on CDs. Information, but not all CDs are available. Websites associated with TV shows often have additional features, blogs, and sometimes comments from actors or creators. Also, in the case of many movies or sports events, There are often video games that are often associated with a linear (four)-issued (in the case of a movie) or (in the case of a sport) that can be closely linked to real-world events (eg, a player's transaction). The colocation service 210 is very It is suitable for delivering linear content when linking disparate forms of related content together. Indeed, delivering a movie movie is not as challenging as delivering a highly interactive video game, and the colocation service 21 can deliver a linear inner valley to the home or office Multiple devices in, or delivered to a mobile device. Figure 23 shows an illustration of a colocation service 210 User interface page, which displays the choice of linear content. 139958.doc -129· 200951728 However, unlike most linear content delivery systems, the colocation service 210 is also capable of delivering relevant interactive components (eg, a menu on a DVD) And features, interactive overlays on HD-DVD, and Adobe Flash on the website (as explained below). Therefore, the client device 415 limits no restrictions on which features are available. The pipe system 210 is capable of dynamically and instantly linking linear content to video game content. For example, if a user is watching a Quidditch game in a Harry Potter movie and decides that he is willing to try to play © Quidditch, then Just click the button and the movie will pause and it will be immediately sent to the Quidditch clip of the Harry Potter video game. After playing the Quidditch game, another click on the button and the movie will start again immediately. In the case of graphics and production technology, the video captured by the camera cannot be distinguished from the live action characters, when the user When performing the transition from the Quidditch game in the live action movie to the Quidditch game in the video game on the colocation service (as described in this article), the two scenes can't actually be distinguished. This is linear content and interaction. The director of both styles (eg 'video games') offers a new creative option 'because the lines between the two worlds become indistinguishable. With the colocation service architecture shown in Figure 14, 3 movies can be made The control of the virtual camera is provided to the viewer. For example, in a scene occurring within the train, it will be possible to allow the viewer to control the virtual camera and look around the train as the story progresses. This assumes that all three objects in the train ("Assets") and enough to be able to reproduce these scenes and original movies in real time. 139958.doc • 130- 200951728 Computational Competency Levels Available β and even for non-computer-generated entertainment, there are very exciting interactive features available In other words, the film "Pride and Prejudice" has many scenes in the beautifully decorated old British mansion. For a particular large scene, the user can pause the video and then control the camera to patrol the generosity, or possible surrounding area. To implement this, a camera with a fisheye lens can be carried through the building '^ when it tracks its position' very similar to implementing prior art

Apple's QuickTinie

❹ VR. The various round frames will then be converted so that the scene/image is not distorted and then stored along with the movie on the RAID 歹J 1511 1512 and played back when the user chooses to continue the virtual tour. In the case of a sporting event, a live event (e.g., a basketball game) can be streamed via the colocation service 21 for viewing by the user (as it would be for a typical TV). After the user views a particular play, the video game of the game (finally the basketball player looks as realistic as the real player's photo) can catch up with the player starting in the same position, and the user (possibly controlling one player each) can replay To see if it can do better than these players. The colocation service 21 described in this article is extremely suitable for supporting this future world's because it can withstand computing power and large-capacity storage resources that are impractical to be installed in the home or in most office settings, and its calculations Resources are always up-to-date (in the case of the latest computing hardware available) 'But in the home background, there will always be families with older generations of pCs and video games. In addition, in the colocation service 21〇, the use of 139958.doc -131- 200951728 is concealed of all this computational complexity, so even if the user may be using a very sophisticated system, from the user's point of view, It is generally simple to change the channel on TV. In addition, the user will be able to access all of the computing power and computing power that will result from any client 415. The multiplayer game to the extent that the game is multiplayer, it will be able to communicate not only to the application/game server i52i_i 525 via the incoming route 1502 but also to the hosted service 2iq. Executing ® server or gaming machine's internet (not shown when the multiplayer game is played on a computer on the universal Internet, the application/game server 1521-1525 will have extremely fast access to the internet The benefits (compared to the performance of the game system on the word processor in the home), but it will be limited by the capabilities of other computers playing games on slower connections, and potentially by the games on the Internet. The scorpion scorpion is designed to accommodate the least commonality (these game vocabularies will be the home computer for relatively slow consumer Internet connections). But when fully hosted on the server 21 server center When playing multiplayer games, you can make a big difference. Hosting the game for the user's game, each app/game feeding device 1521·1525 will be used with other apps/games. 1525 and any server interconnect for central control of multiplayer games with extremely high speed, very low latency connectivity and a large, very fast storage array host. For example, for ultra high speed Ethernet After entering the 1502 network, the application/game server (2) will communicate with each other' and communicate to the potential of only one billion bits per second at 139958.doc -132- 200951728 milliseconds or less. At the time of the delay, the host will host any of the feeders that are centrally controlled by the multiplayer game. In addition, the RAID arrays 1511-1512 will be able to respond very quickly and then transfer the data at a rate of one billion bits per second. The user customizes the character in terms of appearance and clothing so that the character has a large number of geometric shapes and behaviors that are unique to the character, under the prior art system limited to the game client executing on the PC or the game console in the home. If the character enters the field of view of another user, the user will have to wait until a long slow download is completed in order to load all geometry and behavioral data into it. In the computer. Within 21〇 of the colocation service, the same download can be superior to the ultra-high-speed Ethernet network with the RJD array 1 5 JJ _ 15 12 servo at a rate of 1 billion bits per second. Even for home users. With an 8 Mbps internet connection (which is extremely fast based on today's standards), the ultra-high-speed Ethernet is also 100 times faster, so it takes a minute to work on a fast internet connection at SuperSpeed B. It will take less than one second on the Internet. The top player grouping and competition host hosting service 210 is extremely suitable for the tournament. Because no game system is executed in the local client, there is no chance for the user to cheat. Due to the ability of the output route 1540 to multicast UDP streams, the colocation service 210 can simultaneously broadcast larger tournaments to thousands of people in the audience. In fact, when there are so popular that thousands of users are receiving a particular stream of the same stream (for example, showing a larger tournament view), the video stream is sent to a content delivery network (CDN) (such as Akamai). Or Limelight) may be more efficient for distribution to a large number of client devices 415. 139958.doc -133- 200951728 When CDN is used to display the top view player grouping game viewfinder page, similar level efficiency is obtained. For larger tournaments, live celebrity commentators can be used to provide comments during specific competitions. Although a large number of users will be watching the larger championships and a relatively small number will be played in the tournament. The audio from the celebrity commentator can be routed to the host/host of any Bystander mode copy of the game hosted in the tournament and hosted in the tournament/game Servo H 1521.1525 'and the audio can be added Above the game audio. The video of the celebrity commentator can be overlaid on the game (and possibly just on the bystander view). The main transport protocol and hypertext transfer protocol (Ηττρ) of the webpage loading acceleration global information network is conceived and defined in an era in which only commercial high-speed Internet connections are used, and online consumers use dial-up data machines or ISDN. in. At this time, the "gold standard" for fast connection is the T1 line, which symmetrically provides a data rate of 1 > 5 Mbps (i.e., equal data rates in both directions). Today, the situation is completely different. The average home connection speed of a large number of developed worlds connected via dSL or electric data machines has a much higher downlink data rate than the T1 line. In fact, in some parts of the world, fiber-to-the-edge (fiber-to-the- Curb) is introducing data rates of up to 50 Mbpd1 Mbps into the home. Unfortunately, HTTP has not been architected (and not implemented) to make effective use of these rapid speed improvements. The website is a collection of files on the remote server. Quite simply, HTTP requests the first file, waiting to download the file, and then 139958.doc -134- 200951728 Qing seeking the first floor case 'waiting to download the rights and so on. In fact, Hup allows more than one "open connection" (ie, every request - more than one slot), but only allows very much because of agreed standards (and the desire to prevent the network word server from being overloaded). Open connection. In addition, because web pages are structured, Pan browsers are often unaware of multiple simultaneous pages that can be used for immediate downloading (ie, 'only after parsing a page becomes obvious: you need to download a new standard like imagery) . Therefore, the broadcasts on the website are loaded one by one. In addition, due to the request and response protocol used by HTTP, there is a delay of 1 millisecond (about a typical US network server) associated with each file loaded. In the case of a relatively low speed connection, this does not introduce a lot of problems, because the download time for the file itself determines the waiting time of the web page. However, as the connection speed increases (especially in the case of complex web pages), it starts to cause problems. In the example shown in Figure 24, a typical commercial website (this particular website is from a larger athletic shoe trademark) is shown. There are 54 files on the website. Files include HTML, CSS, JPEG, PHP, JavaScript, and Flash files' and include video content. A total of .5 μbytes must be loaded before the live web page (that is, the user can click on it and start using it). There are many reasons for a large number of constructions. First, it is a complex and sophisticated web page 'and secondly' which is a web page that is dynamically combined based on information about the users accessing the page (eg, from which country the user came from, what language s 'users have previously performed Purchase, etc.) and download different bids based on all of these factors. However, it is still a very typical commercial web page. 139958.doc -135- 200951728 Figure 24 shows the amount of time that elapses before the live web page is live as the connection speed increases. At 1.5 Mbps connection speed reading, using a conventional network server with a conventional network browser, it costs Μ privately at 12 Mbps connection speed 24 〇 2 before the live broadcast of the web page, and the loading time is reduced to &; $ seconds, or about fast - times. But at 96 Mbps connection speed _ download time is only reduced to about 5.5 seconds. The reason for this is because at this high download speed, the time to download the standard itself is the smallest, but the number of each standard is approximately 1 〇〇 each, which results in a delay of 54 slots * 100 milliseconds = 5.4 seconds. . Therefore, regardless of the number of connections to the home, the site will spend at least 5.4 seconds before the live broadcast. The other factor is the side of the feeding device; each TTP request is added after the queue, so this will have a significant impact on the busy word server, because for each small one to be obtained from the web server For projects, HTTP requests need to wait for them to return. One way to solve these problems is to discard or redefine Ηττρ. Or, it might be possible for website owners to better merge their files into a single file (for example, in Adobe Flash format. However, as a practical matter, this company and many others have a large investment in their website architecture. Also, despite Some homes have 12-100 Mbps connections, but most homes still have slower speeds, and HTTP does work well at slow speeds. The alternative is to host a web browser host on the app/game feed pirate 1521 On the -1525, and the standard host for the network feeder is on the RAID array 1511-1512 (or potentially hosted on the hosted web browser application/game server 1521_1525) In the RAM or on the local storage). Because of the very fast interconnection via the routing route 15〇2 (or to the local storage 139958.doc • 136·200951728), it is not 100 per file under HTTP. The millisecond delay' is that there will be a minimum latency per file under HTTP. Then 'not the user in the family accessing the web via HTTP,

Instead, the user can access the web page via the client 415. Then, even under a 15 Mbps connection (because this page does not require a lot of bandwidth for its video), the web page will be live broadcast in less than 1 second per line 2400. In essence, there will be no delay before the web browser executing on the application/game server 1521_1525 displays the live broadcast page, and there will be no detectable before the user terminal 415 displays the video output from the web browser. The delay to arrive. When the user uses the mouse to search the webpage and/or type a word on the webpage, 'the user's input information is sent to the web browser executed on the application/game server 1521-1525, and the network accordingly makes Respond. One of the disadvantages of this method is that ^^ 〇 (U ^ material, use 帛 width, even if the web page becomes static. This can be transmitted by configuring the compressor only if (and if) the web page changes The data and then only the data is transferred to the changed page portion to remedy the presence of some web page scams with a changed flash slogan, etc., which tend to be confusing unless there is something to move (eg, For video clips, the reason is: 丨 usually the webpage is static. For these webpages, it may be as follows: =) The host service 210 will transmit less data (with the network word) because only the actual display will be transmitted The image of the helmet, the client's executable code, and I book ..., the simple type of flip image. ^Large objects that have never been viewed (such as Rolling I39958.doc -137- 200951728) So using Host Hosting Service 210 to host old pages, you can reduce the page load time to open the page and change it on TV. The extent of the channel: effectively and instantly broadcast the webpage. Promoting the elimination of games and applications as previously described, 'video games and applications with instant graphics are very complex applications and usually when they are issued to In the field, it contains defects. Although the software developer will get feedback on the defect from the user 'and it may have some way to return the machine state after the crash', but exactly identify what caused the game or instant It is very difficult for an application to crash or improperly execute. When a game or application is executed in the colocation service 210, the video/audio output of the game or application is constantly recorded on the delay buffer 15 15 . The door dog process executes each application/game servo 1521-1525, which will host the service control to the host System 401 periodically reports that application/game server 1521-1525 is executing smoothly. If the watchdog process fails to report, server control system 4-1 will attempt to communicate with application/game server 1521-1525. And if successful, Bay will collect whatever machine status is available. No matter what information is available to be sent to the software developer along with the video/audio recorded by the delay buffer 1515. So 'when the game or application software developer is autonomous When the escrow service 210 gets notified of the crash, it gets a record of the frame of the frame that caused the crash. This information can be extremely valuable in tracking down defects and repairing them. 139958.doc •138· 200951728 Also Note that when the application/game server 1521-1525 crashes, the server is restarted at the most recent restartable time and the message is provided to the user, apologizing for technical difficulties. Resource sharing and cost savings Figure 4a And the system shown in Figure 4b provides multiple benefits for both end users and game and application developers. For example, usually, home And office client systems (for example, PCs or game consoles) are only in use during a small percentage of the week. According to Nieisen Entertainment, "Active Player Benchmark Study" (11 zhong: Boundary Review).卩1<116 outside 3 reviews^.(:0111/.§1- bi n/stories. pi ? ACCT= 104&STORY=/www/story/l 0-05-2006/ 00044461 15&EDATE=) 2006 On October 5th, the interactive game, the active gamer spent an average of 14 hours a week playing on the video game console and playing on the handheld device for about 17 hours a week. The report also states that for all game play activities (including console, handheld device and 1> (: game play), active players average - 13 hours per week. Consider the higher number of console video game play time, exist 24*7 = 168 hours a week, he suggested that in the active gamer's home, the video game console is only used for 17/168 = 1〇〇/. The time is in use. Or, at 9〇% of the time, video The game console is idle. The high cost of a given video game console, and the fact that the manufacturer subsidizes h, etc. 'This is a very inefficient use of expensive resources. t ', the PC is usually only in - Week-times are used in hours, especially for non-portable desktop PCs that are often used by programs such as Autodesk Maya. Do your best to operate at all hours and holidays and some PCs (For example, a portable PC for taking home ^ for work at night) is used at 139958.doc -139- 200951728 for all hours and holidays, but most commercial activities tend to focus on a given business time zone. Week - to Friday About 9 ΑΜ to 5 ρ Μ, less false 曰 and disconnection time (such as lunch), and because most PC usage occurs when the user actively uses pc, it follows: desktop PC utilization tends to follow this Wait for the number of hours of operation. If the PC is used constantly from 9 AM to 5 PM for five days of the fake week, then it will imply that % is used during the hour of 40/168 24/. The high-performance desktop is %. It is a very expensive investment for business, and this reflects a very low utilization. Schools that teach on desktop computers can use computers in weeks or even less, and despite the hours of teaching And change, but most of the teaching occurs during the hours of the week-to-Friday, so in general, the video game console is only used in small weeks (four). It is worth noting that Because many people are in business or at school during the non-holiday week-to-Friday hour, so these people usually wait for hours _do not play video games and therefore when they actually play video games, Usually during other hours (the , evening, weekend and holiday.) Given the configuration of the colocation service shown in the blunt diagram, the usage patterns described in the above two paragraphs lead to very efficient use of resources. Obviously there is a given The time limit is limited by the number of users who are hosting the service, especially if the user needs instant responsiveness for complex applications such as sophisticated 3D video games. However, no, Video game console in the home or PC used by the business (which is usually

: Most of the time is idle.) 'Server 4〇2 can be used by different users. (4) Heart (4) f For example, 'High-performance dual CPU and dual GPU 139958.doc -140· 200951728 and high-performance RAM 4〇 2 It can be used by businesses and schools on non-holiday from 9 AM to 5 PM, but is used by players who play cutting-edge video games at night, on weekends and on holidays. Similarly, low-performance applications can be utilized by businesses and schools during business hours on low-performance servers 4 (2) with (10) coffee cpu, no coffee (or very low-end GPU), and limited RAM. The low performance server 4〇2 is utilized during non-commercial hours. Additionally, in the case of the colocation service configuration described herein, the resources are effectively shared among thousands (if not millions) of users. In general, online services only use a small percentage of their total user base to use the service at a given time. If you consider the Nie丨sen video game usage statistics listed previously, it is easy to understand why. If the active player plays the console game for only 17 hours a week, and if the peak usage time of the game is assumed to be at night (5·12 AM, 7*5 days = 35 hours/week) and weekend (8 Am_12 AM, 16*) During a typical non-working, non-commercial hour of 2=32 hours/week, there is 35+32=65 peak hours for a 17-hour game. It is difficult to estimate the exact peak user load on the system for a number of reasons: some users will play during the extra-peak time, there may be a cluster peak of the user at a particular daytime, and the peak time may be affected by the type of game being played ( For example, a child's game will probably be played at an earlier time in the evening). However, assuming that when the player is likely to play the game, the average number of hours played by the player is much less than the number of hours in the day, then only the number of users of the hosted service 210 will use the colocation at a given time. Service 210. For this analysis, the peak load is assumed to be 12 5%. Therefore, only 12.5. /. The calculation, compression, and bandwidth resources are used at 139958.doc • 141- 200951728 at a given time, resulting in only 12.5% of the hardware cost to support a given user's performance game due to resource reuse. Level. In addition, assuming that some games and applications require more computing power than others, then resources can be dynamically allocated based on the game being played by the user or by the application executed by the user. Therefore, users who choose a low-performance game or application will be assigned a low-performance (lower-cost) server 402, and users who choose a high-performance game or application will be assigned a high-performance (more expensive) feeder 402. In fact, a given game or application may have a lower performance and higher performance area of a game or application, and can switch users from one server 402 to another between games or applications. The '402' is executed on the lowest cost server 402 that maintains the user's needs for the game or application. Note that RAID array 405, which is much faster than a single disk, will be used by even low performance server 402, which has the benefit of faster disk transfer rates. Thus, the average cost of the parent feeder 402 across all of the games played or the applications used is much less than the cost of playing the most expensive servers 402 of the most productive games or applications, however, even the low performance server 402 Disk performance benefits will also be obtained from RAID array 405. In addition, the server 402 in the colocation service 210 may be only a PC motherboard that does not have a disk or peripheral interface (different from the network interface), and just can be integrated down to a fast network just to the SAN 403. A single wafer of the interface. Also, the RAID array 405 may be shared among a much larger number of users than there are disks, so the cost per disk for each active user will be much less than that of a disk drive. All of this device will likely reside in a rack in the server room environment that is controlled by the environment 139958.doc -142- 200951728. If the server 402 fails, it can be easily repaired or replaced at the colocation service 210. In contrast, a PC or game console in a home or office must be sturdy, and must be able to survive a reasonable wear and tear to prevent a single appliance from being hit or dropped, requiring a housing with at least one drive, must It is spared from adverse environmental conditions (for example, being forced into a heated AV cabinet with other appliances), requires a service guarantee, must be packaged and shipped, and is sold by a retailer that may receive retail profits. In addition, the PC or game console must be configured to meet the peak performance of the most computationally intensive expected game or application that will be used at some point in the future, even for lower performance games or applications (or games or applications). District) may also play most of the time. In addition, if a pc or console fails, it can be a costly and time consuming process (which adversely affects manufacturers, users, and software developers). Therefore, it is assumed that the system shown in FIG. 4a provides an experience equivalent to the experience of the local computing resource to the user for the user to experience the computing power in the home, office or school. The architecture shown provides a much lower computational power. Eliminate the need for upgrades. ‘Users no longer have to worry about upgrading PCs and/or consoles to play new games or handle new applications with higher performance. Any game or application on the Hosting Service 21 (regardless of the type of server 402 that such games or applications require) can be used by the user and all games and applications are executed in close proximity (ie, Quickly loaded from the rAID array 405 or the local storage on the server 402) and appropriately updated with the latest updates and defects 139958.doc • 143- 200951728 ❹ 修复 Repair (ie the 'software developer will be able to choose to execute Given the ideal server configuration of the game 402 of the game or application, and then configuring the server 402 with the best driver, and then over time, the developer will be able to provide updates, bug fixes, etc. simultaneously Give the host hosting all copies of the game or application in service 210). In fact, after the user starts using the colocation service 210, the user may find that the game and the application continue to provide a better experience (eg, via update and/or defect repair) and may be in the following situation: the user is one year later Discover that new games or applications can be used to take advantage of computing technology (eg, higher-performance GpU) (which did not even exist a year ago), so it would be impossible for users to purchase The technology of playing games or executing applications a year later because the computing resources for playing games or executing applications are invisible to the user (ie, from the user's point of view, the user only chooses to be close and instantly The game or application that starts executing - more like the user changes the channel on the TV), so the user's hardware will be "upgraded" even if the user is not aware of the upgrade. Eliminate the need for backup, another problem for businesses, schools, and users in the home. The problem is that the right disk is faulty, or if there is an inadvertent erasure, it is stored in the local % or video game console. Information (examples > in the case of the console, the user's game outcomes and ratings) may be lost. There are many available applications that provide backups for K's and the game can be controlled by the server for backup, but the local backup is usually stored in another secure magnetic and organized place. Discs (or other non-volatile storage devices), and because of the slow upstream speeds available through typical low-cost Internet connections, backups for online services are often limited. In the hosted service 21 of Figure 4a, the data stored in the RAID array 405 can be configured using prior art RAID configuration techniques well known to those skilled in the art, such that when the disk fails, The data will not be lost and the technician at the server center hosting the failed disk will be notified, and then the technician will replace the disk, which will then be automatically updated to cause the RAID array to tolerate the failure again. In addition, since all drives are close to each other with a fast local network via SAN 4〇3, all disk systems are configured in the server center to be periodically backed up to secondary storage (which can be stored in It is not difficult to relocate at the center of the server or easily. From the point of view of the hosted service user, the data is always completely secure and never has to be considered for backup. Access to the presentation Users often want to try out games or applications before buying games or applications. As previously described, there is a demonstration (the "verb" form means a trial demo version, and the demo version is also called a "demo", but as a noun) a prior art component of a game and application, but each of its One is subject to restrictions and/or is inconvenient. With the colocation service 21, it is easy for the user and easy to try out the demo. In effect, the user performs the presentation and trials the presentation via a user interface, such as the user interface described below. The presentation will be loaded almost immediately onto the feeder 402 suitable for the presentation' and it will be executed exactly like any other game or application. Regardless of whether the presentation requires a very efficient server 139958.doc -145- 200951728 402 is still a low-performance server 4G2, and regardless of the type of home or office client 415 used by the user, from the user's point of view, the presentation is Will work. A software publisher of a game demo or application demo will be able to precisely control how long the demo and trials are allowed for the user to try, and of course, the demo may include providing the user with a full version of the game or application being demonstrated. The user interface element of the access opportunity. Because the presentation may be below cost or free of charge, some users may try to use repeated presentations (especially repeating the potentially interesting ® game demos). The colocation service 210 can use various techniques to limit the presentation usage for a given user. The most straightforward method is to establish a user ID for each user and limit the number of times the given user ID is allowed to play the presentation. However, the user can set multiple user IDs, especially if they are free. One technique for addressing this problem is to limit the number of times a given client 415 can play a presentation. If the client is a standalone device, the device will have a sequence number and the colocation service 210 can limit the number of times the presentation can be accessed by a client having a serial number. If the client 415 is executing as a software on a PC or other device, the host can be assigned a serial number and stored on the PC and used to limit the presentation, but the PC can be used by the user. To re-stylize 'and the sequence number is erased or changed, another option is that the colocation service 210 maintains the PC network adapter media access control (MAC) address (and/or other machine-specific identifiers, such as The record of the hard disk drive number, etc.) and limit the use of the presentation to the MAC address. Assume that the MAC address of the network adapter can be changed. However, this is not an extremely simple method. Another method is to limit the number of times a presentation can be played to a given IP address. 139958.doc -146- 200951728 The π address can be periodically reassigned by the electrical, local data machine and DSWK 'but it does not happen very frequently in practice, and if it can be determined (eg 'by contacting the ISP IP is in the block of the IP address used for residential DSL or electrical data access, and usually a small number of extensions to the home can be established. x, there may be multiple devices in the home after a NAT router sharing the same ιρ address, but usually in a residential setting, there will be a limited number of such devices. If the 1? address is in the block of the word business, a larger number of presentations for business can be established. However, in the end, all combinations of the previously described methods are the best way to limit the number of presentations on the pc. While there may not be a very simple way to make the determined and technically skilled user likely to be limited in the number of replayed presentations, creating a number of obstacles can create enough obstacles to make most users unworthy of going The demo system is abused, and instead, it uses the demo when it wants to try out new games and applications. Benefits for Schools, Business, and Other Institutions Significant benefits especially occur in commercial, school, and other institutions that utilize the systems shown in Figure 4a. Businesses and schools have substantial costs associated with installing, maintaining, and upgrading PCs, especially when talking about PCs that perform high-performance applications such as May a. As stated previously, ρς is typically utilized only in one-hours per hour of the week, and as in the home, PCs with the ability to position the quasi-performance are much more expensive in the office or school environment than at the server center. The cost in the environment. In the case of larger businesses or schools (eg, large universities), the 1T department of these entities sets up the server center and maintains computers that are accessed via LAN-level connections and remotely accessed by 139958.doc -147-200951728 Actual. There are many solutions for remote access to a computer via a LAN or via a private high frequency wide connection between offices. For example, it can be obtained by Microsoft's Windows terminal server, or by a virtual network computing application (such as VNC from RealVNC Ltd.) or by a thin client component from Sun Microsystems. Remote access by a PC or server has a range of qualities in graphical response time and user experience. In addition, these self-managed server centers are typically dedicated to a single business or school and therefore cannot be utilized when disparate applications (such as 'entertainment and business applications') utilize the same computing resources at different times of the week. Overlap of use. Therefore, many businesses and schools lack the scale, resources, or expertise to independently set up a server center with a network connection to each user's LAN speed. In fact, a large percentage of schools and businesses have the same Internet connection as their homes (eg, 'DSL, TEL, MP). However, such organizations may still have a need for very efficient calculations (either periodically or periodically, for example, a small construction company may have only a small number of architects, and when designing, it is relatively modest. The calculations are needed, but they may periodically require very efficient 3d "10 calculations (eg, when establishing a new building design for the user end). The system shown in Figure 4a is extremely suitable for such organizations. These organizations only need to provide the same type of network connection (e.g., DSL, electrical data modem) to the home and are typically very inexpensive. It can use a low % as the client 415, or no PC at all, and utilizes a dedicated component of the low-cost specializer 139958.doc 200951728 which only implements the control signal logic 413 and the low-latency video decompression 412. These features are particularly attractive to schools that may have problems with the theft of pcs or damage to pc internals. &sets solve many of the problems for such organizations (and many of these advantages are also common for home user sharing for general purpose computing,

The operation f (which must ultimately be passed back to the user in some form to: have a viable business) may be much lower because (4) the computing resources are shared with other applications that have different peak usage times during the week, (9) Organizations may obtain (and incur costs) access to high performance computing resources only when needed, and (c) such organizations do not have to provide resources for backing up or otherwise maintaining high performance computing resources. Elimination of piracy In addition, games, applications, interactive movies, etc. may no longer be pirated as they are today. Since the game is executed at the server center, the user does not have access to the basic code, so there is no piracy. Even if the user is going to copy the source code, the user cannot execute the code on a standard game console or home computer. This opens up markets around the world where standard video games are not available (such as 'China'). Re-locking of used games is also impossible. For game developers, there are fewer market discontinuities as they are today. With the new generation of technology forcing users and developers to upgrade and game developers depending on the current situation of timely delivery of the hardware platform, the colocation service 210 can be gradually updated as the game requirements change over time. Streaming Interactive Video 139958.doc -149- 200951728 The above description provides low latency streaming interactive video based on the Universal Internet (which also implicitly includes audio along with video, as used herein) The novel basic concepts enable multiple applications. Through the Internet, the technology system only has an enabling application that can be implemented by high-latency interaction. For example, basic playback controls (e.g., pause, rewind, fast forward) for linear video work properly at high latency, and it is possible to select among the online video feeds. Moreover, as previously stated, the nature of some video games allows them to be played with high latency. However, the high latency (or low compression ratio) of prior art methods for streaming video severely limits the potential application of streaming video or narrows its deployment to a specialized network environment, and even in such environments. The prior art also introduces a substantial burden on the network. The techniques described herein open up the possibilities for multi-dragon applications under low-latency streaming interactive video over the Internet, especially those enabled via consumer-grade Internet connections. In fact, under the user-side device that is typically small compared to the client 465 of Figure 4c, it is sufficient to provide an extremely fast amount of computing power, an arbitrary amount of fast storage, and an extremely fast network connection between powerful servers. An enhanced user experience that enables a new era of computing. In addition, because the bandwidth requirement does not increase as the computing power of the system grows (that is, because the bandwidth requirement is only about display resolution, quality, and frame rate), the broadband Internet connection is Sexually ubiquitous (eg, covered by a wide range of low-latency wireless coverage), reliable and capable of meeting the display device 422 of all users; i high enough bandwidth, and topical, typical consumption And 139958.doc -150- 200951728 What is necessary for business applications is a thick client (such as execution bias (4),

Li眶, OSX, etc. PCs or mobile phones) are even thin client (such as Adobe Flash or Java). Reconsideration of the assumption of the structure of the cross-talking interactive video (4) structure. This is an example of a colocation center service shown in Figure 15. For the delay path _ and/or the video path of the group video 1550, the multicast video stream of the program/game word processor 1521-1525 is transmitted via the path 1552. Or the light (4) 51 is fed back to the application/game server 1521_1525 after being selected (4). This can be achieved by a variety of practical applications (eg, such as Figure 16, 丄 圃 it) that are not possible or feasible by prior art servers or local computing architectures, such as those illustrated in Figures 17 and 20 As a more general architectural feature, the feedback loop provides a streamlined interactive video position, because the video can be looped indefinitely when the application needs video. There are many application possibilities that have never been available before. The $_key architecture is characterized by a one-way stream of video streams. This effectively enables any degree of multicasting of interactive video streaming (compared to, the bidirectional stream such as TCP/IP streams will grow more and more on the network from back and forth as the number of users increases) The traffic is stagnant). The important ability within the Multicast System Center because it allows the system to respond to the long-term needs of Internet users (and indeed the 'world's population) to communicate on a one-to-many or even multiple-on-the-spot basis . Again, the example of the use of both the motion video recursion and the multicast (as illustrated in the description herein) (such as Figure 139958.doc 151 200951728 16) is only the tip of the very large iceberg. The various functional modules and associated steps described herein may be embodied by a particular hardware component (such as a special application integrated circuit ("ASIC" or programmed) that contains the fixed-line logic used to perform the steps. The computer components are executed in any combination with custom hardware components. In an embodiment, the modules can be implemented in a TMS320X architecture such as 'TMS32〇C6〇〇〇, tm(10)(10)〇,

And so on) a programmable digital signal processor ("Dsp"). A variety of different DSP's can be used while still adhering to these basic principles. Embodiments may include various steps as described above. These steps may be embodied in a machine executable finger 7 that causes a general purpose or special purpose processor to perform particular steps. Various components (such as computers, hidden files, hard drives, input devices) that are not related to these basic principles have been omitted from the figure to avoid confusion. Elements of the subject matter disclosed may also be provided as a machine-readable medium for storing machine-executable instructions. A machine-readable medium can include, but is not limited to, a flash memory, a compact disc, a CD_R〇M, a DVD R〇M, a ram, an EPROM, an EEPROM, a magnetic or optical card, a broadcast medium, or other type suitable for storing electronic instructions. Machine readable medium. For example, the present invention can be downloaded as a computer program that can be accessed from a remote computer via a communication link (eg, a modem or network connection) by means of a data signal embodied in a carrier wave or other communication medium (eg, , the server) is transferred to the requesting computer (for example, the client). It should also be understood that the elements of the subject matter disclosed may also be provided as a computer program product 139958.doc • 152·200951728, which is a medium readable medium on which a command machine can be stored for a stylized computer. (eg, processing the device) to perform-sequence operations. Alternatively, the operations are performed by a combination of hardware and software. The machine readable medium can include two: not limited to: flexible disk, optical disk, cd_r〇m, and magneto-optical disk, a RAM, EPR〇M, EEPR〇M, magnetic card or optical card propagation medium or suitable for storing electronic instructions. Other types of media/machine readable media. For example, the elements of the subject matter disclosed may be downloaded as a computer program product. The program may be via a communication link (eg, a data machine or a network connection) by means of a data signal embodied in a carrier wave or other communication medium. The remote computer or electronic device is transferred to the request process. In addition, while the disclosed subject matter has been described in connection with the specific embodiments, numerous modifications and variations are Accordingly, the specification and drawings are to be regarded as illustrative rather BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 illustrates the architecture of a prior art video game system. 2a-2b illustrate a high order system architecture in accordance with an embodiment. Figure 3 illustrates the actual, rated, and required data rate for communication between the client and the server. Figure 4a illustrates a colocation service and a client 0 used in accordance with an embodiment. Figure 4b illustrates an exemplary delay associated with communication between a client and a colocation service. Figure 4c illustrates a client device in accordance with an embodiment. 139958.doc • 153· 200951728 Figure 4d illustrates a client device in accordance with another embodiment. Figure 4e illustrates an example block diagram of the client device of Figure 4c. Figure 4f illustrates an example block diagram of the client device of Figure 4d. Figure 5 illustrates an example of video compression that may be used in accordance with an embodiment. Figure 6a illustrates one example of video compression that may be used in another embodiment. Figure 6b illustrates the peak in the rate of the resource associated with transmitting a low complexity, low motion video sequence. Figure 6c illustrates the peak in the data rate associated with transmitting a high complexity, high motion video sequence. Figures 7a through 7b illustrate an example video compression technique used in an embodiment. Figure 8 illustrates an additional example video compression technique used in an embodiment. Figures 9a through 9c illustrate example techniques for use in mitigating data rate peaks in an embodiment. ❹ Figures 10a through 1B illustrate an embodiment in which an image image block is effectively packaged within a packet. Figures 11a through 15 illustrate an embodiment using a forward error correction technique. Figure 12 illustrates an embodiment of compression using a multi-core processing unit. Figures 13a-13b illustrate geolocation and communication between colocation services, in accordance with various embodiments. Figure 14 illustrates an exemplary delay associated with communication between the client and the colocation service. 139958.doc • 154- 200951728 Figure 15 illustrates an example colocation service server central architecture.

Figure 16 illustrates an example screen shot of an embodiment of a user interface including a plurality of live video windows illustrating the user interface after selection of a particular video window. Figure 18 illustrates the user interface of Figure 17 after zooming in on a particular video window to the full screen size. Figure 19 illustrates example collaborative user video material overlaid on a multiplayer game screen.

Figure 20 illustrates an example user page for a game player on a colocation service. Figure 21 illustrates an example 3D interactive advertisement. Figure 22 illustrates an example sequence of steps for surface capture from a live broadcast performance to produce a photorealistic image with a textured surface. Figure 23 illustrates an example user interface page that allows selection of linear media content. Figure 24 is a graph illustrating the amount of time elapsed and the connection speed before the live web page is live. [Main component symbol description]

100 CPU/GPU 101 Random Access Memory (RAM) 102 Display Device (SDTV/HDTV or Computer Monitor) 103 Hard Disk Drive 1〇4 Optical Media Driver/Optical Media/Disc 105 Network Connection 139958.doc -155 - 200951728 106 Game Controller 205 Client Device / Client / Client Platform 206 Internet 210 Host Hosting Service / Hosting System 211 User Location 220 Game or Application Software Developer 221 Input Device 222 Display Device / Monitor or TV 301 Nominal Maximum Data Rate 302 Available Maximum Data Rate 303 Required Data Rate 401 Hosted Service Control System / Host Managed Service Control Server 402 Server 403 Storage Area Network (SAN) 404 Video Compressed Logic/Video Compressor 405 Independent Redundant Array (RAID) 406 Control Signal 406a Control Signal 406b Control Signal 409 Routing Logic 410 Internet 412 Low Latency Video Decompression Logic 413 Control Signal Logic 139958.doc -156- 200951728 10 415 Client/Client Device/Customer Computer 420 Display Device 421 Input Device 422 Display Device/Display 440 Internet Bypass Service 441 Presence Point 442 WAN Interface 443 Firewall/Router/NAT (Network Address Translation) Device 444 WAN Interface 451 One-Way Transmission Time for Sending Control Signals 452 Via User Round trip routing/user location routing 453 arrow/round trip delay/user ISP 454 universal internet delay 455 one-way routing 456 maximum one-way delay 457 arrow/compression 458 video decompression 462 ethernet jack / Internet connection 463 HDMI (High Definition Multimedia Interface), Connector 464 Display Capability 465 Client Device / Client 466 Shading Glasses 468 Display Device / SDTV (Standard Definition TV) or 139958.doc • 157· 200951728 HDTV (High Definition Television) / Monitor Input Device Client Device Flash Memory Video Camera Display Device Bluetooth Input Device / Wired USB Input Device Bus

469 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 Ethernet interface WiFi subsystem control CPU / control processor Bluetooth interface USB subsystem video decompressor video output subsystem audio decompression subsystem audio Output Subsystem 490 HDMI Interface

491 DVI-I 492 S-Video 493 Composite Video 494 Digital Interface 495 Stereo Analog Interface 497 Ethernet 139958.doc -158- 200951728

499 Power 501 Uncompressed Video Frame 502 Uncompressed Video Frame 503 Uncompressed Video Frame 511 Compressed Frame 512 Compressed Frame 513 Compressed Frame 520 Compression Logic 559-561 Uncompressed video frame 611 I round frame 612-613 P frame 620 compression logic 621 rated maximum data rate 622 maximum data rate available / available data rate 623 峰值 frame required peak data rate / peak data rate 624 video Stream data rate / video stream data rate sequence 633 I frame peak 634 video stream data rate sequence / video stream data rate 635 P frame peak 636 P frame peak 670 B frame 671 I frame 701-760 uncompressed Frame 711-770 R frame 139958.doc -159· 200951728

721-770 Uncompressed frame 731-780 R frame 805 Mobile character 811 R frame 812 R frame 922 Available maximum data rate 934 Video stream data rate 941 Peak data rate / peak maximum data rate 942 2 times peak data Rate 943 3 times peak data rate 944 4 times peak data rate 952 frame 2 times peak 953 flattened 2 times peak 954 frame 4 times peak 955 flattened 4 times peak 961 uncompressed frame 1 962 without Compressed frame 2 963 Uncompressed frame 3 964 Uncompressed frame 4 965 Uncompressed frame 5 966 Uncompressed frame 6 967 Uncompressed frame 7 968 Uncompressed Box 8 969 Uncompressed frame 9 139958.doc -160- 200951728

970 uncompressed frame 10 981 compressed frame 1 982 compressed frame 2 983 compressed frame 3 985 compressed frame 5 986 compressed frame 6 987 compressed frame 10 991 Transmission time (xmit time) 992 Transmission time 993 Transmission time (2 times peak value) 995 Transmission time 996 Transmission time (4 times peak value) 997 Transmission time 1001-1005 Packet 1010 Image block package package logic 1100 I image block 1101 Forward error Correction Code (FEC) 1103 I Image Block 1104 Forward Error Correction Code (FEC) 1105 Forward Error Correction Code (FEC) 1110 Audio Packet 1111 FEC Code 1112 Audio Packet 1113 FEC Code 139958.doc -161 - 200951728 Lu

1120 User input command 1121 FEC code 1122 User input command 1123 FEC code 1200 Multicore and / or multiprocessor 1201-1208 Core 1205 4x2 configuration / uncompressed frame 1300 Server Center / Hosting Service 1301 Network 1501 Internet access service 1502 Incoming routing/incoming routing network 1511 RAID array 1512 RAID array 1515 RAID array/delay buffer 1521-1525 Application/game server 1529 Uncompressed video/audio 1530 Video Compression/Common Gathering/Common Hardware Compression/Common Hardware Compressor/Common Video Compression Hardware 1539 Compressed Video/Audio Output 1540 Outbound Routing/Outbound Routing Network 1550 Delay Buffer and/or Grouped Video/ Feedback Fun Road 1551 Path 1552 Path 1599 Outgoing Internet Traffic 139958.doc •162- 200951728 1600 "Thumbnail" Video Window 1601 Yellow Selection Box 1602 Game Information 1700 Video Window 1800 Arrow 1801 Overlay 1900 Video and Audio 2000 The video of the person itself β 2001 Live video 2002 partner video clip / self-reward clip / 3D DVR Self-recruiting clips 2004 Digital 2005 Figure 2400 line 2401 1.5 Mbps connection speed 2402 12 Mbps connection speed 2403 96 Mbps connection speed HS1-HS6 Colocation Service Server Center 139958.doc 163-

Claims (1)

200951728 VII. Patent Application Scope: κ A method comprising: executing a game or application on a feeder of a colocation service system; continuously recording video/audio of the game or application in a 9-time buffer Outputting; reporting to the colocation service system by the monitoring program to report whether the game or application is executed; if the monitoring program cannot report during a period interval, the host system is managed by the host , you are, collect any available machine status of the game or application and the video/message recorded by the delayed pound smasher. θ ❹ 139958.doc