TW200952495A - Apparatus for combining aplurality of views of real-time streaming interactive video - Google Patents

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: logically subdividing each of a sequence of images into a plurality of tiles, each of the tiles having a defined position within each of the sequence of images, the defined position remaining the same between successive images; detecting motion within the sequence of images occurring at each of the positions of each of the tiles; and encoding each tile within each image of the sequence of images using a first compression format or a second compression format, wherein the frequency at which a particular tile is encoded according to the first compression format across the sequence of images is based on the detected amount of motion at the position of that tile across the sequence of images.

Description

200952495 VI. Description of the Invention: [Technical Field of the Invention] The present disclosure is generally in the field of a data processing system that improves the ability of a user to manipulate and access audio and video media. This application is entitled "Apparatus and Method for Wireless" for the purpose of wireless video game application on December 10, 2002.

Video Gaming) No. 10/315, 460 Partial Continuation (CIP) application, which has been assigned to the assignee of this CIP application. [Previous technology] Recorded since the era of Thomas Edison The audio and movie media have become one of the clubs. In the early 2nd century, there were widespread distributions of recorded audio media (magnetic columns and records) and movie media (coin-operated jukeboxes and movies), but the two technologies are still in their infancy. In the late 1920s, film and audio were combined on the basis of a bestseller, followed by a combination of color film and audio. Radio broadcasts have evolved into broadcast-selling audio media that largely support the form of advertising. When the television (TV) broadcast standard was established in the mid-1940s, television and radio were joined in the form of broadcast-selling media to bring previously recorded or live-lived movies to the home. By the middle of the 20th century, most American households had a record player (ph〇n〇graph reC() I>d player) for playing ** recorded audio media, and for receiving live broadcast broadcast audio. Radio, and a television set for playing live broadcast audio/video (A/V) media. These three "media players" (recorders, radios and τν) group 139852.doc 200952495 are often combined into a common public speaker cabinet to become the "media center" of the family. Although media choices are limited for consumers, the media “ecosystem” is very stable. Most consumers know how to use the Media Player and enjoy the full range of capabilities. At the same time, publishers of the media (mostly film and television studios, and music companies) are able to distribute their media to both cinemas and families without extensive piracy or "secondary lock sales" (ie, already used) Resale of the media). Usually, publishers won't

Revenues are derived from secondary sales, and as a result, secondary sales reduce the publisher's revenue for new sales of purchasers who would otherwise be able to use the media themselves. Although there are indeed used records sold during the mid-20th century, these: sales will not have a big impact on record publishers, because they are different from movies or video shows (which are usually viewed once or only several times by adults) Music tracks can be listened to hundreds or even thousands of times. Because of &, the music media is far more durable than the film/video media ((4)’ for adult consumers). Once a record is purchased, if the consumer likes the music, the consumer may keep it for a long time. From the mid-20th century to the present, the media ecosystem has undergone a series of fundamental changes for the interests and losses of consumers and publishers. In the case of the widespread introduction of audio recorders (especially cassettes with high quality stereo), there is indeed a high degree of consumer convenience. But it also marks the beginning of :: in the broad consumer media practice _ piracy. Indeed, many = (4) consumers who use W tape to record their own two pieces of convenience for convenience (ie, students in the dorm who are ready to access each other's slap collection) will be pirated. Also, consumers will record music played on the radio by 139852.doc 200952495 instead of buying a record or tape from the publisher. The emergence of consumer VCRs has led to more consumer convenience, as VCRs can now be set up to record TV shows, which can be viewed at a later time, and VCRs have also led to the creation of a video rental industry where movies and TV programming can be designed. Access on an as-needed basis. The rapid development of popular home media devices since the mid-1980s has led to unprecedented choice and convenience for consumers' and has led to a rapid expansion of the media publishing market. Today's consumers are faced with multimedia choices and multimedia devices, many of which are tied to specific forms of media or to specific publishers. A keen media consumer may connect a bunch of devices to the TVs and computers in each room of the house, resulting in a "snake" cable to one or more televisions and/or personal computers (PCs) and a group of remote controls. (In the context of this application, the term "personal computer" or "PC" refers to any type of computer suitable for use in a home or office), including desktop computers, Macintosh(s) or other non-Windows computers. , Windows compatible devices, Unix variants, laptops, etc.). Such devices may include video game consoles, VCRs, DVD players, audio surround sound processors/amplifiers, satellite set-top boxes, and TV set-top boxes. In addition, for enthusiastic consumers, there may be multiple devices with similar functions due to compatibility issues. For example, σ' ‘Shaokes may have both HD-DVD and Blu-ray DVD players, or both Microsoft Xbox® and Sony Playstation® video game systems. In fact, due to the incompatibility of several games across several versions of the game console, consumers may have both XBox and later versions (such as Xbox 360®). Frequently, consumers are confused about which video 139852.doc 200952495 input and which remote end to use. Even after placing the disc in the correct player (eg DVD, HD-DVD, Blu-ray, xb〇x or Playstati〇n), selecting the video sfl and audio input for the device and finding the correct remote control 'Consumers still face technical challenges. For example, in the case of Wide Wing. DVD, the user may need to first determine the correct aspect ratio. (eg, 4:3, full, zoom, wide zoom, cinema wide, etc.) and then on their TV or surveillance Set the correct aspect ratio on the screen. Similarly, users who use the device may need to first determine the correct audio surround sound system format (for example, AC-3, Dolby Digital, DTS, etc.) and then set the correct audio surround sound system format. Often, 'consumers are unaware that they may not have access to the media content of their television or audio system's full capabilities (for example, watching movies that are squeezed in the wrong aspect ratio) or listening to stereo audio rather than surround sound. . Increasingly, Internet-based media devices have been added to the stack of devices. Audio devices like the Sonos® digital music system allow audio to stream directly from the Internet. Similarly, devices like the SlingboxTM entertainment player record video and stream it through the home network or via the Internet, where the video can be viewed remotely on the PC. And Internet Protocol Television (IPTV) services provide cable TV-like services via digital subscriber line (DSL) or other home Internet connections. There have also been recent efforts to integrate multiple media functions into a single device, such as the Moxi® Media Center and the implementation of the Windows XP Media Center version, although each of these devices provides a convenience to the functions it performs. However, each lacks universal and simple access to most media. In addition, it is often the case that the device often costs hundreds of dollars to manufacture due to the high processing and/or local storage requirements of the 139852.doc 200952495. These modern consumer electronics devices typically consume a lot of power and consume a lot of power even when idle, which means that they are expensive and waste energy with the day (4). For example, if the consumer forgets to cut the device Or switching it to a different video input, the device may continue to operate. In addition, because the device is not a complete solution, it must be integrated with other devices in the family. Leaving the sniffer line and many remote controls. In addition, when many newer Internet-based devices work properly, Usually offer more - as compared to its original shape of the possible forms available) of the media. For example, devices that stream video over the Internet often stream only video material, rather than being able to accompany dvd interactive "extra billing" streams, such as video "production", games, or director reviews. . This is due to the fact that turbulent materials are often made in a specific format intended for the particular device that handles interactivity at the local end. For example, each of DVD, HD_DVD, and Blu-ray Disc has its own specific interactive format. "Any home media device or local computer (which may be developed to support all popular formats) will require some degree of sophistication. (sophistication) and flexibility, which may be too expensive and complicated for consumer operations. This problem is exacerbated. If a new format is introduced in the future, the local device may not have the hardware capability to support the new format, which would mean that the consumer would upgrade the known media device. For example, if a higher resolution video or stereoscopic video is introduced later in the future (for example, per-eye view • 39852.doc 200952495 traffic), the local device may not have the decoding to decode the video. Capabilities' or it may not have hardware for outputting the video in a new format (eg, false-twisting to achieve stereo vision by 120 fps video synchronized with shuttered glasses) where 60 fpS is delivered to each In the eye, if the consumer's video hardware can only support 60 fpS video, this option will not be available in the absence of upgraded hardware purchases. When it comes to cutting-edge interactive media (especially video games), the problem of media device obsolescence and complexity is a serious problem. Modern video game applications are basically divided into four main non-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2, and PS3); Microsoft Xbox® and Xbox 360®; and Nintendo Gamecube® and

WiPM; and PC-based games. Each of these platforms is different from the others, so that games written to be executed on one platform are typically not executed on another platform. There may also be compatibility issues between a generation of devices and next-generation devices. Even though most software game developers build software games designed to be independent of the specific platform, in order to execute specific games on a specific platform, a proprietary software layer (which is often referred to as "game development") is required. Adapt the game to use on a specific platform. Each platform is in the form of a "console" (ie, a separate box attached to a TV or monitor/speaker) «for the consumer or itself - PC. Typically, video games are sold on optical media such as Blu-ray DVD, DVD-ROM or CD-ROM, which contain video games embodied in sophisticated real-time software applications. As home broadband speeds increase, video games are increasingly Become available for loading. :下下J39852.doc 200952495 Due to the immediacy and high computing requirements of high-end video games, the special requirements for achieving platform compatibility with video game software are extremely demanding. For example, we may expect full game compatibility from a generation of video games to next-generation video games (for example, from XBox to XBox 360, or from PlayStation 2 ("ps2") to Pbystation 3 ("PS3")). As is the case, there is a universal compatibility of productivity applications (such as 'Microsoft Word) from one PC to another with a faster processing unit or core. However, this is not the case for _ video games. Because when a video game is released, video game manufacturers often seek the highest possible performance for a given price point, so the system is often dynamically architected so that many games written for the previous generation are in the next generation. Not working on the system. For example, s ’ XBox is based on the χ86 series processor, while the χΒ〇χ 36〇 is based on the PowerPC series. Techniques can be used to simulate the previous architecture, but assuming that the video game is an instant application, it is often impractical to achieve the exact same behavior in the simulation. φ This is a loss to consumers, video game console manufacturers and video game software publishers. For the consumer, it means keeping the old generation of 5-hole game consoles and the new generation of video game consoles connected to the TV so that it is enough for all games. For console manufacturers, it means the costs associated with the simulation and slower adoption of the new console. And for publishers, it means that multiple versions of new games may have to be released to cover all potential consumers - not only for each commercial (eg, XBox, Playstation) version of the video game, but also often The version used for each version of a given trademark (for example, pS2 and ps3). For example, 139852.doc 200952495 For the purpose of developing a separate version of the electronic art "Crazy" for XBox, XBox 36, ps2, pS3, Gam_be, side and % platforms and other platforms. Portable devices such as cellular phones and portable media players also challenge game developers. Increasingly, these devices are connected to a wireless data network and are capable of downloading video games. However, there are a variety of cellular phones and media reference devices in the market that have a variety of display resolutions and computing capabilities. Also, because such devices typically have power consumption, cost, and weight constraints, they typically lack high-level graphics/acceleration hardware like graphics processing units ("GPUs") (such as those manufactured by NVIDIA of Santa Clara, CA). ). Therefore, game software developers typically develop a given game title that is used for many different types of portable devices. The user can find that the given game title is not available for its particular cellular phone or portable media player. In the case of a home game console, hardware platform manufacturers typically ask software game developers for a version of the tax on their ability to publish games on their platform. Honeycomb telephony wireless communication companies often also ask game publishers for royalties to download games to cellular phones. In the case of % games, there is no royalties paid for publishing games, but game developers often face high costs due to the high consumer service burden used to support multiple PC configurations and possible installation issues. Moreover, PCs often have fewer obstacles to piracy of game software because they can be easily reprogrammed by technically savvy users and games can be more easily pirated and more easily distributed (eg, via the Internet). . Therefore, for software game developers 139852.doc, 200952495, there are costs and disadvantages found in game consoles, cellular phones and PCs. For game publishers of consoles and PC software, the cost is not limited to this. In order to distribute the game via the retail channel, the publisher asks the retailer for a lower wholesale price of the sale price to give the retailer a profit margin. Publishers must also pay for the cost of manufacturing and distributing physical media for games. Retailers often also ask publishers for "price protection fees" to cover possible out-of-pocket expenses (such as games not being sold, or the price of the game is reduced, or the retailer must refund some or all of the wholesale price and / or purchase Retract the game). In addition, retailers often ask publishers for help in selling games in advertising flyers. In addition, retailers are increasingly purchasing back games from users who have completed the game, and then the games are sold as used games, and the revenue of the used games is typically not shared with the game publisher. The following facts add to the cost burden imposed on game publishers: games are often pirated and distributed via the Internet for users to download and perform free copying. As the Internet broadband speed increases and broadband connectivity becomes more widespread in the United States and the world (specifically, the "Internet cafes" to the home and to the PCs that lease the Internet) are increasingly being Assigned to a PC or console via download. X, wideband connections are increasingly used to play multiplayer and massively multiplayer online games (both of which are referred to in this disclosure by the acronym "MMOG" to mitigate the association with retail distribution Some of the costs and issues. Downloading online games solves some of the disadvantages of game publishers because the distribution costs are usually small and there are fewer or no 139852.doc 12 200952495 There is a cost of unsold media. But the downloaded games Still subject to piracy, and due to /, size (often a large number of billions of bytes), it may take a very long time to download. In addition, multiple games can be magnetic: such as with a portable computer Or they are sold together with the video game console. However, 'the game or MMOG needs to be connected online. To make the game playable, the piracy problem is alleviated because it usually requires effective use by the user. Account. It is different from linear media that can be copied by camera ❹ _ display screen video or by microphone recording audio from the speaker (eg Video and music), each video game experience is only $, and can't be copied using single video/audio recordings. Therefore, MMOG can be protected from piracy even in areas where copyright law is not enforced and pirated. And thus can support business. For example, Vivendi SA's "World of Warcraft" mm〇g has been successfully deployed, and has not been pirated around the world. And many online *MM0G games (such as Linden Lab's "Second Life" MM〇G) generates revenue from game operators via the economic model built into the game. The assets can be brought, sold and even built using online tools. Therefore, software games other than the familiar game can be used. Mechanisms other than buying or ordering to pay for the use of online games. Although online piracy is often mitigated due to the nature of online or MMOG, the online game operators still face the remaining challenges. Many games require a large percentage (ie, Processing resources for online or appropriate work. If the user has a low-performance local computer (for example, a computer that does not have σρυ, such as the low end "Upper computer") 'They may not be able to play the game. In addition, the game console ages, which is far behind the current state of the art I39852.doc -13- 200952495 and may not be able to handle higher-order games. Even if assumed The native PC can handle the calculation requirements of the game, and often there is installation complexity. There may be drive incompatibility (for example, if a new game is downloaded, a new version of the graphics driver may be installed, which results in relying on the old version. The previously installed game of the graphics drive is not operational. As more games are downloaded, the console may run out of local disk space. When bugs are found and fixed, or if the game is modified (for example, if the game developer It is found that the level of the game is too difficult or too easy to play. Complex games usually receive download patches from game developers over time. These patches require a new download. But sometimes not all users complete all patch downloads. At other times, the downloaded patch introduces additional compatibility or disk space consumption issues. Also, during game play, large data downloads may be required to provide graphical or behavioral information to the local PC or console. For example, if a user enters a room in a MMOG and encounters a scene or character consisting of graphic material or having an action that is not available on the user's home machine, then the information of the scene or person must be downloaded. P If the internet connection is not fast enough, this can result in substantial delay during game play. In addition, if the scene encounters a character who needs more than the storage space or computing power of the local PC or console, it may generate a situation in which the user cannot continue in the game, or must The graphics of reduced quality continue. Therefore, online or MMOG games often limit their storage and/or computational complexity requirements. In addition, they often limit the amount of data transfer during the game. Online MMOG games can also narrow the market for users of playable games. 3 139852.doc 200952495 In addition, technically knowledgeable users are increasingly reversing the native copy of the game and modifying the game to make it cheating. Cheating can be as simple as pressing a repeat button that is faster than possible with human power (for example, to shoot very quickly). In games that support in-game asset trading, cheating can lead to fraudulent transactions involving fraudulent transactions involving assets of real economic value. When online or MMOG economic models are based on such asset transactions, this can result in substantial harmful consequences for game operators. • The cost of developing new games grows as PCs and consoles become more sophisticated (such as 'having more realistic graphics (such as instant light tracking), and more realistic behaviors such as real-time physics simulation. In the early days of the video game industry, video game development was very similar to the development of application software; that is, most of the development costs were in the development of software (as opposed to the development of graphics, audio and behavioral elements or "assets"). ), such as software development that can be developed for use in films with a wide range of special effects. Today, many cutting-edge video game development efforts are more similar to software development than software development. For example, many Video games provide simulations of the 3-D world, and produce characters, props, and environments that are increasingly realistic (that is, computer graphics that look like real-life images of live action shots). Photo-realistic game development is the most One of the challenges is to create a human face that cannot be distinguished from the human face of a living person. Techniques such as the ContourTM Authenticity Captive System developed by CA's San Franeisc〇 capture the precise geometry of the performer's face and track the performer's face with high resolution while the performer is in motion Geometry. This technology allows the reproduction of a 139852.doc -15- 200952495 3D face on a PC or game console. The 3D face can't actually be distinguished from the captured live action face. Accurately capture and reproduce "photo-realistic Human faces are useful in several ways. First, highly recognizable celebrities or athletes are often used in video games (often hired at high cost), and imperfections may be obvious to the user's and thus distracting the viewing experience or Unpleasant. Often, it requires a high degree of detail to achieve a high degree of photorealism - potentially the need to reproduce a large number of polygons and high-resolution textures (with polygons and/or textures moving over the frame based on the frame In the case of a change) 支援 A PC or game that supports the game when the high-polygon count scene with detailed texture changes quickly The console may not have enough shakuhachi "to store enough polygons and textures for the required number of frames generated in the game clip. In addition, it can usually be used on a single disc player on a PC or game console. Or a single disk drive is usually much slower than RAM, and usually cannot keep up with the maximum data rate that the GPU can accept in rendering polygons and textures. Current games usually load most polygons and textures into RAM, which means 〇 Given the complexity and duration of a given scene is largely limited by the capacity of the RAM. In the case of facial animation, for example, this may limit the PC or game console to a low-resolution face that is not realistic. Or limited to making a realistic face in a limited number of frames before the game is paused and the polygons and textures (and other bedding) for more frames are loaded. The field PC or console displays a message similar to "Loading...". Watching the progress bar moving slowly across the screen is recognized by users of today's complex video games as an inherent disadvantage. The next scene is from the disk (unless there is another J39852.doc 200952495 piece) otherwise the "disk" in this article refers to non-volatile optical media or magnetic media, as well as non-magnetic media such as semiconductor "flash" memory. The delay in loading can take several seconds or even a few minutes. This wastes time and can make gamers quite frustrated. As previously discussed, 'large or all delays may be due to loading of polygons, textures, or other material from the disk. Time may also be the case when the processor and/or GPU in the pc or console is ready for use. When loading the data of the scene, it takes a part of the loading time. For example, a soccer video game allows players to choose among a large number of players, group sports fields, and weather conditions. Therefore, depending on the particular combination selected, different polygons, textures, and other materials (collectively "objects") may be needed for the scene (for example, different groups have different colors and patterns on their uniforms). It is possible to enumerate many or all of the various arrangements and pre-calculate many or all of the objects in advance and store the objects on a disk for storing the game. However, if the number of arrays is large, the storage required for all objects may be too large to be mounted on the disk (or too unrealistically impossible to download). As a result, existing PC and console systems are typically constrained in both the complexity and playback duration of a given scene and suffer long loading times for complex scenes. Another significant limitation of prior art video game systems and application software systems is that they increasingly use large databases such as 3D objects (such as polygons and textures) that need to be loaded into ^ Or in the game console for processing. As described above, when the databases are stored on the disk at the 2nd end, the f library can take a long time to load. The right database is stored in a remote location and stored via the Internet. 139852.doc -17- 200952495 The load time is usually much more severe. It may take a few minutes, a few hours or even a few days to drop in the library. = Capital::::: A lot of money (for example, for games, movies or calendars: == fine 3D model of tall sailing ships) And intended to be sold to users. However, once the database is downloaded to the local user, it is used by Wm, (4) == Estimate the database to see if it is suitable for the user (for example, when

It is desirable to download the database for the purpose of whether the user's buckled clothing for the game character has a satisfactory appearance or appearance when performing the specific movement. Long load times can be a hindrance to users who estimate the 3D database before deciding to make a purchase.

Similar problems arise in MMOGs (specifically, games that allow users to take advantage of increasingly customized characters). For pcs or game consoles that display characters, they need to be able to access data with 3D geometry (polygons, textures, etc.) and the behavior of the characters (for example, if the character has a shield, then the shield is strong enough to deflect the spear) Library. Typically, when MM〇G is played for the first time by a user, a large database for the character is available under the initial copy of the game, and the initial copy of the game is available on the game disc or downloaded to the disk at the local end. However, as the game progresses, if the user encounters a person or object that the database is not available at the local end (for example, if another user has created a custom character), the user or the object must be downloaded before the person or object can be displayed. database. This can result in a substantial delay in the game. Given the sophistication and complexity of video games, another challenge for video game developers and publishers in the case of prior art video game consoles was at 139852.doc -18· 200952495

Yu: It takes 2 to 3 years to develop video games, and the cost is tens of millions of dollars. Assuming that the new video game console platform is introduced at a rate that is roughly owed every five years, game developers need to start developing their games in the years before the new game console is released, so that when a new platform is released. Make video games available at the same time. Several consoles from competing manufacturers are sometimes released at about the same time (for example, within - or two years of each other), but it remains to be seen that each console is popular (for example, which console will produce maximum video) Game software sales). For example, in recent console loops, Microsoft χΒοχ 36〇, s〇ny piaystati〇n 3, and 〇 〇 jobs are scheduled to be introduced in approximately the same general time frame. But in the years leading up to these introductions, game developers must essentially “press” which console platforms will be more successful than others and invest their development resources accordingly. Film production companies must also share their limited production resources for a long time before the film is released based on the film that it estimates may be successful. Given the degree of investment required for video games, game production is becoming more and more similar to film production, and game production companies are routinely investing in their production resources based on their estimates of the future success of a particular video game. However, unlike film companies, this bet is not based solely on the success of the production itself; the truth is that it is based on the success of the game console on which the game is intended to execute. Simultaneously launching games on multiple consoles mitigates the risk, but this extra effort increases costs and often delays the actual release of the game. The application software and user environment on the PC are becoming more computationally intensive, dynamic and interactive, making it more visually appealing to the user and making it more useful and intuitive. For example, the new Windows VistaTM operating system 139852.doc -19- 200952495 and the successive versions of the Macintosh® operating system have visual effects. Advanced graphics tools (such as MayaTM from Autodesk) provide very sophisticated 3D rendering and animation capabilities (which push the limits of current state of the art CPU and GPU). However, the computational requirements for these new tools are Users of the product and software developers have many practical problems. Because the visual display of the operating system (OS) must work on a variety of computers (including those that are no longer sold but can still be upgraded with the new OS), the graphics requirements are largely influenced by the computer that OS intends to use. The minimum commonality limit (which typically includes computers that do not include GPUs). This severely limits the graphics capabilities of 〇s. In addition, battery powered portable computers (e.g., laptops) limit visual display capabilities because high computing activity in the CPU or GPU typically results in higher power consumption and shorter battery life. Portable computers typically include software that automatically reduces processor activity to reduce power consumption when the processor is not in use. In some computer models, the user can manually reduce processor activity. For example, 'The Sony VGN-SZ280P laptop has a side labeled "Stamina" on one side (for low performance, longer battery life) and the other side is labeled "Speed" (for high performance, Switch for shorter battery life). An OS executing on a portable computer must be able to function even when the computer is executed at one of its peak performance capabilities. As a result, OS graphics performance often remains available at much lower computing power than current state of the art. Frequently selling high-end computing-intensive applications (such as Maya), expecting these applications to be used for high-performance (:. This usually produces much higher efficiency 139852.doc •20- 200952495 can, and more expensive They are less portable and have the least commonality. Therefore, these applications have a much lower target audience than general-purpose os (or universal productivity applications, similar to Microsoft 〇ffice) and are usually more general-purpose OS software or general-purpose applications. The software is sold at a much lower volume. Potential audiences • Further limited because it is often difficult for prospective users to try out these intensive applications in advance. For example, suppose students want to know how to use Maya or already know Potential purchasers of such applications will try to use Maya before purchasing the investment (this may involve also purchasing (4) executing a high-end computer from Maya as a demo version that can be downloaded by a student or potential purchaser or a physical media that is available in a Maya demo version. At the time of the copy, if it lacks the ability to perform Maya to its full potential (for example, to handle complex 3D scenes) , it will not be able to conduct a comprehensive evaluation of the product. This essentially limits the audience of these high-end applications. It also makes the sale price higher because the development cost is usually purchased much less than the number of purchases of the generic application. The number of applications is also paid off. Θ High-priced applications also generate more incentives for individuals and businesses using pirated copies of the application software. Therefore, high-end application software suffers from piracy and piracy, although publishers of the software have made great efforts to Various technologies mitigate this piracy. However, even when using pirated high-end applications, it is impossible for users to exclude the need to invest in the current state of the art pc to perform pirated copies. Therefore, although users can use software applications One of the actual retail prices is used by software applications, but users of pirated software still need to purchase or get expensive PCs to fully utilize the application. 139852.doc -21 · 200952495 This is a high-performance pirated video game. The user is also established. Although the pirates can play the actual game Get a game at a rate, but still need to buy the expensive computing hardware needed to play the game properly (for example, GBu•Enhanced PC, or a high-end video game console like XBox 360). Assume video game Often for consumer entertainment, the extra cost for high-end video game systems can be prohibitively expensive. This situation is in countries where the average annual income of current workers is quite low (relative to the current average annual income of workers in the US) (eg ' China) is even worse. As a result, a much smaller percentage of the population has a high-end video game system or a high-end PC. In these countries, users can pay for the use of computers connected to the Internet. Coffee is quite common. Often, these cybers have older models or low-end PCs that do not have high-performance features, such as GPUs that would otherwise enable players to play computationally intensive video games. This is a key factor in the success of games executed on low-end PCs (such as Vivendi's World of Warcraft, which is highly successful in China and is usually played in Chinese Internet cafes). In contrast, computationally intensive games φ (such as "Second Life") are less likely to play such games on PCs installed in Chinese Internet cafes, which actually do not achieve the low performance of only accessing Internet cafes. User of pc. There are also obstacles to users who are considering a video game and are first willing to try out a demo version of the game by downloading/displaying it to the court via the Internet. Video game demos are often a versatile version of the game, some of which are deactivated or impose limits on the amount of game play. This may involve a long process (possibly several hours) of downloading billions of bytes of data before the game can be installed on a PC or console and executed on a Pc or console. In the case of pc 139852.doc •22· 200952495, it may also involve calculating which special drivers (such as DirectX or 0penGL drivers) the game needs, downloading the correct version, installing the correct version, and then determining if the PC can play the game. The latter steps may involve determining if the PC has sufficient processing (cpu and (iv)) capabilities, sufficient suffixes, and compatible ss (for example, some games are executed on wind〇ws • XP and not on Vista). . Therefore, after attempting to perform the video game demonstration process, the user may find that the video game demo cannot be played (for the user's PC configuration). Worse, once the user has downloaded a new drive to try the demo, these drive versions may be incompatible with other games or applications that the user is accustomed to using on the PC, so the installation of the demo may render the previous operation The game or application cannot be operated. These obstacles not only frustrate users, but they also create obstacles for video game software publishers and video game developers to sell their games. Another problem that leads to non-economic benefits is related to the fact that a given pc or game console is typically designed to accommodate a certain degree of performance requirements for the application and/or game. For example, some PCs have more or less RAM, a slower or faster CPu, and a slower or faster GPU (if they have a GPU). ^ Some games or applications use a given pc or console. Full computing power' while some games or applications don't take advantage of the full computing power of a given PC or console. If the user's game or application selection does not meet the peak performance capabilities of the local PC or console, the user may be wasting money on the PC or console due to unused features. In the case of a console, the console manufacturer may pay more than the cost of funding the console. 139852.doc •23· 200952495

Another problem that exists in the sale and enjoyment of video games involves allowing the user to watch others play the game before the user implements the purchase of the game. There are several prior art methods for recording video games for playback at a later time. For example, U.S. Patent No. 5,558,339 teaches that game state information (including game controller actions) is recorded during a "game play" on a video game client computer (owned by the same or different users). This status information can be used at a later time to replay some or all of the game actions on a video game client computer (e.g., a pc or console). A significant disadvantage of this method is that for a user who views a recorded game, the user must have a video game client computer capable of playing the game and must have a video game application executing on the computer to enable playback. The game play status, etc. when the game state has been recorded (4). In addition to this, the video game application must be written in such a way that there is no possible difference in execution between the recorded game and the played back game. For example, the game graphics system is calculated based on the frame of the frame. For many games, depending on whether the scene is particularly complex to perform other delays (for example, on-the, another-process may be on the I line, so that the CPU loop is taken away from the game application), the game logic may sometimes cost a frame The time is short or longer than a frame time to calculate the graphic displayed as the next frame. In such a game, a "threshold value" frame calculated with less time than a frame time (for example, a few CPU clock cycles) may eventually appear. It may be easy to use the wear order S to calculate the same scene again using the game state information of all the relatives; ^ This time I know a few more than a frame time

CPU clock cycle time (for example, if the internal cPU bus is slightly externally 139852.doc •24- 200952495

The RAM bus is out of phase' and even if there is no large delay from another process that takes a few milliseconds of coffee time from the game processing, it also has a delay of several CPU cycles. Therefore, when the game is played back, the frame becomes calculated in two frame times instead of a single frame time. Some behaviors are based on the frequency at which the game calculates new frames (for example, when the game samples from the game controller's input). This deviation in the time reference for different behaviors does not affect the game play when the game is played, but it can cause the game being played back to produce different results. For example, if the basketball track is calculated at a stable fps rate, but the game controller input is sampled based on the calculated frame rate, the calculated frame rate may be 53 when the game is recorded. Fps, and when replaying the game, the calculated frame rate may be 52 fps, which can cause differences in whether basketball is prevented from entering the basket, resulting in different results. Therefore, the use of game state recording video games requires a very good 1"true game software design to ensure that the same game state information playback produces exactly the same results. Another prior art method for recording video games is to record only the video output of the pc or video game system (e.g., to a VCR, a DVD recorder, or to a video capture board on a PC). The video can then be rewinded and replayed, or alternatively the recorded video can be uploaded to the Internet (usually after video compression). The disadvantage of this method is that the user is limited to viewing only the self-test viewpoint (the sequence is recorded from it) when playing back the 3D game sequence. In other words, the user cannot change the view point of the scene. In addition, when the compressed video of the recorded game sequence played on the home PC or the game console via the Internet is available to other users 139852.doc -25- 200952495, even if the video system is compressed immediately, It is not possible to upload compressed video to the Internet in real time. The reason for this is because many homes connected to the Internet in the world have highly asymmetric broadband connections (for example, DSL and cable modems typically have a much lower downlink bandwidth than the upload bandwidth). High-resolution video sequences that are compressed and often have a higher bandwidth than the bandwidth of the network's upload bandwidth, making it impossible to upload on the fly. Therefore, after playing the game sequence (which may take a few minutes or even hours), there will be a significant delay before another participant on the Internet can view the game. Although this delay can be tolerated in certain situations (eg, viewing the outcome of a game player that appeared at a previous time), it eliminates the ability to watch a live game of the game (eg, a basketball tournament played by a winning player) or live broadcast. The "immediate playback" capability when playing a game. Another prior art method allows a viewer with a television receiver to watch a video game live broadcast 'but only under the control of the television producer. Some TV channels in the United States and other countries provide video game detection video channels, in which φ TV viewers can view specific video game users on the video game channel (for example, participate in the top players of the tournament) β by using the video game system ( The video output of the pc and/or console is fed into the video distribution and processing device for the television channel. This is the case when the TV channel broadcasts a live broadcast of the basketball game, several of which provide live feeds from different angles around the basketball court. The television channel can then utilize its video/audio processing and effects devices to manipulate the output from various video game systems. For example, a television channel can overlay text indicating the status of different players on top of the video from the video game (as it can be overwritten during the live broadcast of the basket 139852.doc -26-200952495 ball game), and the television The channel can add audio from commentators who can discuss the actions that occur during the game. In addition, the video game output can be combined with a camera that records the video of the actual player of the game (e.g., exhibiting the player's emotional response to the game). One of the problems with this approach is that the live video feeds must be made available instantly to the video distribution and processing equipment of the television channel to make it stimulating for live broadcasts. However, as previously discussed, when the video game system is from home execution (especially when a broadcast packet is included in a live broadcast video from a camera that captures the real world video of the game player), this is often impossible. . In addition, in the case of tournaments, the focus is on the game in the home where the player can modify the game and cheat, as described previously, for these reasons, the video game broadcasts on the television channel are often configured to be gathered at a common location (eg A player and video game system at a television studio or in an arena where the television production device can accept video feeds from multiple video game systems and potentially live broadcast cameras. Although such prior art video game television channels can provide a very stimulating performance for a television viewer (which is the same experience as a live video event (eg, an athlete) presented by a video game player, not only based on its video game 纟The movement in the world # 'and according to its actions in the real world), but these video game systems are often limited to situations where players are physically close to each other. In addition, because the television channels are broadcast, each broadcast channel can only display a video stream selected by the producer of the television channel. Due to these restrictions and broadcast time, production equipment and production staff, 139852.doc -27· 200952495 high cost 'these TV channels usually only show top players participating in the top tournaments. In addition, a given television channel that broadcasts a full screen image of a video game to all television viewers displays only one video game at a time. This severely limits the choice of television viewers. For example, a TV viewer may not be interested in a game displayed at a given time. Another viewer may only be interested in watching a particular player's game play that is not shown by the video channel at a given time. In other

In the case of a 'viewer' may only be interested in seeing how an insider handles a particular level in the game. Other viewers may wish to control the view point (from which the video game is viewed), which is different from the view point selected by the production team or the like. Jane S, TV viewers may have countless preferences in watching video games (even if several different TV channels are available, the special broadcast of the TV network should not be suitable for preference for all of the above reasons, making the prior art video game TV channel There are significant limitations in presenting video games to television viewers. Another disadvantage of prior art video game systems and application software systems is that they are complex and often suffer from errors, crashes, and/or unwanted and unwanted behaviors ( Collectively referred to as "defects") ° Although games and programming programs usually undergo debugging and grading processes (called "software quality assurance" or SQA), they are almost unchanged: - games or applications are Defects will suddenly appear in a wide range of audiences issued to the field. Unfortunately, software developers have difficulty identifying and tracking many defects after they are released. Software developers may find it difficult to recognize defects, even when they understand a defect. It can also be (4) that there is a limited amount of 139852.doc •28·200952495 information that can be caused by (4). For example, the user can call the game developer's consumer service hotline and leave a message stating that when playing the game, the glory begins to flash, then changes to solid blue and the PC freezes. It is the Sqa group. Provides very little information useful in tracking defects. One of the online connections • Some games or applications sometimes provide more information under certain conditions. For example, sometimes a “watchdog” can be used. The process of monitoring whether a game or application "crashes." The watchdog process collects information about the state of the game or application process when the game or application crashes (for example, the state of use of the memory stack, the game or application progresses to it) Statistics of the degree, etc., and then uploading the information to the SQA team via the Internet. However, in a complex game or application, the information can take a very long time to decrypt, in order to accurately determine the user at the time of the crash. What is going on. However, it is impossible to determine what sequence of events caused the crash. Another question associated with pc and game console It is: it suffers from service problems that are extremely inconvenient for consumers. Service issues also affect manufacturers of PCs or game consoles because they usually need to send special boxes to safely ship damaged PCs or consoles, and thus incur repairs. Cost (if the pc or console is under warranty). The game or application software publisher may also be affected by the loss of sales (or online service usage) in the repaired state of the pc and/or console. For example, Sony Playstation(g) 3, Micr〇s〇ft xb〇x 36〇®, Nintendo WiiTM, WWind〇ws-based PC or APPle Macintosh's prior art video game system. Each of these systems includes Central processing unit for executing code (cp is usually 139852.doc -29· 200952495

A graphics processing unit (GPU) for performing high-level graphics operations, and multiple forms of input/output (I/O) for communicating with external devices and users. For the sake of simplicity, these components are shown as being grouped together into a single unit. The prior art video game system of FIG. 1 is also shown to include an optical media drive 104 (eg, a DVD-ROM drive); a hard drive 103 for storing video game code and data; for playing a multiplayer game, for downloading Network connection for games, patches, demos, or other media; a random access memory (RAM) 101 for storing code currently being executed by the CPU/GPU 100; used to receive from during game play The game controller 106 that inputs the command by the user; and the display device 1〇2 (for example, SDTV/HDTV or computer monitor). The prior art system shown in Figure 1 suffers from several limitations. First, the optical disk drive 1 〇 4 and the hard disk drive 1 〇 3 tend to have a much slower access speed than the RAM 101 access speed. When working directly via RAM 1〇1, since RAM 101 typically has a much higher bandwidth and does not suffer from the relatively long search delay of the disc mechanism, CPU/GPU 1 can be processed in practice than directly from The number of polygons per second that the hard disk drive 103 or the optical disk drive 104 can read a lot more than the number of polygons per second. However, only a limited amount of RAM is provided in such prior art systems (e.g., 256 512 dead bits). Therefore, a sequence of "Loading..." is often required, where rami〇i is periodically filled with data for a scene under the video game. - Some systems attempt to overlap the loading of code and game play at the same time, but this can only be done when there is a known order event (4) If, if you are driving along the road, you can carry the road while driving the car. The geometry of the building 139852.doc •30- 200952495). For complex and/or fast scene changes, this type of overlay usually does not work. For example, in the case where the user is in the middle of the campaign and the RAM 101 is completely filled with the data representing the object in the view of the moment, the user rightly moves the view to the left to view that the current is not loaded. The object in the RAM 101 will cause a discontinuity in motion because there is not enough time to load the new object from the hard disk drive 1〇3 or the optical disk drive 1〇4 into the RAM 101. Another problem with the system of Figure 1 is due to the limitations of the storage capacity of the hard disk drive 103 and the optical media 1〇4. Although a disk storage device can be fabricated to have a relatively large storage capacity (eg, 50 billion bytes or more than 500 million bytes), it does not provide the specifics encountered in current video games. Sufficient storage capacity for the situation. For example, as previously described, a British football video game may allow a user to select among many groups, players, and sports fields around the world. For each group, each player, and each field, a large amount of texture mapping and environment mapping is required to characterize the 3D surface defects in the world (for example, each group has a unique jersey, each requiring a unique texture mapping) 》 Used to solve this latter problem—Technology: For games, users select text, texture, and environment mapping to pre-calculate texture and environment mapping. This can involve a number of tens of intensive processes, including dust-removing images, 3D mapping, and organizational data structures. Therefore, when video games perform such impulses, there may be a delay for the user. One of the method principles of delay is to perform all of these calculations when the game is initially developed - including the roster of rosters and the per-arrangement of the stadium. The release version of the game thus 139852.doc .31 - 200952495 will be stored on optical media 104 or All pre-processed data on one or more servers on the Internet, when the user makes a selection, pre-processes only the selections for a given group, player roster, and sports field selection via the Internet. The data is downloaded to the hard disk drive 1〇3. However, as a practical matter, the pre-loaded material of each permutation that may be played in the game play may easily be a terabyte of data, which is far away. Exceeding the capacity of today's optical media devices. In addition, the information used for a given group, playing the φ family roster, and playing field selections may easily be hundreds of millions of bytes of data or numbers. Information above the byte. In the case of a home network connection (for example, 10 Mbps), downloading this data via the network connection 1〇5 takes longer than calculating the data on the local end. Therefore, in Figure 1 The prior art game architecture demonstrated exposes users to significant delays between large scene transitions of complex games. Another problem with prior art methods such as the prior art method shown in Figure 1 is that video games tend to It becomes higher order and requires more CPU/GpU processing power. Therefore, even with an unlimited amount of RAM, the video game hardware requirements exceed the peak level of processing power available in these systems. Therefore, users are required. Upgrade game hardware every few years to stay in sync (or play newer games at a lower quality level). One consequence of the trend of higher-order video games over the past is that machines for home video games are usually not available. Economic efficiency, because its cost is usually determined by the requirements of the two most effective games it can support. For example, XBox 360 may be used to play similar battles. "Gears War" game, which requires high-performance CPU, GPU and hundreds of millions of bytes of RAM, or 139852.doc 32· 200952495 may use XBox 360 to play Pac Man. It is a game from the 1970s that requires only a few thousand bytes of RAM and a very low-performance CPU. In fact, XBox 36 has enough computing power to host many of the same "Pacman" games at the same time. In most hours of the week, the video game console is usually cut off. According to • July 2006 Nlelsen Entertainment's research on active gamers aged 3 and over, on average, active players spend 14 hours a week or only 12 weeks in a week. To play console video games. This means that the average video game console is idle 88% of the time, which is the inefficient use of expensive resources. It is assumed that the video game console is often funded by the manufacturer to reduce the purchase price (hope that the funding will come from the future) It’s especially meaningful to earn back the royalties of video game software purchases.) Video game consoles also incur costs associated with virtually any consumer electronics device. For example, the electronics and mechanisms of the system need to be housed in a housing. Manufacturers need to provide service guarantees. Selling the retail of the system φ Merchants are required to receive sales for the system and/or for the sales of video game software. All of these factors add to the cost of the video game console, which must be funded by the manufacturer, delivered to the consumer, or funded by both the manufacturer and the consumer. Another piracy is a major problem in the video game industry. In fact, the security agencies used on each of the larger video game systems have been "broken" over the years, resulting in unauthorized copying of video games. For example, the Xb〇x 360 security system ruptured in July 2006 and users are now able to download illegal copies online. Downloadable games (for example, games for PC or Mac) Special 139852.doc -33· 200952495 Don't be vulnerable to piracy. In certain regions of the world (where piracy is not strong), there is virtually no viable market for independent video game software, as users can easily purchase pirated copies at a fraction of the cost with legitimate copies. Moreover, in many parts of the world, the cost of the game console is a high percentage of revenues so that even if piracy is controlled, few people can afford the current state of the art gaming system. In addition, the market for used games has reduced the revenue of the video game industry. When the user becomes bored with the game, they can sell the game to a store that resells the game to other users. This unauthorized but widespread practice significantly reduces the revenue of game publishers. Similarly, when there is a platform change every few years, there is usually a sales reduction of about 50%. This is because when the user knows that a newer version of the platform is about to be released, the user stops buying the game for the older platform (for example, when the Playstati〇n 3 is about to be released, the user stops purchasing the Playstation 2 game). The combined loss of sales and the increased development costs associated with the new platform can have a significant adverse impact on the profitability of game developers. The new game console is also very expensive eXbox 36〇, Nintend〇忉丨 and

Sony Playstation 3 is retailed for hundreds of dollars. High-capacity PC gaming systems can cost up to $8000. This represents a significant investment by the user, in particular, considering the fact that the hardware became obsolete in a few years and many systems were purchased for the child. The above-mentioned method is an online game in which the game code and data host are hosted on the device and delivered to the good machine as required, and the compressed video and audio are streamed via a digital broadband network. . Some of the 139852.doc •34·200952495 divisions (such as F-land's G-Cluster, which is now a subsidiary of SOFTBANK Broadmedia in Japan) currently offer these services online. Similar gaming services become available in local networks such as hotels and their networks provided by dsl and electric TV providers. The biggest drawback of these systems is the delay, the signal, the signal travels to the game server and the _ from the game server. The game server is usually located in the "head end" of the operator. Fast motion video games (also known as "twitch" video games) require very low latency between when the user performs the action by the game controller and when the display screen is updated to show the result of the user action. A low latency is required to make the user feel the game respond "immediately". For the type of visual game and the proficiency of the user, the user is satisfied at different delay intervals. For example, for a slow informal game (like a backgammon) or a slow action role-playing game, 100 milliseconds. The delay may be tolerable, but in a fast action game, a delay of more than 70 milliseconds or 80 milliseconds may cause the user to perform worse in the game and is therefore unacceptable. For example, in a game that requires fast response time, 'when the delay increases from 50 milliseconds to 1 millisecond, there is a sharp drop in accuracy." When the game or application server is installed in a nearby controlled network environment or to When the user's network path is predictable and/or tolerant of bandwidth peaks in the network environment, control delays are much easier in terms of maximum latency and latency consistency (eg, so users observe from the network) The degree of stability of the digital video stream can be controlled as follows: between the τν network head end and the cable TV user's home, or from the DSL central office 139852.doc -35· 200952495 to the DSL user family 'Or in a commercial office area network (LAN) environment from a server or user. Also, it is possible to obtain a specific hierarchical point-to-point private connection between businesses with guaranteed bandwidth and latency. However, the game console is hosted in a feeder connected to the universal Internet and then the compressed video stream is streamed to the user via a broadband connection. In applications, many factors contribute to latency, leading to severe limitations in the deployment of prior art systems. In a typical broadband-connected home, users can have DSL or cable modems for broadband piggyback services. Banding services typically result in up to 25 milliseconds of travel delay (and sometimes more) between the user's home and the universal Internet. In addition, there is a problem caused by routing data to the server center via the Internet. Round trip delay. The delay through the Internet changes based on the route given the data and the delay caused by the data being routed. In addition to the routing delay, it also passes through the fiber that interconnects most of the Internet. The inconvenience leads to a round trip delay. For example, ❹ For every 1000 miles, a round trip delay of about 22 milliseconds is incurred due to the speed of light passing through the fiber and other consuming. Additional delays may be due to the internet via the Internet. In the case of the data rate of the data of the stream, for example, 'If the user has a DSL service sold under the "6 Mbps dsl service", then In practice, the user will most likely get at most less than 5 Mbps under the line throughput 'that is likely to be periodically seen due to various factors (such as peak load time at the Digital Subscriber Line Access Multiplexer (DSLAM)) Hanging) the resulting connection is degraded. A similar problem can arise if there is congestion in the shared coaxial cable at the local end of the neighboring loop or there is congestion in other places in the cable modem system network 139852.doc • 36 - 200952495 Reduce the data rate of the cable modem used for the connection sold by the "6 Mbps Cable Data Server Service" to much less than the data rate. If the data packet at a stable rate of 4 Mbps is made to the user datagram protocol (UDp) The format is self-serving unidirectionally. The server center streams through these connections. If everything works properly, the data packet will pass without incurring additional delay, but if there is congestion (or other obstruction) and only 3.5 Mbps can be used to stream data to the user, then in the pattern (4), the 'packet will be discarded' resulting in lost data, or the packet will be queued at the congestion point Which may be sent to the far, thereby introducing additional latency. Different congestion points have different secondary queue capacity for storing delayed packets, so in some cases, the congestion cannot be successfully resolved. In other cases, millions of bits of data are queued and eventually sent. However, in almost all cases, the queue at the 'crowding point' has a capacity limit, and once it exceeds these limits, the queue will overflow and the packet will be discarded. Therefore, in order to avoid incurring additional delays (or worse, loss of packets), data rate capacity beyond the game or application server to the user must be avoided. Delays are also incurred due to the time required to compress the video in the server and decompress the video in the client device. When the video game executed on the server is calculating the next frame to be displayed, further delay is incurred. Currently available video compression algorithms suffer from high data rates or high latency. For example, Motion JPEG is an in-frame lossy compression algorithm characterized by low delay =. Each frame of the video is compressed independently of each other frame of the video. When the client device receives the compressed motion; 1^ (3 frames of video), its 139852.doc -37· 200952495 can immediately decompress the frame and display the frame, resulting in very low latency. Because each frame is compressed separately, the algorithm cannot take advantage of the similarity between successive frames, and therefore only the in-frame video compression algorithm suffers from very high data rates. For example, 60 fps (per second) Frame number) 640M80 Motion JPEG video may require 40 Mbps (megabits per second) or 40 Mbps (megabits per second) data. These high data rates for these low resolution video windows It will be too expensive in many broadband applications (and indeed for most consumer Internet-based applications). Also, because each frame is independently compressed, it may be due to lossy compression. The artifacts in the resulting frame may appear at different locations in successive frames. This can result in visual artifacts that appear to the viewer when the video is decompressed. Other compression algorithms (such as, From Microsoft Corporation's MPEG2, H.264, or VC9) when used in prior art configurations, high compression ratios can be achieved 'but at the expense of high latency." These algorithms utilize inter-frame compression and in-frame compression. The algorithm performs only in-frame compression of the frame. Such a frame is called a key frame (commonly referred to as an "I" frame). Then, these algorithms usually have an I frame and a previous image. The box is compared with successive frames. Instead of compressing the previous frame and the successive frames independently, the algorithm determines how the image changes from the I frame to the previous frame and the subsequent frame, and then changes them. Save as: "B" frame (for changes before I frame) and "P" frame (for changes after I frame). This results in a much lower data rate than just in-frame compression. Usually at the expense of higher latency. The I frame is usually much larger (often 10 times larger) than the B or P frame, and therefore, 139852.doc -38- 200952495 is proportionally costly for a given data rate transfer. For a longer period of time. Consider (for example) a situation where the frame is B The size of the box and the p frame is 10 times, and for each single frame, there are 29 8 frames + 3 卩 卩 frames = 59 frames, or for each "frame group" (G〇 p) A total of 6 frames. Therefore, at 6〇fps, there is one (9) frame G〇p per second. Assume that the (4) channel has a maximum data rate of 2 Mbps. In order to achieve the highest quality in the channel. Video, the compression algorithm will generate a 2 Mbps stream, and given the above ratio 'this will result in 2 million bits per frame (Mb) / (59 + 10) = 30,394 bits and each I frame 303, 935 One bit. When receiving a compressed video stream by a decompression algorithm, in order to play video stably, it is necessary to decompress and display each frame at regular intervals (for example, '60 fps). In order to achieve this result, if any frame is subjected to a transmission delay, all frames need to be delayed by at least one delay'. Therefore, the worst case frame delay will define the delay for each video frame. Because the frame is the largest, the I frame introduces the longest transmission delay' and the entire I frame will have to be received before the I frame can be decompressed and displayed (or between any frames in the 1 frame). Assume that the channel data rate is 2

At Mbps, transmitting an I frame will cost 303,935/2 Mb=145 milliseconds. An inter-frame video compression system (as described above) that uses a large percentage of the bandwidth of the transmission channel will experience long delays due to the large size of the I frame relative to the average size of the frame. Or 'in other words' when the prior art interframe compression algorithm achieves a lower average data rate per frame than the in-frame compression algorithm (eg '2 Mbps vs. 40 Mbps), it still suffers high due to the large frame Peak data rate per frame (for example, 3〇3,935*6〇=18 2 Mbps). But keep in mind that the above analysis assumes that both the p-frame and the B-frame are much smaller than the I-frame. 139852.doc •39· 200952495 Although this is generally true, 'this is not true for _ with high image complexity that is not related to previous frames, high motion or scene changes. In these cases, the 'P frame or B frame can become as large as 1 frame (SP frame or B frame becomes larger than 1 frame, then the tip compression algorithm will usually be "forced"! Frame and replace 1> frame or B frame with 1 frame. Because, the peak data rate of the I frame size can appear in any video stream, so for compressed video, when the average video data rate is close to the transmission channel.

At the rate rate capacity (often in this case, given the high data rate requirements for video), high peak data rates from I frames or large ρ frames or frames result in high frame delays.

Of course, the above discussion only characterizes the compression algorithm delays produced by large Β, ρ or I frames in the GOP. If you use a frame, the delay will be higher. The reason for this is because all the frames and I frames after the frame must be received before the 6 frame can be displayed. Therefore, in a sequence of picture groups (g〇P) such as ΒΒΒΒΒΙΡΡΡΡΡΒΒΒΒΒΙΡΡΡΡΡ, where there are 5 frames before each I frame, the video can only be decompressed after receiving the subsequent β frame and I frame. The first frame is displayed. Therefore, if the video is streamed at 60 fps (ie, 16.67 ms/frame), before the first B frame can be decompressed, no matter how fast the channel bandwidth is, five B frames and I frames are received. It will cost ι6.67*6=1〇〇 milliseconds, and this is only the case of 5 β frames. A compressed video sequence with 3 frames of B is quite common. In addition, at low channel bandwidths such as 2 Mbps, the effects of delay due to the size of the I frame are largely added to the delay caused by waiting for the B frame to arrive. Therefore, on the 2 Mbps channel, in the case of a large number of B 139852.doc -40 - 200952495 frames, it is quite easy to use a prior art video compression technique with a delay of more than 5 milliseconds or more than 500 milliseconds. If you do not use the b-frame (for a given quality level, at the expense of a lower compression ratio), then the B-frame delay is not incurred, but the delay caused by the peak frame size described above is still incurred. • The problem is aggravated by the nature of many video games. The video compression algorithm utilizing the GOP structure described above is largely optimized for use with live video or movie material that is intended to be used for passive viewing. Usually 'cameras (real cameras, or virtual cameras in the case of computer-generated animations) and scenes are relatively stable, just because the camera or scene moves too far and bumpy, video or movie material (a) is usually unpleasant to watch. And (b) if it is being viewed, when the camera suddenly bumps back and forth, the viewer is usually unable to follow the action closely (for example, if the camera is disturbed and suddenly when shooting a child who is smouldering on the birthday cake) When bumping back and forth between the cakes, the viewers usually focus on the children and the cake, and ignore the brief interruption when the camera suddenly moves. In the case of video conferencing or video conferencing, the camera can be held in a fixed position and not moved at all, resulting in very little data spikes. But 31) high-motion video games are characterized by constant motion (for example, considering 3D competitions where the entire frame is in fast motion during the duration of the competition, or considering a first-person shooter game where the virtual camera is constantly Move back and forth bumpyly). These video games can produce a sequence of frames with large and frequent peaks, where the user may need to clearly see what happens during their sudden movements. Therefore, in 3D high motion video games, compression artifacts 139852.doc -41 · 200952495 are far intolerable. Therefore, the video output of many video games (due to their nature) produces a compressed video stream with very high and frequent peaks. Assuming that the user of the fast motion video game has a small tolerance for high latency, and given all of the above delay reasons, there have been limitations to video games hosted by server hosts that stream video over the Internet. In addition, users of such applications suffer from similar _ restrictions if highly interactive application hosts are hosted on a common Internet and video is streamed. These services require network configuration, where the colocation server is placed directly at the headend (in the case of cable broadband) or in the central office (in the case of Digital Subscriber Line (DSL)), or in a commercial context Within the LAN (or on a specially graded private connection) to control the route and distance from the client device to the server to minimize latency and adapt to peaks without incurring delays. LANs (typically rated at 1 Mbps to 1 Gbps) and leased lines with sufficient bandwidth typically support peak bandwidth requirements (for example, 18 Mbps peak bandwidth is one of the 100 Mbps LAN capacity fractions). • If special adaptation is made, the peak bandwidth requirement can also be accommodated by the residential broadband infrastructure. For example, on a cable TV system, a dedicated bandwidth can be given for digital video traffic. This dedicated bandwidth can handle peaks such as large I frames. In addition, on DSL systems, higher speed DSL modems (considering high peaks) can be supplied, or specially graded connections that can handle higher data rates can be supplied. However, conventional electronic data processors and DSL infrastructure attached to the universal Internet are far from tolerable for peak bandwidth requirements for compressed video. Therefore, the online service (hosting the video game or application host in a server center that is long distances from the client device, and 139852.doc • 42- 200952495 is then connected via the Internet via a conventional residential broadband connection. The compressed video output stream) suffers from significant latency and peak bandwidth requirements - especially for games and applications that require very low latency (eg %, first-person shooter and other multi-user, interactive action games, & requires an application that responds quickly.) The present disclosure will be more fully understood from the following description of the embodiments and the accompanying drawings, which are not intended to limit the scope of the disclosure to the particular embodiments shown, but only Explain and understand. Specific details (such as device types, system configurations, communication methods, etc.) are set forth in the following description in order to provide a thorough understanding of the disclosure. However, it will be appreciated by those skilled in the art that the specific embodiments described in the practice may not require such specific details. 2a to 2b provide a high-level architecture of the two embodiments, in which the video game and the software application are hosted by the host hosting service 21 and hosted by the user service under the subscription service (note, "user “Location” means that the user device 205 at any location where the user is located 'including the mobile device if it is outdoor” is stored via the Internet 2〇6 (or other public network or private network). take. The client device 2〇5 may be a general-purpose computer having an internal or external display device 222 with a wired or wireless connection to the Internet (such as a PC based on Mi(10)oft Window^Lim^ or a Macintosh computer of the APple company), It can be a dedicated client device (such as a wired or wireless connection to the Internet) that outputs video and audio to a monitor or television 222, such as a set-top box, or it can be speculated as 139852.doc -43· 200952495 A mobile device with a wireless connection to the Internet. Any of these devices may have its own user input device (eg, keyboard, button, touch screen, track pad or inertial-sensing wand, video capture camera and/or Or a motion tracking camera, etc., or it can be connected by a wire connection or wirelessly connected to the input device 221 (eg, keyboard, mouse, game controller, inertial sensing stick, video capture camera, and/or motion) Tracking cameras, etc.) ^ As described in more detail below, the "hosting service 210" includes servers of various performance levels (including those with high-capacity CPU/GPU processing capabilities). During playback of the game or use of the application on the colocation service 210, the home or office client device 205 receives the keyboard and/or controller input from the user' and then transmits the controller input via the Internet 206 to The colocation service 210, the colocation service 21, in response to this, executes the game code and generates successive frames for the video output (video sequence) of the game or application software (eg 'if the user presses The button that directs the person on the screen to move to the right will then generate a sequence of video images showing the person moving to the right. The video image sequence is then compressed using a low latency video compressor, and the colocation service 21 then transmits the low latency video stream over the internet 206. The home or office client device then decodes the compressed video stream and reproduces the decompressed video image on a monitor or τν. Therefore, the calculation and graphics hardware requirements of the client device 2〇5 are significantly reduced. The client 205 only needs to have the processing capability for forwarding the keyboard/controller input to the Internet 206 and decoding and decompressing the compressed video stream received from the Internet 2〇6, actually 139852 .doc 200952495 Any personal computer can do this in software on its CPU (for example, Intei's dual-core cpu running at about 2 GHz can decompress 72 〇p using compressor encoding such as H.264 and Windows Media VC9 Hdtv).

In addition, in the case of any client device, the dedicated chip can also perform the power of these standards in real time at a much lower cost than the general purpose CPU and much less power consumption than the general purpose CPU (such as required by modern PCs). Video decompression. Notably, in order to perform the transfer controller input and decompress the video, the client device 205 does not require any special graphics processing unit (GPU), CD player or hard disk drive (4), as shown in the figure. Prior art video game system). As gaming and programming software become more complex and more photo-realistic, they will require higher performance CPUs, Gpus, more rams, and larger and faster drives, and will enable The computing power at the service 2H) is constantly being upgraded' but the end user will not need to upgrade the home or office user: platform 205, as the court or office client platform will be enabled by a given video decompression algorithm 2 The processing requirement of 〇5 is fixed for the display resolution ^ round frame rate. Therefore, the figure "The system described in the diagram does not have the hardware _ and compatibility issues seen today. In addition, because the game and application software is only executed in the colocation service (4) Home or office (unless otherwise the "office" as used herein will include any ==, including, for example, classrooms), there will never be a copy of the game or application software (in the form of optical media, or downloaded) Software). This significantly reduces the possibility of illegal copying (piracy) of games or application software. 139852.doc •45· 200952495 sex and the possibility of mitigating the piracy of valuable libraries that can be used by games or application software. In fact, if you need a specialized server (for example, a very expensive, large or noisy device) to play games or application software that is not feasible for home or office use, even if you get a game or application software. A copy of the pirated copy will not be operated in the home or office. In one embodiment, the colocation service 210 provides software to a game or application software developer (which is generally referred to as a software development company, a game or movie studio, or a game or application software publisher) 220 that designs a video game. The development tool 'so that it can design a game that can be executed on the colocation service 21 。. These tools allow developers to take advantage of the features of colocation services (the features are usually not available in a standalone PC or game console) (for example, very fast access to very large databases of complex geometries (unless otherwise conditional) Otherwise, "geometry" will be used herein to refer to the polygons, textures, rigging, lighting, behavior, and other components and parameters that define the 3D dataset). Under this architecture, different business models are possible. Under a model, the host hosting service 210 charges a subscription fee from the end user and pays a royalty to the developer 22, as shown in Figure 2a. In an alternate implementation (not shown in the figure), the developer 220 charges the subscription fee directly from the user and pays the colocation service 210 for the cost of hosting the game or application content. These basic principles are not limited to any particular business model used to host an online game or application host. Compressed video features 139852.doc -46· 200952495 As previously discussed, one of the significant issues with online video game services or application software services is latency. The 70 ms _ 8 〇 millisecond delay (the time from when the input device is actuated by the user to when the response is displayed on the display device) is the upper limit for games and applications that require fast response times. However, this is very difficult to achieve in the case of the architecture shown in Figure 2a and the hole due to a large number of actual and physical constraints. As indicated in Figure 3, when a user subscribes to an internet service, the connection is typically rated at a nominal maximum data rate 301 to the user's home or office. Depending on the provider's policy and routing device capabilities, the maximum data rate may be more or less strictly enforced', but typically the actual available data rate is lower due to one of many different reasons. For example, there may be too many network traffic at the DSL central office or on the local cable modem loop 'or there may be noise on the cable, causing the dropped packets' or the provider may establish per user The maximum number of digits per month. Currently, the maximum downlink data rate for cable and DSL services is often in the range of hundreds of kilobits per second (Kbps) to 30 Mbps. Honeycomb services are typically limited to hundreds of Kbps of data. However, the speed of broadband services and the number of users subscribed to broadband services will increase dramatically over time. Currently, some analysts estimate that 33% of US broadband users have data rates below 2 Mbps or above 2 Mbps. For example, some analysts predict that by 2010, more than 85% of US broadband users will have data rates of 2 Mbps or more. As indicated in Figure 3, the actual available maximum data rate of 3 〇 2 may fluctuate over time. Therefore, it is sometimes difficult to predict the actual available data rate for a particular video stream in low latency, online gaming or application software situations 139852.doc •47· 200952495. The right is for a given amount of scene complexity and motion to maintain a given resolution (eg, 64〇><480 @ 60 fps) for a given number of frames per second (fpS) A quasi-required data rate of 3〇3 rises above the actual available maximum data rate 302 (as indicated by the peak in Figure 3), and several questions may arise. For example, some Internet services will only discard packets, causing lost data and distorted/lost effects on the user's video screen. The service will temporarily buffer (ie, queue) additional packets. These packets are provided to the client at the available data rate, resulting in an increase in latency - an unacceptable result for many video games and applications. Finally, some Internet service providers view the increase in data rate as a malicious attack (such as a denial of service attack (a well-known technique used by computer hackers to disable network connections)) and will be cut in a specific time period. Disconnect the user's internet connection. Thus, the embodiments described herein seek to ensure that the required data rate for a video game does not exceed the maximum available data rate. Master Managed Service Architecture Figure 4a illustrates the architecture of a hosted service 21〇 according to an embodiment. The colocation service 210 can be located in a single server center or can be spread across multiple server centers (to provide a low latency connection to users with a lower latency to a particular server center than others) to Provides load balancing between users' and provides redundancy in the event of one or more server center failures. The colocation service 21〇 can eventually include tens of thousands or even millions of servers 402, so that the servo is very large. 139852.doc -48· 200952495 The user base. The colocation service control system 401 provides overall control over the host escrow service 210 and directs routers, servers, video compression systems, billing and accounting systems, and the like. In one embodiment, the hosted service control system 401 is implemented on a Linux-based distributed processing system. • The processing system is bound to storage for user information, server information, and system statistics. The RAID array of the database. In the above description, the various actions implemented by the colocation service 21 are initiated and controlled by the colocation service control system 401 unless otherwise attributed to other specific systems. The colocation service 210 includes a number of servers 402, such as those currently available from Intel, IBM, and Hewlett Packard, among others. Alternatively, the server 402 can be assembled into a custom component configuration, or the server 402 can ultimately be integrated to implement the entire server as a single wafer. Although this figure shows a small number of servers 402 for purposes of illustration, in actual deployments there may be as few as one server 402 or up to millions or tens of millions of servers 402. Server 402 can be configured in the same way (as an example of some configuration parameters) with the same CPU type and performance; with or without GPU, and with GPU, with the same GPU type and performance; with the same number of CPUs And GPU; RAM with the same amount and the same type/speed; and with the same RAM configuration), or various subsets of the server 402 can have the same configuration (for example, 25% of the servers can be configured in a specific way) 50% of the servers are configured in a different way, and 25% of the servers are configured in another way), or each of the feeders 4 〇2 can be different. 139852.doc -49- 200952495 In one embodiment, the server 402 is free of magnetic disks, that is, does not have its own native mass storage device (which is an optical or magnetic storage device, or a semiconductor-based storage device). 'Other large-capacity storage components such as flash memory or servo-like functions, each server accessing a shared mass storage via a fast backplane or a network connection. In one embodiment, the _quick connection is a storage area network (SAN) 403 connected to a series of independent redundant disk arrays (RAID) 405, with a parameter between the devices implemented using the ultra-high speed Ethernet connection. As is known to those skilled in the art, s AN 403 can be used to combine many RAID arrays 405, resulting in extremely high bandwidths that are close to or possibly exceed those available from current game consoles and RAM in pC. bandwidth. Moreover, while RAID arrays based on rotating media such as magnetic media often have significant seek time access latency, RAID arrays based on semiconductor memories can be implemented with much lower access latency. In another configuration, some or all of the servers 4〇2 provide some or all of their own mass storage at the local end. For example, the server ❹ 402 can store frequently accessed information (such as a copy of its operating system and video game or application) on a low-latency local flash memory-based storage. The SAN can be used to access a RAID array 405 with higher seek latency based on rotating media to access a large database of geometry or game state information less frequently. In addition, in one embodiment, the colocation service 210 uses the low latency video compression logic 404 described in detail below. The video compression logic 410 can be implemented in software, hardware, or any combination thereof (the specific embodiments thereof are described below). Video compression logic 404 includes logic for compressing audio and visual material 139852.doc -50-200952495. In operation, when a video game is played via a keyboard, mouse, game controller or other input device 421 or an application at the user location 211 is used, the control signal logic 413 on the client 415 will indicate A push button press (and other types of user input) control signals 406a_b (typically in the form of a UDP packet) are transmitted to the colocation service 210 to route control signals from a given user to the appropriate server (or Multiple servos are routed to multiple servers 402 in response to the user's input device. As illustrated in Figure 4a, control signals 406 & can be routed to server 402 via san. Alternatively or additionally, control signals 4〇6b may be routed directly to server 402 via a hosted service network (e.g., an Ethernet-based local area network). Regardless of how the control signals 4〇6a-b are transmitted, the or the servers execute the game or application software in response to the control signals 406a-b. Although not illustrated in FIG. 43, various network connection components (such as 'firewalls and/or gateways') can handle the φ edge of the colocation service 21 (eg, colocation service 210 and internet 410) Incoming and outgoing traffic at the edge of the user premises 211 (between the Internet 41〇 and the home or office client 415). The graphics and audio output of the executed game or application software (ie, the new video image sequence) is provided to the low-latency video compression logic 4〇4, and the low-latency video compression logic 4〇4 is based on the low-latency video compression technique (such as The video image sequence is compressed and compressed via the Internet 41 (or, as described below, via the best high speed network service bypassing the universal Internet). The video stream (usually with compressed or uncompressed audio) is transmitted back to 139852.doc • 51- 200952495 to the client 415. Next, the low latency video decompression logic 412 on the client 415 decompresses the video and audio streams and reproduces the decompressed video stream, and typically plays the decompressed audio stream on the display device 422. Alternatively, the audio may be played on a separate speaker from the display device 422 or not played at all. Note that although input device 421 and display device 422 are shown as stand-alone devices in Figures 2a and 2b, they can be integrated into a client device such as a portable computer or mobile device. ❹ Home or office client 415 (described previously as home or office client 205 in Figures 2a and 2b) can be a very low cost and low power device with very limited computational or graphical performance and possibly very limited The local mass storage device does not have a local large-capacity storage. In contrast, each of the servers 402 coupled to the SAN 403 and the plurality of RAIDs 405 can be an exceptionally efficient computing system, and in fact, if multiple servers are cooperatively used in a side-by-side configuration, almost There are no restrictions on the amount of computation and graphics processing that can be tolerated. In addition, the computational power of the server 402 is provided to the user due to the low latency video compression 4〇4 and the low latency video decompression 412 (perceptually perceived by the user). When the user presses the button on the input device 42 i , the image on the display 422 is updated in response to the button press (with no meaningful delay in perception) as if the game or application software was executing at the local end. Thus, for a very low-performance computer or a home or office client 415 that is only a low-cost chip that implements low-latency video decompression and control signal logic 413, it is effectively available to the user from a remote location that appears to be available at the local end. Provide any computing power. This gives the user the ability to play the most network-level, processor-intensive (usually new) video games and applications with the highest performance 139852.doc •52· 200952495. Figure 4c shows a very basic and inexpensive home or office client device 465. This device is an embodiment of the home or office client 415 from Figures 4a and 4b. It is about 2 inches long. It has an Ethernet jack 462 that interfaces with an Ethernet power over Ethernet (PoE), which receives its power from the Ethernet jack 462 and its connection to the Internet. Sex. The device is capable of performing ❹ NAT over a network that supports Network Address Translation (NAT). In an office environment, many new Ethernet switches have

PoE and bring the PoE directly to the Ethernet jack in the office. In this way

In the shape of If, the required Ethernet cable from the wall jack to the client 465 is required. If the available Ethernet connection does not carry power (for example, in a home with a dsl or cable modem but no PoE), there is a low-cost wall-type "brick" (ie, power supply) available. It will accept an Ethernet-free electrical meter without power and output an Ethernet network with P〇E. The client 465 contains control signal logic 413 (FIG. 4a) coupled to the Bluetooth wireless interface with a Bluetooth input device 479 such as a keyboard, mouse, game controller and/or microphone and/or headset. Interface. Moreover, one embodiment of the client 465 can output video at 120 fps with the display device 468. The display device 468 can support 120 fps video and signal a pair of shading glasses 466 (typically via infrared) for each One successive frame alternately shades one eye and then covers the other eye. The effect that the user feels is "jumping out" to show the image of the screen. One of the operations is supported. The display device 468 is a Samsung HL-T5076S. Since the video stream for each eye is separate, in an embodiment where the two independent video streams are compressed by the host 139852.doc • 53·200952495 pipe service 210, the frames are interleaved in time, and The box is decompressed in client 465 in two separate decompression processes. The client 465 also includes low-latency video decompression logic 412, which decompresses incoming video and audio and outputs via HDMI (High Definition Multimedia Interface), connector 463, HDMI (High Definition Multimedia Interface), connector 463 It is inserted in SDTV (Standard Definition Television) or HDTV (High Definition Television) 468 to provide video and audio to the TV, or to be inserted into the monitor 468 supporting HDMI. If the user's monitor 468 does not support HDMI ’, HDMI to DVI (digital visual interface) can be used, but the audio will be lost. Under the HDMI standard, display capabilities (eg, supported resolution 'frame rate') 464 are conveyed from display device 468 and then transmitted back to colocation service 21 via Internet connection 462, thus The colocation service 210 can cause the compressed video to be streamed in a format suitable for the display device. Figure 4d shows a home or office client device 475 that is identical to the home or office to client device 465 shown in Figure 4c, except that the client device 475 has more external interfaces. Again, the client 475 can accept P〇E to power 'or it can occupy an external power adapter (not shown) that is plugged into the wall. The video camera 477 provides the compressed video to the client 475 using the client 475 USB input, and the compressed video is uploaded by the client 475 to the colocation service 210 for use as described below. The low latency compressor will be built into the camera 477 using the squeezing technique described below. In addition to having an Ethernet connector for its Internet connection, the client 475 also has an 8 〇 211 g wireless interface to the Internet. Both interfaces 139852.doc -54· 200952495 are capable of using NAT within a network that supports NAT. In addition to the HDMI connector for outputting video and audio, the client 475 also has a dual-link DVI-I connector, and the dual-link DVI-I connector includes an analog output (and has a standard that will provide VGA output). Adapter • Cable). It also has an analog output for composite video and S video. For audio, the client 475 has a left/right analog stereo RCA jack and has a TOSLINK output for digital audio output. In addition to the Bluetooth wireless interface to input device 479, it also has a USB jack for interfacing to the input device. Figure 4e shows one of the internal architectures of the client 465. Embodiments Q. All of the devices or devices shown in the figure can be implemented in a field programmable logic array, a custom ASIC, or several discrete devices (custom design or off-the-shelf). In the middle. Ethernet 497 with PoE is attached to the Ethernet interface 481. Power 499 is derived from Ethernet 497 with P〇E and is connected to the remaining devices in user terminal 465. Bus 480 is a common bus for communication between devices. The control CPU 483 that executes the small client control application from flash memory 476 (almost any small CPU is suitable, such as the MIPS R4000 series CPU at 100 MHz with embedded RAM) is implemented for the network (ie The protocol of the Ethernet interface is stacked and also communicates with the colocation service 21' and configures all devices in the client 465. It also processes the interface with the input device 469 and sends the packet (along with the forward error correction " user controller data) back to the colocation service 21 . 139852.doc • 55· 200952495 In addition, the control CPU 483 monitors the packet traffic (e.g., whether the packet is lost or delayed, and its timestamp). This information is sent back to the colocation service 210 so that it can constantly monitor the network connection and adjust its inner valley accordingly. The control program for controlling the CPU 483 and the unique serial number for the particular client 465 unit are initially loaded for the flash memory 476 at the time of manufacture. This sequence number allows the colocation service 210 to uniquely identify the user 465 unit. The Bluetooth interface 484 wirelessly communicates to the input device 469 via its antenna (within the client 465). Video decompressor 486 is a low latency video decompressor configured to implement the video decompression described herein. A large number of video decompression devices exist either as off-the-shelf products or as intellectual property (Ip) with designs that can be integrated into FPga or custom ASICs. One company that provides IP for the H 264 decoder is Ocean Logic of Manly of NSW Australia. The advantage of using IP is that the compression technique used in this article does not match the compression standard. Some standard decompressors are flexible enough to be configured to accommodate the compression techniques in this article' but some standard decompressors may not. However, in the case of IP, there is full flexibility in redesigning the decompressor as needed. The output of the video decompressor is coupled to the video output subsystem 487, which couples the video to the HDMI interface. 490 video output. The audio decompression subsystem 488 can be implemented using an available standard audio decompressor, or it can be implemented as, or can be, for example, implemented 139852.doc -56- 200952495

The audio decompression is implemented in the control processor 483 of the Vorbis audio decompressor. The device that implements the audio decompression is coupled to an audio output subsystem 489 that couples the audio to the audio output of the HDMI interface 490. Figure 4f shows an embodiment of the internal architecture of the client 475. As can be seen, in addition to the additional interface and optional external DC power from the power adapter plugged into the wall (and if so, the optional external DC power replacement will be from the Ethernet PoE 497), The architecture is the same as that of the client 465. The functionality common to the client 465 will not be repeated below, but the additional functionality will be described below. The CPU 483 communicates with additional devices and configures additional devices.

The WiFi subsystem 482 provides wireless internet access via its antenna as an alternative to Ethernet 497. The WiFi subsystem is available from a number of manufacturers, including the Atheros Communications ° USB Subsystem 485 of Santa Clara, CA, which provides an alternative to Bluetooth communication for wired USB input device 479. The USB subsystem is fairly standard and can be easily used in FPGAs and ASICs, and is often built into existing devices that perform other functions such as video decompression. The video output subsystem 487 produces a wider range of video outputs than the video output in the client 465. In addition to providing HDMI 490 video output, it provides DVI-I 491, S-Video 492 and Composite Video 493. Also, when the DVI-I 49 1 interface is used for digital video, the display capability 464 is passed back from the display device to the control CPU 483 so that it can notify the host escrow service 210 of the capabilities of the display device 478. The 139852.doc -57· 200952495 interface provided by the video output subsystem 487 is a fairly standard interface and is readily available in many forms. The audio output subsystem 489 digitally outputs audio via the digital interface 494 (S/PDIF and/or Toslink) and outputs an analog form of audio via the stereo analog interface 495. Round trip delay analysis Of course, in order to achieve the benefits of the previous paragraph, the user's action of using the input device 421 and the effect of seeing the motion on the display device 420 should be no more than 7 〇·8 〇 milliseconds. This delay must account for all factors in the path from the input device 421 in the user premises 211 to the colocation service 21A and back to the user premises 211 to the display device 422 again. Figure 4b illustrates the various components and networks (signals must travel through them and above the components and the network are timetables that list the exemplary delays that can be expected in actual implementation. Note that Figure 4b is simplified so that Only important path routing is shown. Other routes for other features of the system are described below. A double-headed arrow (eg, arrow 453) indicates a round trip delay and a single = head (10) such as 'arrow 457' indicates a one-way delay 'and' ~" indicates approximate measurement. Should be the purpose of iH 'will exist in the shape of (four) unreachable shape, but in the freshman in the United States == machine connection:, these delays can be achieved in the following paragraphs, the idea, although to the Internet Weng/Μ-β The cellular wireless connection of the network will indeed work in the system that is not being implemented. The nested data system (such as EVD〇) incurs the delay that non-American bees can achieve the money shown. However, at 2 o'clock, the future honeycombs that cannot be used to deplete this (4) material can be used at 139852.doc •58· 200952495 ❹

From the user device 421 at the user location 421, the user-inducing input device 421 '$ sends a user control signal to the client that can be a stand-alone device such as a set-top box, or it can be at (iv) The software or hardware executed in the other device of the device is subscribed and packetized (in a real format) and the destination address is given to the packet to reach the colocation service 210. The packet will also contain information indicating which user the control signal is from. The control signal packet is then forwarded to the wan interface 442 via a firewall/router/NAT (Network Address Translation) device 443. The WAN interface 442 is an interface device provided by the user's lsp (Internet Service Provider) to the user premises 211. The WAN interface 442 can be a cable or DSL modem, a WiMax transceiver, a fiber optic transceiver, a cellular data interface, an Internet pr〇t〇e_i-over-powerline interface, or an Internet connection. Any of a number of other interfaces. Additionally, firewall/router/NAT device 443 (and (possibly) WAN interface 442) can be integrated into client 415. An example of this would be a mobile phone that includes software for implementing the functionality of the home or office client 415, and for wirelessly routing and connecting to the Internet via a certain standard (eg, 802.11g). member. The WAN interface 442 then routes the control signals to the "presence point" 441 WAN interface 442 for the user's Internet service provider (ISSP) herein to provide a WAN conveyor that is connected to the user premises 211. A facility between the Internet or a private network. The existence of a broken temple will vary depending on the nature of the Internet service provided. For DSL, it will usually be the central office of the telephone company located in the DSLAM. For the 139852.doc • 59- 200952495 cable modem, which usually It will be the head end of the cable multi-system operator (Ms〇). For a honeycomb system, it will typically be the control room associated with the honeycomb tower. But regardless of the nature of the point, it will then route the control signal packet to the Universal Internet. The control signal packets are then routed to the WAN interface 444 to the host escrow service 210 via the interface that is most likely to be the fiber optic transceiver interface. WAN 444 will then route the control signal packets to routing logic 409 (which can be implemented in many different ways, including Ethernet switches and routing servers), routing logic 4〇9 to estimate the user's address and control signals Routed to the correct server 4〇2 for a given user. The server 402 then treats the control signals as inputs to the game or application software executing on the server 4〇2 and uses the control signals to process the next frame of the game or application. Once the next frame is generated, the video and audio are output from the server 402 to the video compressor 4〇4. Video and audio can be output from the server 402 to the compressor 404 via various components. Compressor 404 can be built into server 402 first, so compression can be performed at the local end within server 402. Alternatively, it may be connected via a network to a network (either a private network between the server 402 and the video compressor 404, or a network via a shared network such as the SAN 403) (such as B Too network connection) Output video and/or audio in a packetized form. Alternatively, the video may be output from the server 402 via a video output connector, such as a DVI or VGA connector, and then captured by the video compressor 404. Alternatively, the audio can be output from the server 402 as digital audio (e.g., via a TOSLINK or S/PDIF connector) or analog audio, and the analog audio is digitized and encoded by the audio compression logic within the video compressor 404. 139852.doc -60- 200952495 Once the video compressor 404 has captured the video frame from the server 402 and the audio generated during the frame time, the video compressor will compress the video and audio using the techniques described below. Once the video and audio are compressed, it is packetized by a single address to send it back to the user's client 415 and routed to the WAN interface 444, the WAN interface 444 then • via the universal Internet The path 410 routes the video and audio packets, and the universal Internet 410 then routes the video and audio packets to the user's lsp presence point. 0 441 The presence point 441 routes the video and audio packets to the WAN interface 442 'WAN at the user's premises. Interface 442 routes the video and audio packets to firewall/router/NAT device 443 'Firewall/Router/NAT device 443 and then routes the video and audio packets to client 415. The client 415 decompresses the video and audio, and then displays the video on the display device 422 (or the built-in display device of the user terminal) and sends the audio to the display device 422 or to a separate amplifier/speaker or to the user terminal. Amplifier/speaker. 〇 In order for the user to feel that the entire process just described is perceptually free of lag, the round trip delay needs to be less than 70 milliseconds or 80 milliseconds. Some of the delay delays described in the return path are controlled by the colocation service 210 and/or the user, while other delay delays are not controlled by the colocation service 210 and/or the user. Nonetheless, based on the analysis and testing of a large number of real-world situations, the following are approximate measurements. The one-way transmission time 451 for transmitting control signals is usually less than! In milliseconds, routing 452 via the round-trip route of the place is typically completed within approximately 139852.doc • 61· 200952495 seconds using an easy-to-use 4 level firewall/router/nat switch on the Ethernet. User ISPs widely change their round trip delay 453, but in the case of DSL and cable modem providers, they typically see a round trip delay video on the universal Internet 410 between 1 and 25 milliseconds. How to be routed and whether there are any faults on the route (and these questions are discussed below) and greatly changed 'but usually the universal Internet • Provides the best route and the delay is largely due to light Determined by the speed of the fiber (given the distance to the destination). As discussed further below, 1000 has determined 1000 miles as the approximate farthest distance from which the colocation service 210 is expected to be placed away from the user premises 211. At 1 mile (2,000 miles back and forth), the actual transmission time for signals via the Internet is approximately 22 milliseconds. The WAN interface 444 to the colocation service 210 is typically a commercial grade fiber optic high speed interface with negligible latency. Therefore, the universal internet latency 454 is typically between 丨 milliseconds and 10 milliseconds. The one-way routing 455 delay via the colocation service 210 can be achieved in less than one millisecond. Server 4〇2 will typically calculate a new frame for the game or application in less than a frame time (which is 167 milliseconds at 60 fps), so 16 milliseconds is the reasonable maximum size that will be used. The delay is 456. In the preferred hardware implementation of the video compression and audio compression algorithms described herein, compression 457 can be accomplished in less than 1 millisecond. In the next best version, compression can take up to 6 milliseconds (of course, a less desirable version can take longer, but these implementations will affect the total latency of the round trip and will require other delays to be shorter (eg, subtractable) Small via the allowable distance of the universal Internet) to maintain the 70ms-80ms delay target). The round trip delay of Internet 454, user lsp 453 and user location. Route 452 has been considered, so the remaining video decompression 458 extension 139852.doc • 62- 200952495, the video decompression 458 delay depends on the video solution Whether the compression 458 is implemented in a dedicated hardware or in a software implemented on a client device 415 (such as a mobile device), the size of the visual display and the performance of the decompressing CPU are changed. Generally, decompression 458 takes 1 millisecond and Time between 8 milliseconds. • Therefore, it is possible to delay the round-trip travel of the worst-case conditions that the user of the system shown in Figure 4a can expect to experience by delaying all the worst-case delays seen in practice. It is: 1 + 1+25+22+1 + 16+6+8=8〇春*卜Μ 'In fact' in practice (with the explanation of misunderstanding discussed below) 'This is roughly the use of Figure 4a The prototype version of the system shown in the show (the use of off-the-shelf Windows PCs in the US as a client device and home DSL and cable modem connections) sees the round trip delay. Of course, situations that are better than the worst can lead to much shorter delays, but you can't rely on them to develop widely used business services. In order to achieve the delays listed in the figure via the universal internet, the video compressor 4〇4 and the video decompressor 412 (from Figure φ 4a) in the client 415 are required to generate a packet stream with very specific characteristics. So that the sequence of packets generated via the entire path from the colocation service 210 to the display device 422 is not subject to delay or excessive packet loss, and in detail, consistently falls over the Internet connection via the user (via WA) N interface 4 4 2 and firewall / router / NAT 433) can be used within the constraints of the user's bandwidth. In addition, the video compressor must generate a sufficiently robust packet stream to withstand the inevitable packet loss and packet reordering that occurs in normal Internet and network transmissions. Low-Temperature Video Compression 139852.doc 63. 200952495 To accomplish the above objectives, an embodiment employs a new video compression method that reduces the delay and peak bandwidth requirements for video transmission. Prior to describing such embodiments, an analysis of current video compression techniques will be provided with respect to Figures 5 and 6a through 6b. Of course, if the user has sufficient bandwidth to handle the data rate required by such techniques, then these techniques can be used according to the basic principles. Note that this does not address audio compression, but rather states that audio compression is implemented concurrently and synchronously with video compression. Prior art audio compression techniques that meet the requirements for this system exist. Figure 5 illustrates a particular prior art for compressing video in which each individual video frame 501-503 is compressed by compression logic 520 using a particular compression algorithm to produce a series of compressed frames 511-513. One embodiment of this technique is "Motion JPEG" where each frame is compressed based on a discrete cosine transform (DCT) according to a joint image expert group (jpeg) compression algorithm. A variety of different types of compression algorithms can be used, however, these basic principles are still adhered to (e. g. 'wavelet based compression algorithms such as JpEG_❹ 2000). One problem with this type of compression is that it reduces the data rate of each frame' but it does not utilize the similarity between successive frames to reduce the data rate of the total video stream. For example, as illustrated in Figure 5, assume a frame rate of 64〇χ48〇χ24 bits/pixel=640*480*24/8/1024=900 kbytes/frame (KB/frame) , for a given quality image, the motion JpEG may only compress the stream by 1/10, resulting in a stream of 90 KB/frame. At 60 frames per second, this would require 90 KB * 8 bits * 60 frames / sec = 42 2 Mbps channel bandwidth, which is now available for almost all home Internet connections in the United States today 139852.doc 200952495 It will be extremely high bandwidth and too high bandwidth for many office internet connections. In fact, assuming that it requires a constant data stream in such a high bandwidth, and it will only serve one user, it will consume 1 Mbps 6 too wide LAN LAN bandwidth even in an office LAN environment. A large percentage and a heavy Ethernet switch that supports the LAN. Therefore, compression for motion video is inefficient when compared to other compression techniques, such as those described below. In addition, a single frame compression algorithm (such as JpEG& 1^〇} 2〇〇〇) using a lossy pressure collapse algorithm produces compression artifacts that may be unobtrusive in still images (eg, in a scene) The artifacts in the dense leaves may not appear as artifacts, because the eyes do not know exactly how the dense leaves should be presented. ^ However, once the scene is in motion, the artifacts may protrude because the eye detects the self. The artifacts that change from frame to frame, although the artifacts are in the area of the scene where artifacts may not be noticeable in still images. This results in the perception of "background noise" in the sequence of frames. The appearance and edge analogy of the "background noise" is similar to the "snow" noise visible during TV reception. Of course, this type of compression can still be used in the specific embodiments described herein, but in general, in order to avoid background noise in the scene, a high data rate (ie, a low compression ratio) is required for a given perceived quality. ). Other types of compression (such as H.264, or Windows Media VC9, MPEG2, and MPEG4) are more efficient in compressing video streams because they exploit the similarities between successive frames. These techniques all rely on the same general techniques for compressing video. Thus, although the H.264 standard will be described, the same general principles apply to various other compression algorithms. A large number of H.264 compressions 139852.doc -65- 200952495 Restrictors and decompressors are available, including the x264 open source software library for compression Η·264 and the FFmpeg open source software library for decompressing H.264. 6a and 6b illustrate an exemplary prior art compression technique in which a series of uncompressed video frames 5〇1_503, 559_561 are compressed by compression logic 620 into a series of 图ι frames 611, 671; "P-frames" 612 -613; and. "B frame" 670. The vertical axis in Figure 6a generally represents the resulting size of each of the encoded frames (although the frames are not drawn to scale). As described above, those skilled in the art are well aware of the video writing codes using the I frame, the B frame, and the P frame. Briefly, block I 611 is a DCT-based compression of frame 501 that is completely uncompressed (similar to a compressed JPEG image as described above). The size of P-frames 612-613 is typically significantly smaller than the size of I-frame 611 'because it utilizes the material of the previous I-frame or p-frame; that is, it contains an indication between the previous I-frame or P-frame Changed information. The B-frame 670 is similar to the p-frame except that the B-frame uses the frame of the subsequent reference frame and (possibly) the frame of the previous reference frame. φ For the following discussion, 'will assume that the required frame rate is 60 frames per second, each I frame is about 160 Kb, the average P frame and B frame are 16 Kb and a new I is generated every second. Frame, under this set of parameters, the average data rate will be: 160 Kb + 16 Kb * 59 = ll Mbps. This data rate falls appropriately within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid background sfL problems from only intra-frame coding. Because the difference between the P-frame and the B-frame tracking frame, the compression artifact tends to appear and disappear from the frame to the frame. To reduce the background noise problem described above. 139852.doc -66 - 200952495 One of the problems with compression of the above type is that although the average data rate is relatively low (e.g., '1.1 Mbps), a single I-frame may take several frame times to transmit. For example, using prior art, a 2.2 Mbps network connection (eg, a DSL or cable modem with a 2.2 Mbps peak from the maximum available data rate 302 of Figure 3a) would typically be sufficient to stream video at 丨1 Mbps ' One 160 Kbps I frame per 60 frames. This will be done by having the decompressor put a 1 second video into the queue before decompressing the video. Within j seconds, 1.1 Mb of data will be transmitted 'which will be easily accommodated by the 2.2 Mbps maximum available data rate' even if it is assumed that the available data rate may periodically drop by as much as 50%. Unfortunately, this prior art method would result in a 1 second delay in video due to the 1 second video buffer at the receiver. This delay is sufficient for many prior art applications (e.g., playback of linear video), but it is an extremely long delay for fast motion video games that cannot tolerate delays greater than 70 milliseconds _8 milliseconds. If an attempt is made to eliminate the 1 second video buffer, it will still not result in a sufficient delay reduction for fast motion video games. For example, as previously described, the use of a B-frame will require receipt of all b-frames and I-frames before the I-frame. If it is assumed that approximately 59 non-work frames are split between the P frame and the B frame, there will be at least 29 B frames and the I frame received before any B frames can be displayed. Therefore, regardless of the available bandwidth of the channel, a delay of 1/6 sec duration for each of the 29+1=30 frames, or a delay of 5 〇〇 milliseconds is required. Obviously, his time is extremely long. Therefore, another method would be to eliminate the B frame and only use the I frame and the P frame. (One of the consequences is that for a given quality level, the data rate will increase by I39852.doc •67- 200952495. But for consistency in this example, continue to assume that each i frame is 160 in size. Kb and the average P frame size is 16 Kb, and therefore the data rate is still 1.1 Mbps). This method eliminates the unavoidable delay introduced by the frame because the decoding of each P-frame depends only on the previously received frame. The problem with this approach is that the I frame is much larger than the average p frame so that on the low frequency wide channel (as is typical in most homes and many offices), the transmission of the I frame adds substantial delay. This is illustrated in Figure 6b. The video stream data rate 624 is lower than the available maximum data rate 621 (except for the I frame), where the peak data rate 623 required for the I frame is well above the maximum available data rate 622 (and even exceeds the nominal maximum data rate 621). ^ The data rate required for the P frame is less than the maximum available data rate. Even if the peak maximum data rate peak at 2.2 Mbps is steadily maintained at its 2 2 Mbps peak rate, it will take 16 〇Kb/2.2 Mb=71 ms to transmit the I frame, and if the maximum data rate 622 is available, it drops 50°/ . (1.1 Mbps), it will take 142 milliseconds to transmit the I frame. Therefore, the delay in the transmission 1 frame will fall somewhere between 71 milliseconds and 142 milliseconds. This delay is added to the delay identified in the map (the delays are up to 70 milliseconds in the worst case), so this will result from the moment the user actuates the input device 421 until the image is presented on the display device 422. The total round trip delay of 141-222 milliseconds is extremely high. And if the maximum available data rate drops below 2.2 Mbps, the delay will increase further. It is also noted that ‘blocking" by ISPs that exceed the peak data rate 623 of the available data rate 622 typically has serious consequences. Devices in different ISPs will behave differently, but when the data rate is much higher than the available data rate 622, 139852.doc .68· 200952495 = the following behavior is quite equal in the mess and the data ISP = (4) by The packet is queued to delay the packet (introduction delay), some or all packets, (C) the connection history may be disabled because the ISP is worried that it is a sub-sound 4

hit). Therefore, transmitting packet streams at full data rate (having characteristics such as those in Figure 6b, which are not their characteristics) is not a viable option. The peak 623 can be queued at the host agent service 21G and sent at a bet rate lower than the maximum available data rate, thereby accepting the delay of the m-segment. J "Additionally" the video stream data rate sequence shown in Figure 6b appears to be a very "tamed" (Ume) 9 stream data rate sequence, and will be expected due to the shrinking of video from the video sequence. Data rate sequence, the video sequence does not change very A and has very little motion (for example, as will be common in X video conference calls, in video conference calls, the camera is in a fixed position and has very little motion And the objects in the scene (for example, the seated person's 5 words) show less movement). The video stream data rate sequence 634 shown in Figure 6c is a typical sequence that is expected to be visible from video with much more motion, such as may be produced in a movie or video game or in an application software. Note that in addition to the peak 633 of the worksite, there are also P-frame peaks (such as 635 and 636) that are quite large and in many cases exceed the maximum available data rate. Although the peaks of these P-frames are not quite as large as the I-frame peaks, they are still so large that they cannot be carried by the channel at full data rate' and like the I-frame peaks, the P-frame peaks must be transmitted slowly (borrowed) This increases the delay). \39S52.doc -69- 200952495 On a South Broadband channel (for example, 100 Mbps LAN, or a high-bandwidth 1 (9) Mbps private connection), the network will be able to tolerate large peaks such as 1 frame peak or p frame peak 636 However, in principle, low latency can be maintained. However, these networks are often shared among many users (for example, in an office environment), and the "peak" data will affect the performance of the LAN, especially • The network traffic is routed to a private shared connection ( For example, from a remote data center to an office. First, remember that this example is an example of a relatively low resolution video stream of 640x480 pixels at 6〇 fps. The 1920x1 080 HDTV stream at 6〇 fps is easily handled by modern computers and monitors, and the 2560x1440 resolution display at 60 fps is increasingly available (for example, Apple's 30" display). With h_264 compression, a high motion video sequence of 1920X1080 at 60 fps may require 4 5 Mbps to achieve a reasonable quality level. If the peak of the frame is assumed to be 1〇 of the nominal data rate, it will produce a 45 Mbps peak and a small but still large p-frame peak. If several users are receiving video streams on the same 100 Mbps network (for example, a private network connection between the office and the data center), it is easy to see how the peak of the video stream from several users can happen to The bandwidth of the network is flooded and may overwhelm the bandwidth of the backplane of the switch supporting the user on the network. Even in the case of ultra-high-speed Ethernet, if enough users have enough peaks for simultaneous alignment, they can flood the network or network switch. "In addition, once the 2560x1440 resolution video becomes more common, the average The video stream data rate may be 9·5 Mbps, which may result in a 95 Mbps peak data rate. Needless to say, the 100 Mbps connection between the data center and the office (which is now an extraordinarily fast connection) will be completely defeated by the peak traffic of 139852.doc -70- 200952495 from a single user. Therefore, even if LAN and private network connections are more tolerant to peak-streaming video, streaming video with high peaks is not required and may require special planning and adaptation of the office's Ting department. For standard linear video applications, these issues are not a problem 'because the data rate is "smoothed" at the transmission point and the data for each frame is below the maximum available data rate 622, and in the decompressed frame, Before the p @ frame and B frame sequence, the buffer in the client side stores the frame, the p frame and the B frame sequence. Therefore, the data rate on the network remains close to the average data rate of the video stream. Unfortunately, this introduction delay, even if the B-frame is not used, is unacceptable for low-latency applications such as video games and applications that require fast response times. A prior art solution for mitigating video streams with high peaks is to use techniques often referred to as "constant bit rate" (CBr) encoding. Although the term CBR appears to imply that all frames are compressed to have the same bit rate (ie, 'size'), it is often referred to as a compression paradigm, in which a certain number of frames are allowed to be crossed (in In the case of our person, the maximum bit rate of a frame). For example, in the case of Figure 6, if a CBR constraint is applied to the code that limits the bit rate to, for example, 70% of the nominal maximum data rate 621, the compression algorithm will limit the frames. Each of the frames is compressed such that any frame that would normally be compressed using more than 70% of the nominal maximum data rate 621 will be compressed with fewer bits. The result of this is that it will normally require more bits to maintain the "extremely lacking" bit of the frame of a given quality level and the image quality of these frames will be better than the 139852.doc •71 · 200952495 The image quality of other frames rated for more than 70% of the maximum data rate of 621 is poor. This method produces acceptable results for a particular type of ablation (whose towel (4) is expected to have less motion or scene change and (b) the user can accept periodic grades. A good example of a suitable CBR application is a video conference call, because there are fewer peaks, and in the case of temporary quality degradation (such as 'if the camera is glanced' & causing significant scene motion and large peaks, it may be during the glance There are not enough bits for high quality 影像 image compression 'which will result in a degraded image f), which is acceptable to a number of users. Unfortunately, CBR is not well suited for many applications with high complexity scenes or a large amount of motion and/or the need to align the f-level. 1 The low latency compression logic 404 used in one embodiment uses a number of different techniques to solve the problem of streaming low latency compressed video while maintaining high quality. H low latency compression logic 4G4 is only generated! The frame and the p-frame, thereby alleviating the need to wait for several frame times to decode each call frame. In addition, as illustrated in Figure 7a, in one embodiment, low latency pressure: logic 404 divides each uncompressed frame 7gi_7 into a series of "image blocks" and individually encodes each image block as Work frame or p-frame: In this paper, the I frame and P frame of the group are referred to as "R round frame 711_770. In the specific example shown in Figure 7 &, each will be uncompressed. The frame is subdivided into 16 image blocks u of the 4M matrix. The base principle is not limited to any specific sub-dividing mechanism. In the column, the low-latency compression logic 4°4 divides the video frame into Xu image blocks, and will be from each One of the frames of the image block (ie, 139852.doc •71·200952495 compression) is the i frame (ie, 'the image block is compressed as if it were 1/16 of the size of the full image. Frame, and the compression for this "mini" frame is I frame compression) and encodes the remaining image blocks into p frames (ie, for each "mini" / 16 frame compression Compressed for the p frame). Compressed • Image blocks that are 1 frame and image blocks that are compressed into P frames will be referred to as *1 image block and p image block, respectively. The image block to be encoded as an I image block is changed with each successive video frame. Therefore, in a given frame time 仅, only one of the image blocks in the video frame is an I image block, and the remaining ones of the image blocks are P image blocks. For example, in FIG. 7a, the image block of the uncompressed frame 701 is encoded as 1 image block 1 and the remaining 1 - 15 image blocks are compiled into P image blocks (P 1 to P ! 5) to generate R frame 711. In the next uncompressed video frame 7〇2, the image block 1 of the uncompressed frame 701 is encoded as an image block and the remaining image blocks 2 and 2 to 15 are encoded as P image blocks ( P〇' and P2 to Pi5) to generate R block 712. Therefore, the I image block and the P image block used for the image block are gradually _ time-interleaved on successive frames. The process continues until an R image block 770 is generated (the last image block in the matrix is encoded as an I image block (i.e., '115)). The process then resumes' to produce another R frame such as frame 711 (i.e., for the image block ’ 'encoded I image block). Although not illustrated in the figures, in one embodiment, the first R frame of the video sequence of the R frame contains only j image blocks (ie, such that subsequent P frames have reference image data (from which Computational motion)) or 'In one embodiment, the start sequence uses the same I picture block pattern as normal, but does not include p picture blocks for their image blocks that have not been encoded with the I picture block. In other words, the specific image block is not encoded with any data before the arrival of the first image i39852.doc • 73· 200952495, thereby avoiding the start-up peak in the video stream data rate 934 in Figure 9a, which is further detailed below. Description. In addition, as described below, various sizes and shapes can be used for the image blocks while still complying with these basic principles. • The video decompression logic 412 executing on the client terminal 41 5 decompresses each image. The image block appears as a separate video sequence of the small I frame and the P frame, and then each image block is reproduced to the drive display. The frame buffer of device 422. For example, image block 0 of the video image is decompressed and reproduced using 1〇 and P〇 from R frames 711 to 770. Similarly, the image block 1' and the like are reconstructed using the work 1 and Pi from R frame 7 to 77. As described above, the decompression of the I frame and the P frame is well known in the art, and the decompression of the image block and the p image block can be performed by causing multiple execution entities of the video decompressor to be at the user end. Execute on 415 to complete. Although the multiplication process appears to increase the burden on the UE, it does not actually increase the computational burden on the client 415 because the image block itself is proportionally smaller (relative to the number of additional processing) Therefore, the number of pixels displayed is the same, as if there is a processing and using a conventional full-size J-frame and p-frame. This R-frame technique significantly mitigates the peak bandwidth typically associated with the 5 frame (as illustrated in Figures 6b and 6c), since any given frame is primarily

Can be roughly 1 Kb. The final result is about 1 〇 Kb + 15M - P frame for each Kb = 25 Kb R map 139852.doc .74 · 200952495 So the parent 60 frame sequence will be 25 Kb * 60 = 1.5 Mbps. Therefore, at 6 frames/sec, this would require a channel capable of maintaining a bandwidth of i.5 Mbps which has a much lower peak due to the dispersion of the I picture block through the 6 frame interval. Note that in the previous example, the average data rate was hl Mbps in the case of the same assumed data rate with the frame and the p-frame. This is because in the previous example, only a new 1 frame is introduced every 60 frame time, and in this example, the 6 image blocks constituting the I frame are cycled in the 6 frame time, and Therefore, an equalization of one frame is introduced every 16 frame times, resulting in a slightly higher average data rate. Despite this, in practice, the introduction of more frequent I frames does not linearly increase the data rate. This is due to the fact that the P-frame (or p-image block) mainly encodes the difference from the previous frame to the next frame. Therefore, if the previous frame is similar to the next frame, the P frame will be very small. If the previous frame is quite different from the next frame, the P frame will be very large. However, because the p-frame is largely derived from the previous frame, rather than being derived from the actual frame, the resulting coded frame may contain more errors than I frames with a sufficient number of bits (for example, Visual artifacts). In addition, when a P frame follows another P frame, error accumulation can occur (when there is a long P frame sequence, it gets worse now, the sophisticated video compressor will detect the quality of the image in a sequence The fact that the p-frame is demoted afterwards, and if necessary, it assigns more bits to subsequent P-frames to improve quality, or if it is the most efficient action process, replaces the P-frame with a 1 frame. When using a long P-frame sequence (for example, 59 p-frames, as in the previous example above), the specific scene has a large number of complex 139852.doc -75 - 200952495 noise and / or motion 'usually, More bits are needed for the p-frame (because it becomes farther away from the I-frame or P-frame from the relative view point, the P-frame that closely follows the J-frame tends to require a distance of 1 The farther P frame has fewer bits. Therefore, in the example shown in Figure 7a, the p-free frame is 15 frames away from the 1 frame. • Farther (before the 1 frame) In the previous example, the P frame can be separated from the I frame by 59. Therefore, in the case of the more frequent frame, p The frame is smaller than @. Of course, the exact relative size will vary based on the nature of the video stream, but in the example of Figure 7 & if the image block is 10 Kb, the size of the P block can be only 0.75 kb ' Resulting in 1〇Kb+15*0.75 Kb=21.25 Kb, or at 60 frames per second, the data rate will be 21.25 Kb*60 = l.3 Mbps, or with an i frame at 1.1 Mbps followed by The data rate of the 59 p-frame streams is about 16% higher. Once again, the relative results between the two methods for video compression will vary depending on the video sequence, but usually, we have discovered by experience that To set the quality level, it takes about 20% more bits to use the R frame than to use the I/p frame sequence. However, of course, the R frame sharply reduces the peak value. This makes the video sequence much smaller than I/p. The delay of the frame sequence is available. The R frame is configured in a number of different ways depending on the nature of the video sequence, the reliability of the channel and the available data rate. In an alternative embodiment, a different from 16 is used in the 4χ4 configuration. Number of image blocks. For example, 2 image blocks can be used in a 2χ1 or 1x2 configuration 4 image blocks can be used in 2x2, 4x1 or lx4 configurations, 6 image blocks can be used in 3x2, 2x3, 6x1 or 1x6 configurations or can be 4χ2 (as shown in Figure 7b), 2M, 8x1 or 8 image blocks are used in the 1x8 configuration. Note that the image block does not need to be square, and the video frame 139852.doc •76- 200952495 does not have to be square or even rectangular. The image block can be decomposed into the best fit. Regardless of the shape of the video stream and the application. In another embodiment, the loop of the image block and the P image block is not locked to the number of image blocks. For example, in the 8 image block 4x2 configuration, 16 cycle phases can still be used as shown in Fig. 7b. Sequential uncompressed frames • 721, 722, and 723 are each divided into 8 image blocks 0_7, and each image block is individually compressed. From R frame 731, only the image block is compressed into an image block, and the remaining image blocks are compressed into P blocks. For subsequent R frame 732, all eight image blocks are compressed into p image blocks, and then for subsequent R frame 733, the image blocks are compressed and the other image blocks are compressed into P image blocks. In addition, the 16 frames continue to be sorted so that only one image block is generated every other frame, so during the 15th frame time (not shown in FIG. 7b) and during the 16th frame time The last I image block for image block 7 is generated (using all p image blocks to compress R frame 780). Then, the sequence starts again by compressing the image block by the image block and compressing the other 影像 image blocks into P image blocks. As in the previous embodiment, the first frame of the entire video sequence will typically be a J-picture block to provide a reference for the P-picture block forward from the other point. ! The loop of image blocks and p-image blocks does not even need to be an even multiple of the number of image blocks. For example, in the case of 8 image blocks, another one is used; [before the image block, each frame having a J image block can be followed by 2 frames of all P image blocks] In one embodiment, if, for example, a particular area of the screen is known to have more motion (requiring more frequent I image blocks) and other areas are more static (eg, showing game scores) (requires less frequent I) Image block), and other image blocks can be sorted together with the image block more often than 139852.doc •77·200952495. In addition, each frame is illustrated in Figures 7a-7b. single! The image block, but η, u_ , encodes multiple I image blocks in the frame (depending on the bandwidth of the transmission channel). Conversely, a particular frame or frame sequence can be transmitted without an I-picture block (i.e., only a p-picture block). Then. The reason why the method works properly is that although the image block that does not have a spread across each single-round frame seems to cause a large peak, the behavior of the reference is not so simple because each image block is Separate compression from other image blocks' so that when the image block becomes smaller, the encoding of each image block can become less effective because the compressor of a given image block cannot utilize similar image features and similar motion from other image blocks. . Therefore, dividing the screen into 影像6 image blocks will generally result in less efficient coding than dividing the screen into 8 image blocks. However, if the screen is divided into 8 image blocks and it causes the introduction of a full I frame every 8 frames (rather than every 6 frames), it results in a much higher overall data. rate. _ Therefore, by introducing a full j frame every 16 frames instead of every 8 frames, the total data rate is reduced. Moreover, by using eight larger image blocks (rather than 16 smaller image blocks), the total data rate is reduced, which also reduces the peak of the data caused by the larger image blocks to some extent. In another embodiment, the low latency video compression logic 404 of FIGS. 7a and 7b is configured by pre-configuration or based on image quality in each image block by virtue of known characteristics of the video sequence to be compressed. The ongoing analysis automatically controls the allocation of bits to each of the image blocks in the R frame. For example, in some competitive video games, the player's car (which is relatively motionless in scene 139852.doc -78· 200952495) occupies most of the lower half of the screen before the top half of the screen. It is completely filled with roads, buildings and landscapes that are approaching, which is almost always in motion. If compression logic 404 assigns an equal number of bits to each image block, then the image block (image block 4-7) on the lower half of the screen in uncompressed frame 72i in Figure 7b will typically It is compressed at a higher quality than the image block (image block 〇-3) in the upper half of the screen in the uncompressed frame 721 in Fig. 7b. If it is known that this particular scenario of the particular game or game has such characteristics, the operator of the colocation service 210 can configure the compression logic 404 to allocate more bits to the image block at the top of the screen (and to assign Alternatively, compression logic 404 may estimate the compression quality of the image block after compressing the frame (using one or more of a number of compression quality metrics such as peak signal to noise ratio (PSNR) compared to the bit of the image block at the bottom of the screen. ))) and if it is determined that a particular image block consistently produces better quality results over a particular time window, it gradually assigns more bits to the image block that produces lower quality results until the various image blocks are similar In an alternative embodiment, the compressor logic 404 assigns a bit to achieve a higher quality in a particular image block or image block group. For example, the view is at the center of the glory than at the edge. High quality. In one embodiment, 'in order to improve the resolution of a particular region of the video stream, the video compression logic 404 uses a smaller image block to encode the video stream with relatively more. The complexity of the scene and/or the area of motion (compared to the area of the video stream that has relatively little scene complexity and/or motion). For example, as illustrated in Figure 8 'in the -R frame 811 ( It is possible to use smaller image blocks around the moving characters 8〇5 in one of the areas with the same image block size 139852.doc -79·200952495 small one series R frame (not shown). Then, when When the character 8〇5 moves to a new area of the image, a smaller image block is used around the new area in the other R frame 812 as described. As described above, various sizes and shapes can be used as the “image block”. While still adhering to these basic principles. Although the cyclic I / P image blocks described above substantially reduce the peak in the data rate of the video stream, they do not completely eliminate the peaks, especially in fast changes or highly complex In the case of video images (such as in movies, video games, and an application software). For example, during a sudden scene transition, a complex round frame may be followed by another complex frame that is completely different. Even Several I shadows Blocks can be only a few frames away from the scene transition, which is not helpful in this case, because the material of the new frame is independent of the previous I image block. In this case (and even if not everything changes, In other cases where a large number of images are also changed) 'Video Compressor 4〇4 will determine that many, if not all, P-picture blocks will be more efficiently coded for image blocks and result in very high data rates for the frame. The peak of the big. As previously discussed, it is only for most consumer-grade Internet connections (and many office connections), it can only be "blocked" beyond the maximum data rate available as shown at 622 in Figure 6c and Information for the rated maximum data rate of 621. Note that the rated maximum data rate of 62 丨 (for example, "6 Mbps DSL") is essentially a sales figure for users considering purchasing an internet connection' but usually does not guarantee performance levels. . For the purposes of this application, it is irrelevant because we only focus on the maximum data rate 622 available when streaming video over a connection. Thus, in Figures 9a and 9c, when the solution to the peak problem is described in 139852.doc -80-200952495, the nominal maximum data rate is omitted from the graph and only the maximum data rate 922 is shown. The video streaming data rate must not exceed the maximum available data rate of 922. To solve this problem, the first thing the video compressor 404 does is to determine the peak data rate 941, which is the data rate at which the channel can be processed steadily. This rate can be determined by a number of techniques. One of the technologies will be more

❹ The increased data rate test stream is gradually sent from the colocation service 21〇 to the client 415 (in Figures 4a and 4b), and the UE is provided with feedback on the level of packet loss and delay to the colocation. service. When the packet is lost and/or the delay begins to show a sharp increase, it is indicative of the maximum available data rate 922. Thereafter, the colocation service 21 can gradually reduce the data rate of the test stream until the client 415 reports that the test stream has been received within a reasonable period of time (the packet loss and the acceptable level of the delay are near minimum). This determines the peak maximum data rate 941, which will then be used as the peak data rate for streaming video. Over time, the peak data rate 9 fluctuates (eg, if another in the family - the user begins to seriously use the internet connection), and the client 415 will need to constantly monitor the peak data rate 941 to see packet loss. Or whether the delay is increased (indicating that the maximum data rate 922 is available to fall below the peak f rate 941 determined by ^), and if so 'the peak data rate 94 is similarly, if the user 415 finds over time Packet loss and delay remain at the optimal level, then it can request video compression H to slowly increase the f rate to see if the maximum available data rate is increased (for example, if the family is severely connected to the Internet connection), And wait again until the packet goes to 139852.doc 200952495 and/or the higher delay indication has exceeded the maximum data rate 922 available, and the lower level for the peak data rate 941 can be found again, but the lower level may Above the level before the increased data rate of the test. Thus, the peak data rate 941' can be found by using this technique (and other techniques like it) and periodically adjusted as needed. The peak data rate 941 determines the maximum data rate that can be used by the video compressor 4〇4 to stream the video to the user. The logic for determining the peak data rate can be implemented at the user's location 211 and/or on the colocation service 21'. At the user premises 211, the client device 415 performs computations to determine the peak data rate and transmits this information back to the colocation service 21; at the colocation service 210, the courier service server 4〇 2 Performing a calculation to determine the peak data rate based on statistics received from the client 415 (eg, peak loss, delay, maximum data rate, etc.). Figure 9a shows an example video stream data rate 934 with substantial scene complexity and/or motion, using the cyclic I/p image block compression techniques previously described and illustrated in Figures 7a, 8 and 8 produce. The video buffer 404 is configured to output the compressed video at an average data rate lower than the peak data rate 941, and note that most of the time, the video stream data rate remains below the peak data rate 941. A comparison of the data rate 934 with the video stream data rate 634 shown in Figure & which is generated using I/p/B or I/p frames shows that the cyclic P/P image block compression produces much smoother data. rate. However, at frame 2 times peak 952 (which is close to 2 times peak data rate 942) and frame 4 times peak 954 (which is close to 4 times peak data rate 944), the data rate still exceeds the peak-beat rate 941 ' unacceptable. In practice, even for 139852.doc -82- 200952495 high-motion video from fast-changing video games, peaks above the peak data rate 941 appear in frames less than 2〇/〇, more than 2 times the peak data rate The peak of 942 is rare, and the peak of more than 3 times the peak data rate of 943 is rare. However, when it does occur (for example, during a scene transition), the data rate required must produce a good quality video. Image. One way to solve this problem is to simply configure the video compressor 404 to output its maximum data rate to the peak data rate 941. Unfortunately, the resulting video output quality during the peak frame is poor because the compression algorithm is extremely lacking. What is caused is that compression artifacts occur when there is a sudden transition or rapid motion, and in time, the user begins to realize that artifacts always appear suddenly when there is a sudden change or rapid motion, and it can become quite annoying. Although the human visual system is quite sensitive to visual artifacts that occur during sudden changes or rapid motion, it is not very sensitive to the reduction in the detection of the frame rate in such situations. In fact, when these sudden changes occur, it seems that the human visual system is focused on tracking these changes, and if the frame rate temporarily drops from 60 fps to 30 fps and then immediately returns to 6〇fps, then: The vision system won't notice. In addition, in the case of very sharp transitions (such as sudden scene changes), if the frame rate drops to 2 〇 fps or even 15 fps and then immediately returns to 60 fps, the human visual system will not notice. As long as the frame rate reduction occurs only occasionally, it seems to the human observer that the video system is continuously executed at 60 fps. The use of the human visual system is exploited by the technique illustrated in Figure 9b. Server 402 (from Figures 4a and 4b) produces an uncompressed video output stream at a stable frame rate (in one embodiment, at 60 fps). The timetable shows the output of each frame 961_97 per 1/60 second. Starting from frame 961, each uncompressed video frame is output to the low latency video compressor. 404, the low latency video compressor 404 compresses the frame for less than one frame time. The compressed frame 丨 981 in the first frame. The data produced for the compressed frame 丨 981 may be larger or smaller depending on a number of factors as previously described. If the data is small enough to transmit it to the client 415 at a frame time (1 / 60 seconds) or less than one frame time, then the transmission time (xmit time) is 99 丨 (indication) It is transmitted during the length of the arrow of the duration of the transmission time. In the next frame time, the server 402 generates an uncompressed frame 2 962, which is compressed into a compressed frame 2 982, and at a peak data rate 941 during a transmission time 992 of less than one frame time. It is transmitted to the client 415. Next, in the next frame time, the server 4〇2 generates an uncompressed frame 963. When the uncompressed frame 3 963 is compressed by the video compressor 404, the resulting compressed frame 3 983 is more data than can be transmitted at a peak data rate 941 in a frame time. Therefore, it is transmitted during the transmission time (2 times peak) 993, which occupies one of the frame time and the next frame time. Now, during the next frame time, the server 402 generates another uncompressed frame 4 964 and outputs it to the video compressor 404, but the data is ignored and illustrated by 974. This is because video compressor 404 is configured to ignore other uncompressed video frames that arrive when they are still transmitting previously compressed frames. Of course, the video decompressor of the client's stomach 84 139 852.doc 200952495 415 will fail to receive frame 4, but it simply continues to display frame 3 on display device 422 for 2 frame times (ie, Temporarily reduce the frame rate from 60 fpS to 3〇fp〇. For the next frame 5, the server 4〇2 outputs the uncompressed frame $.965, which is compressed into a compressed frame 5 985 And transmitting it in frame 1 during transmission time 995. The video decompressor of client 415 decompresses frame 5 and displays it on display device 422. Then the server outputs uncompressed At block 6 966, video compressor 404 compresses it into compressed frame 6 986, but the resulting data is very large. The compressed frame is transmitted at peak data rate 941 during transmission time peak 996. But it takes almost 4 frames to transfer the frame. During the next three frame times, the video compressor 404 ignores the three frames from the server 4〇2, and the decompressor of the client 41 5 stably holds the frame 6 on the display device 422 for a duration. 4 frame times (ie, temporarily reduce the frame rate from 6〇fps to 15 fps). Finally, the server 4〇2 outputs the frame 1〇97〇, which is compressed by the video compressor 404 into the compressed frame 1〇987, and transmitted during the transmission time 997, and the user terminal 415 The decompressor decompresses the frame ίο and displays it on the display device 422 and restarts the video again with 6 prints 3. / idea 'Although the video compressor 4〇4 discards the video frame from the video stream generated by the server 4〇2' but it does not discard the audio material (regardless of the form of the audio system), and discards the video When the frame is compressed, the video compressor 4〇4 continues to compress the audio data and transmits it to the client 415. The client 415 continues to decode the audio data and provides the audio to the user for playback of the sound 139852.doc •85- 200952495 News of '·,,》 «What device. Therefore, during the period of discarding the frame, the audio continues without subtracting the reduced, compressed video phase & the compressed audio consumes a relatively small percentage of the bandwidth, and thus does not have a significant impact on the total data rate. . Although not illustrated in any of the data rate maps, U in the peak data rate 941 is the data rate capacity reserved for the compressed audio stream. Select the example just described in the ® 9b towel to illustrate how the frame rate drops during the (iv) rate peak, but not illustrated when using the previously described cyclic p/p image block technique, 'the data rate peaks and Discarding frames that occur are rare 'even during high scene complexity/high action sequences (such as those found in video games, movies, and some-application software). Therefore, the reduced frame rate is rare and temporary, and the human visual system does not measure them. If the frame rate reduction mechanism just described is applied to the video stream data rate described in 09a, then Figure 9. The resulting video stream data rate is described. In this example, the '2 times peak 952 has been reduced to the flattened 2 times peak 953, and the 4 times peak 955 has been reduced to the flattened * times the peak milk, and the entire video stream data rate 934 彳 car surname μ + Year 934 remains at or below the peak data rate of 941. ^ Therefore, using the techniques described above, high motion video streams can be transmitted with low latency via a common Internet and consumer Internet connection. In addition, in an office environment on a LAN (for example, (10) Ethernet or Nen line) or private network (for example, = Mbps connection between data center and office), it can be transmitted without peaks. 139852.doc -86 · 200952495 Motion video streaming so that multiple users (for example, 1920 χ 1080 at 6 〇 fps at 45 Mbps) can use LAN or shared private data connections without flooding the overlapping peaks Or network switch backplane. Data Rate Tuning In one embodiment, the colocation service 21 initially evaluates the availability of the channel. The maximum data rate 622 and the delay determine the appropriate data rate for the video stream and then dynamically adjust the data rate in response thereto. To adjust the data rate, the colocation service 210 can, for example, modify the image resolution and/or the number of frames per second of the video stream to be sent to the client 415. In addition, the colocation service can adjust the quality level of the compressed video. When changing the resolution of the video stream (e.g., from 128 〇 x 720 resolution to 640 x 360), video decompression logic 412 on client 41 5 can scale the image to maintain the same image size on the display glory. In one embodiment, the colocation service 210 suspends the game in the event that the channel is completely exited. In the case of a multiplayer game, the colocation service φ reports to other users that the user has quit the game and/or paused the game for other users. Discarded or delayed packet In one embodiment, if the data is lost due to packet loss between the video compressor 4〇4 and the client 415 in FIG. 4a or 4b, or cannot be decompressed due to being too late to arrive. And the packets satisfying the delay requirements of the decompressed frame are lost without being received in order, and the video decompression logic 412 can mitigate visual artifacts. In the streaming/P-frame implementation, if there is a lost/delayed packet, the entire screen is affected, which may cause the screen to freeze completely. 139852.doc -87- 200952495 A time period or display other screen wide visual artifacts . For example, if a lost/delayed packet is caused! If the frame is lost, you will receive a new one! Before the frame, the decompressor will lack a reference for all the p-frames to follow. If the P frame is lost, it will affect the p frame that is used for the entire screen. Depending on how long before the frame appears, this will have a longer or shorter visual impact. Using the interleaved Ι/p blocks shown in Figure 7a and the following, the lost/delayed packets are less likely to affect the entire screen because it only affects the image blocks contained in the affected packets. If the data of each image block is sent in an individual packet, if the packet is lost, it affects only one image block. Of course, the duration of the visual artifact will depend on whether the image block is lost and how many frames will be spent before the image block appears in the event of a loss of the image block. However, assuming that the different image blocks on the screen are updated very frequently (perhaps with each frame), even if one of the image blocks on the screen is affected, the other image blocks may not be affected. In addition, if an event causes several packets to be lost at the same time (for example, a temporary interrupt in the power of the DSL line, the spike of the data stream Φ), some of the image blocks will be larger than other image blocks, but because of some images The block will be updated quickly by the new j image block' so it is only temporarily affected. Moreover, in the case of the implementation of the streaming I/p frame, not only the I frame is the most critical frame, but also the frame is extremely large, so if there is an event causing the discarded/delayed packet, it is much smaller. Phosphorus 2 ratio, I frame is affected by the high probability (that is, if any part of the frame is lost, it is impossible to decompress the J frame). For all of these reasons, the use of Ι/P blocks results in much smaller visual artifacts when the packet is discarded/delayed compared to the Ι/p frame. 139852.doc -88- 200952495 An embodiment attempts to reduce the effects of lost packets by intelligently encapsulating compressed video blocks into TCP (Transmission Control Protocol) packets or UDP (User Datagram Protocol) packets. For example, in one embodiment, the image block is aligned with the packet boundary whenever possible. Figure 10a illustrates how the image block can be encapsulated within a series of packets 1001_ • 1005 without implementing this feature. Specifically, in the figure, the image block crosses the packet boundary and is inefficiently packaged such that the loss of a single packet results in the loss of multiple frames. For example, if the packet 1003 or 1004 is lost, the three image blocks are lost' resulting in visual artifacts. In contrast, Figure 10b illustrates image block packaging logic for intelligently encapsulating image blocks within a packet to reduce the effects of packet loss. First, image block encapsulation logic 1010 aligns the image block with the packet boundary. Therefore, the image blocks ΤΙ, T3, T4, T7, and T2 are aligned with the boundaries of the packets 1001_1005, respectively. Image block encapsulation logic also attempts to combine image blocks into packets in a potentially more efficient manner without crossing the packet boundaries. Combining image blocks T1 and T6 in one packet 1〇〇1 based on the size of each Φ in the image block; combining T3 and T5 in one packet 1〇〇2; combining image block T4 with D8 In a packet 1003; the image block T8 is added to the packet 1004; and the image block T2 is added to the packet 1005. Therefore, under this scheme, a single packet loss will result in the loss of no more than 2 image blocks (rather than the 3 image blocks as illustrated in Figure 〇a). An additional benefit of the embodiment shown in Figure 10b is that the image blocks are transmitted in a different order in which they are displayed within the scene; This approach produces less noticeable artifacts on the display if the adjacent packets are lost due to the same event that interferes with the transmission (which would affect areas that are not in close proximity to each other on the screen 139852.doc -89 - 200952495 domain). An embodiment uses forward error correction (FEC) techniques to protect a particular portion of the video stream from channel errors. As is known in the art, FEC techniques such as Reed-Solomon and Viterbi generate error correction information and attach it to the material transmitted via the communication channel. If an error occurs in the basic data (for example, I frame), FEC can be used to correct the error. The φ FEC code increases the data rate of the transmission, so ideally it is used only when it is most needed. If the material is being sent and it will not result in a very noticeable visual artifact, then the FEC code may preferably not be used to protect the material. For example, a P-Picture block immediately before a lost I-picture block will only produce a 1/60 second visual artifact on the screen (ie, the image block on the screen will not be updated). This visual artifact can hardly be detected by the human eye. As the p-image block is further backwards from the I-image block, the loss of the P-image block becomes more and more noticeable. For example, if the image block cycle pattern is that an I image block is followed by 15 P image blocks before the I image block is available again, if the P image block immediately after the image block is lost, it leads to The image block shows an incorrect image for 15 frame times (at 60 fps, it will be 250 milliseconds. The human eye will easily detect a 250 millisecond flow interruption. Therefore, the p image block is a new I image block. The more backward (that is, the closer the P image block follows the 丨 image block), the more noticeable the artifact is. As discussed earlier, in general, the closer the P image block follows the I image block, the closer it is to The smaller the data of the image block, the smaller the P block that follows the I image block is not only protected from loss 139852.doc 90- 200952495 but § more critical' and its size is smaller. In addition, in general, protection is required. The smaller the data, the smaller the FEC code needed to protect it. Thus, as illustrated in Figure 11a, in one embodiment, only the I image block has an FEC code due to the importance of the j image block in the video stream. , FEC 1101 contains an error for I image block 1100 N and Ke Fang 11〇4 containing code for an I image block 1103 an error correction code. In this embodiment, Jie Movies for the FEC as no blocks.

In one embodiment illustrated in Figure lib, P-blocks that are most likely to cause visual artifacts when lost are also generated. In this embodiment, FEC 11 05 provides error correction codes for the first 3 p-image blocks but not for the following p-image blocks. In another embodiment, an FEC code is generated for the p-image block with the smallest data size (which would tend to be the P-block that was originally generated after the j-image block, which is most critical for protection). In another embodiment, instead of transmitting the FEC code along with the image block, the image block is transmitted twice, each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment shown in Figure 11c, coffee codes 1111 and 1113 are generated for the audio packets (1) for simultaneous transmission from the colocation service to the video. Maintaining the integrity of the audio in the video stream is especially important because distorted audio (such as 'clicks or clicks') can result in a user experience that is particularly undesirable. The FEC code helps ensure that the audio content is reproduced without distortion at 415. Ringing the Computer In another embodiment, instead of sending the FEC code along with the audio material, the audio data is transmitted twice, each time in a different packet. 139852.doc •91- 200952495 If one packet is lost/delayed, another packet is used. In addition, in one embodiment illustrated in the figure, the feC codes 1121 and 1123 are used for user input commands (eg, pressing by slaves) 1120 and 1122 for uplink transmission from the client terminal 415 to the colocation service 21, respectively. . This is important because 'missing button presses or mouse movements in video games or applications can lead to an undesirable user experience. In another embodiment, instead of transmitting the FEC code along with the user input command material, the user input command data is transmitted twice, each time in a different packet. If a packet is lost/delayed, another packet is used. In one embodiment, the colocation service 210 evaluates the quality of the channel with the client 415 to determine whether to use the FEC, and if so, determines which portion of the video, audio, and user commands should be applied for FEC. The "quality" of the evaluation channel may include functions such as estimating packet loss, delay, etc. as described above. If the channel is particularly unreliable, the colocation service 2 can apply FEC to all I image blocks, P blocks, audio and user commands. In contrast, if the channel is reliable, the colocation service 2 can apply FEC only for audio and user commands, or FEC may not be applied to audio or video, or FEC may not be used at all. Various other arrangements of FEC applications can be used while still adhering to these basic principles. In an embodiment, the colocation service 210 constantly monitors the status of the channel and changes the FEC policy accordingly. In another embodiment, referring to FIG. 4a and FIG. 4b, when the packet is lost/delayed, resulting in loss of image block data, or if the FEC is unable to correct the lost image block data due to the particularly bad packet loss, The 139852.doc -92-200952495 client 415 evaluates how many frames remain before receiving a new i-picture block and compares it to the round-trip delay from the client 415 to the colocation service 210. If the round trip delay is less than the number of frames before the new image block should arrive, the client 415 sends a message to the colocation service 21, requesting a new 1 image block. This message is routed to the video compressor 4〇4, and instead of generating a P-picture block for the image block in which the data has been lost, a work image block is generated. Assuming that the system shown in Figures 4a and 4b is designed to provide a round trip delay of typically less than 80 milliseconds, this results in the image block being corrected to within 8 milliseconds (at 60 fps, the frame has 16 67 milliseconds) Duration, so in full frame time, an 80 millisecond delay will result in a corrected image block within 83.33 milliseconds, and 83.33 milliseconds is 5 frame time, which is an attractive interruption, but far less than (for example) 15 frames 25 〇 millisecond interrupts attract attention) 'When the compressor 404 is out of its normal loop order to generate such an I image block', if the I image block will cause the bandwidth of the frame to exceed the available bandwidth, then compression The 404 will delay the looping of other image blocks so that other image blocks will receive the P image block during the frame time (even if the image block will normally be an I image block during the frame), and then the usual loop The image frame will continue to be continued from the next frame, and the image block that has received the block in the previous frame will typically receive the I image block. Although this action temporarily delays the phase of the R frame cycle, it will usually be visually unobtrusive. Video and Audio Compressor/Decompressor Implementation Figure 12 illustrates a particular embodiment in which multiple cores and/or multiple processors 1200 are used to compress eight image blocks in parallel. In one embodiment, a dual-core processor running at 2.66 GHz or higher, a quad-core Xe〇n cpu computer 139852.doc • 93· 200952495 system, each core is implemented as an independent process open source χ264 H.264 compressor . However, a variety of other hardware/software configurations can be used while still adhering to these basic principles. For example, each of the CPU cores can be replaced by an H.264 compressor implemented in an FPGA. In the example shown in Figure 12, cores 1201 - 1208 are used to process j shadow blocks and P blocks as eight independent threads. As is well known in the art, current multi-core and multi-processor computer systems are inherently integrated with multi-threaded processing operating systems such as Microsoft Windows XP Professional (64-bit or 32-bit) and Linux. Ability to perform multi-thread processing. In the embodiment illustrated in Figure 12, since each of the eight cores is only responsible for one image block, it operates largely independently of the other cores' each performing a separate instantiation of x264. Use a PVI Express xl-based DVI capture card (such as 'Sender from Microtronix 〇f 〇osterhout in Netherlands. Video Imaging ip Development Board) to capture uncompressed φ video at 640x480, 800x600 or 1280x720 resolution, and The FPGA on the card uses Direct Memory Access (DMA) to transfer the captured video to the system ram via the DVI bus. The image blocks are configured in a 4x2 configuration 1205 (although this is illustrated as a square image block, in this embodiment 'which has a 160x240 resolution). Each instantiation of χ264 is configured to compress one of the eight 160x240 image blocks, and is synchronized such that each core enters a loop after initial I image block compression, each frame and another The frames are out of phase to compress an I image block followed by seven P image blocks, as illustrated in FIG. The resulting transmitted 139852.doc • 94·200952495 reduced image blocks are combined into a packet stream at each frame time using the previously described technique and the compressed image block is then transmitted to the destination client terminal 4 1 5 . Although not illustrated in Figure 12, if the data rate of the combined 8 image blocks exceeds the specified peak data rate 941, all 8 x264 processes will temporarily suspend the necessary frame time until the 8 for the combination has been transmitted. The data of the image block. In one embodiment, the client 415 is implemented as a software on a PC that implements 8 instantiations of FFmpeg. The receiving process receives eight image blocks and routes each image block to FFmpeg for instantiation, and FFmpeg instantiates the decompressed image block and reproduces it to the appropriate image block location on display device 422. The client 415 receives the keyboard, mouse or game controller input from the input device driver of the PC and transmits it to the server 402. The server 402 then receives the input device data received by the application and applies it to the game or application executed on the server 402. The server 402 is a PC that executes Windows using an Intel 2.16 GHz dual-core CPU. Server 402 then generates a new frame and exports the new frame to the motherboard based graphics system via its DVI output or via the DVI output of the NVIDIA 8800GTX PCI Express card. At the same time, the server 402 outputs audio generated by the game or application via its digital audio output (eg, S/PDIF), which is coupled to a dual quad-core Xeon-based PC that implements video compression. Digital audio input. The Vorbis open source audio compressor is used to compress audio simultaneously with video, regardless of the core that can be used to process the thread. In one embodiment, the core of the image block is first compressed. First, the audio pressure is 139852.doc -95-200952495. The compressed audio is then transmitted along with the compressed video and the Vorbis audio decompressor is used on the client 41 5 to decompress the compressed audio. The main sock escrow server center distributes light through a glass (such as an optical fiber) at a fraction of the speed of light in the vacuum, and thus the exact rate of propagation of light in the fiber can be determined. However, in practice, considering the routing delay, transmission inefficiency, and other elapsed time, we have observed that the optimal delay on the Internet reflects a transmission speed that is closer to 50% of the speed of light. Therefore, the optimal 1 mile round trip delay is about 22 milliseconds, and the optimal 3000 mile round trip delay is about 64 milliseconds. Therefore, a single server on the US coast will be too far apart to serve the client on the other coast (which may be up to 3 miles away) with the desired delay. However, as illustrated in Figure 13a, if the colocation service 21 is located at the center of the United States (e.g., Kansas, Nebraska, etc.), the distance to any point in the continental United States is about 15 〇〇. Miles or 1500 miles or less 'round trip internet latency can be as low as 32 milliseconds. See Figure 4b. Note: Although the worst case delay allowed by the user ISP 453 is 25 milliseconds, 'usually, in the case of DSL and cable modem systems, we observed a delay of closer to 10-15 milliseconds. Further, Fig. 4b assumes that the maximum distance from the user premises 211 to the colocation center 210 is 1 mile. Thus, in the case of the typical 15 millisecond user isp round trip delay used and the maximum internet distance of 1500 miles for a 32 millisecond round trip delay, the time from when the user actuates the input device 42 1 The total round trip delay for seeing a response on display device 422 is 139852.doc 96· 200952495 1 + 1+ 15+32 + 1+ 16+6 + 8 = 80 milliseconds. As a result, 8 milliseconds of response time is typically achieved over an Internet distance of 15 miles. This will allow any user location in the US mainland with a sufficiently short user ISP delay of 453 to be taken from a single server center located at the center. - In another embodiment illustrated in Figure Ub, cHS1-HS6 in the colocation service 210 server is strategically located around the United States (or other geographic area), a larger hosted service server center Close to the high population of 10 ~ while the 疋 position (for example, HS2 and HS5). In one embodiment, the server center HS1-HS6 exchanges information via the network 1301, which may be the Internet or a private network or a combination of the two. In the case of multiple server centers, services can be provided to users with high user lsp delays 453 at a lower latency. Although the distance on the Internet is indeed a factor in the latency of travel back and forth via the Internet, other factors that are largely unrelated to latency also work. Sometimes the packet flow is routed over the Internet to a distance away and back again, resulting in delays from long loops, sometimes routing devices that are not operating properly on the path, resulting in delays in transmission. There are sometimes traffic that overloads the path, which introduces delays. In addition, sometimes, there is a failure to prevent the user's Isp from routing to a given destination. Therefore, although the universal Internet usually provides connections from one point to another with fairly reliable and optimal routing and latency, this fairly reliable and optimal route and delay is largely determined by distance (especially In the case of a long-distance connection that leads to the outside of the user's local area, but the reliability and delay are not guaranteed and often cannot be used by 139852.doc •97· 200952495 Achieving β for a given destination on the universal Internet In one embodiment, when the user client 415 is initially connected to the colocation service 210 to play a video game or use an application, the client is Each of the available colocation service server centers HS1-HS6 communicates (e.g., using the techniques described above). If the delay is low enough for a particular • connection, use the connection. In one embodiment, the client communicates with all of the colocation service server centers or a subset of the colocation service servers to select the courier service center with the lowest latency connection. The client can select the service with the lowest latency connection ~' or the server center can identify the server center with the lowest latency connection and provide this information (for example, in the form of an internet address) to the client. The client 415 can be redirected if the specific colocation service server center is overloaded and/or the user's game or application can tolerate the delay to another, less loaded colocation service server center. Go to another host _ server service server center. In this case, the user will be executed

The game or application is suspended on the server 402 at the user's overloaded server center and the game or application status data is transferred to the server 4〇2 at the center of the other host hosting server. The game or application will then restart. In an embodiment, the colocation service 21 will wait until the game or application reaches a natural pause point (eg, between in-game levels < after the user initiates a "save" operation in the application =) Only transfer. In still another embodiment, the colocation service 21 will wait until the user activity ceases for a specified period of time (e.g., W 139852.doc -98 - 200952495) and then the transmission will begin at that time. As described above, in one embodiment, the colocation service 210 subscribes to the Internet bypass service 440 of FIG. 14 in an attempt to provide a guaranteed delay to its client. The Internet Bypass Service, as used herein, provides services from a point in the Internet to another point with a guaranteed private network route (eg, 'delay, data rate, etc.'). For example, if the host hosting service 210 is using the AT&T DSL service provided by San Francisco to receive a large amount of traffic from the user (rather than routing to the central office based in AT&T San Francisco), The colocation service 210 will then lease from the service provider between one or more of the central office based in San Francisco and the server center for the colocation service 210 (possibly AT&T itself or another A high-capacity private data connection for a provider). Then, if all the hosted service server centers HS1-HS6 use the 网τ&τ DSL user's route through the universal Internet to San Francisc〇 to cause excessive delay, then the private information can be used instead. connection. Although the private data connection is generally more expensive than the route through the universal Internet, as long as it maintains a small percentage of the colocation service 210 connected to the user, the total cost impact is low and the user will experience more consistent Service experience. In the event of a power failure, the feeder center often has two spare power layers. The first layer is typically backup power from a battery (or from an alternate ready-to-use energy source, such as a flywheel that remains operational and attached to the generator) that provides power immediately when the mains fails and maintains the center of the feeder Running. If the power failure is temporary and the mains return quickly (eg 139852.doc -99- 200952495 as in the knife), the required battery remains. However, if the power failure lasts for a long period of time, then 2 heart transport machines (such as 'diesel power supply') replace the battery and send = power generation ^ a machine Angbei' because it must be able to produce more of this feeder center usually self-powered The electricity obtained by the main line. Two =, each of the colocation services is ==, so that in the event of a power failure in a server center, the game and application of the second == are suspended, and then the game or application status data is fed from each The device is transmitted to the feeder 402 at the center of the other server, and then the user is notified to each user to direct it to the new word processor. Assuming that such occasions occur occasionally, the user is transferred to a colocation service feeder center that is unable to provide the optimal delay (ie, the user will only have to tolerate the duration of the higher latency duration power failure). Acceptable, it will allow for the option of transferring a much wider range of users. For example, given the time difference across the United States, users on the East Coast may be sleeping at u:3 pm while users on the West Coast are starting to peak in video game usage at 8:30 PM. If there is a power failure in the colocation service (4) center on the West Coast, there may not be enough West Coast servers to handle all users at the other colocation service center. In such a case, some users may be transferred to the colocation service server center with the available server 402 on the East Coast, and the only consequence for the user will be a higher latency. Once the user has transferred from the center of the feeder that lost power, the server center can then start its server and set it up in order to drain the battery (or other immediate power backup). In this way, it can be avoided that the server for the server is in the embodiment, during the time of the severe loading of the colocation service 21 (or because the peak user loads 'or because of one or more words The device center fails. The user is transferred to another server center based on the delay of the game or application being used by the user. Therefore, a preference will be given to users of games or applications that require low latency to have available low latency server connections with limited supply. Hosted Hosting Service Features Figure 15 illustrates an embodiment of the components of the server center for the hosted hosting service 210 utilized in the following description of the features. As with the colocation service 210 illustrated in Figure 2a, the components of this server center are controlled and coordinated by the colocation service 210 control system 4〇1 unless otherwise conditional.

The orderly cut off, before cutting off the cost of all equipment motors. The incoming Internet traffic 15〇1 from the user client 415 is directed to the incoming route 1502. Usually, the Internet traffic 15〇1 will enter the center of the feeder via a high-speed fiber connection to the Internet, but any network connection component with sufficient bandwidth, reliability and low latency will be sufficient. . Inbound routing 15〇2 network (which can be implemented as an Ethernet, fiber channel network 'or via any other transport component) switch and a system that supports the routing servers of the switches The arriving packets are routed to the appropriate application/game server 1521-1525. In one embodiment, the packet delivered to the particular application/game server represents a subset of the data received by the client 139852.doc -101 - 200952495 and/or may be other components within the data center (eg, Road connection components, such as gateways and routers, are used to translate/change. In some cases, for example, if a game or application is concurrently executed on multiple servers, each time the packet is routed to more than one server: 1521·1525. RAID array 15ii-1512 is connected to the inbound routing network

1502, so that the application/game server 1521-1525 can read the rAID arrays 1511-1512 and write the 1105 arrays 1511_1512. In addition, 1^11) 阵列 array 1515 (which can be implemented as multiple RAID arrays) is also coupled to incoming routing 1502, and data from RAID array 1515 can be read from application/game servers 1521-1525. Incoming routing 15〇2 can be implemented in a variety of prior art network architectures (including tree-structured switches, inbound Internet traffic) in its roots; in interconnecting the mesh structures of all various devices Implemented in ; or as a sub-network sequence of interconnects (concentrated traffic among interworking devices is isolated from the set of traffic in other devices). One type of network is configured as S AN, which, although commonly used for storage devices, can also be used for general high-speed data transfer between devices. Also, the application/game servers 1521-1525 can each have multiple network connections to the incoming route 15〇2. For example, the feeders 1521-1525 can have a network connection to a subnet attached to the RAID arrays 1511-15 12 and another network connection to a subnet attached to other devices. The application/game server 1521_1525 can be configured identically, somewhat differently, or all differently, as previously described with respect to the feeder 402 in the embodiment illustrated in Figure 4a. In one embodiment, each user typically has at least one application/game servo when using the colocation service 139852.doc • 102· 200952495

1521-1525. For simplicity of description, it will be assumed that a given user is using the application/game server 1521, but multiple servers may be used by the user, and multiple users may share a single application/game server 1521 -1525. The user's control input sent from the client 415 (as previously described) is received as an incoming Internet service 15〇1 and routed to the application/game server 1521 via the incoming route 1502. The application/game server 1521 uses the user's control input as a control input to the game or application executing on the server and calculates the video and the next frame of the audio associated therewith. The application/game server 1521 then outputs the uncompressed video/audio 1529 to the shared video compression 1530. The application/game server can output uncompressed video via any component (including one or more ultra-high speed Ethernet connections), but in one embodiment, the video is output via a DVI connection, and the audio and Other compression and channel status information is output via a Universal Serial Bus (Usb) connection. The shared video compression 1530 compresses uncompressed video and audio from the application/game feeder 1521_1525. This abolition can be implemented entirely in hard or hard hardware. There may be a dedicated compressor for each application/game server 1521 - 525' or if the compressor is fast enough, a given compressor may be used to compress from more than one application/game server 1521-1525 Video/audio. For example, at 6〇 fps, the video frame time is 16.67 milliseconds. If a compressor can compress 1 frame in i milliseconds, then the compressor can be used to compress from up to 16 applications/game servers 139852.doc -103 by taking input from one server after another. - 200952495 1521-1525 video/audio, which stores the state of each video/audio compression process and switches the background as it cycles through the video/audio stream from the server. This results in substantial cost savings for the compressed hardware. Since different servers will complete the frame at different times, in one embodiment, the compressor resources are in a shared state with shared storage components (eg, RAM, flash memory) for storing the state of each compression process. In the cluster 153, and when the server 1521-1525 is complete and ready to be compressed, the control memberΟ

Determining which compression resource is available at that time, providing the compression resource with the state of the compression process of the server and the uncompressed video/audio frame to be compressed. Note that one of the states of the compression process for each server The packet encapsulation information about the >1 shrink itself, such as the decompressed frame buffer data of the previous frame (which can be used as a reference for the p image block), the quality of the resolution of the video output; the image block structure (4) The allocation of bits in the block is limited to the quality and audio format (for example, stereo, surround sound, secret y® AC-3). However, the compression process state also includes information about the following communication channel status: peak data rate 941, and whether the previous frame (as illustrated in the flap) is currently being output (and therefore the current frame should be ignored), and potentially There are _ characteristics that should be considered in I (such as excessive packet loss) that affect compression decisions (eg, in terms of the frequency of the video block), because the peak data rate 941 or other channel characteristics change over time, such as The application/game server 1521 is determined by the application/game server 1521, which is determined by the application/game server 1521, which is used by each user to use the information sent from the client 415, so the application/game server 152 1 -1 525 The relevant information is sent to the shared hardware compression 139852.doc -104. 200952495 1530. The shared hardware compression 1530 also encapsulates the compressed video/audio using components such as those previously described, and Where appropriate, the application will 'copy specific data, or take other steps to fully ensure that the video/audio data stream is received by the client 415 and is feasible. Quality and reliability/sex decompression capabilities, some applications (such as the applications described below) require video/audio output from a given application/game server 1521-1525 simultaneously at multiple resolutions (or in multiple other formats). If the application/game server 1521-1525 thus notifies the shared hardware compression 1530 resource, the uncompressed video audio 1529 of the application/game server 1521-1525 will be Simultaneous compression in different formats, different resolutions, and/or in different packet/error correction structures. In some cases, some compression resources may be shared among multiple compression processes that compress the same video/audio (eg, in many compression calculations) In the method, there is a step of scaling the image to multiple sizes before applying compression. If it is necessary to output images of different sizes, this step can be used to simultaneously servo several compression processes.) In other cases, for each case The format will require a separate compression resource. In any case 'will be used for a given application/game server 1521·15 25 (one or more) of all the various resolutions and formats of compressed video/audio W39 are simultaneously output to the outgoing route 154. In one embodiment, the output of the compressed video/audio 1539 is UDP format, so it is a one-way packet flow. The outgoing routing network 1540 contains a series of routing servers and switches. The I39852.doc 200952495 series routing server and switch will stream each compressed video/audio stream. Internet traffic 1599 interface (which will typically be connected to the optical interface to the Internet) directed to the intended user or other destination and/or back to delay buffer 1515, and/or back to incoming routing 15〇2, and/or output via a private network (not shown) for video distribution. Note that a given video/audio stream can be simultaneously output to multiple destinations as described below. In one embodiment, this is implemented using Internet Protocol (ιρ) multicast, where the broadcast is intended At the same time, _ flows to a given UDP stream of multiple destinations' and the broadcast is repeated by the routing server and switch in the outgoing route 1540. The multiple destinations of the broadcast may be used over the Internet to multiple uses. The client 415, via the incoming route 15〇2, to the plurality of application/game 4 server 1521-1525 'and/or to one or more delay buffers 1515. Therefore, the server 1521 will be given The output of 1522 is compressed into one or more formats' and each compressed stream is directed to one or more destinations. In addition, in another embodiment, if multiple application/game servers 1521-1525 are simultaneously The user uses (for example, in a parallel processing configuration for generating a 3D output with a complex scene) and each word processor produces a portion of the resulting image', which can be compressed by a shared hardware 15 3 0 Video output combination of 1521-1525 Form a combo frame and process the combined frame as described above from the other side as if it were from a single application/game server 1521-1525. Note that in an embodiment, it will be used by the application/game server 1521_ A copy of all video generated by 1525 (at least at a resolution or higher resolution of the video viewed by the user) is recorded in delay buffer 1515 for a certain number of minutes to 139852.doc 200952495 (in one embodiment) 15 minutes). This allows each user to "rewind" the video from each session in order to check previous work or performance (in the case of the game). Therefore, in one embodiment, the route will be routed to the user. Each compressed video/audio output 1539 stream of the client 415 is also multicast to the delay buffer 1515. When the video/audio is stored on the delay buffer 1515, the directory on the delay buffer 1515 provides the application/game server. The network address of 1521-1525 (which is the source of the delayed video/audio) and the location of the delayed video/audio location on the delay buffer 1515 are referenced. The live, instantly viewable, instantly playable game application/game server 1521-1 525 can be used not only to execute a given application or video game of the user, but also can be used to establish support for hosting via the host. A user interface application of the host service 210 of the navigation and other features of the service 210. A screen shot of the user interface application is shown in Figure 16 ("game viewfinder" screen for this particular φ user interface screen Allow users to view 15 games played (or delayed) by other users. Each of the "thumbnail" video windows (such as '1600) is a live video window in motion, showing A video from a user's game. The view shown in the thumbnail may be the same view that the user is looking at, or it may be a delayed view (eg, if the user is playing a fighting game, the user may not want other users to see where they are hidden and It can choose to delay any view of its game play for a period of time (10), such as age clock). _ can also be a camera view of the game with different views of the user. The user can select the game selection to be simultaneously viewed based on a plurality of criteria via the I39852.doc -107. 200952495 menu selection (not shown in this description) as a small sample of the exemplary selection, and the user can select the random game. Choose (such as those shown in Figure 16), all categories of games* (both played by different players), only top players in the game, players of a given level in the game' or lower level players ( For example, if the player is learning the basics, the player who is "right" (or a competitor), the game with the most number of viewers, and the like. Note that 'usually, each user will decide whether the video from their game or application can be viewed by others, and if so, decide which other people can view and when they can be viewed by others, and decide whether the video can only have View in case of delay. The application/game server 1521-1 525 that produces the user interface screen shown in FIG. 16 is provided to each user (the application/game server 1521-1525 is requesting a game from the user) The application/game server 1521-1 525 sends the message to retrieve the 15 video/audio feeds. ❹ The message is sent via the incoming route 1502 or another network. The message will include the size and format of the requested video/audio and will identify the user viewing the user interface screen. A given user may choose to select the "Piracy" mode and not allow any other user to view the video/audio of their game (from their view point or from another view point), or as described in the previous paragraph, the user may choose Allow viewing of video/audio from its games, but delay the video/audio being viewed. The user application/game server ( (10) (which receives and accepts requests for permission to view its video/audio) will therefore confirm with the requesting server, and it will also notify the shared hardware compression 1530 that 139852.doc -108 is required. - 200952495 Generates an additional compressed video stream that is formatted by the use or fired by the size (assuming the format and screen size are different from the size and screen size produced), and it will also display the compressed video. Destination (ie, request server). If the requested video/audio is only delayed, the requesting application/game server 1521-1525 will be notified as such, and will find the location of the video/audio in the directory on the delay buffer 1515 as a delay. The video/audio source of the application/game server 1521_1525 is derived from the delay buffer 15 15 to obtain delayed video/audio. Once all such requests have been generated and processed, up to 15 live broadcast thumbnail-sized video streams are routed from the outgoing route 1540 to the incoming route 15 〇 2, to the application/game that generates the user interface. The server 1521_1525 will be decompressed and displayed by the server. The delayed video/audio stream may be at an excessive screen size, and if so, the application/game server 152i-1525 will decompress the streams and scale the video stream down to the thumbnail size. In one embodiment, the request for audio/video is sent to a central r management service (not shown in Figure 5) similar to the colocation service control system of Figure 4a (and managed by a central "management" service) The central "management" service then redirects the requests to the appropriate application/games feeders 1521-1525. Further, in an embodiment, it may not be necessary to request 'because the thumbnails are "pushed" to the user of the user who is allowed to do so. All of the simultaneous audio from 15 games can produce harsh sounds. The user can choose to mix all the sounds together in this way (maybe to get the feeling of "喧嚣" generated by all the actions being viewed), or the user can choose to listen to the audio from one game at a time. 139852.doc -109^ 200952495 The choice of play is done by moving the yellow selection box 16G1 to the given game (the yellow frame can be moved by using the arrow keys on the keyboard, by moving the mouse, by moving the joystick) Or by pushing a direction button on another device such as a mobile phone). Once a single game is selected, it is only played from the audio of the game. Also, show game information 16〇2. In the case of this game, for example, the publication of the trademark ("EA") and the game label "Need for Speed Card Canyon" and the orange bar indicate the number of people who play or view the game at a particular time under relative conditions ( In this case, a lot, so the game is "hot". In addition, a "status" is provided indicating that there are 80 different instantiations in which 145 players are actively playing Need for Speed games (ie, the game can be played by individual player games or multiplayer games), and there are 68 views. (This user is one of them). Note that such statistics (and other statistics) are collected by the hosted service control system 4.1 and stored on the RAID arrays 1511-1512' for maintaining the hosted service 21 operations. The user is billed appropriately and paid to the publisher who provided the content. Some statistics are recorded as a result of actions performed by the service control system 4, and some statistics are reported to the service control system 401 by the individual application/game servers 1521-1525. For example, when the game is being viewed (and when the game is stopped viewing), the application/game server 1521-1525 executing the game viewfinder application sends a message to the colocation service control system 401 to The colocation service control system 401 can update how many games are in view of the statistics "some statistics can be used by user interface applications (such as this game viewfinder application). 139852.doc -110· 200952495 When the right user clicks on the activation button on his input device, he will see the thumbnail video in the yellow frame and the thumbnail video will be live broadcast as the screen size. This effect is shown in the process of Figure 17. Note that the size of the video window 1700 is increased. In order to implement this effect, the application "game feed device 1521, 1525 from the execution of the axis of the heart / game feeder 152M525 request has a full screen size routed to it (with the user's display device 422 Resolution of the video stream. Execution of the game application/game (4) 1521_1525 notification hard (four) retractor (10) Replica of the thumbnail size of the game is no longer needed (unless another application/game server 1521-1525 Such a thumbnail is required, and then it directs the shared hardware compressor 1530 to send a copy of the full screen size of the video to the application/game server 1521_1525 that amplifies the video. The user playing the game may or may not have resolution. The display device 422 has the same resolution as the display device of the user who enlarges the game. In addition, other viewers of the game may or may not have the same display device 422 as the resolution to use the game. Different audio playback components, such as stereo or surround sound.) Therefore, the shared hardware compressor 153 determines whether a request has been satisfied. A suitable compressed video/audio stream is required by the user of the video/audio stream and if a suitable compressed video/audio stream does exist, the shared hardware compressor 1530 notifies the routing 154 to copy the stream. Routing to the application/game server 1521-1525 that amplifies the video, and if the appropriate compressed video/audio stream does not exist, compressing the video is suitable for another copy of the user and directing the routing to the stream Send back to Route 15 02 and the application/game server 1521_1525 to enlarge the video. 139852.doc • 111 - 200952495 This server that receives the full screen version of the selected video will now decompress the full screen version and Gradually scale up to full size. Figure 18 illustrates the full zoom of the game to full screen and display by the user. The full resolution of device 422 shows how the screen looks after the game (as directed by the arrow 1800) Instructed to execute the game viewfinder application/game server 1521-1525 to other applications/game servers 1521-15 that provide thumbnails 25 sends a message to indicate that the thumbnails are no longer needed and sends a message to the colocation service control server 4〇i to indicate that the other games are no longer viewed. At this time, the only display generated is the top of the screen. 1801, which provides information and menu control to the user. Note that as the game progresses, the viewer grows to 2, 5, 3 viewers. In the case of so many viewers, there must be a display device 422 Many viewers have the same or close resolution (the ability of each application/game server 1521-1525 to scale the video for adjustment of fit). 〇 Because the game shown is a multiplayer game, the user can decide to join the game at some point. The colocation service 210 may or may not allow users to join the game for a variety of reasons. For example, the user may have to pay for the game and choose not to pay, the user may not have enough

A game that is played remotely (in fact, so it can be viewed on another continent without delay), but for games played 139852.doc -112- 200952495, the delay must be low enough Having the user (a) enjoy the game 'and (b) is in the same position as other players who may have a lower latency connection) ° If the user is not allowed to play, the user is provided with the application of the game viewfinder user interface The program/game server 1521-1 525 will request the colocation service control server 401 to initiate (ie, locate and launch) the application/game server 1521 that is suitably configured for playing a particular game.

1525 loads the game from the RAiD array ι511_1512, and then the colocation service control server 401 will direct the routing control 1502 to transmit the control signal from the user to the application/game server of the current colocation game and Now the hosted game application/game server will guide the shared hardware dust reduction 1530 self-compression video/audio from the hosted game viewfinder application/game server switch to compression from the current colocation Video/audio of the game application/game server. The vertical synchronization of the game viewfinder application/game service with the new application/game server of the colocation game is not synchronized, and thus there may be a time lag between the two synchronizations. Since the shared video compression hardware 153 will start to freeze the video after the application/game server 1521-1525 completes the video frame, the first frame from the new server can be completed than the full frame time of the old server. Earlier, the first frame from the new server may be before the transmission is completed in the previously compressed frame (for example, consider the transmission time outside the picture 992: if the uncompressed frame 3 963 is completed earlier) One and a half of the time will impact the transmission time 992). In this case, the shared video compression hardware 1530 will be ignored as 'Ignore (974) frame 4 964), the first frame of the new server (for example and the client 41 5 will come from the old servo 139852. Doc •113· 200952495 The last frame of the device maintains an extra frame time, and the shared video compression hard camera 1530 will begin to compress the next frame time video from the new application/game server of the hosted game. For the user, visually, the transition from one application/game server to another application/game server will be seamless. The colocation service control server 401 will then notify the host to host the game framing. The app/game server 1521_1525 switches to the idle state until it is needed again. 使用者 The user can then play the game. In addition, the exception game will be played instantly in the sense (because the game has been ten The bite/second speed is from 1^1 (1 array 1511-512 is loaded onto the application/game server 1521_1525)' and will be configured by the ideal driver and scratchpad (in the case of trade In the case of loading the game along with the operating system configured for the game, to the server that is exactly suitable for the game, and other applications that are unlikely to compete with the operation of the game on the server Also, as the user progresses through the game, each of the segments of the game will be loaded from the RMD array at a rate of one billion bits per second (ie, 8 seconds per billion bytes). The 1511-1512 is loaded into the server and due to the huge storage capacity of the raid array WU-BU (because it is a shared resource for many users, it can be very large, but still cost effective), allowing pre-computation of geometry Set or other game clip settings and store them on the RAID arrays 1511-1512 and load them very quickly. In addition, because the hardware configuration and computing power of each application 'game server 1521_1525 is known So the pixels and vertex shading can be pre-computed. So the game can be launched almost instantly 'it will be executed in an ideal environment, 139852.doc -114- 200952495 and the subsequent clips will load almost instantly. In addition to these advantages, the user will be able to view other people's playing games (via the previously described game viewfinder, and other components), and both determine whether the game is interesting, and if so, learn from watching others. In addition, 'the user will be able to demonstrate the supplement immediately, without having to wait for a large download and/or installation' and the user will be able to play the game instantly (perhaps on a less expensive trial basis, or in a longer term) In addition, the user will be able to play the game on a Windows PC, Macintosh, on a television, at home, while traveling, and even on a mobile phone with a wireless connection of sufficiently low latency. In addition, this can be done without having a physical copy of the game. As previously stated, the user can decide not to allow his game to be played by others. 'Allow his game to be viewed after a delay, allowing his game to be Select a user view or allow their games to be viewed by all users. In any embodiment, in one embodiment, the video/audio is stored in the delay buffer 10 1515 for 15 minutes, and the user will be able to "rewind" and view the first.

The previous game is played, and the game is paused, the game is played back slowly, the game is fast forwarded, etc. as it is when viewing τν with a digital video recorder (DVr). Although in this example t, the user is playing the game, the same "dvr" capability is available if the user is using the application. This may be useful in verifying previous work and in other applications as detailed below. In addition, if the game is designed to have the ability to fall back based on the use of game state information so that the camera view can be changed, it will also support this "3D DVR" capability but it will need to design the game to support r3D 139852.doc • 115· 200952495 Ability. Using the delay buffer called "please" capability will work with any game or application (of course, limited to the video generated when using the game or application), but in the case of a 3d earning stomach The user can control the 3D "travel" of the previously played clip and cause the delay buffer 1515 to record the resulting video and record the game state of the game segment. Therefore, the specific "traversal" is recorded as compressed video, but since the game state will also be recorded, different crossings may be at a later date in the same segment of the game. As described below, users on the colocation service 2 will each have a user page' in which the user can post information about himself and other materials. One of the things that the user will be able to announce is a video clip from a game that has been saved. For example, if the user has overcome the particular difficulty of the game, the user may just "rewind" to the location where he or she has achieved great results in the game, and then instruct the colocation service 210 to The video clip φ of duration (for example, 3 seconds) is saved on the user's user page for viewing by other users. The only thing to do to implement this application/game server 1521_1525 that the user is using is to play back the video stored in the delay buffer 1515 to the RAID array 15 11-1 5 12 and then index the video segments. On the user's user page. If the game has the ability to have a 3D DVR, as described above, the user can also record the game state information required by the 3D DVR and make it available to the user's user page. In the case where the game is designed to have "bystanders" in addition to active players, 139852.doc -116 - 200952495 (that is, users who can travel in the 3D world without observing and observe the action) The game viewfinder app will enable the user to join the game as a bystander and player. Self-examination implementation point of view For the colocation system 210, there is no difference between the user being a bystander and not the active player. The game is loaded onto the application/game word server 1521-1525 and the user will control the game (e.g., control the virtual camera viewing the world). The only difference will be the user's gaming experience. © Multiple User Cooperation Another feature of the colocation service 210 is the ability of multiple users to collaborate while viewing live video broadcasts (even if they are viewed using a different device). This is useful when playing games and when using an application. Many PCs and mobile phones are equipped with video cameras and have the ability to perform instant video compression (especially when the image is small). Also, 'small camera is available, attachable to the TV' and it is not difficult to implement instant compression in software or using one of many hardware compressors for compressing video. Also, many PCs and all mobile phones have a microphone and the headset is available with a microphone. The cameras and/or microphones combined with native video/audio compression capabilities (specifically, using the low latency video compression techniques described herein) will enable the user to combine video and/or audio with input device control data. Together, it is transmitted from the user premises 211 to the colocation service 21〇. When using these techniques, the capabilities illustrated in Figure 19 can be achieved: the user can have their video and audio 1900 appear on the screen of another user's game or application. This example is a multiplayer game where teammates collaborate in the car. Video/audio using 139852.doc • 117- 200952495 can only be selectively viewed/heard by their teammates. Moreover, because there is effectively no delay in using the techniques described above, the player will be able to talk to each other or exercise at once without an appreciable delay. • This video/audio integration is accomplished by having compressed video and/or audio from the user's camera/microphone as incoming Internet traffic 1501. Next, the incoming route 1502 routes the video and/or audio to the application/game server 1521-1525 that is permitted to view/hear the video and/or audio. Then, the user selecting the respective application/game server 1521-1525 using video and/or audio decompresses the video and/or audio and integrates it as needed to appear in the game or application, such as by 1900 explained. The example in Figure 19 shows how to use this collaboration in a game, but this collaboration can be an extremely powerful tool for applications. Consider a scenario where a large building is being designed by the architects in Chicago for a New York-based real estate developer for New York City, but the decision involves financial investors who are on the move and happen to be at Miami Airport, and need Make decisions about specific design elements of the building (how it relates to the buildings in the vicinity) to meet both investors and real estate developers. Suppose the construction company has a high-resolution monitor with a camera attached to the PC in Chicago, the real estate developer has a laptop with a camera in New York, and the investor has a mobile phone with a camera in Miami. The construction company can use the colocation service 210 to host a powerful architectural design application that is highly realistic 3D reproduction' and it can utilize the large database of New York City buildings and the library of buildings being designed. The architectural design application will execute on one of the applications / 139852.doc - 118 · 200952495 game server 1521-1525 (or on several if it requires a lot of computing power). Each of the three users at the disparate location will be connected to the colocation service 21〇, and each will have a simultaneous view of the video output of the architectural design application, but it will be targeted for each A user has a given device and network connectivity characteristics that are appropriately sized by the shared hardware compression 1530 (eg, a building company can see 2560xl440 6〇fps display via the 2 Mbps commercial internet connection. Production developers can see 1280x720 60 fpS images via a 6 Mbps DSL connection on their laptops, and investors can see 32〇x18〇6〇fps images via a 250 Kbps cellular connection on their mobile phones. One party will hear the voice of the other party (the conference call will be handled by any of the many widely available conference call suites in the app/game calendar i52i_ 1525), and via the input device The user of the button will be able to use their local camera to make the video appear. As the meeting proceeds, 'architects will be able to demonstrate with 3D renderings that are photorealistic and realistic. #其使建筑的建筑进行进行进行进行进行进行的建筑物的建筑物的建筑物的建筑物的,其所有的方的的相相的相相的相相。 The party will rotate the building and make it adjacent to the other in the area - the building will look like the building, and all parties will see the same video at the resolution of the display devices of the parties. Any one of the local devices used can handle 3D animation with this realism without asking a, not to use

Say the huge library of resources needed to download or even reproduce the buildings surrounding New York City. Since the user's ’' but it will simply be incredibly realistic. In addition, when a party wants its face to be seen

Despite the disparate native device, it has a seamless experience. When the good news conveys its emotional state 139852.doc •119· 200952495 Developers or investors want to control the building program and use their own input devices (the keyboard, mouse, keypad or touch screen), then it can , and its "T responds with no perceived latency (assuming its network connection does not have unreasonable delays). For example, in the case of a mobile phone, if the mobile phone is connected to the WiFi network of the airport, it will have a very low latency. However, if it uses a cellular data network available in the United States today, it is likely to suffer a noticeable lag. However, for most of the purposes of the conference (where investors are watching architects controlling building crossings or talking about video conference calls), even honeycomb delays should be acceptable. Finally, at the end of the cooperative conference call, the real estate developer and investor will make their comments and stop the colocation service, and the construction company will be able to "rewind" the video of the conference that has been recorded on the delay buffer 1515 and Review comments, facial expressions and/or actions applied to the 3D model of the building during the meeting. If there are specific segments that it wishes to save, then the segments of the video/audio can be moved from the delay buffer 1515 to the RAID arrays 1511-1512 for archival storage and later playback. Also, from a cost perspective, if an architect only needs to use New York City's computing power and a large database for a 15-minute conference call, then it only needs to pay for the time when the resources are used, without having to have high capabilities. Workbench does not have to purchase expensive copies of large databases. FX Rich Community Services Hosting Services 210 is an unprecedented opportunity to build video-rich community services on the Internet. Figure 2A shows an exemplary user page for a game player on the colocation service 21〇. Like the (10) viewfinder application 139852.doc • 120- 200952495 program-like user page is an application executed on one of the application/game word processor mi-1525. All thumbnails and video windows on this page show video that you hate to move (if the clip is short, it loops). By using a video camera or by uploading video, the user (the user name is "KILLHAZARD") can publish his own video 2 (other users can view the video), and the video is stored on the RAID array 1511-512. . Also, when other users come to the user page of KILLHAZARD, if KILLHAZARD is using the colocation service 210 at this time, Beij will display the live video of whatever it is doing 2001 (assuming it is allowed to view its use) The user of the page views it). This will request whether the KILLHAZARD is active from the service control system 401 by the host/host server application application/game server 1521-1525 (and if so, request the application/game server 1521- 1525) to complete. The compressed video stream of the appropriate resolution and format is then sent to the application/game server 1521-1525 executing the user page application and displayed using the same method used by the game finder application. If the user selects a window with a live broadcast of KILLHAZARD and then clicks on their input device appropriately, the window will be enlarged (using the same method as the game viewfinder application again) and the live video will be The resolution of the display device 422 of the user (suitable for viewing the characteristics of the user's internet connection) is filled in the screen. The key advantage of this prior art approach is that the user viewing the user page can see the live broadcast of the game that the user does not own, and may not have a local computer or game console capable of playing the game. It is 139852.doc -121 - 200952495. The user provides an excellent opportunity to view the user's page towel exhibition (4) "active towel" for users to play games. It is an opportunity to understand the game that the user may wish to try or be good at. . Video clips recorded or uploaded from the camera of KILLHAZARD's partner 2002 are also displayed on the user's page and below each video clip is the text indicating whether the slot is playing online or not (example #, (相正玩游戏" Dragon Knight (Eragon) and MrSnuggles99 offline, etc.) By clicking on the menu item (not shown), the partner video clip switches from displaying the recorded or uploaded video to the game currently playing the colocation service 21〇 The partner is in the live broadcast of the content being played in his game at that time. Therefore, it becomes a game viewfinder that is grouped by the partner. If the game of the partner is selected and the user clicks on the game, the game will be enlarged to Full screen, and the user will be able to watch the game played live on the full screen. Again, the user of the game viewing the partner does not own the copy of the game, nor does it have the local computing/game console resources for playing the game. The game view is valid instantaneous. As described earlier, when the user plays the game on the colocation service 210, the user can " Pour the game and find the video clip it wishes to save, and then save the video clip to its user page. These are called "Brag Clip" e video clips 2〇〇3 are all by KILLHAZARD The self-reward clip saved from the previous game played by it. The number 2004 shows how many times the self-review clip has been viewed, and when the self-respect clip is viewed 'the user has the opportunity to rate it, and the shape of the orange keyhole The number of graphs 2005 indicates how high the rating is. When the user views the 139852.doc • 122· 200952495 user page, the self-review clip 2003 is continuously cycled along with the rest of the video on the page. If the user selects and clicks One of the clips 2〇〇3 is zoomed in to present the clip 2〇〇3, and the DVR control that allows playback, pause, back/reverse, fast forward, step, etc. the clip. The 2003 playback system loads the compressed video clips stored on the RAID array 15 jj _ 15 12 by the application/game server ι521_ 1525 and decompresses them and plays them back. 0 ❹ Self-recruiting clip 2003 can also be a "3D DVR" video clip from a game that supports 3D DVR capabilities (ie, a sequence of game states from a game that can be played back and allows the user to change the camera view point). In this case, in addition to the specific "crossing" compressed video recordings that the user performs while recording the game segment, the game state information is also stored. When the user page is being viewed and all thumbnails and video windows are constantly looped The 3d DVR self-reward clip 2003 will cause the user to record the "travel" of the game clip as a compressed video of the self-reward clip 2003 constantly looping. But 'when the user selects the 3D DVR self-reward clip 2003 and clicks When the 3D DVR is self-recommended with Clip 2003, in addition to allowing DVR control of the compressed video capture clip, the user will be able to click on the button that gives its 3D DVR capability for the game clip. It will be able to independently control the camera "crossing" during the game segment, and if it wishes (and the user of the user page so allows), it will be able to record the alternate self-reward clip in the form of compressed video. The alternative self-reward clip "crossing" will then be available to other viewers of the user page (immediately) or after the owner of the user page has 139852.doc -123- 200952495 with the opportunity to verify the clip ). This 3D DVR Self-Camp Clip 2003 capability is enabled by launching a game that will replay the recorded game state message on another application/game server 1521-1 525. Because the game can be launched almost instantaneously (as previously described), so it is activated (its playback is limited to the game state recorded by the self-reward clip) and then the user is allowed to record the compressed video to the delay buffer 1515. It is not difficult to "cross" with a camera. Once the user completes the "crossing", the game is deactivated. 〇 From the user's point of view, launching the “crossing” with the 3D DVR Self-Camp Editing 2003 is no more difficult than controlling the DVR control of the Linear Self-Review Clip 2003. It does not know the game or even know how to play the game. It is only a virtual camera operator staring at the 3D world during the game segment recorded by another operator. Users will also be able to add their own audio to the self-reward clips (or record or upload them from the microphone in this way, and use the self-reported clips to create custom animations using characters and actions from the game.) Production techniques are often referred to as "machinima." As the user progresses through the game, they will reach different skill levels. The played game reports the results to the service control system 4〇1, and these skill levels It will also be displayed on the user page. Interactive animated online advertising has changed from text to still images, video, and now to interactive clips 'usually using an animated thin client like Ad〇be Fiash. The reason for using the animated thin client is that the user 139852.doc -124- 200952495 is usually less impatient with the privilege of selling products or services to them. Also, the thin client is on a very low performance PC. Execution, and therefore advertisers can have a high level of confidence: interactive ads will work properly. Unfortunately, 'animated like Adobe Flash The short client is limited in the duration of the interactive process and experience (to reduce download time). Figure 21 illustrates an interactive ad where the user will select the outside of the car while the car is spinning in the showroom. And internal color, while simultaneous ray tracing

Show how the car looks. The user then selects the avatar to drive the car, and then the user can pick up (4) the car for driving on a race track or through a country other than M〇naco. The user can select a larger engine or a better tire and then see how the changed configuration affects the car's ability to accelerate or maintain stability. Of course, advertising is effectively a cutting-edge 3D video game. But for such ads that can be played on a PC or video game console, it will require 100 MB downloads, and in the case of a PC, it may require a special drive and may lack sufficient CPU or GPU computing power on the PC. It is not executed at all. Therefore, such advertisements are impractical in prior art configurations. In the colocation service 210, the advertisements are placed almost immediately, and are preferably executed regardless of the capabilities of the user's client 415. As a result, it delivers more quickly than a streamlined, user-friendly ad, with a richer experience and a high degree of reliability. Streaming geometry RAID arrays 1511-512 and inbound routing 1502 during instant animation provide such fast data rates with such low latency that it is possible to design depending on 139852.doc -125- 200952495 RAID array 15 11-1 5 12 and routing 1502 to reliably deliver geometric video games and applications directly in the middle of the game play or in an application during instant animation (eg, traversal with complex databases). In prior art systems, such as the video game system shown in Figure 1, the large number of storage devices available (especially in practical home devices) is so slow that it is not possible to stream geometry during game play (except for the required In addition to the slightly predictable geometry, for example, in a driving game where a designated road exists, the geometry of the building used to enter the field of view can be reasonably properly predicted and a large number of storage devices can search for the upcoming geometry in advance. The location where the shape is located. But in complex scenes with unpredictable changes (for example, in a battle scene with complex characters around), if the RAM on the PC or video game system is completely filled for the current view The geometry of the object, and then after the user suddenly turns his character to view his character, if the geometry is not preloaded into the RAM, there may be a delay before the geometry can be displayed. Medium RAID array 1 5 11 -1 512 can exceed the speed data of ultra-high speed Ethernet speed, and Under the SAN network, it is possible to achieve a speed of 10 billion bits per second that is better than 10 billion bit Ethernet or better than other network technologies. 1 billion bits per second will be less than one. Loads 1 billion 7G data in seconds. It can load about 170 million bits (21 MB) in 6〇fps frame time (16 67 ms). Of course, even in RAID configuration, Rotating media will still incur a delay greater than one frame time, but the flash memory based on the flash memory will eventually be larger than the rotating 139852.doc -126- 200952495 media RAID array and will not incur this high latency. In one embodiment, a very low latency access is provided using a cache of large RAM writes. Thus, with a sufficiently high network speed and a large amount of memory with low latency, the CPU and / or the GPU can process 3D data to quickly stream geometry to the application/game server 1521-1525. Thus, in the previously given example, where the user suddenly turns his character and looks backwards, Loads all characters behind them before the character finishes spinning The geometry, and therefore, for the user, will appear to be in a world that is as realistic as the live action of the live action. As discussed earlier, in the final border of a photorealistic computer animation One is a human face, and because of the sensitivity of the human eye to incompleteness, the slightest error from a photo-realistic face can lead to negative reactions from the viewer. Figure 22 shows the use of ContourTM authenticity capture technology (below) Subject of the same application: Application No. 10/942,609, "Apparatus and method for capturing the motion of a performer", September 15th, 2004; September 10, 2004, Application No. 10/942,413 "Apparatus and method for capturing the expression of a performer"; February 25, 2005 App Application No. 11/066,954 "Apparatus and method for improving marker identification within a motion capture system"; March 10, 2005 11/077,628 "Apparatus and method for performing motion capture using shutter synchronization" Application No. 11/255,854, October 20, 2005, “Apparatus and 139852.doc -127- 200952495 method for performing motion capture using a random pattern on capture surfaces”; 1st of application on June 7, 2006 / 449,131 "System and method for performing motion capture using sulphur application techniques"; "System and method for performing motion capture by strobing a fluorescent lamp"; June 7, 2006 Japanese Application No. 1/449, 127 "System and method for three dimensional capture of stop-motion animated characters", each of which has been captured by the assignee of this CIP application) How the live broadcast performance results in a very smooth capture surface that achieves a high polygon count tracking surface (i.e., the polygon motion accurately follows the motion of the face). Finally, photo-realistic results are produced when the live broadcast of the live video is mapped onto the tracking surface to create a textured surface. Although current GPU technology is capable of reproducing many polygons in the tracking surface and texture and illuminating the surface in real time, if the polygon and texture each frame time changes (which will produce the most photorealistic result), it will quickly All available RAM of a modern PC or video game console is consumed. Using the stream geometry technique described above, the geometry is continuously fed into the application/game server 1521-1525 so that it can continually animate a photo-realistic face to allow for almost indistinguishable The video game of the facial face of the live action becomes practical. Integration of Linear Content and Interactive Features 139852, doc -128· 200952495 Movies, TV shows and audio materials (collectively, “Linear Content”) are widely available in many forms for home and office users. Linear content is available on physical media such as CD, DVD, HD-DVD and Blu-ray media. It can also be recorded by DVRs from satellite and cable TV broadcasts. In addition, it can be used via pay-as-you-go (PPV) content for satellite and cable τν and video on demand (VOD) on cable TV. Increasing linear content is available via the Internet for downloaded content and streaming content. Today, there is really no place to experience all the features associated with linear media. For example, DVD and other video optical media typically have interactive features that are not available at other locations (such as director's "flat, slap" shorts, etc. p-line music sites have cover art that is not normally available on cD. And song information, but not all CDs are available. Websites associated with TV shows often have additional features, blogs, and sometimes comments from actors or creators. Also, in many movie or sports events Often, there are often video games that are often associated with linear media—from distribution (in the case of a movie) or (in the case of sports) to real-world events (eg, player transactions). 210 is very suitable for delivering linear guilt when connecting related content in disparate form. (4) Delivering movie movies is not as challenging as delivering interactive video games, and colocation service 2(7) can line I· Delivery to a variety of devices in a home or office, or delivery to a mobile device. Figure 23 shows an illustration for a colocation service 21 User interface page, which displays the choice of linear content. Ϊ 39852.doc -129. 200952495 However, unlike most linear content delivery systems, colocation service 210 is also capable of delivering relevant interactive components (eg, on DVD) Menus and features, interactive overlays on HD-DVD, and Adobe Flash on the website (as explained below). Therefore, the client device 415 limits no restrictions on which features are available. The escrow system 210 is capable of dynamically and instantly linking linear content to video game content. For example, if a user is watching a Quidditch game in a Harry Potter movie and decides that he or she is willing to try to play Quidditch, then Just click the button and the movie will pause and it will be immediately sent to the Quidditch clip of the Harry Potter video game. After playing Quidditch, click another button and the movie will start over immediately. Photorealistic In the case of graphics and production technology, the video captured by the camera cannot be distinguished from the live action characters, when the user From the transition of the Quidditch game in the live action movie to the Quidditch game in the video game on the colocation service (as described in this article), the two scenes can't actually be distinguished. This is linear content and interaction. The director of both styles (eg, video games) offers a new creative option because the lines between the two worlds become indistinguishable. Using the colocation service architecture shown in Figure 14, you can put in 3D movies. The control of the virtual camera is provided to the viewer. For example, in a scene occurring within the train, it will be possible to allow the viewer to control the virtual camera and look around the train as the story progresses. This assumes all 3D objects in the train ("assets ") and the ability to instantly reproduce these scenes as well as the original movie 139852.doc -130- 200952495 computing power level is available. And even for non-computer-generated entertainment, there are very stimulating interactive features that can be provided. For example, the 2005 film "Pride and Prejudice" has many scenes in the ornately decorated old British mansion. For a particular building, the user can pause the video and then control the camera to patrol the building, or possibly the surrounding area. To accomplish this, a camera with a wide, eye lens can be carried through the building, and when it tracks its position, it is very similar to implementing the prior art φ Apple's QuickTime VR. The various frames will then be converted so that the image is not distorted and then stored along with the movie on the shakuhachi arrays 1511-1512 and played back when the user chooses to continue the virtual tour. In the case of a sports event, a live broadcast of a sports event (such as a basketball game) can be streamed via the colocation service 21 for the user to view (as it would be for the user to see p as desired for a typical TV) After a specific play, the video game of the game (finally the basketball player looks as realistic as the real player Φ normal photo) can catch up with the player starting in the same position, and the user (possibly controlling one player each) can replay to view it. Can it be better than these players? The colocation service 210 described in this article is extremely suitable for supporting this future world because it can withstand the computing power that is impractical to be installed in the home or in most office settings. And large-capacity storage resources, and their computing resources are always up-to-date (in the case of the latest computing hardware available), but in the home background, there will always be households with older PCs and video games. In addition, in the colocation service 210, those using B9852.doc -131- 200952495 are concealed of all this computational complexity, therefore, Even if the user may be using a non-technical system, it is as simple as changing the channel on the TV from the user's point of view. In addition, the user will be able to access all computing power - and computing power will be from any client 415 The experience brought. Multiplayer to the extent that the game is multiplayer, it will be able to communicate not only to the application/game server 1521-1525 via the portal/route 1502 network but also by the network bridge To the Internet with a server or gaming machine that is not executing in the colocation service 210 (not shown when the multiplayer game is played by a computer on the universal Internet, the application/game server 1521-1525 will Has the benefit of extremely fast access to the Internet (compared to what the game system does on the feeder in the home) 'but it will be limited by the capabilities of other computers playing games on slower connections, and potentially The game servers on the Internet are designed to accommodate the least commonalities (the game servers will be home computers on a relatively slow consumer Internet connection) φ Really limited. But when playing a multiplayer game entirely in the colocation service 21 server center, then a great difference can be achieved. The host hosts each application/game server 1521-1525 for the user's game. It will be interconnected with other application/game servers 1521-1525 and any server that centrally controls multiplayer games with extremely high speed, very low latency connectivity and a large, very fast storage array host. For example, if SuperSpeed Ethernet is used to enter the 1502 network, then the application/game server ι521_1525 will be communicated to each other and communicated to a potential of only 1 139852 at a bite/second speed. .doc -132- 200952495 Any server that centrally controls the central control of multiplayer games in milliseconds or less than 1 millisecond. In addition, RAID arrays 1511_1512 will be able to respond very quickly and then transfer data at a gigabit per second rate. As an example, if a user customizes a character in appearance and clothing so that the character has a large number of geometric shapes and behaviors unique to the character, the game user is limited to playing on a PC or a game console in the home. In the prior art system, if the character enters the field of view of another user, the user Φ will have to wait until a long slow download is completed in order to load all geometry and behavioral data into his computer. In the colocation service 210, the same download can be superior to the ultra-high-speed Ethernet from the RAID array 1511·1512 at a rate of one billion bits per second. Even if the home user has an 8 Mbps Internet connection (which is extremely fast based on today's standards), the ultra-fast Ethernet network is 1x faster. Therefore, it takes less than a second to spend a minute on a fast internet connection on a super-fast Ethernet. Top Player Grouping and Tournament φ Colocation Service 210 is ideal for tournaments. Since no game is executed in the local client, there is no chance for the user to cheat. Again, due to the ability of the output route 1540 to multicast the UDp stream, the colocation service 210 is sufficient to simultaneously broadcast a larger tournament to thousands of viewers. In fact, when there are so popular that thousands of users are receiving a particular stream of the same stream (for example, showing a larger tournament view), the video stream is sent to a content delivery network (CDN) (such as Akamai). or

Limehght) may be more efficient for distribution to a large number of client devices 415. 139852.doc 133· 200952495 When CDN is used to display the game viewfinder page of the top player group, a similar level of efficiency can be obtained. For larger tournaments, live celebrity commentators can be used to provide comments during the special competition. Although a large number of users will be watching a larger tournament, a relatively small number will be played in the tournament. The audio from the celebrity commentator can be routed to the application/φ game server 1521-1525, which can host any of the bystander mode replicas of the game in the tournament_player and hosted tournaments, and the audio can be Added to the game audio. The video of the celebrity commentator can be overlaid on the game (and possibly just on the bystander view). Web page loading accelerates the World Wide Web's main delivery agreement, hypertext transfer protocol (HTT^ is conceived and defined in an era where only commercial high-speed Internet connections are used, and online consumers use dial-up modems or ISDN At this time, the "gold standard" for fast connection is the τι line, which symmetrically provides a data rate of 15 Mbps Mbps (that is, equal data rates in both directions). Today's situation is completely different "a large number of developed worlds" The average home connection speed via a dSL or cable modem connection has a much higher downlink data rate than the Dingsian line. In fact, in some parts of the world, fiber-to-the-curb is positive Introduce data rates up to 50 ]^1)1)3 to 1 Mbps into the home. Unfortunately, HTTP has not been architected (and not implemented) to effectively utilize these dramatic speed improvements. The website is a collection of files on the remote server. Quite simply, HTTP asks for the first configuration, waiting to download the broadcast, and then 139852.doc -134- 200952495 requests the second broadcast 'waiting for the next case. In fact, Ηττρ allows more than one "open connection" (ie, more than one file at a time), but because of the agreed standards (and the desire to prevent the web server from being overloaded), only a very small number of open connection. In addition, because web pages are structured, Pan browsers are often unaware of multiple simultaneous pages that can be used for immediate downloading (ie, only after analyzing a page: it is necessary to download new files like images) . Therefore, the files on the website are essentially loaded one by one. In addition, due to the request and response protocol used by HTTP, there is a roughly 100 millisecond delay associated with each file loaded (accessing a typical US network server). In the case of a relatively low speed connection, this does not introduce a lot of problems, because the download time for the file itself determines the waiting time of the web page. However, as the connection speed increases (especially in the case of complex web pages), it starts to cause problems. In the example shown in Figure 24, a typical commercial website is shown (this particular φ website is from a larger sports shoe trademark website with 54 files. The files include HTML, CSS, JPEG, PHP, JavaScript, and Flash files, and include Video content. A total of 15 bytes must be loaded before the live web page (that is, the user can click on it and start using it). There are many reasons for a large number of files. First of all, it is a complex and sophisticated web page. 'and secondly, it is a webpage based on a combination of information about the user accessing the page (eg, from which country the user came from, what language the user had previously purchased, etc.), and depending on all of these factors And download different slots. However, it is still a very typical commercial web page. 139852.doc -135· 200952495 Figure = Show does not increase with the connection speed before the live broadcast of the web page. At 1.5 Mbps connection speed 24 〇1, using a conventional web server with a conventional web browser, it takes 13.5 before the live webpage, at 2 Mbps connection speed 2402, The entry time is reduced to 6.5 seconds, or approximately doubled, but at 96 Mbps connection speed 2403, the load time is only reduced to approximately 5.9 seconds. This is because at this high download speed, the file itself is downloaded. The time is the smallest, but each file is approximately 100 milliseconds. The delay is still maintained, resulting in a delay of 54 files *1〇〇 milliseconds = 5.4 seconds. Therefore, no matter how fast the connection to the family, this site is before the live broadcast. It will always take at least 5.4 seconds. Another factor is the server side of the queue; each HTTP request is added after the queue, so on the busy server, this will have a significant impact, because for the network Each small item that the server gets is 'HTTP requests need to wait for it to return. One way to solve this problem is to discard or redefine HTTP. Or 'may make the site owner better merge their files into a single file 10 ( For example, in Adobe Flash format, however, as a practical matter, this company and many others have invested heavily in their website architecture. In addition, although some families have 12-100 Mbps connections But most families still have slower speeds, and HTTP does work well at slow speeds. An alternative is to host a web browser host on the app/game server 1521-1525 and The file host for the web server is hosted on the RAID array 1511-1512 (or potentially hosted in the RAM on the application/game server 1521-1 525 hosted on the web browser) On the end of the storage). Because of the very fast interconnection via the incoming route 1502 (or to the local storage 139852.doc -136-200952495), it is not a delay of 1 millisecond per file under HTTP. There is a minimum latency per file under HTTP. Next, instead of allowing the user in the family to access the web page via ΗΤτρ, the user can access the web page via the user terminal 41 5 . Then, even under a 5 Mbps connection (because this page does not require a lot of bandwidth for its video), the web page will be live broadcast in less than 1 second per line 2400. In essence, there will be no delay before the web browser displayed on the application/game server 1521_1525 displays the live broadcast page, and will not exist until the user terminal 415 displays the video output from the web browser. The delay measured by the debt. When the user uses the mouse to search the webpage and/or type a word on the webpage, 'the user's input information is sent to the web browser executed on the application server 1521-1525, and the web browser will Respond accordingly. One of the disadvantages of this method is that if the compressor is arbitrarily transmitting video data, then the bandwidth is used, even if the web page becomes static. This can be remedied by the group state compressor to transmit data only when (and if) the web page changes and then only transfer the data to the changed page portion. When there are some web pages with a flashing slogan or the like that are constantly changing, the web pages tend to be annoying' and unless the reason is to move something (e.g., a video clip), the web page is typically static. For such web pages, it may be that the use of colocation service 210 will transfer less data (compared to the conventional web server), since only the actual displayed image will be transmitted, no thin client executable code And there are no large objects that may never be viewed (such as scrolling over images). 139852.doc -137· 200952495 Therefore, using the colocation service 210 to host the old webpages can reduce the webpage loading time to the extent that the webpage is similar to changing the frequency on the TV: effectively live broadcast instantly The page. Promoting the Debugging of Games and Applications As mentioned earlier, video games and applications with instant graphics are very complex applications and often have defects when they are released into the field. Although the software developer will get feedback from the user about the defect and it may have a way to return the machine state after the crash, it is exactly what is causing the game or instant application to crash or not. Proper execution is very difficult. When the game or application is executed in the colocation service 21, the video/audio output of the game or application is recorded on the delay buffer 1515. In addition, the watchdog process executes each application/game server 1521-1525, which will periodically report to the colocation service control system 401 that the application/game server 1521·1525 is smoothly executing. Row. If the watchdog process fails to report, the server control system 401 will attempt to communicate with the application/game server 1521-1 525, and if successful, will collect whatever machine state is available. Information about whatever is available is sent to the software developer along with the video/audio recorded by the delay buffer 1515. Therefore, when the game or application software developer gets a notification of a crash from the colocation service 210, it gets a record of the frame of the frame that caused the crash. This information can be extremely valuable in tracking down defects and fixing them. 0 139852.doc -138- 200952495 Also note that when the app/game server 1521_1525 crashes, the server is restarted at the most recent restartable time. And apologize for technical difficulties by providing the message to the user. Resource sharing and cost savings

The systems shown in Figures 4a and 4b provide multiple benefits to both end users and game and application developers. For example, in general, home and office client systems (e.g., PCs or game consoles) are only in use for a small percentage of the week. According to Nielsen Entertainment "Active Player Benchmark Study" (http://www.prnewswire.com/cgi-bin/stories.pl?ACCT=104&STORY=/www/story/l 0-05-2006/ 〇 〇〇4446115&EDATE=) October 5th, 2006, the active gamer spends an average of 14 hours a week on the video game console and plays on the handheld device for about 17 hours a week. The report also states that active gamers average 13 hours a week for all game play activities, including consoles, handheld devices, and PC game play. Considering the higher number of console video game play time, there is 24*7;=168 hours a week, he implies that in the middle of active players, the video game console is only 17/168 = 1〇〇/〇 in a week. The little cockroach is in use +. Or, at 9G% of the time, the high cost of the video game console's idle m-game console and the fact that the manufacturer is funding the devices' is a very inefficient use of resources. PCs in the business are usually only used in weeks-minutes, especially for non-portable desktop PCs that are often required by door-end applications (such as 'Aut()desk Maya). Hour and holiday operations and some c (for example, the portable system used for work at night is used in 139852.doc • 139· 200952495 for all hours and holidays 'but most business activities tend to be given The business time zone is concentrated from about 9 am to 5 PM from Monday to Friday, with less false and disconnected time (such as 'lunch), and because most PC usage occurs when the user actively uses the PC, so Follow: Desktop PCs tend to follow these hours of operation. If assuming a constant use of pc from 9 AM to 5 PM for five days of the week, then he would imply that pc is 4〇/168=24% of the week. It is used in the hour. High-performance desktop PCs are very expensive investments for business, and this reflects very low utilization. Schools that teach on desktops can be even smaller in a week. Use computer in the rate, and although it is regarded as the hour of teaching And change, but most of the teaching takes place during the hours of the week from Friday to Friday. Therefore, in general, pc and video game consoles are only used in small fractions of a week. Worth / idea, because many people They work in business or at school during the hours of the non-holiday Monday-Friday, so these people usually do not play video games during these hours, and therefore when they do play video games, they are usually tied to other Hours (such as evenings, weekends, and fakes).

Given the configuration of the colocation service shown in Figure 4a, the usage patterns interspersed in the above two paragraphs result in very efficient use of resources. Obviously, there is a limit to the number of users who can be served by the hosted service at a given time, especially if the user needs instant responsiveness for complex applications such as cutting-edge 30 video games. In case. However, not in the video game console in the family or by the commercial use of pc (which usually = most days (four) idle placement), the feeding machine can (4) reuse with the user in 5 no. For example, the high-performance server with high-performance dual CPU and dual GPU H9852.doc -140- 200952495 and a large amount of RAM 402 can be used by businesses and schools on non-holiday 9 AM to 5 PM, but by playing cutting-edge video games. The player uses it at night, on weekends and on holidays. Similarly, low-performance games can be utilized by businesses and schools during low-performance servers 402 with Celeron CPUs, GPU-free (or very low-end GPUs), and limited RAM during business hours and low-performance games can be used in non-commercial hours. Use low-performance server 4〇2 during the period. In addition, in the case of the colocation service configuration described herein, the © resource is effectively shared among thousands (if not millions) of users. In general, online services only use a small percentage of their total user base to use the service at a given time. If you consider the Nieisen video game usage statistics listed previously, it is easy to understand why. If the active player plays the console game for only 17 hours a week, and if the peak usage time of the game is assumed to be at night (5-12 AM, 7*5 days = 35 hours/week) and weekend (8 AM-12 AM, During a typical non-working, non-commercial hour of 16*2=32 hours/week, for a 17-hour game, there are 35 + 32=65 peaks in a week. It is difficult to estimate the exact peak user load on the system for a number of reasons: some users will play during the extra-peak time, there may be a cluster peak of the user at a particular daytime, and the peak time may be affected by the type of game being played ( For example, a child's game will probably play at an earlier time in the evening.) However, it is assumed that when the player is likely to play the game, the average number of hours played by the player is much less than the number of hours in the day, then only the hosted service A user of one of the 210 rate fractions will use the colocation service at a given time. For this analysis, the peak load will be assumed to be 12 5%. Therefore, only 12.5% of the computing, compression and bandwidth resources use 139852.doc -141 - 200952495 at a given time, resulting in only 12.5% of the hardware cost to support a given user due to resource reuse. One of the performance games is given a level. In addition, it is assumed that some games and applications require more computing power than others, and resources can be dynamically allocated based on the game played by the user or the application executed by the user. Therefore, users who choose a low-performance game or application will be assigned a low-performance (lower-cost) server 402, and users who choose a high-performance game or application will be assigned a high-performance (more expensive) vocalist 402. In fact, a given game or application may have a lower performance and higher performance area of the game or application, and the user can switch from one server 402 to another between games or applications. 402, to maintain the user's execution on the lowest cost server 402 that satisfies the needs of the game or application. Note that rajd array 405, which is much faster than a single disk, will be used by even low performance server 402, which has the benefit of faster disk transfer rates. Thus, the average cost per server 402 across all played games or applications used is much less than the cost of playing most of the most expensive servers 402 of the highest performance game or application, however, even the low performance server 402 Disk performance benefits will be obtained from RAID array 405. In addition, the server 402 in the colocation service 210 may be only a PC motherboard that does not have a disk or peripheral interface (different from the network interface), and just can be integrated down to a fast network just to the SAN 403. A single wafer of the interface. Also, the RAID array 405 may be shared among a much larger number of users than there are disks, so the cost per disk for each active user will be much less than that of a disk drive. All of this device will likely reside in a rack that is controlled by the environment 139852.doc -142- 200952495 in the server room ring && + β 〇 mourning. If the server 402 fails, it can be repaired or replaced at the host escrow service 2 ι. In contrast, the console in the home or office must be sturdy, must be able to 'freely wear and tear to prevent being hit or dropped by a separate appliance' and have a shell with at least one disk Machines must survive unfavorable environmental conditions (for example, being forced into a heated AV cabinet with other appliances), require service assurance, must be packaged and shipped, and sold by retailers who may receive retail profit (5). In addition, the PC or game console must be configured to meet the peak performance of the most computationally intensive expected game or application that will be used at some point in the future, even for lower performance games or applications (or games or applications). District) may also play most of the time. In addition, if a PC or console fails, it can be a costly and time consuming process (which adversely affects manufacturers, users, and software developers). Therefore, it is assumed that the system shown in FIG. 4a provides the user with an experience equivalent to the experience of the local computing resource for the user to experience the computing power in the home, office, or school, via FIG. 4a. The architecture shown in the middle provides a much lower computational power. Eliminate the need for upgrades. Users don't have to worry about upgrading their PCs and/or consoles to play new games or handle newer applications. Any game or application on the colocation service 210 (regardless of the type of server 402 that the game or application requires) can be used by the user and all games and applications are executed in close proximity (ie, fast) Loaded from the RAID array 405 or the local storage on the server 402) and suitably with the latest updates and defects 139852.doc -143- 200952495 fixes (ie, the software developer will be able to choose to execute the given game) Or the ideal server configuration of the application server 402, and then the server 402 is configured with the best driver, and then over time, the developer will be able to provide updates, bug fixes, etc. to the host generation over time. Manage all copies of the game or application in the 21st). In fact, after the user starts using the colocation service 21, the user may find that the game and the application continue to provide a better experience (eg, via update and/or defect repair) and may be the following: user A year later, it was discovered that new games or applications could be used on services 210 that utilized computing technology (eg, higher performance GPlJ), which did not even exist a year ago, so it would be impossible for users to purchase A year ago technology that played games or executed applications after one year. Because the computing resources for playing games or executing applications are for the user. Invisible (that is, 'from the user's point of view, the user only chooses a game or application that is close to being executed immediately - more like the user changes the channel on the TV), so the user's hardware will be in the user. It has not been "upgraded" even if it is not known to be upgraded. Eliminating the need for backups Another major problem for users in business, schools, and homes is backup. If the disk fails, or if there is an inadvertent erasure, the information stored in the local pC or video game console (for example, 'in the case of the console, the user's game results and ratings) may be lost. There are many available applications that provide manual or automatic backup for PCs, and the game console status can be uploaded to an online server for backup, but the local backup is usually copied to a safe and organized one. The other end of the magnetic butterfly (or 139852.doc 200952495 other non-volatile storage devices), and because of the slow upstream speed available via a typical low-cost Internet connection, the backup of online services is often limited. In the hosted service 21 of Figure 4a, the data stored in the RAID array 405 can be configured using prior art RAID configuration techniques well known to those skilled in the art, such that when the disk fails, The data will not be lost and the technician at the server center hosting the failed disk will be notified, and then the technician will replace the disk, which will then be automatically updated to cause the RAID array to tolerate the failure again. In addition, since all drives are close to each other with a fast local network via SAN 4〇3, all disk systems are configured in the server center to be periodically backed up to secondary storage (which can be stored in It is not difficult to relocate at the center of the server or easily. From the point of view of the hosted service user, the data is always completely secure and never has to be considered for backup. Access to the presentation Users often want to try out the game or 10 applications before purchasing a game or application. As previously described, there are demonstrations (the "verb" form means a trial demo version, and the demo version is also called a "demo", but as a noun) a prior art component of a game and application, but each of them Subject to restrictions and/or inconvenience. With the colocation service 21, it is easy and convenient for the user to try out the demo. In effect, the user performs the presentation and trials the presentation via a user interface, such as the user interface described below. The presentation will load almost instantly onto the server 402 suitable for the presentation, and it will execute exactly like any other game or application. Regardless of whether the presentation requires a very efficient server 139852.doc -145- 200952495 402 is still a low-performance server 402, and regardless of the type of home or office client used by the user, from the perspective of the user, the presentation Will work. A software publisher of a game demo or application demo will be able to precisely control how long the demo and trials are allowed for the user to try, and of course, the demo may include providing the user with a full version of the game or application being demonstrated. The user interface element of the access opportunity. Because the decision may be lower than the cost price or provided free of charge, some users may try to use repeated presentations (especially to repeatedly play interesting game demos). The colocation service 210 can use various techniques to limit the presentation usage for a given user. The most straightforward method is to establish a user ID for each user and limit the number of times the given user ID is allowed to play the presentation. However, the user can set multiple user IDs, especially if they are free. One technique for solving this problem limits the number of times a given client 4 5 can play a presentation. If the client is a standalone device, the device will have a serial number, and the colocation service 210 can limit the presentation to have a φ The number of client accesses of the serial number. If the client 415 is executing as a software on a PC or other device, the host can be assigned a serial number and stored on the PC and used to limit the presentation usage, but the PC is assumed to be available to the user. Reprogramming, and the sequence number is erased or changed, then another option is that the colocation service 210 maintains the PC network adapter media access control (MAC) address (and/or other machine specific identifiers, such as hard Record the number of the disc player, etc.) and limit the use of the demo to the MAC address. It is assumed that the MAC address of the network adapter can be changed, however, this is not an extremely simple method. Another method is to limit the number of times a presentation can be played to a given IP address. 139852.doc -146- 200952495 Although Ip addresses can be reassigned periodically by eDonkey modems and DSL providers', they do not occur very frequently in practice, and if determinable (eg 'by contacting an ISP The IP is in a block of IP addresses for residential DSL or cable modem accesses' and typically can be used for a small number of presentations for a given home. Also, there may be multiple devices in the home following a ΝΑτ router sharing the same Ip address, but typically in a residential setting, there will be a limited number of such devices. If the IP address is in the Servo Business Area 则 block, then a larger number of presentations for the business can be established. However, in the end, all combinations of the previously described methods are the best way to limit the number of presentations on the PC. While there may not be a very simple way to make the determined and technically savvy user likely to be limited in the number of replayed presentations, creating a large number of obstacles can create enough obstacles to make most pc users not worth the trouble To abuse the demo system, and instead, use the demo when it wants to try out new games and applications. Benefits for Schools, Business, and Other Institutions 显 Significant benefits especially occur in commercial, school, and other institutions that utilize the systems shown in Figure 4a. Businesses and schools have substantial costs associated with installing, maintaining, and upgrading PCs, especially when talking about PCs that perform high-performance applications such as Μ. As previously stated, pcs are typically only utilized in the hourly rate of force, and as in the home, the cost of a PC with a given monthly performance is much higher in an office-to-school environment than in a business. The cost in the server center environment. In the case of larger businesses or schools (eg, large universities), the IT gates of these entities sigh at the feeder center and maintain computers that are accessed via LAN-level connections and remote access 139852.doc •147- 200952495 Can be practical. There are many solutions for remote access to a computer via a LAN or via a private high frequency wide connection between offices. For example, 'through Microsoft's Windows terminal server, or through a virtual network computing application (such as VNC from RealVNC Ltd.) or through a thin client-side component from Sun Microsystems, users can get Remote access by a PC or server has a range of qualities in graphical response time and user experience. In addition, these self-managed server centers are typically dedicated to a single business or school and therefore cannot be utilized when disparate applications (eg, entertainment and business applications) utilize the same computing resources at different times of the week. The overlap of possible uses. Therefore, many businesses and schools lack the scale, resources, or expertise to independently set up a server center with a network connection to each user's LAN speed. In fact, a large percentage of schools and businesses have the same Internet connections as homes (eg, 'DSL, cable modems'). However, these organizations may still have the need for very high performance calculations (either periodically or periodically). For example, a small construction company may have only a small number of architects, with relatively modest computing needs when designing, but it may periodically require very efficient 3D calculations (eg 'when built for users When the new building is designed for 3D crossing). The system shown in Figure 4a is extremely suitable for such organizations. These organizations only need to provide the same type of network connection (e.g., DSL, cable modem) to the home and are typically very inexpensive. It can be used as a client 415, or no PC at all, and utilizes an inexpensive specializer 139852.doc 200952495 which only implements control nickname logic 413 and low latency video decompression 412. These features may have pC. The school of stealing or the problem of damage to specialized components within the PC is particularly attractive. This configuration solves many of the problems for such organizations (and many of these * points are also shared by home users for general purpose computing). For example, operating costs (which ultimately must be passed back to the user in some form to have a viable business) may be much lower because (a) computing resources are tied to other applications that have peak usage times in a week. Sharing, (b) these groups can obtain (and incur costs) access to high-performance computing resources only when needed, and (c) such organizations do not have to provide backup or other maintenance of high-performance computing resources. Resources. Elimination of piracy Other games, applications, interactive movies, etc. may no longer be pirated as they are today. Since the game is executed at the server center, the user does not have access to the basic code, so there is no piracy. Even if the user is going to copy the source code, the user cannot execute the code on a standard game console or home computer. This opens up markets around the world where standard video games are not available (such as 'China'). Resale of games that have already been used is also impossible. For game developers, there are fewer market discontinuities as they are today. With the new generation of technology forcing users and developers to upgrade and game developers depending on the current situation of timely delivery of the hardware platform, the colocation service 210 can be gradually updated as the game requirements change over time. Streaming Interactive Video 139852.doc • 149· 200952495 The above description provides low-latency streaming interactive video based on the Universal Internet (which also implicitly includes audio along with video, as used in this article) The new I-be basic concept enables multiple applications. Prior art systems that provide streaming video over the Internet have only those applications that can be implemented by high latency interactions. For example, basic playback controls for linear video (e.g., pause, rewind, fast forward) work properly at high latency' and it is possible to select among the linear video feeds.

As stated earlier, the nature of some video games allows them to be played at high latency. However, the high latency (or low compression ratio) of prior art methods for streaming video severely limits the potential application of streaming video or narrows its deployment to a specialized network environment, and even in such environments, The prior art also introduces a substantial burden on the network. The techniques described in this article open the door to a variety of applications that are likely to be possible via low-latency streaming interactive video over the Internet, especially through consumer-grade Internet connections. In fact, it is sufficient to provide enhancements by effectively any amount of computing power, any amount of fast storage, and extremely fast network connections between powerful ships II under the user-side device that is typically small with the client 465 of Figure 乜. The user experience 'enables a new era of computing. [Because the bandwidth requirements do not increase as the computing power of the system grows (ie, because the bandwidth requirements are only about display resolution, quality, and frame rate), once the broadband Internet connectivity, The ubiquitous (eg, m-wide low-latency wireless coverage), reliable, and high enough bandwidth to meet the needs of all users' display devices 422, will be typical consumers and 139,852.doc 200952495 Business applications are necessary for thick clients (such as executing wind〇ws,

PCs or mobile phones such as Linux, OSX, etc. are even thin client (such as Adobe Flash or Java). The emergence of streaming interactive video has led to a reconsideration of assumptions about the structure of the computing architecture. This is an example of the colocation service 2U) word processor center embodiment shown in FIG. A video path feedback loop for the delay buffer buffer and/or the group video 1550, wherein the application/game word processor (10)-ribbed multicast stream interactive video output is immediately or via the path via the path (10) The 1551 is fed back to the application/games feeder 1521_1525 after a selectable delay. This can be achieved by a variety of practical applications (such as 'such as those illustrated in Figures 16, 17, and 2) that are not possible or feasible by prior art servers or local computing architectures. In general architectural features, the feedback loop 1550 provides recursion for streaming interactive video levels because the video can be looped indefinitely when the application requires video. There are many application possibilities that have never been available before. Another key architectural feature is that the video stream is unidirectional. This effectively enables any degree of multicasting of streaming interactive video (in contrast, bi-directional streams such as TC:/IP streams will be generated on the network of incoming port L as the number of users increases) More and more traffic stagnation). The ability to do so within the multicast system is because it allows the system to respond to the growing needs of Internet users (and indeed the 'world's population) to communicate on a pair = even many-to-many basis . Again, the examples discussed in this article ^Examples of the use of both H-Video Recursion and Multicast (such as Figures 139852.doc -151 - 200952495 16) are only the tip of a very large iceberg with the possibility. The various modules and associated steps may be performed by a specific hardware component (such as a special application integrated circuit ("ASIC") containing the fixed-line logic for performing the steps. ) or by any combination of stylized computer components and custom hardware components. > In an embodiment, the modules can be implemented in a programmable digital signal processor such as the TMS320X architecture (e.g., TMS32® C6®, TMS32〇c5〇(9), ❹, etc.) such as an instrument. ("DSP"). A variety of different DSPs can be used while still adhering to these basic principles. Embodiments may include various steps as (4) above. These steps can be embodied in machine executable instructions that cause a general purpose or special purpose processor to perform particular steps. Various components (such as computer memory, hard disk drives, and input devices) that are not related to these basic principles have been omitted from the circle to avoid mixing the relevant aspects. Elements of the subject matter that are not disclosed may also be provided as a machine-readable medium for storing machine executable fingers. A machine-readable medium can include, but is not limited to, a flash memory, a compact disc, a CD_R〇M, a DVD R〇M, a ram, an EPROM, an EEPR0M, a magnetic or optical card, a broadcast medium, or other type suitable for storing electronic instructions. Machine readable medium. For example, the present invention can be downloaded as a computer program that can be accessed from a remote computer via a communication link (eg, a data modem or a network connection) by means of a data signal embodied in a carrier wave or other communication medium ( For example, the server) is transferred to the requesting computer (eg, the client). It should also be understood that the elements of the subject matter disclosed may also be provided as a computer program product 139852.doc • 152·200952495, which may include a readable medium on which the instructions 15 are stored, and such instructions are available On a stylized computer (for example, processing Ϊ = his electronics) to perform - sequence operations. Alternatively, the operations are performed by a combination of hardware and software. The machine-readable medium can include, but is not limited to, a flexible disk, a compact disk, a guilloche, and a magneto-optical disk, a rom, a ram EPROM, an EEPRQM, a magnetic or optical card, a communication medium, or other types of media suitable for: storing electronic instructions. / Machine-readable medium. For example, the elements of the disclosed subject matter can be downloaded as a computer program product, the program can be transmitted via a communication link (for example, a data machine or a network connection) by means of a carrier wave or other communication medium. The data signals are transmitted from the remote computer or electronic device to the requesting process. Further, although the subject matter disclosed has been described in connection with the specific embodiments, numerous modifications and changes are appropriately within the scope of the present disclosure. The specification and drawings are to be regarded as illustrative and not restrictive. FIG. 1 illustrates the architecture of a prior art video game system. Figures 2a-2b illustrate a high level system architecture in accordance with an embodiment. Figure 3 illustrates the actual, nominal and required data rate for communication between the client and the server. Figure 4a illustrates an embodiment in accordance with an embodiment. Host colocation services and clients used. Figure 4b illustrates an exemplary delay associated with communication between the client and the colocation service. Figure 4c illustrates a client device in accordance with an embodiment. 139852.doc -153- Figure 4d illustrates a client device in accordance with another embodiment. Figure 4e illustrates an example block diagram of the client device of Figure 4c. Figure 4f illustrates an example block diagram of the client device of Figure 4d. An example form of video compression used in the embodiment. Figure 6a illustrates an example form of video compression that may be used in another embodiment. Figure 6b illustrates the sanding associated with transmitting a low complexity, low motion video sequence. The peak value in the bucket rate illustrates the peak in the data rate associated with transmitting a high complexity, high motion video sequence. Figures 7a through 7b illustrate an example video compression technique used in an embodiment. Additional example video compression techniques used in the embodiments. Figures 9a through 9e illustrate example techniques used in the embodiment for mitigating data rate _ peaks. Figure 1a to Figure 1 〇b illustrates an embodiment of effectively packaging image image blocks in a package. Figures 11a through Ud illustrate an embodiment using forward error correction techniques. Circle 12 illustrates one embodiment of compression using a multi-core processing unit. Figures 13a-13b illustrate geolocation and communication between colocation services in accordance with various embodiments. Figure 14 illustrates an illustration associated with communication between a client disk master and its host. 139852.doc -154- 200952495 Figure 15 illustrates an example colocation service server central architecture. Figure 16 illustrates an example screen shot of an embodiment of a user interface including a plurality of live video windows. This user interface is shown after selecting a particular video window. Figure 18 illustrates the user interface of Figure 17 after a particular video window has been enlarged to full screen size. Figure 19 illustrates example collaborative user video material overlaid on a multiplayer game screen. Figure 2 shows (4) the instance user page of the game player on the colocation service. Figure 21 illustrates an example 3D interactive advertisement. Figure 22 illustrates an example sequence of steps for surface capture from a live broadcast performance to produce a photorealistic image with a textured surface. Figure 23 illustrates an example user interface page that allows selection of linear media content. Figure 24 is a graph illustrating the amount of time elapsed and the connection speed before the live web page is live. [Main component symbol description] 100 CPU/GPU 101 Random access memory (RAM) 102 Display device (SDTV/HDTV or computer monitor) 103 Hard disk drive 104 Optical media drive/Optical media/CD player 105 Network connection 139852 .doc -155· 200952495 ❹ ❿ 106 Game Controller 205 Client Device/Client/Client Platform 206 Internet 210 Hosting Service/Hosting System 211 User Location 220 Game or Application Software Developer 221 Input Device 222 Display Device/Monitor or TV 301 Nominal Maximum Data Rate 302 Available Maximum Data Rate 303 Required Data Rate 401 Hosted Service Control System/Host Managed Service Control Server 402 Server 403 Storage Area Network Road (SAN) 404 Video Compression Logic/Video Compressor 405 Independent Redundant Disk Array (RAID) 406 Control Signal 406a Control Signal 406b Control Signal 409 Routing Logic 410 Internet 412 Low Latency Video Decompression Logic 413 Control Signal Logic 139852 .doc -156· 200952495 ❹ 415 Client/Client Device/User Computer 420 Display Device 421 Input Device 422 Display Device/Display 440 Internet Bypass Service 441 Presence Point 442 WAN Interface 443 Firewall/Router/NAT (Network Address Translation) Device 444 WAN Interface 451 Single for Sending Control Signals To the transmission time 452, the route to/from the user's premises route/user route 453 arrow/round trip delay/user ISP 454 universal internet delay 455 one-way route 456 maximum one-way delay 457 arrow/compression 458 video decompression 462 Ethernet jack/internet connection 463 HDMI (High Definition Multimedia Interface), Connector 464 Display Capability 465 Client Device / Client 466 Shading Glasses 468 Display Device / SDTV (Standard Definition TV) or 139852 .doc -157· 200952495 HDTV (High Definition Television) / Monitor 469 Input Device 475 Client Device 476 Flash Memory 477 Video Camera 478 479 480 481

Display Devices Bluetooth Input Devices / Wired USB Input Devices Bus Bar Ethernet Interface 482 WiFi Subsystem 483 Control CPU / Control Processor 484 Bluetooth Interface 485 USB Subsystem 486 487 488 489

Video Decompressor Video Output Subsystem Audio Decompression Subsystem Audio Output Subsystem 490 HDMI Interface

491 DVI-I 492 S-Video 493 Composite Video 494 Digital Interface 495 Stereo Analog Interface 497 Ethernet 139852.doc -158- 200952495 499 Power 501 Uncompressed Video Frame 502 Uncompressed Video Frame 503 Compressed video frame 511 compressed frame 512 compressed frame 513 compressed frame 520 compressed logic Ο ^ 559-561 uncompressed video frame 611 I frame 612-613 Ρ frame 620 Compression 621 Rated Maximum Data Rate 622 623 φ 624 633 Available Maximum Data Rate / Available Data Rate I Frame Required Peak Data Rate / Peak Data Rate Video Stream Data Rate / Video Stream Data Rate Sequence I Frame Peak 634 635 Video Stream data rate sequence/video stream data rate Ρ frame peak 636 Ρ frame peak 670 Β frame 671 I frame 701-760 uncompressed frame 711-770 R frame 139852.doc -159- 200952495

721-770 Uncompressed frame 731-780 R frame 805 Mobile character 811 R frame 812 R frame 922 Available maximum data rate 934 Video stream data rate 941 Peak data rate / peak maximum data rate 942 2 times peak data Rate 943 3 times peak data rate 944 4 times peak data rate 952 frame 2 times peak 953 flattened 2 times peak 954 frame 4 times peak 955 flattened 4 times peak 961 uncompressed frame 1 962 without Compressed frame 2 963 Uncompressed frame 3 964 Uncompressed frame 4 965 Uncompressed frame 5 966 Uncompressed frame 6 967 Uncompressed frame 7 968 Uncompressed Box 8 969 Uncompressed frame 9 139852.doc -160- 200952495

970 uncompressed frame 10 981 compressed frame 1 982 compressed frame 2 983 compressed frame 3 985 compressed frame 5 986 compressed frame 6 987 compressed frame 10 991 Transmission time (xmit time) 992 Transmission time 993 Transmission time (2 times peak value) 995 Transmission time 996 Transmission time (4 times peak value) 997 Transmission time 1001-1005 Packet 1010 Image block package package logic 1100 I image block 1101 Forward error Correction Code (FEC) 1103 I Image Block 1104 Forward Error Correction Code (FEC) 1105 Forward Error Correction Code (FEC) 1110 Audio Packet 1111 FEC Code 1112 Audio Packet 1113 FEC Code 139852.doc -161 - 200952495 ❹

1120 User input command 1121 FEC code 1122 User input command 1123 F EC code 1200 Multicore and / or multiprocessor 1201-1208 Core 1205 4x2 configuration / uncompressed frame 1300 Server Center / Host Escrow Service 1301 Network Road 1501 Internet Access 1502 Inbound Routing/Incoming Routing Network 1511 RAID Array 1512 RAID Array 1515 RAID Array/Delay Buffer 1521-1525 Application/Game Server 1529 Uncompressed Video/Audio 1530 Shared Video Compression/Common Cluster/Common Hardware Compression/Common Hardware Compressor/Common Video Compression Hardware 1539 Compressed Video/Audio Output 1540 Outbound Routing/Outbound Routing Network 1550 Delay Buffer and/or Grouping Video/Feedback Loop 1551 Path 1552 Path 1599 Outgoing Internet Traffic 139852.doc 162· 200952495

1600 "thumbnail" video window 1601 yellow selection box 1602 game information 1700 video window 1800 arrow 1801 over 1900 video and audio 2000 user's own video 2001 live video 2002 partner 2003 video clip / self-reward clip / 3D DVR from Reward clip 2004 number 2005 icon 2400 line 2401 1.5 Mbps connection speed 2402 12 Mbps connection speed 2403 96 Mbps connection speed HS1-HS6 colocation service server center 139852.doc 163-

Claims (1)

200952495 VII. Patent Application Range: 1. A device comprising: a plurality of servers arranged in a parallel processing architecture to generate a plurality of streams of streaming interactive video controlled by a plurality of users; 70' combines a subset of the plurality of strings generated by the plurality of servers and generates a new stream of low latency stream interactive video; ❹ used to select the single 4 to be operated to provide a user interface Allows a P-interactive operation of low-latency streaming interactive video for multiple strings 139852.doc