TW200949567A - Apparatus for combining recorded application state with application streaming interactive video output - Google Patents

Abstract

An apparatus is described comprising a Power-over-Ethernet interface, a High-Definition Media Interface (HDMI), a Bluetooth interface, a means for decompressing low-latency streaming interactive video and audio from an Internet-based hosting service and outputting the decompressed audio and video through HDMI, and a means for transmitting to the Internet-based hosting service user control information and statistics from received packets. Further, the apparatus is able to receive two interleaved streams of low-latency streaming interactive video.

Description

200949567 VI. Description of the Invention: [Technical Field of the Invention] The present disclosure is generally in the field of a data processing system that improves the ability of a user to manipulate and access audio and video media. This application is the 10th, 315, 460 part of the continuation (CIP) application filed on December 10, 2002, entitled "Apparatus and Method for Wireless Video Gaming". The assignee of the CIP application has been granted. [Prior Art] Since the days of Thomas Edison, recorded audio and film media have become one aspect of society. In the early 20th century, there were widespread releases of recorded audio media (magnetic columns and records) and movie media (coin-operated jukeboxes and movies), but the two technologies are still in their infancy. In the late 1920s, films and audio were combined on the basis of bestsellers, followed by color films and audio. Radio broadcasts have evolved into broadcast-selling audio media that largely support the form of advertising. When the television (TV) broadcast standard was established in the mid-1940s, television and radio were joined in the form of broadcast-selling media to bring previously recorded or live-lived movies to the home. By the middle of the 20th century, most American households had a phonograph record player for playing recorded audio media, a radio for receiving live broadcasts of broadcast audio, and a broadcast audio for live broadcasts. / Video (A/V) media TV. These three "media players" (recorders, radios and TVs) group 139849.doc 200949567 are often combined into a common public speaker cabinet to become the "media center" of the family. Although media choices are limited for consumers, the media “ecology, first” is very stable. Most consumers know how to use Media Play * 且 and enjoy the full range of their abilities. At the same time, publishers of the media, (Λ#for film and television studios' and music companies) are able to distribute their media to both cinemas and families without extensive piracy or “secondary sales” (ie, already Use media resale). Often, publishers do not receive revenue from secondary sales, and therefore 'secondary sales reduce publishers' revenue for new sales of buyers who would otherwise be able to use their own media. Although there are indeed used records sold during the mid-20th century, such sales will not have a major impact on record publishers, because they are different from movies or video spots (which are usually viewed by adults - once or only a few times) Music tracks can be listened to hundreds or even thousands of times. ^ Cognac, therefore, music media is far more "lasting" than film/video media (that is, for adult consumers, it has lasting value). Once you have purchased ^日Η ^, (4) 1 "If the consumer likes the music, then the fee can be kept for a long time. From the middle of the 20th century to the present, the media, the body, the system for the consumer And publishing: 4 and knowing the loss have experienced a series of fundamental changes. In the case of the widespread introduction of audio recorders (especially cassettes with high quality body), there is indeed a high degree of Consumers nowadays a wide range of consumer media, Ren Ba, also bought the "Practical Practice - The Beginning of Piracy." The philanthropist makes the record for the sake of convenience, but the increased consumption * The student of the album collection will enter: Take the clothes and the 'movers will record 139849.doc 200949567 music broadcasted by radio instead of buying the album or tape from the publisher. The emergence of consumer VCR leads to more consumption Convenience, because the VCR can now be set to record TV programs, which can be viewed at a later time, and the VCR also leads to the establishment of the video rental industry, in which movies and TV program designs can be accessed on an "as required" basis.The rapid development of popular home media devices since the mid-1980s has led to unprecedented choice and convenience for consumers, and has led to a rapid expansion of the media publishing market. Today, consumers face multimedia choices and multimedia devices, many of which are tied to specific forms of media or to specific publishers. A keen media consumer may connect a bunch of devices to the TVs and computers in each room of the house, resulting in a "snake" cable to one or more televisions and/or personal computers (PCs) and a group of remote controls. (In the context of this application, the term "personal computer" or "PC" refers to any type of computer suitable for use in a home or office), including desktop computers, Macintosh® or other non-Windows computers, and Windows compatible devices, Unix variants, laptops, etc.). Such devices may include video game consoles, VCRs, DVD players, audio surround sound processors/amplifiers, satellite set-top boxes, cable TV set-top boxes, and the like. In addition, for enthusiastic consumers, there may be multiple devices with similar functions due to compatibility issues. For example, consumers may own both HD-DVD and Blu-ray DVD players, or both Microsoft Xbox® and Sony Playstation® video game systems. In fact, due to the incompatibility of several games across several versions of the game console, consumers may have both XBox and later versions (such as Xbox 360®). Frequently, consumers are confused about which video is used and which remote end is used. Even after placing the disc in the correct player (eg DVD, HD-DVD, Blu-ray, Xbox or Playstation), selecting the video and audio inputs for the device and finding the correct remote control, the consumer still faces Technical challenges. For example, in the case of a wide-screen DVD, the user may need to first determine the correct aspect ratio (eg, 4:3, full, zoom, wide magnification, cinema wide, etc.) and then on their TV or monitor screen. Set the correct aspect ratio. Similarly, the user may need to first determine the correct audio surround sound system format (e.g., AC-3, Dolby Digital, DTS, etc.) and then set the correct audio surround sound system format. Often, consumers are unaware that they may not have access to the media content of their television or audio system (for example, watching a movie that is squeezed in the wrong aspect ratio, or listening to stereo audio instead of surround sound) . Increasingly, Internet-based media devices have been added to the stack of devices. Audio devices like the Sonos® digital music system stream audio directly from the Internet. Similarly, devices like the 8111^13 entertainment player record video and stream it through the home network or via the Internet, where the video can be viewed remotely on the PC. And Internet Protocol Television (IPTV) services provide cable TV-like services via digital subscriber line (DSL) or other home Internet connections. There have also been recent efforts to integrate multiple media functions into a single device, such as the Moxi® Media Center and the PC running the Windows XP Media Center version. While each of these devices provides a little convenience for the functions they perform, each lacks universal and simple access to most media. In addition, often due to the need for expensive processing and / or local materials, these n pieces often cost a number of 7 US dollars to install k ^ another 'such modern consumer electronics devices usually consume a lot of power' Even when idle, it consumes a lot of power, which means it is expensive and wastes energy over time. For example, if the device forgets to cut a device or switch it to a different video input, the device may continue to work. In addition, because none of these devices is a complete solution, it must be integrated with other devices in the home (4), which still leaves the mouse with a mouse line and many remote controls. Moreover, when many (four) work properly on the Internet, they typically provide a more general form of media (compared to the form that might otherwise be available). For example, f 'devices that stream video over the Internet often only stream video material, and can't often accompany dvd interactive "extra billing" streams, such as video "production", games or director reviews. . This is due to the fact that interactive materials are often produced in a specific format intended for the specific device that handles interactivity at the local end. For example, each of Qing, HD-DVD, and Blu-ray Disc has its own specific interactive format. Any home media device or local computer (which may be developed to support all popular formats) will require some degree of sophistication and flexibility that would be too expensive and complicated for consumer operations. 〇 This problem is exacerbated. If a new format is introduced in the future, the local device may not have the hardware capability to support the new format. This means that consumers will have to purchase the upgraded local media device. For example, the old device may not have a higher resolution video or stereo video (for example, each eye 139849.doc 200949567 stream), then the local device may not have the decoding to decode the video. Capabilities, or they may not have hardware for outputting the video in a new format (for example, to achieve stereo vision by 12 〇 fps visual synchronization with shuttered glasses), where 6 〇 fPs Delivered to each eye, if the consumer's video hardware can only support 60 fps video, this option will not be available in the absence of an upgraded hardware purchase). When it comes to cutting-edge interactive media (especially video games), the problem of obsolescence and complexity of media is a serious problem. Modern video game applications are basically divided into four main non-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2, and PS3);

Microsoft Xbox (g^Xb〇x 36〇8; and Nintend〇 8 and

WiiTM; and PC-based games. Each of these platforms is different from the others, so that games written to be executed on one platform are typically not executed on another platform. There may also be compatibility issues between a generation of devices and next-generation devices. Even though most software game developers build software games designed to be independent of the specific platform, in order to execute specific games on a specific platform, a proprietary software layer (which is often referred to as the "game development engine") is needed to adapt the game. Used on a specific platform. Each platform is sold to the consumer in the form of a "console" (i.e., a separate box attached to the TV or monitor/speaker) or itself as a PC. Typically, video games are sold on optical media such as Blu-ray DVD, DVD-ROM or CD_ROM, which contains video games embodied in sophisticated real-time software applications. As home broadband speeds increase, video games are becoming increasingly available for download. 0 139849.doc 200949567 Due to the immediacy and high computing requirements of high-end video games, the special requirements for platform compatibility with video game software are extremely extreme. harsh. For example, we may expect full game compatibility from a generation of video games to next-generation video games (for example, from XBox to XBox 360, or from playstati〇n 2 ("ps2") to Playstation 3 ("PS3")). Sex, as there is a general compatibility of productivity applications (eg, MiCrosoft Word) from pc to another pc with a faster processing unit or core. However, this is not the case for video games. Because #release-generation video game, view

Game manufacturers often seek the highest possible performance for a given price point, so dynamic architectural changes are made to the system so that many games written for use in previous generation systems do not work on later generation systems. For example, 5, based on the χ86 series processor, and XBox 360 is based on

PowerPC series. Techniques can be used to simulate the previous architecture, but it is often impractical to assume that video games are instant applications that achieve the same behavior in the simulation. This is a loss to the publishers of the "Shaokes", video game console manufacturers and video game software publishers. For consumers, it means keeping the old generation of S, the game console and the new video game console connected to the TV to: Play enough for all games. For console manufacturers, their monthly cost associated with the simulation and slower adoption of the new console. And for publishers, +, Gan Li 5, it may be necessary to issue multiple versions of the new game in order to cover the ancient 1 + potential consumers · not only issued each trademark for video games (for example, Λϋ〇Χ , Playstation) version, and often issued for a given quotient ;); eve > < mother version (for example, PS2 and PS3) version. Example 139849.doc -10- 200949567 For the purpose of developing a separate version of the electronic art "Crazy Rugby 08" for XBox, XBox 360, PS2, PS3, Gamecube, &quot;Wii and PC platforms and other platforms. Portable devices such as cellular phones and portable media players also challenge game developers. Increasingly, these devices are connected to a wireless data network and are capable of downloading video games. However, there are a variety of cellular phones and media devices in the market that have many different display resolutions and computing capabilities. Also, because such devices typically have power consumption, cost, and weight constraints, they typically lack high-level graphics acceleration hardware like a graphics processing unit ("GPU") (such as those manufactured by NVIDIA of Santa Clara, CA). Therefore, game software developers typically develop a given game title that is used for many different types of portable devices. The user can find that the given game title is not available for its particular cellular phone or portable media player. In the case of a home game console, hardware platform manufacturers typically ask software developers for a version of the tax on their ability to publish games on their platforms. Honeycomb telephony wireless communication companies often also ask game publishers for royalties to download games to cellular phones. In the case of pc games, there is no royalties paid for publishing games, but game developers often face high costs due to the high consumer service burden used to support multiple PC configurations and possible installation problems. Also, pcs generally have fewer obstacles to piracy of game software because they can be easily re-programmed by technically savvy users and games can be more easily pirated and more easily distributed (eg, via the Internet). Therefore, for software game developer 139849.doc 200949567, 'there are costs and disadvantages found in game consoles, cellular phones and PCs. For game publishers of consoles and PC software, the cost does not stop there. In order to distribute the game through the retail channel, the publisher asks the retailer for a wholesale price below the sale price to give the retailer a profit margin. The publisher usually has to pay for the cost of manufacturing and distributing the physical media that holds the game. Ask the publisher for a "price protection fee" to cover possible unforeseen expenses (such as a game not being sold, or the price of the game is reduced, or the retailer must refund some or all of the wholesale price and / or withdraw the game from the purchaser). 'Retailers usually also ask publishers for the cost of helping to sell games in advertising flyers. The retailer increasingly purchases the game from the user who has completed the game, and then sells the game as a used game, usually not sharing the revenue of the used game with the game publisher. The following facts are added to the game publisher. Cost burden: Games are often pirated and distributed via the Internet for users to download and copy for free. As the Internet broadband speed increases and broadband connectivity becomes more widespread in the US and around the world (specifically In the case of homes and to "network cafes" that lease PCs connected to the Internet, games are increasingly being distributed to PCs or consoles via downloads. Moreover, broadband connections are increasingly used to play multiplayer and large-scale Online game (the two are referred to in this disclosure by the acronym "MMOG"). These changes alleviate some of the costs and issues associated with retail distribution. Downloading online games solves game publishers' The second disadvantage is that the cost of distribution is usually small and there is little or no 139849.doc 200949567 there is a cost of unsold media. Downloaded games are still subject to piracy, and due to eight sizes (often many billions of bytes) it can take a very long time to download. In addition, multiple games can fill up small drives such as Detachable computers are sold together with or together with the tfl game console. However, insofar as the game or MMOG requires an online connection to make the game playable, the piracy problem is alleviated because it is usually required Have a valid user account. Unlike video media (such as video and music) that can be copied by camera ❹ # 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频 视频It is unique and cannot be copied using simple video/audio recordings. Therefore, MMOG can be protected from piracy and thus can support commerce even in areas where copyright law is not enforced and pirated. For example, Vivendi SA's World of Warcraft mm〇g has been successfully deployed and has not been pirated throughout the world. And many online or MM〇G games (such as to use (10) Lab's "Second Life" MM〇G) generate revenue from game operators through the economic model built into the game, where assets can be used with online tools. Brought, sold and even established. Therefore, mechanisms other than the purchase or subscription of conventional game software can be used to pay for the use of online games. Although piracy is often mitigated due to the nature of the online or MMOG, online game operators still face the remaining challenges. Many games require a large number of local (i.e., in-home) processing resources for proper operation on the line or MM〇G. If the user has a low-performance local computer (for example, a computer that does not have an αρυ, such as a low-end laptop), it may not be able to play the game. In addition, as the game console ages, it lags far behind the current state of the art and may not be able to handle higher-order games. Even if it is assumed that the user's own MPC can handle the calculation requirements of the game, there is often installation complexity. ::: Sub-drive incompatibility (for example, if a new game is downloaded, it may be a graphics driver 11, which renders the game dependent on the old version of the graphics driver inoperable). As more games are downloaded, the console may run out of local disk space. When a defect is found and repaired, or if the game is repaired, if the game developer finds that the level of the game is too difficult or too easy to play, the complex game usually receives the downloaded patch from the game developer over time. (patch). These patches require a new download. But sometimes not all users complete all patch downloads. At other times, the downloaded patch introduces additional compatibility or disk space consumption issues. 〃 Also, during game play, large data downloads may be required to provide graphical or behavioral information to the local PC or console. For example, if a user enters a room in a MMOG and encounters a scene or character that is composed of graphical material or has behavior that is not available on the user's home machine, then the material of the scene or character must be downloaded. If the internet connection is not fast enough, this can result in substantial delays during game play. In addition, if the scene or character encountered needs more than the storage space or computing power of the storage space or computing power of the local PC or console, then it may generate a situation in which the user cannot continue in the game, or must be of quality The reduced graphics continue. Therefore, online or MMQG games often limit their storage and/or computational complexity requirements. In addition, it often limits the amount of data transfer during the game. Online or MMOG games can also narrow the market for users of playable games. 139849.doc -14- 200949567 In addition, technically knowledgeable users are increasingly reversing the native copy of the game and modifying the game to make it cheating. Cheating may be as simple as pressing a button with a quicker human hand (for example, to shoot non-$ quickly). The game that supports asset trading in games '=#Defense^ causes fraudulent transactions to involve fraudulent assets with real economic value. On the t-line or the MM0G economic model system, based on these asset transactions, this can have substantial harmful consequences for the game operator. φ, the cost of developing new games, as the console can produce more sophisticated games (such as 'having more realistic graphics (such as instant light tracking), and more realistic behavior (such as real-time physics simulation)) increase. In the early days of the video game industry, video game development and application software development were very similar, that is, most of the development costs were in the development of software (as opposed to the development of graphics, audio and behavioral elements or "assets"). More than that, they can be developed for the development of their software for films with a wide range of special effects. Today, many cutting-edge video game development efforts are more like a software development with special effects than software development. For example, many video games simulate the world of D, and produce characters, props, and environments that are increasingly real (that is, computer graphics that look like real-life images of live action movies). One of the most challenging aspects of photo-realistic game development is to create a human face that can't be distinguished from a living person's face. Facial capture techniques, such as the c〇ntourTM authenticity capture system developed by San Francisco's San Francisc〇, capture the precise geometry of the performer's face and track the performer's face with high resolution while the performer is in motion Precise geometry. This technique allows reproduction on a pc or game console. 139849.doc -15· 200949567 3D Faces The 3D face is virtually indistinguishable from the captured live action face. Accurately capturing and reproducing "photo-realistic" faces is useful in several ways. First, highly recognizable celebrities or athletes are often used in video games (often employed at high cost), and imperfections may be apparent to the user, making the viewing experience distracting or unpleasant. Often, high levels of detail are required to achieve a high degree of photorealistic sensation - potentially requiring the reproduction of large numbers of polygons and high resolution textures (in the case where polygons and/or textures change with the movement of the face on the basis of the frame) .

When a high polygon count scene with detailed texture changes quickly, the game-enabled pc or game console may not have enough ram to store enough polygons and texture data for the required number of animated frames produced in the game clip. . In addition, a single disc drive or a single drive on a poem PC or game console is typically much slower than RAM and typically cannot keep up with the maximum data rate that the GPU can accept in rendering polygons and textures. Current games usually load most polygons and textures into the ram, which means that the given scene is largely affected by the ram's complexity in complexity and duration.

limit. In the case of, for example, facial animation, this may limit the PC or game console to a low resolution without realism that can be paused in the game and loaded into polygons for more frames, barren) Previously created in a limited number of frames, the real-world facet ϋ11 is not similar to the "Loading..." message: the slower the screen is moved by the use of today's complex video games. u is an inherent disadvantage. The next scene is a disk (unless there is another article 139849.doc • 16 - 200949567), otherwise the term “disk” refers to non-volatile optical media or magnetic media and non-semiconductor “flash” memory. Disk media) can take a few seconds or even a few minutes to load. This wastes time and can make gamers quite frustrated. As discussed previously, a large or all delay may be due to the loading time of polygons, textures, or other material from the disk, but may also be the case when the processor and/or GPU in the pc or console is ready for use. When loading the data of the scene, it takes a part of the loading time. For example, a football video game allows players to choose among a large number of players, groups, sports fields, and weather conditions. Therefore, depending on which combination of choices you choose, you may need different polygons, textures, and other assets for the scene (collectively, "objects") (for example, different groups have different colors f and patterns on their uniforms). It is possible to enumerate many or all of the various arrangements and to pre-calculate many or all of the objects and store the objects on a disk for storing games. However, the number of rows of awnings is too large, and the storage of the objects may be too large to be mounted on the disk (or too hard to download). Thus, existing (10) and console systems are typically constrained in both the complexity and playback duration of the 2 scenes and suffer long loading times for complex scenes. The limitations of the prior art videoconferencing system and application software are: the basin, the power, the ./, and the increased use of (for example) the large database 'polygons and textures of 3D objects'. These large databases need to: Used in the console for processing. As mentioned above, when the database 2 = is stored on the disk, the f library can take a long time. However, the 4 database is stored in the remote location and stored via the Internet. 139849.doc 200949567 It is often much more serious when it is worn. In such cases, downloading a large repository can take minutes, hours, or even days. In addition, the capital; ', the library often generates a large amount of fees (for example, a detailed high-altitude sailing 3D model used in games, movies, or historical recordings) and is intended for sale to local end users. However, once the database is downloaded to the local user, there is a risk of being pirated. In many cases, the user only uses the estimated database to see if it is suitable for the user (for example, whether the user's job title has a satisfactory appearance or appearance when the user performs a particular move). And hope to download the database. Long load times can be a hindrance to users who estimate the 3D database before deciding to make a purchase. Similar problems arise in MMOGs (specifically, games that allow users to take advantage of increasingly customized characters). For a PC or game console displaying a character, it needs to be able to access the behavior with 31) geometry (polygon, texture, etc.) and the character's behavior (eg, if the character has a shield, then the shield is strong enough to deflect the spear) database. Usually, when 1^11^〇(} is played by a user for the first time, a large database for characters is available under the initial copy of the game, and the initial copy of the game is available on the game disc or downloaded to the local end. Disk. However, as the game progresses, if the user encounters a character or object that is not available to the database at the local end (for example, if another user has established a squat character), before the person or object can be displayed, The database must be downloaded. This can result in a substantial delay in the game. To defy the cutting-edge and complexity of §fl games, another challenge for video game developers and publishers in the case of prior art video game consoles is 139849.doc -18. 200949567 Yu: Developing video games often takes 2 to 3 years and costs tens of millions of dollars. Assuming that the new video game console platform is introduced at roughly every five years, the game developer The development of these games needs to be started several years before the release of the new game console to make video games available simultaneously when the new platform is released. Several from competitive manufacturers The consoles are sometimes released at about the same time (for example, within a year or two of each other), but it remains to be seen that each console is popular (for example, which console will produce the largest video game software sales). In the recent console loop, 'M1Crosoft χΒοχ 36〇, s〇ny playstati〇n 3 and chests are scheduled to be introduced in about the same general time frame. But in the years before the introduction, game development In essence, the business must “inject” which console platforms will be more successful than others, and invest their development resources accordingly. Film production companies must also share their limited time based on their estimated possible success before the film is released. Production resources. Given the growth in the demand for video games, game production becomes more and more similar to film production, and game production companies routinely invest in their production resources based on their estimates of the future success of specific video games. However, unlike film companies, this bet is not based solely on the success of the production itself; the truth is that it is based on the game. The success of the game console on which you want to execute. Simultaneously playing games on multiple consoles can alleviate the risk, but this extra effort increases costs and often delays the actual release of the game. Application software and users on the PC The environment is becoming more computationally intensive, dynamic and interactive, not only making it more visually appealing to users, but also making it more useful and intuitive. For example, the new Wind〇ws vistaTM operating system is 139849.doc 19 200949567 Both the sequential versions of the Macintosh® operating system have visual effects. High-end graphics tools such as MayaTM from Autodesk offer very cutting-edge 3D rendering and animation capabilities that drive the current state of the art CPU and GPU Limit). However, the calculation requirements for these new tools create many practical questions for users and software developers of such products.

question. No D Because the visual display of the operating system (〇S) must work on a variety of computers (including those that are no longer sold but can still be upgraded with the new OS), 图形3 graphics requirements are largely influenced by 5 The least common point limit for computers used, which typically include computers that do not include a GPU. This severely limits the graphics capabilities of the OS. In addition, battery powered portable computers (e.g., laptops) limit visual display capabilities because high computing activity in the CPU or GPU typically results in higher power consumption and shorter battery life. Portable computers typically include software that automatically reduces processor activity to reduce power consumption when the processor is not being utilized. In some computer models, the user can manually reduce the handling activity. For example, s〇ny's VGN-SZ280P laptop has one side marked "" (for low performance, longer battery life) and the other side marked "Speed" (using For high performance, short battery life). The (4) implementation on a portable computer shall be capable of functioning even if the computer is executed at a rate of one of its peak performance capabilities. Therefore, '(10) graphics performance often remains available for computing power well below current state of the art. High-end, computationally intensive applications such as Maya are often sold, and it is expected that such applications will be used on high-performance PCs. This usually yields much higher efficiency, and is more expensive, less portable, and least common. As a result, such applications have a much more limited target audience than a generic OS (or generic productivity application, similar to Mi(10)oft 0ffice) and are typically sold in much lower quantities than the generic OS software or software. Potential audiences are limited in their steps because it is often difficult for prospective users to try out second-tier computing-intensive applications in advance. For example, suppose students want to know how to use or already know that potential buyers of such applications want to try Maya before they invest in the purchase (this may involve buying a high-end computer that can execute Maya). When a student or potential purchaser can download a copy of the physical media of the version of the City or the aya (4) version, if it is sufficient to perform Maya to its full potential (for example, to handle complex scenes), then it will It is not possible to conduct a comprehensive evaluation of the product. This essentially creates the application, which also increases the selling price because the development cost is usually paid off through a much smaller number of purchases than the general-purpose application. High-priced apps also generate more excitement for individuals who use pirated copies of the application software. Therefore, the high-end application software has been hunted by a publisher of a version of the software, and a lot of efforts have been made to reduce the piracy. However, even when using pirated high-end applications, it is impossible for users to exclude PCs that invest in expensive current state of the art to perform pirated copies. Therefore, although the user can use the software application at the actual retail price of the software application and the score of the shell gun, the user of the 7 software still needs to purchase or obtain an expensive PC in order to fully utilize the application. . 139849.doc •21 · 200949567 ❹ This is also true for users of high-performance pirated video games. Although the second edition can get the game at a rate that is the actual price of the game, it still needs to purchase the expensive computing hardware needed to play the game properly (for example, GPU_ bare PC, or high-end video game control like ΧΒox 360) station). Assuming that video games are usually entertainment for consumers, the extra cost for high-end video game systems can be too expensive. "This situation is in a country where the average annual income of current workers is quite low (relative to the current average annual income of workers in the US). (for example, China) is even worse. As a result, a much smaller percentage of the population has high-end video game systems or high-end PCs. In these countries, users can pay Z to use computers connected to the Internet. "And often, Internet cafes have features that are not highly efficient (such as: ^ allows players to play computationally intensive video) Older models of games (Gpu) or low-end PCs. This is a key factor in the success of games executed on low-end PCs (such as Vivendi's "World of Warcraft", which is highly successful in China and is usually "in the first In the country's Internet cafes.) In contrast, computationally intensive games (first-life) are less likely to be played on PCs installed in Chinese Internet cafes. These games do not actually reach users who are only able to access the low performance % of Internet cafes. There are also barriers to users considering purchasing a video game and first being willing to try out a demo version of the game by downloading = not downloading to their home via the Internet. Video game demos are often a versatile version of the game, with a limit on the amount of game play. This may involve a long process (possibly several hours) of downloading billions of bytes of data before the game can be installed on a PC or console, on the Pc* console. In the case of PC 139849.doc -22- 200949567, it may also involve calculating which special drivers (such as DirectX or OpenGL drivers) the game needs, downloading the version of the original, installing the correct version, and then determining if the PC can play. The game. The latter step may involve determining if the PC has sufficient processing (CPU and GPU) capabilities, sufficient RAM, and a compatible OS (for example, some games are executed on Windows XP and not on Vista). Therefore, after attempting to perform the video game presentation process, the user may find that the video game demo is not playable (given the user's PC configuration). Worse, once the user has downloaded a new drive to try the demo, these drive versions may be incompatible with other games or applications that the user is accustomed to using on the PC, so the installation of the demo may render the previous operation The game or application cannot be operated. These obstacles not only frustrated users, but they also created obstacles for video game software publishers and video game developers to sell their games. Another problem that is not economically viable is related to the fact that a given PC or game console is typically designed to accommodate a particular level of performance requirements for the application and/or game. For example, some PCs have more or less RAM, slower or faster CPUs, and slower or faster GPUs (if they have GPUs). Some games or applications take advantage of the full computing power of a given PC or console, while some games or applications do not take advantage of the full computing power of a given PC or console. If the user's game or application selection does not meet the peak performance capabilities of the local PC or console, the user may be wasting money on the PC or console due to unused features. In the case of a console, the console manufacturer may pay more than the cost of funding the console. 139849.doc -23- 200949567 Another problem that exists in the sale and enjoyment of video games involves allowing (4) people to watch other people's play before the user implements the purchase of the game. There are several prior art methods for recording video games for playback at a later time. For example, U.S. Patent No. 5,558,339 teaches that game state information (including game controller actions) is recorded during a "game play" on a video game client computer (owned by the same or different users). This status information can be used at a later time to replay some or all of the game actions on a video game client computer (e.g., a pc or console). A significant disadvantage of this method is that for viewing the usage of the recorded game I, the user must have a video game user % computer capable of reducing the game and must have a video signal on the computer to make the # replay When recording the state of the game, Lai (4) is expected to be the same. In addition, the Visual Studio application must be written in such a way that there is no possible difference in execution between the recorded game and the played back game. For example, the game graphics system is calculated based on the frame of the frame. For many games, depending on whether the scene is particularly complex or if there are other delays that slow down execution (for example, on a PC, another process may be executing, so that the CPU loop is taken away from the game application), the game logic may sometimes cost - The frame time is short or longer than the frame time to calculate the graphic displayed for the next frame. In such a game, a "threshold value" frame calculated with less time than a frame time (for example, a few CPU clock cycles) may eventually appear. When using the same game state information to re-manage the same scene, it may be easier to spend a few more CPU clock cycles than a frame time (for example, if the internal cpu bus is slightly external to the 139849.doc • 24· 200949567 DRAM busses are out of phase, and even if there is no large delay from another process that takes a few milliseconds of CPU time from game processing, it introduces several cpu cycle time delays). Therefore, when the game is played back, the frame becomes calculated in two frame times instead of a single frame time. Some behaviors are based on the frequency at which the game calculates new frames (for example, when the game samples input from the game controller). This deviation in the time reference for different behaviors does not affect the game play &amp; when playing the game, but it can be expected that the played back game will produce different results. For example, if the basketball track is calculated at a steady (10) fps rate, but the game controller input is sampled based on the calculated frame rate, then when the game is recorded, the calculated frame rate may be 53 fps, and when replaying the game, the calculated frame rate may be 52 fps, which can cause differences in whether basketball is prevented from entering the basket, resulting in different results. Therefore, the use of game state recording video games requires a very cautious game software design to ensure that the same game state information playback produces exactly the same results.另一个 Another prior art method for § recorded video games is to record only the video output of a PC or video game system (eg, to a VCR, DVD recorder, or to a video capture board on a PC). The video can then be replayed and replayed, or alternatively, the recorded video can be uploaded to the Internet (usually after the video is pressed, after the post). The disadvantage of this method is that when playing back the 3D game sequence, the user is limited to only self-test viewpoints (sequences are recorded from it) to view the sequence. In other words, the user cannot change the view point of the scene. In addition, when the recorded video sequence of the recorded game sequence played on the home PC or the game console via the Internet is used by other users, even if the video system is compressed immediately, It may be possible to instantly reduce the video to the Internet. The reason for this is that the world is connected to the Internet. The broadband connection is asymmetrical (for example, dSL and environmental data machines usually have a much lower downlink bandwidth than the upload bandwidth). Compressed high-resolution video sequences often have a higher bandwidth than the network's upload bandwidth, making it impossible to upload H immediately after playing the game sequence (4 b or even hours) or so on the Internet. There will be a significant delay before another user on the road can view the game ^ although this delay can be tolerated in certain situations (eg 'watching the outcome of a game player that appeared at a previous time), but it eliminates the live broadcast of the watching game ( For example, the ability to play a basketball tournament by a winning player or the "immediate playback" capability when playing a game live. Another prior art method allows viewers with television receivers to watch video game live broadcasts, but only under the control of the television producer. Some television channels in the United States and other countries provide video game detection video channels in which television viewers can view specific video game users (e.g., top players in the tournament) on the video game channel. This is accomplished by feeding the video output of the video game system (PC and/or console) to a video distribution and processing device for the television channel. This is the case when the TV channel broadcasts a live broadcast of the ball game, several of which provide live feeds from different angles around the basketball court. The television channel can then utilize its video/audio processing and effects devices to manipulate the output from various video game systems. For example, a TV channel can overlay text indicating the status of different players on the video from the video game (as it can be overwritten during the live broadcast of the basket 139849.doc -26 - 200949567 during the ball game). The channel can add audio from commentators who can discuss the actions that occur during the game. In addition, the video «output and camera that records the video of the actual player of the game (eg, showing the player's emotional response to the game) can be combined. One of the problems with this approach is that the live video feeds must be made available instantly to the video distribution and processing equipment of the television channel to make it stimulating for live broadcasts. However, as previously discussed, when the video game system is from home execution (especially when a broadcast packet is included in a live broadcast of a camera from a real-world video that is capturing a game player), this is often not the case. may. In addition, in the case of tournaments, the concern is that the player in the family can modify the game and cheat as previously described. Due to such originals and purposes, such video game broadcasts on television channels are often configured with players and video game systems that are gathered at a common location (eg, at a television studio or in an arena), where the television production equipment A video feed from multiple video game systems and potential live broadcast cameras is acceptable. Although these prior art video game television channels can provide a very stimulating performance for a television viewer (which is similar to a live broadcast of a sports event (eg, similar to a video game player presented as an "athlete"), not only based on its experience The actions in the video game world, and based on their actions in the real world, but such video game systems are often limited to situations where players are physically close to each other. In addition, because the television channels are broadcast, each broadcast channel can only display a video stream selected by the producer of the television channel. Due to these restrictions and broadcast time, production equipment and production staff, 139849.doc -27· 200949567 high cost 'these TV channels usually only show top players participating in the top tournaments. In addition, a given video channel of a full screen image of a video game is broadcast to all television viewers to display only one video game. This severely limits the choice of TV viewers. For example, a TV viewer may not be interested in a game displayed at a given time. Another viewer may only be interested in watching a particular player's game play that is not shown by the video channel at a given time. In other situations, the viewer may only be interested in seeing how the player is dealing with a particular level in the game. Other viewers may wish to control the view point (from which the video game is viewed), which is different from the view point selected by the production team or the like. In short, TV viewers may have countless preferences in watching video games (even if several different TV channels are available, the particular broadcast of the TV network should not be suitable for preference). For all of the above reasons, the first video game TV channel has significant limitations in presenting video games to television viewers. Another disadvantage of prior art video game systems and application software systems is that they are complex and often suffer from errors, crashes, and/or undesired and undesirable behaviors (collectively "defects"). Although games and applications are usually undergoing debugging and tuning processes (often referred to as "software quality assurance" or SQA), they are almost unchanged: - games or applications are sent to the vast majority of the field Audience, defects will suddenly appear. Unfortunately, it is difficult for software developers to identify and track many defects after the release. Software developers may have difficulty recognizing defects. Even when it is aware of a defect, there may be only information that it can be used to identify what is causing the defect. 139849.doc 200949567 参 Parameters. For example, the user can call the game developer's consumer service hotline and leave a message stating that when playing the game, the screen begins to flash, then changes to solid blue and the PC freezes. It provides the sqa team with very little information that is useful in tracking defects. One of the online connections Some games or applications sometimes provide more information under certain conditions. For example, the "Watchdog" process can sometimes be used to monitor whether a game or application "crashes". The watchdog process collects information about the state of a game or application process when a game or application crashes (for example, memory). Statistics of the usage status of the stack, the extent to which the game or application progresses, etc., and then uploading the information to the SQA team via the Internet. But in a complex game or application, the information can take a very long time to decrypt, in order to accurately determine what the user is doing during the crash. Despite this, it is impossible to determine what sequence of events caused the crash. The problem associated with PCs and game consoles is that they suffer from service problems that are extremely inconvenient for consumers. Service issues also affect manufacturers of PCs or game consoles because they typically need to send special boxes to safely ship damaged PCs or consoles, and thus incur the cost of repairs (if the pc or console is under warranty). Game or application software publishers may also be affected by sales losses (or online service usage) caused by PCs and consoles in the repair state.

Figure 1 illustrates a prior art video game system such as Sony PIaystati〇n8 3, I 36_, Nintend0 WiiTM, a Windyws-based PC or Apple Macintosh. Each of these systems includes a central processing unit (cafe) for executing code (usually 139849.doc -29-200949567 graphics processing unit (GPU) for performing high-level graphics operations), and for Multiple forms of input/output (1/〇) for external devices and user communication. For the sake of simplicity, these components are shown as being combined into a single unit 100. The prior art video game system of FIG. 1 is also shown to include an optical media drive 104 (eg, a DVD-ROM drive); a hard drive 103 for storing video game code and data; for playing a multiplayer game, for downloading Network connection for games, patches, demos, or other media 丨〇5; random access memory (RAM) 101 for storing code currently being executed by CPU/GPU 100; used to receive during game play The game controller 106 that inputs the command by the user; and the display device 102 (eg, SDTV/HDTV or computer monitor). The prior art system shown in Figure 1 suffers from several limitations. First, compared to the access speed of ram ιοί, the optical drive 1〇4 and the hard drive 1〇3 tend to have much slower access speeds. When working directly via RAM 1〇1, since RAM 101 typically has a much higher bandwidth and does not suffer from the relatively long search delay of the disc mechanism, CPU/GPU 1 can be processed in practice than directly from The number of polygons per second that the hard disk drive 103 or the optical disk drive 1 〇 4 can read when the code and data are read. However, only a limited amount of RAM is provided in such prior art systems (e.g., 256-512 megabytes). Therefore, a "Loading..." sequence is often required, where ram ι〇ι is periodically filled with data for a scene under the video game. Some systems attempt to overlap code loading and game play at the same time, but this can only be done when there is a known sequence of events (for example, if driving along a road, you can load the roadside while driving the car. The geometry of the building 139849.doc -30- 200949567 is approaching). For complex and/or fast scene changes, this type of overlay usually does not work. For example, if the user is in the middle of a campaign and the RAM 101 is completely filled with information indicating the object in the view of the moment, 'If the user moves the view to the left quickly to view the current not loaded' The object in RAM 1 〇1 will result in a discontinuity in motion because there is not enough time to load the new object from the hard disk drive 103 or the optical drive 1〇4 into the RAM 101. Another problem with the system of Fig. 1 is due to the limitation of the storage capacity of the hard disk drive 103 and the optical medium 1〇4. Although the disk storage device can be manufactured to have a relatively large storage capacity (for example, 500 million bytes or more than 500 million bytes), it is not provided for use in current video games. Sufficient storage capacity for a particular situation. For example, as previously described, a British football video game may allow a user to select among many groups, players, and sports fields around the world. For each group, each player, and each field, a large number of texture maps and environment maps are required to characterize the 3D surface of the world (for example, each group has a unique jersey, each requiring a unique texture map) ). One of the techniques used to solve this latter problem: For games, once the user has selected texture and environment mapping, the texture and environment mapping is pre-computed. This can involve a number of computationally intensive processes, including decompressing images, smearing maps, shadows, and organizational data structures. Therefore, when the video game performs such calculations, there may be a delay for the user. One way to reduce this delay is, in principle, to perform all of these calculations when the game is first developed _ including the group 'player roster and each of the sports fields'. The release version of the game thus 139849.doc •31- 200949567 will include all such pre-processed material stored on optical media 1〇4 or on the internet—or multiple servers, when the user makes a choice, The selected pre-processed data for a given group, player roster, and sports field selection is downloaded to the hard disk drive 103 via the Internet only. However, as a practical matter, the pre-loaded material for each permutation possible in game play may easily be a terabyte of data that far exceeds the capacity of today's optical media devices. In addition, the information used for a given group, play roster, and sports field selection may easily be hundreds of millions of bytes of information or hundreds of millions of bytes of information. In the case of a home network connection (for example, 10 Mbps), downloading this data via the network connection 1〇5 takes longer than calculating the data on the local side. Thus, the prior art gaming architecture shown in Figure 1 allows the user to experience significant delays between large scene transitions of complex games. Another problem with prior art methods such as the prior art method shown in Figure 1 is that video games tend to become higher order and require more CPU/GPU processing power over the years. Therefore, even with an unlimited amount of RAM, video game hardware requirements exceed the peak levels of processing power available in such systems. Therefore, users are required to upgrade their game hardware every few years to stay in sync (or play newer games at a lower quality level). One consequence of the trend of higher-order video games over the past is that machines for video games for home use are generally not economical because their cost is usually determined by the requirements of the most powerful game they can support. For example, you might use XBox 360 to play a game like Gears 〇f War, which requires a high-performance CPU, GPU, and hundreds of millions of bytes of ram, or 139849.doc -32- 200949567 You can use the XBox 360 to play "pac", which is a game from the 1970s. It only requires thousands of bytes of RAM and very low = cpu. In fact, XBgx 36() has enough computing power to host many of the same "Pacman" games at the same time. In most hours of the week-week, the video game console is usually cut off. According to Nielsen Entertainment's research on active players over the age of 13 and above in July 2006, on average, active players - spending 14 hours in midweek or - 28 weeks in the middle of the week, only 12% of the time To play console video games. This means that the average video game console is idle 88% of the time, which is an inefficient use of expensive resources. It is particularly meaningful to assume that a video game console is often funded by a manufacturer to lower the purchase price (which is expected to be earned back by royalties from future video game software purchases). Video game consoles also incur costs associated with virtually any consumer electronics device. For example, the electronics and mechanisms of the system need to be housed in a housing. Manufacturers need to provide service guarantees. Selling the retail of the system φ 胄 requires a profit on the sale of the system and/or on the sales of the video game software. The cost of adding these video game consoles to all of these factors must be financed by the manufacturer, delivered to the consumer, or funded by both the manufacturer and the consumer. In addition, 'piracy is a big problem for video game workers. In fact, the security agencies used on each of the larger video game systems have been "broken" over the years, resulting in unauthorized copying of video games. For example, the xb〇x 360 security system crashed in July 2nd and users can now download illegal copies online. Downloadable games (Zhuang, games for pqMac) Special 139849.doc -33- 200949567 Don't be vulnerable to piracy. In certain regions of the world (where piracy is not strong), there is virtually no viable market for independent video game software, as users can easily purchase pirated copies at a cost-small rate with legitimate copies. Moreover, in many parts of the world, the cost of the game console is a high percentage of revenue, so that even if piracy is controlled, it is reported that few people can afford the current state of the art game system. In addition, the market for used games has reduced the revenue of the video game industry. When the user becomes bored with the game, they can sell the game to a store that resells the game to other users. This unauthorized but widespread practice significantly reduces the revenue of game publishers. Similarly, when there is a platform change every few years, there is typically a 50% reduction in sales. This is due to &amp; : When the user knows that a newer version of the platform is about to be released, the user stops buying the game for the older platform (for example, when the piaystati〇n 3 is about to be released, the user stops buying the Playstation 2 game) ). In combination, the loss of sales and the increased development costs associated with the new platform can have a significant adverse effect on the profitability of game developers. The new game console is also very expensive. Xb〇x 360, Nintendo Medical and

Sony Playstation 3 is available in hundreds of dollars. The PC gaming system can cost up to $8. This represents a significant investment by the user, in particular, considering the fact that the hardware became obsolete in a few years and many systems were purchased for the child. The above-mentioned problem is the 'online game, in which the game code and data host are hosted on the server and delivered to the client machine as required. The video and audio through the digital video are contending through the digital broadband network. flow. Some of the 139849.doc 200949567 divisions (such as G_cluster of 'Finland, which is now a subsidiary of SOFTBANK Broadmedia in Japan) provide such services to the winter needles. Similar game services become available in local networks such as hotels and their networks provided by cable TV providers. The larger drawback of these systems is the delay, that is, the signal travels to the game feeder and the cost of the game bribe. (4), the game server is usually located in the "head end" of the operator. The fast motion video game (also known as the "tWitCh" video game) requires very much between the time when the user performs the action by the game controller and the time when the update is not displayed to show the result of the user's action. Low latency. A low latency is required to make the user feel the game respond "immediately". The user is satisfied with different delay intervals depending on the type of visual game and the proficiency of the user. For example, for slow informal games (like backgammon) or slow action role-playing games, the '100 millisecond delay may be tolerable, but in fast action games 'more than 70 milliseconds or 8 milliseconds. The delay can cause the user to perform worse in the game and is therefore unacceptable. For example, in a game that requires fast response time, when the delay increases from 50 milliseconds to 100 milliseconds, there is a sharp drop in accuracy. When the game or application server is installed in a nearby controlled network environment or in a network environment where the network path of the field is predictable and/or tolerant of peak bandwidth, 'at maximum delay and delay In terms of consistency, the control delay is much easier (for example, 'so the user observes the steady motion from the digital video string &quot;IL via the network). This level of control can be achieved either between the cable TV network head end and the cable TV user's home, or from the DSL Central Office 139849.doc -35- 200949567 to the DSL user's home, or from the server or user The commercial office area network (LAN) environment has the potential to obtain a point-to-point private connection with a specific level of bandwidth and latency between businesses. However, in the game or application system in which the game console is hosted in a server center connected to the universal Internet and then the compressed video stream is streamed to the user via a broadband connection, many factors cause delays, resulting in delays. A severe limitation in the deployment of prior art systems.

• In a typical broadband wideband home, users can have a DSL or electrical data modem for broadband π services. These broadband services typically result in up to 25 milliseconds of round-trip travel between the user's home and the universal Internet. In addition, there is a round trip delay due to routing data to the server center via the Internet. The delay through the Internet changes based on the delay in the route of the given data and the data being routed. In addition to the route +, Tatian delay, but also due to the light through the fiber optic ball interconnecting most of the Internet, the number of trains and the singularity of the squadrons caused a delay in travel. For example, for every 1000 miles, the speed of the first through the fiber and other consumption

Incurs a round trip delay of approximately 22 milliseconds. In the case of L, the data rate of the data streamed via the Internet may be cited as an example. 5, the user has a DSL service sold under the "6 Mbps DSL service", in practice, the user will most likely A line throughput of less than 5 Mbps is obtained, and it will be possible to periodically see connection degradation due to various factors (crowding at the Digital Subscriber Line Access Multiplexer (DSLAM) during h载入 value loading time). If looping through neighbors

A similar problem can arise with the local shared coaxial cable + L power present in the crowded or cable modem system network 139849.doc -36 - 200949567, which will be used for "6 Mbps cable The data rate of the connected cable modem sold by the Data Machine Service is reduced to much less than the data rate. If the data packet at a stable rate of 4 Mbps is unidirectionally streamed from the server center via the connection in the User Datagram Protocol (UDP) format, if everything works properly, the poor packet will pass. Without incurring additional delays, but if there is congestion (or other obstruction) and only 3.5 Mbps can be used to stream data to the user, then in the case of the _ type, the packet will be discarded, resulting in lost data, or the packet will be An extra delay is introduced by arranging the queue at the point of congestion until it can be sent. Different congestion points have different capacity for storing the delayed packets, so in some cases, the packets that cannot be successfully resolved are discarded immediately. In other cases, millions of bits of data are queued and eventually sent. However, in almost all cases, the queue at the point of congestion has a capacity limit, and once it exceeds these limits, the queue will overflow and the packet will be discarded. Therefore, in order to avoid incurring additional delays (or worse, participation, loss of packets), data rate capacity beyond the game or application server to the user must be avoided. Delays are also incurred due to the time required to compress the video in the server and decompress the video in the client device. When the video game executed on the server is calculating the next frame to be displayed, further delay is incurred. Currently available video compression algorithms suffer from high data rates or high latency. For example, Motion JPEG is a frame-only lossy compression algorithm characterized by low latency. Each frame of the video is compressed independently of each other frame of the video. When the client device receives a frame of compressed motion jPEG video, its 139849.doc •37· 200949567 can immediately decompress the frame and display the frame, resulting in very low latency. However, because each frame is compressed separately, the algorithm cannot take advantage of the similarity between successive frames, and therefore only the in-frame video compression algorithm suffers from very high data rates. For example, 60 fps (frames per second) 640 χ 480 motion JPEG video may require 40 Mbps (megabits per second) or 40 Mbps (megabits per second) or more. These high data rates for such low resolution video windows will be too expensive in many broadband applications (and indeed for most consumer Internet-based applications). In addition, because each frame is independently compressed, 假 the artifacts in the frame that may be due to lossy compression may appear at different locations in successive frames. This can result in a visual artifact that appears to the viewer as moving when the video is decompressed. Other compression algorithms (such as MPEG2, H.264, or VC9 from Microsoft Corporation) can achieve high compression ratios when used in prior art configurations, but at the expense of high latency. These algorithms utilize inter-frame compression and in-frame compression. Periodically, the algorithms perform only in-frame compression of the frame. This type of frame is called a key frame (commonly called a Γι frame). Then, this G algorithm usually compares the I frame with the previous frame and the subsequent frame. Instead of compressing the previous frame and the successive frames independently, the algorithm determines how the image changes from the I frame to the previous frame and the subsequent frame, and then stores the changes as: "Β" frame (for 1 change before the frame) and "Ρ" frame (for changes after the I frame). This results in a much lower data rate than just in-frame compression. However, it usually comes at the expense of higher latency. The I frame is usually much larger (usually 1x larger) than the frame or frame, and therefore, the 139849.doc -38- 200949567 , , ', data rate transmission takes a prolonged proportion of time. Consider, for example, a situation where the J frame is 10 times the size of the B frame and the p frame, and for each single 1 frame, there are 29 B frames + 30 P frames - 59 boxes A total of 60 frames, or for each "Frame Group" (GOP). Therefore, at 60 fPs, there is one 60-frame GOP per second. Assume that the transmission channel has a maximum data rate of 2 Mbps. In order to achieve the most south quality video in the channel, the compression algorithm will generate a 2 Mbps data stream, given a ratio of φ above, which will result in 2 million bits per frame (Mb) / (59 + 10) = 30,394 One bit and 303, 935 bits per I frame. When the compressed video stream is received by the decompression algorithm, in order to stably play the video, each frame needs to be decompressed and displayed at regular intervals (e.g., 60 fps). To achieve this result, if any frame is subjected to a transmission delay, all frames need to be delayed by at least one delay' so the worst case frame delay will define the delay for each video frame. Because the frame is the largest, the j frame introduces the longest transmission delay' and the entire frame will have to be received before the I frame can be decompressed and displayed (or depending on any frame between the frames). Assuming a channel data rate of 2 Mbps, transmitting a j frame will cost 3 〇 3,935/2 隐 = 145 milliseconds. An inter-frame video compression system (as described above) that uses a large percentage of the bandwidth of the transmission channel will experience long delays due to the large size of the I frame relative to the average size of the frame. Or, in other words, 'When the prior art interframe compression algorithm achieves an average lower data rate per frame than the in-frame compression algorithm (eg '2 Mbps vs. 40 Mbps), it still suffers high due to the large frame Exaggerate the parent frame data rate (for example, 303,935*60 = 18.2 Mbps). But keep in mind that the above analysis assumes that both the p-frame and the B-frame are much smaller than the frame. 139849.doc .39· 200949567 Although this is generally true, this is not true for frames with high image complexity that are not related to previous frames, high motion, or scene changes. In this case, the P frame or the B frame can become larger than the j frame (if the p frame or frame becomes larger than the I frame, the tip compression algorithm will usually "force" the frame. And replace the P frame or the B frame with an I frame). Therefore, the peak data rate of a frame size can appear in the digital video stream at any time. Thus, for compressed video, when the average video data rate is close to the data rate capacity of the transmission channel (often for this condition, given the high data rate requirements for video), from the I frame or the large P frame or B The high peak data rate of the frame results in a high frame delay. Of course, the above discussion only characterizes the compression algorithm delay generated by the large b-frame, p-frame or I-frame in G〇p. If you use the B frame, the delay will be higher. The reason for this is because all B frames and I frames after the B frame must be received before the B frame can be displayed. So in such as

In the picture group (GOP) sequence of BBBBBIPPPPPBBBBBIPPPPP, there are 5 B frames before each I frame, and the first B picture can be displayed by the video decompressor only after receiving the subsequent B frame and I frame. frame. Therefore, if the video is streamed at 60 fpS (ie, 16.67 ms/frame), before the first B frame can be decompressed, no matter how fast the channel bandwidth is, five B frames and I frames are received. It will cost i6 67*6 = 1 〇〇 milliseconds, and this is only the case of $ b frames. A compressed video sequence with 30 B-frames is quite common. In addition, in the low channel bandwidth such as 2 MbpSi, the delay effect due to the size of the I frame is largely added to the delay caused by waiting for the B frame to arrive. Therefore, on the 2 Mbps channel, in the case of a large number of B 139849.doc 200949567 frames, it is quite easy to use a prior art video compression technique with a delay of more than 5 milliseconds or more than 500 milliseconds. If the b-frame is not used (at the expense of a lower compression ratio for a given quality level), then the B-frame delay is not incurred' but still causes the delay due to the peak frame size described above. ❿ The problem is aggravated by the nature of many video games. The video compression algorithm utilizing the G Ο P structure described above is largely optimized for use with live video or movie material that is intended for passive viewing. Usually, the camera (real camera, or the virtual camera in the state of the computer generated by the computer) and the scene are relatively stable, just because if the camera or scene moves too far and bumpy, the video or movie material (a) usually looks good. Unpleasant, and (b) if it is being watched' #Camera suddenly, back and forth, the viewer is usually unable to follow the action closely (for example, if the camera is shooting a child who is turbid on the sizzling Being motivated ^ suddenly bumped back and forth between the cakes, the viewers usually focus on the children and the cake, regardless of the simple and sinister when the camera suddenly moves). In the case of video conferencing or video conferencing, the camera can be held in a fixed position and not moved at all, resulting in very little data spikes. However, the high-definition action video game is characterized by motion (for example, considering the deduction of the game, and the entire frame is in a fast motion during the duration of the competition, or considering a personal shooting game in which the virtual camera is constantly Back and forth: These: The game can produce frames with large and frequent values. Users may need to clearly see what happened during their sudden movements. Therefore, in 3D high-motion video games, Compression artifacts 139849.doc -41 - 200949567 are far intolerable. Therefore, the video output of many video games (due to their nature) produces a compressed video stream with very high and frequent peaks. Assume that users of fast motion video games There is a small tolerance for high latency' and given all of the above-mentioned delay reasons, there is still a limitation on video games hosted by server hosts that stream video over the Internet. In addition, if a highly interactive application is required The host is hosted on the universal Internet and the video stream is streamed', and users of such applications suffer similar restrictions. Network configuration is required, where the colocation server is directly placed at the head end (in the case of a wide-spectrum) or central office (in the case of a digital subscriber line (DSL)), or a commercial background. Medium lan (or specially graded private connection) to control the route and distance from the client device to the server to minimize latency and adapt to peaks without incurring delays. LAN (usually rated at 1〇〇) Mbps-1 Gbps) and leased lines with sufficient bandwidth typically support peak bandwidth requirements (for example, 18 Mbps peak bandwidth is one of the 100 Mbps LAN capacity fractions). Peak bandwidth if special adaptation is applied. The requirements can also be accommodated by the residential broadband infrastructure. For example, on a cable Tv system, a dedicated bandwidth can be given for digital video traffic, which can handle peaks such as large frames. DSL systems can supply higher speed DSL modems (taking into account high peaks) or can be supplied with specially graded connections that can handle higher data rates. However, conventional cables attached to the universal Internet The machine and DSL infrastructure is far from tolerant for peak bandwidth requirements for compressed video. Therefore, online services (hosting video games or application hosts in server centers that are long distances from the client device) And 139849.doc • 42- 200949567 then subject the compressed video output stream via the Internet to a significant delay and peak bandwidth requirement via a conventional residential broadband connection - especially for very low latency Games and applications (for example, the first I-called shooting game and other multi-users, interactive action games, or applications that require fast response time). [Embodiment] This disclosure will be from the following [Embodiment] and attached The drawings are more fully understood, and Φ ❿ ', , and 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 The specific details (such as device type, system group, method, etc.) are set forth in the following description in order to provide a thorough understanding of the present disclosure. However, those skilled in the art should understand that the practice describes the actual: Specific details may not be required. Figures 2a through 2b provide a high-level architecture of two embodiments in which video games and software applications are hosted by a colocation service 21 and hosted at a subscription service = by a user location 211 (Note that “Make (4) place” means wherever the user is located. 'If the mobile device is used, it includes outdoor.” The client device 2G5 is connected to the Internet 2G6 (or other H network or network). The client device 2〇5 can be a general-purpose computer with an internal or external display device 222 having a &quot;wire or wireless connection to the Internet&quot; (e.g., based on Microsoft’s partial deduction) Or the company's Macintosh computer, or it may be a dedicated client device such as a set-top box that turns video and audio out to a monitor or television 222 (with wired or wireless connection to the Internet) Or its 139S49.doc -43- 200949567 to have a mobile device to the Internet wireless connection. Any of these devices may have its own user input device (eg, keyboard, button, touch mode) a screen, a track pad or an inertial-sensing wand, a video capture camera, and/or a motion tracking camera, or the like, or an external input device 221 connected by wire or wirelessly ( For example, a keyboard, a mouse, a game controller, an inertial sensing stick, a video capture camera, and/or a motion tracking camera, etc. As described in more detail below, the colocation service 210 includes servers of various performance levels (including having The high-capacity CPU/GPU processing capabilities of the server. During playback of the game or use of the application on the colocation service 210, the home or office client device 205 receives keyboard and/or controller input from the user. And then it transmits the controller input to the colocation service 210 via the Internet 2〇6, and the colocation service 210 executes the game code in response thereto. Successive frames for the video output (video sequence) of the game or application software (for example, if the user presses a button that will direct the person moving on the screen to the right), the game program will then generate the display character to the right The video image sequence is then compressed using a low-latency video compressor, and the colocation service 21 then transmits the low-latency video stream via the Internet 206. The home or office client device then decodes the compressed video. The video stream and the decompressed video image are reproduced on the monitor or TV. Therefore, the calculation and graphics hardware requirements of the client device 2〇5 are significantly reduced. The client 205 only needs to have the processing capability for forwarding the keyboard/controller input to the Internet 206 and decoding and decompressing the compressed video stream received from the Internet 2〇6, in fact today B9849 .doc 200949567 Any personal computer can do this in software on its CPU (for example, the lntel &amp; dual-core cpu executing at about 2 GHz can decompress 72 using compressor encoding such as H.264 and Windows Media VC9 〇p hdtv). In addition, in the case of any client device, the dedicated chip can also be executed at a much lower cost than the general-purpose CPU and much less power consumption than the general-purpose CPU (such as that required by modern PCs). Standard video decompression. Notably, in order to perform the transfer control, the input and the uncompressed view

For the sake of this, the client device 2〇5 does not require any specialized graphics processing unit (GPU), optical drive or hard drive (such as the prior art video game system shown in Figure 1). As games and application software become more complex and more photo-like, they will need a higher-performance CPU, Gpu, more ram, and more fast drives. The management of the escrow service can be upgraded continuously, but the end user will not need to upgrade the home or office/end platform 205 because the video will be decompressed by a given video (4) (4) The processing requirements of 5 are constant for the display resolution shoulder and frame rate. Therefore, there are no hardware limitations and compatibility problems seen today in the system described in Figures h to 2b. In addition, because the game and application spearware software is only executed in the server g of the host hosting service, it is in the user's home or office (unless otherwise conditional, the "office" as in this article will include any There is never a copy of the game or application software (in the form of optical media, or downloaded software) in the home background, including, for example, the classroom. This significantly reduces the possibility of illegal copying (piracy) of games or application software dragons, and the possibility of pirating valuable libraries that can be used by games or application software. In fact, if you need a specialized server (for example, a very expensive, large or noisy device) to play games or application software that is not feasible for home or office use, even if you get a game or application software. A copy of the pirated copy will not be operated in the home or office. In one embodiment, the colocation service 2 10 is directed to a game or application software developer (which generally refers to a software development company, a game or movie studio) or a game or application software publisher that designs video games. A software development tool is provided to enable it to design games that can be executed on the colocation service 21〇. These tools allow developers to take advantage of the features of colocation services (these features are usually independent? (or will not be available in the game console) (for example, very fast access to very large databases of complex geometries (unless otherwise Condition, otherwise "geometry" will be used herein to refer to the definition of polygons, textures, rigging, lighting, behavior, and other components and parameters of the 3D dataset)) ❹ Under this architecture, different business models are possible Under a model, the colocation service 21 charges the subscription fee from the end user and pays royalties to the developer, as shown in Figure 2a. In an alternative implementation (Figure 2), the developer 22G is directly The self-collected person charges the subscription fee and pays the courier service 21 for the cost of hosting the game or the application content. ί等f The raw material limit (4) is to provide any specific business model for the online game or the helper host agency. The video characteristics of compressed air compression 139849.doc -46· 200949567 As one of the first words, there is a significant problem in providing video game services or application software services online. Delay. The 70ms-80ms delay (from the time the input device is actuated by the user for 4 minutes to the time the response is displayed on the display device) is the upper limit for games and applications that require fast response times. Due to the large number of internal and physical constraints, this is very difficult to achieve in the case of the architecture shown in Figures 2a and 2b. As indicated in Figure 3, when the user subscribes to the Internet service, the connection φ is usually The nominal maximum data rate to the user's home or office is 3.1. Depending on the provider's strategy and routing equipment capabilities, the maximum data rate may be more or less strictly enforced, but usually due to many differences One of the reasons is that the actual available data rate is low. For example, there may be too many network traffic at the DSL central office or at the local computer data path, or there may be noise on the SEM line. , thereby causing the discarded packet, or the provider may establish a maximum number of bits per user per month. When the maximum downlink data rate for cable and DSL services is φ Hundreds of kilobits per second (KbPs) to 3 Mbps. Honeycomb services are typically limited to hundreds of Kbps of data. However, the number of users of broadband service speed and subscription broadband services will vary. Time has increased dramatically. Currently, some analysts estimate that 33% of US broadband users have data rates below 2 Mbps or above 2 Mbps. For example, some analysts predict that by 2010, more than 85% of US broadband The user will have a data rate of 2 Mbps or more. As indicated in Figure 3, the actual maximum data rate of 3 〇 2 can fluctuate over time. Therefore, in low latency, online gaming or application software 139849 .doc • 47- 200949567, it is sometimes difficult to predict the actual data rate available for a particular video stream. The right is the data rate required for a given amount of scene complexity and motion to maintain a given quality level at a given resolution (eg, 640x480 @ 60 fps) for a given number of frames per second (fpS). 3 The rise is higher than the actual available maximum data rate of 3 0 2 (as indicated by the peak in Figure 3), and several problems can arise. For example, some Internet services will only discard packets, resulting in lost data and distorted/lost images on the user's video screen. Other services will temporarily buffer (ie, queue) extra packets and provide them to the client at the available data rate, resulting in increased latency - unacceptable results for many video games and applications . Finally, some Internet service providers view the increase in data rate as a malicious attack (such as denial of service attacks (well-known techniques used by computer hackers to disable network connections)) and will be cut in a specific time period. Disconnect the user's internet connection. Accordingly, the embodiments described herein seek to ensure that the required data rate for video games does not exceed the maximum available data rate. 主机 Hosted Service Architecture Figure 4a illustrates the architecture of a hosted service 21 根据 according to an embodiment. The colocation service 21 can be located in a single server center or can be spread across multiple server centers (think of having a lower latency than others: a low latency connection to a user to a specific dragon center path to Provides load balancing between users and provides redundancy in the event of one or more server center failures). The colocation service 21() can eventually include tens of thousands or even millions of servers 4〇2, so that the servo is very large. 139849.doc •48- 200949567 The user base. The colocation service control system 4-1 provides overall control over the host escrow service 2H) and directs routers, feeders, video compression systems, billing and accounting systems, and the like. In one embodiment, the host escrow control system is implemented on a Linux-based distributed processing system, and the processing system is bound to store information for user information and server information. And a RAID array of database of system statistics. In the above description, various actions implemented by the colocation service 21 are initiated and controlled by the colocation service control system 401 unless attributed to other specific systems. The colocation service 210 includes a number of servers 4, such as those currently available from Intel, IBM, and Hewlett Packard, among others. Alternatively, the server 402 can be assembled into a custom component configuration, or the server 402 can ultimately be integrated to implement the entire server as a single wafer. Although this figure shows a small number of servers 4〇2 for illustrative purposes, in actual deployments there may be as few as one server 402 or as many as millions or _ millions of servers 402. . Server 4〇2 can be configured in the same way (as an example of some configuration parameters, with the same CPU type and performance; with or without GPU, and with GPU, with the same GPU type and performance; with the same number Cpu and GPU; RAM with the same amount and the same type/speed; and with the same RAM configuration), or various subsets of the server 402 can have the same configuration (eg, 25./〇 server can be in a specific way Configuration, 50% of the servers are configured in a different way, and the 25°/server is configured in another way), or each server 402 can be different 0 139849.doc -49- 200949567 in one implementation In the example, the server itself is too @4·~I ... the disk, that is, the local large-volume storage device (which is an optical or magnetic storage device, or a semiconductor-based storage device) ^ ^ yL 陕 S 己 或 或 或 或 或 或 或 或 或 或 或 或 或 或 或 或 己 己 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服 伺服In an embodiment, the 'speed connection is a storage area network (SAN) 4〇3 connected to a series of independent redundant disk arrays (raid), between devices implemented using ultra-high speed Ethernet connection. As is known to those skilled in the art, the SAN 403 can be used to combine many R fine arrays together, resulting in extremely high bandwidth - close to or possibly beyond what can be used in current game consoles and PCs. The bandwidth. Moreover, while RAID arrays based on rotating media such as magnetic media often have significant seek time access latency, RAID arrays based on semiconductor memory can be implemented with much lower access latency. In another configuration, some or all of the servers 4〇2 provide some or all of their own mass storage at the local end. For example, the server 402 can store frequently accessed information (such as a copy of its operating system and video game or application) on a low-latency local flash memory-based storage, but A SAN is used to access a large database of rotationally-oriented RAID arrays 405' with south search latency to access geometry or game state information less frequently. Additionally, in one embodiment, the colocation service 21 uses the low latency video compression logic 404 described in detail below. The video compression logic 410 can be implemented in software, hardware, or any combination thereof (the specific embodiments thereof are described below). Video compression logic 404 includes logic for compressing audio and visual material 139849.doc -50-200949567. In operation, 'when playing a video game via a keyboard, mouse, game controller or other input device 421 or using an application at the user location 211, the control signal logic 413 on the client 415 will indicate The control button 4〇6a_b (usually in the form of a UDP packet) pressed by the button (and other types of user input) is transmitted to the colocation service 21〇. In the future, the control signal from a given user is routed to the appropriate server (or multiple servos)

The server is routed to a plurality of server devices 402 in response to the user's input device. As illustrated in the figure, the control signal 4〇6&amp; can be routed to the server 402 via the SAN. Alternatively or additionally, the service network can be hosted via the host.

The control message ______ is routed directly to the server 402 by WJU ^yJOD (e.g., an Ethernet-based local area network). Regardless of how the control signals 406a-b are transmitted, the or the servers execute the game or application software in response to the control signals 4〇6ab. Although not illustrated, various network connectivity components, such as firewalls and/or gateways, may handle the edge of the colocation service 210 (eg, between the colocation service 210 and the Internet 41〇) And/or incoming and outgoing traffic at the edge of the user premises 211 (between the Internet and the home or office client 415). The graphics and audio output of the executed game or application software (ie, the new video image sequence) is raised to: low latency video compression logic 4G4, low latency video compression logic based on low latency video compression technology (described in this article) And their technology) compresses the video image sequence to the slave (4) and via the Internet 4H) (or, as described below, via the bypass of the Universal Internet Service) The video stream (usually with or without fluffy audio) is transmitted back to 139849.doc 200949567 to the client 415. Next, the low latency video decompression logic 412 on the client 415 decompresses the video and audio streams and reproduces The decompressed video stream, and typically the decompressed audio stream is played on display device 422. Alternatively, the audio may be played on a separate speaker from display device 422 or not at all. Note that despite input device 421 and Display device 422 is shown as a stand-alone device in Figures 2a and 2b, but can be integrated into a client device such as a portable computer or mobile device. The room client 415 (described previously as home or office client 205 in Figures 2a and 2b) can be a very low cost and low power device with very limited computational or graphical performance and possibly #常常本本End-capacity storage or no local-capacity storage. By contrast, each server 4〇2 coupled to SIS 403 and multiple RAID 405 can be an extra-high-performance computing system, and actually If multiple feeding devices are used cooperatively in a side-by-side configuration configuration, there is almost no limit on the amount of computational and graphics processing that can be tolerated. In addition, due to low-latency video compression 4〇4 and low-latency video decompression 412 (perceived by the user), so the computing power of the server 4〇2 is provided to the user. When the user presses the button on the input device, the image on the display 422 is pressed in response to the button. Update (no meaningful delay in perception), like game or application software, executed at the local end so 'for very low-performance computers or just low-latency video decompression and control signals The low-cost family of the 413 chip or the client-to-client 415' effectively provides the user with any computing power at the remote location available to the local end. This is given to the user for playing the highest order, processor. The ability of intensive (usually new) video games and applications with maximum performance of 139849.doc -52- 200949567. Figure 4c shows a very basic and inexpensive home or office client device 465. This device is from the blunt and graph Park's home or office client 415. t - embodiment. It is about 2 pairs long. It has an Ethernet jack 462 that interfaces with an Ethernet cable (Ethernet over Ethernet) (PoE). The device derives its power and its connectivity to the Internet from the Ethernet jack 462. The device is capable of executing φ NAT over a network that supports Network Address Translation (NAT). In the environment, many new Ethernet switches have PoE and bring P〇E directly to the Ethernet jack in the office. In this case, what is required is an Ethernet cable from the wall jack to the customer terminal 465. If the available Ethernet connection does not carry power (for example, in a home with a dsl or cable modem but no P0E), there is an inexpensive wall-type "brick" (ie, power supply) available. It will accept the powerless Ethernet and thin and output the Ethernet with p〇E. The client 465 contains control signal logic φ 413 (Fig. 4a) coupled to the Bluetooth wireless interface with a Bluetooth input device 479 such as a keyboard, mouse, game controller and/or microphone and/or earphone. Interface. Moreover, one embodiment of the client 465 can output video at 120 fps with the display device 468 coupled, and the display device 468 can support 12 〇 fps video and signal a pair of opaque glasses 466 (typically via infrared) for Each successive frame alternately shades one eye and then obscures the other eye. The effect felt by the user is to "jump out" the stereoscopic 31) image of the display screen. One of the operations is supported. The display device 468 is a Samsung HL-T5 〇 76S. Since the video stream for each eye is separate, in one embodiment where the two independent video streams are compressed by the host 139S49.doc -53 - 200949567 tube service 210, the frames are interleaved in time and the graph The box is decompressed in client 465 in two separate decompression processes. The client 465 also includes low-latency video decompression logic 412, which decompresses incoming video and audio and outputs via HDMI (High Definition Multimedia Interface), connector 463, HDMI (High Definition Multimedia Interface), connector 463 It is inserted in SDTV (Standard Definition Television) or HDTV (High Definition Television) 468 to provide video and audio to the TV, or to be inserted into the monitor 468 supporting HDMI. If the user's monitor 468 does not support HDMI, HDMI to DVI (Digital Visual Interface) can be used, but the audio will be lost. Under the HDMI standard, display capabilities (eg, supported resolution, frame rate) 464 are communicated from display device 468, and then transmitted back to the colocation service 2 via Internet connection 462, The colocation service 210 can therefore compress the compressed video in a format suitable for the display device. Figure 4d shows a home or office client device 475 which is identical to the home or office to client device 465 shown in Figure 4c, except that the client device 475 has more external interfaces. Again, the client 475 can accept p〇E to power, or it can occupy an external power adapter (not shown) that is plugged into the wall. The video camera 477 provides the compressed video to the client 475 using the client 475 USB input, and the compressed video is uploaded by the client 475 to the colocation service 210 for use as described below. The low latency compressor will be built into camera 477 using the compression techniques described below. In addition to having an Ethernet connector for its Internet connection, the client 475 also has an 8 〇 211 g wireless interface to the Internet. Both interfaces 139849.doc -54- 200949567 are capable of using NAT within a network that supports NAT. In addition to the HDMI connector for outputting video and audio, the client 475 also has a dual-link DVI-I connector, and the dual-link DVI-I connector includes an analog output (and has a standard that will provide VGA output). Adapter cable). It also has an analog output for composite video and S video. For audio, the client 475 has a left/right analog stereo RCA jack and has a TOSLINK output for digital audio output. In addition to the Bluetooth wireless interface to input device 479, it also has a USB jack for interfacing to the input device. Figure 4e shows an embodiment of the internal architecture of the client 465. All or some of the devices shown in this figure can be implemented in field programmable logic arrays, custom ASICs, or several discrete devices (custom design or off-the-shelf). Ethernet 497 with PoE To the Ethernet interface 48 1. Power 499 is derived from Ethernet 497 with PoE and is connected to the rest of φ in user terminal 465. Bus 480 is a common bus for communication between devices. The control CPU 483 that executes the small client control application from flash memory 476 (almost any small CPU is suitable, such as the MIPS R4000 series CPU at 100 MHz with embedded RAM) is implemented for the network (ie The protocol of the Ethernet interface stacks and also communicates with the colocation service 210 and configures all of the devices in the client 465. It also processes the interface with the input device 469 and sends the packet (along with the user controller data subject to forward error correction protection) back to the colocation service 210. 139849.doc -55- 200949567 In turn, the control CPU 483 monitors the packet traffic (e.g., whether the packet is lost or delayed, and the timestamp of its arrival). This information is sent back to the colocation service 21〇' so that it can strangely monitor the network connection and adjust its intranets accordingly. Initially, at the time of manufacture, the flash memory (4) is loaded with a control program for controlling the CPU 483 and a serial number for the specific client 465 unit. This sequence number allows the colocation service (10) to uniquely identify the user 465 unit. The Bluetooth interface 484 communicates wirelessly to the input device 469 via its antenna (within the client's milk). Video decompressor 486 is a low latency video decompressor configured to implement the video decompression described herein. A large number of video decompression devices exist, either as off-the-shelf products or as a smart product with a design that can be integrated into FpGA or custom ASICs (4) p). - The company that provides the h(10) decoder is the eE L〇gic 〇f Manb of NSW Australia. The advantage of using IP is that the compression techniques used in this article do not match the compression standards. Some indicate that the quasi-decompressor is flexible enough to be configured to accommodate the compression techniques in this article, but some standard decompressors may not. But hey, in the case of IP, there is complete flexibility in redesigning the decompressor as needed. The output of the view compression coupling is coupled to the video output subsystem 487, which couples the video to the video output of the HDMI interface 49. The audio decompression subsystem 488 may be implemented using an available standard audio decompressor, or it may be implemented as an Ip, or may be implemented, for example, 139849.doc • 56 - 200949567

The audio decompression is implemented in the control processor 483 of the Vorbis audio decompressor. The device that implements the audio decompression is coupled to an audio output subsystem 489 that couples the audio to the audio output of the HDMI interface 490. Figure 4f shows an embodiment of the internal architecture of the client 475. As can be seen, in addition to the additional interface and optional external DC power from the power adapter plugged into the wall (and if so, the optional external DC power replacement will be from the Ethernet PoE 497), The architecture is the same as the architecture of the client 465. The functionality common to the client 465 will not be repeated below, but the additional functionality will be described below. The CPU 483 communicates with additional devices and configures additional devices.

The WiFi subsystem 482 provides wireless internet access via its antenna as an alternative to Ethernet 497. The WiFi subsystem is available from a number of manufacturers, including Atheros Communications of Santa Clara, CA. USB subsystem 485 provides an alternative to Bluetooth φ communication for wired USB input device 479. The USB subsystem is fairly standard and can be easily used in FPGAs and ASICs, and is often built into existing devices that perform other functions such as video decompression. The video output subsystem 487 produces a wider range of video outputs than the video output in the client 465. In addition to providing HDMI 490 video output, it provides DVI-I 491, S-Video 492 and Composite Video 493. Also, when the DVI-I 491 interface is used for digital video, the display capability 464 is passed back from the display device to the control CPU 483 so that it can notify the host escrow service 210 of the capabilities of the display device 478. The 139849.doc -57- 200949567 interface provided by the video output subsystem 487 is a fairly standard interface and is readily available in many forms. The audio output subsystem 489 is via the digital interface 494 (s/pDlF and/or

Toslink) outputs audio digitally and outputs analog video via stereo analog interface 495. Round trip delay analysis ❹ Of course, in order to achieve the benefits of the front-end, the user's action of using the input device 421 and the back-and-forth delay between seeing the action on the display device should be no more than 7 milliseconds _8 milliseconds. . This delay must take into account all factors in the path from the input device 421 in the user premises 2U to the colocation service 2ι〇 and the return of the person to the user premises 211 to the display device 422. Figure 4b illustrates the various components and networks through which signals must travel, and above these components and the network is a timetable that lists exemplary delays that can be expected in actual implementation. Note that the map is simplified to only extend the important path routing. The following describes the routing of information for other features of the system. A double-headed arrow (eg 'arrow 453') indicates a round trip delay and a single: arrow (eg 'arrow 457') indicates a one-way delay, and "~" indicates an approximate measurement. ° should indicate that there will be a delay in the listed World situation i. In the case of the Big 1 situation, in the United States, use the cable modem connection to the user's location η 1 , these delays can be noted in the next paragraph III _ into ° X ' Note, although to the Internet Honeycomb wireless networks work in the systems shown, but most current US bee=bedding systems (such as EVD〇) incur very high latency and will not be able to achieve the delays shown by the money. However, these basic principles can be implemented on future cellular technologies that may be able to implement this level of delay. 139849.doc • 58- 200949567 Beginning with the input device 421 at the user premises 211, once the user actuates the input device 421, a user control signal is sent to the client 415 (which may be a standalone device such as a set top box, or It may be software or hardware implemented in another device such as a pc or mobile device, and it is packetized (in UDP format in one embodiment) and given a destination address for the packet to reach the colocation Service 210. The packet will also contain information indicating which user the control signal is from. The control signal packets are then forwarded to the WAN interface 442 via a firewall/router/φ NAT (Network Address Translation) device 443. The WAN interface 442 is an interface device provided to the user premises 211 by the user's ISP (Internet Service Provider). The WAN interface 442 can be a TEL or DSL modem, a WiMax transceiver, a fiber optic transceiver, a cellular data interface, an Internet 〇ver-P〇werline interface, or many interfaces to the Internet. Any other interface in . Additionally, firewall/router/NAT device 443 (and (possibly) WAN interface 442) can be integrated into client 415. One such φ instance would be a mobile phone that includes software for implementing the functionality of the home or office client 415' and for wirelessly routing and connecting to the Internet via a standard (eg, 802.11g) Components. The WAN interface 442 then routes the control signals to a "presence point" 441 for the user's Internet service provider (ISSP), which is a WAN transporter that provides connectivity to the user premises 211. A facility with an interface between a universal Internet or a private network. The nature of the point of existence will vary depending on the nature of the Internet service provided. For DSL, it will typically be the central office of the telephone company to which the DSLAM is located. For the 139849.doc •59- 200949567 cable modem, it will typically be the cable multi-system operator (MS〇) headend. For a honeycomb system, it will typically be the control room associated with the honeycomb tower. But regardless of the nature of the point, it will then route the control signal packet to the universal internet 410. The control signal packets are then routed to the WAN interface 444 to the host escrow service 210 via the interface that is most likely to be the fiber optic transceiver interface. WAN 444 will then route the control signal packets to routing logic 409 (which can be implemented in many different ways, including the Ethernet parent and routing servers), routing logic 4〇9 to estimate the user's address and will control Route L is routed to the correct server for a given user.伺服 The server 402 then treats the control letter i, such as the chorus, as input to the game or application software executing on the vocabulary 4 〇 2 and uses the control signals to process the next frame of the game or application. Once the next frame is generated, the video and audio self-feeding suit H4()2 is output to the video compressor 4〇4. Video and audio can be output to the compressor program via various components. First, the attenuator 404 can be placed in the feeder 4〇2 so that compression can be performed at the local end within the feeder 402. Alternatively, it may be connected via a network to the network (either a private network between the server and the video compressor 404, or a network via a shared network such as the SAN 403) (such as, The Ethernet connection) outputs video and/or audio in a packetized form. Alternatively, the video content benefit 402 can be output via a video output connector (such as a DVI or a steal connector) and then captured by the video compressor 4〇4. &amp;, can be used to output audio from the server as digital audio (for example, via t〇sunk or S_F connectors) or analog audio ' analog audio is digitized and encoded by the audio compression logic within the video compressor. 139849.doc • 60. 200949567 / Once the video compressor 404 has captured the video frame from the server 4〇2 and the audio generated during the frame time, the video compressor will use the technique described below. Compress video and audio. Once the video and audio are compressed, they are packetized by an address to send it back to the user's client 415 and routed to the WAN interface. The wan interface 444 then goes through the universal Internet. The path 41 and the audio packet are routed, and the universal network 41 0 routes the video and audio packets to the presence point _ 441 of the user, and the presence point 441 routes the video and audio packets to the user's place. The WAN interface 442, the WAN interface 442 routes the video and audio packets to the firewall/router/NAT device 443, and the firewall/router/NAT device 443 then routes the video and audio packets to the client 415. The user 4 41 5 decompresses the video and audio, and then displays the video on the display device 422 (or the built-in display device of the user terminal) and sends the audio to the display device 422 or to a separate amplifier/speaker or to the user terminal. In the amplifier / speaker. ❹ In order for the user to feel that the entire process just described is perceptually free of lag, the 'round trip delay' needs to be less than 70 milliseconds or 8 milliseconds. Some of the delay delays described in the return trip path are controlled by the colocation service 21 and/or the user&apos; while other delay delays are not controlled by the colocation service 21 and/or the user. Nonetheless, based on the analysis and testing of a large number of real-world situations, the following are approximate measurements. The one-way transmission time 45 1 for transmitting control signals is typically less than 1 millisecond, and the round-trip routing 452 via the user premises is typically performed using an easy-to-access consumer-grade firewall/router/NAT switch on the Ethernet. i 139849.doc • 61 - 200949567 completed in seconds. The user ISP extensively changes its round trip delay 453, but in the case of DSL and cable modem providers, it is typically seen between 1 and 25 milliseconds. The back-and-forth delay of the Universal Internet 41 is how the routing is routed and whether there are any faults on the route (and these issues are discussed below), but the general Internet is usually quite optimal. Routing and delay are largely determined by the speed at which light passes through the fiber (the distance given to the destination). As discussed further below, 1 mile has been determined as the approximate farthest distance from which the host escrow service 21 is expected to be placed away from the user premises 211. At 1 mile (round trip ❿ 2000 miles), the actual transmission time for signals via the Internet is approximately 22 milliseconds. The WAN interface 444 to the hosted service is typically a commercial-grade fiber optic high-speed interface with a decisive delay. Therefore, the universal internet latency 454 is typically between 丨 milliseconds and 1 () milliseconds. The one-way routing 455 delay via the colocation service 21 can be achieved in less than 1 millisecond. The server will usually be used for a new frame of the game or application at less than one frame time (which is 16 7 milliseconds at 6 〇 fps), so 16 milliseconds is the reasonable maximum that will be used. One-way delay 456. In the best hardware implementation of the video compression and compression algorithm described herein, compression 7 can be completed in 1 millisecond. In the next best version, 'compression can take up to 6 milliseconds (of course, the version of L can take longer, but these implementations will affect the round trip, delay and will require other delays shorter (for example, can be reduced) Small through the general 'internal distance of the weekly network to maintain the 7G milliseconds - 8G milliseconds delay target). The round trip delay of Internet 454, user ISP 453 and user location route 452 has been considered, so the rest is video decompression 458 extension 139849.doc • 62· 200949567 犄, video decompression 458 delay depends on video decompression Whether the 458 is implemented in a dedicated hardware or in a software implemented on a client device 415 (such as a PC or a mobile device), the size of the visual display and the decompression of the cpU2 performance are changed. Generally, decompression 458 takes 1 millisecond and 8 milliseconds. Between the time. • Therefore, the worst-case back-and-forth delay that the user of the system shown in Figure 4a can expect will be expected by adding the delays of all the worst conditions seen in practice. It is: 1 + 1+25+22+i + i6+6+8=8〇 φ milliseconds. In addition, in practice, in practice (with the misunderstanding described below) 'This is roughly a prototype version of the system shown in Figure ( (using the ready-made Windows PC as a client device and home DSL in the US) Cable data machine connection) See the round trip delay. Of course, situations that are better than the worst can lead to much shorter delays, but you can't rely on them to develop widely used business services. In order to achieve the delays listed in FIG. 4b via the universal internet, the video compressor 4〇4 and the video decompressor 412 (from FIG. Φ 4a) in the client 415 are required to generate a packet stream having very specific characteristics. So that the sequence of packets generated via the entire path from the colocation service 210 to the display device 422 is not subject to delay or excessive packet loss, and in detail, consistently falls over the Internet connection via the user (via The WAN interface 442 and firewall/router/NAT 433) can be used within the constraints of the user's bandwidth. In addition, the video compressor must generate a sufficiently robust packet stream to withstand the inevitable packet loss and packet reordering that occurs in normal Internet and network transmissions. Low Latency Video Compression 139849.doc -63- 200949567 In order to accomplish the above objectives, an embodiment employs a new video compression method that reduces the delay and peak bandwidth requirements for video transmission. Prior to describing such embodiments, an analysis of current video compression techniques will be provided with respect to Figures 5 and 6a through 6b. Of course, if the user has a bandwidth sufficient to handle the data rate required by such techniques, then such techniques can be used in accordance with the basic principles. Note that the audio compression is not addressed in this document, but rather that the audio compression is implemented simultaneously and synchronously with video compression. Prior art audio compression techniques that meet the requirements for this system exist. Figure 5 illustrates a particular prior art for compressing video in which each individual video frame 501-503 is compressed by compression_logic 520 using a particular compression algorithm to produce a series of compressed frames 511-513. One embodiment of this technique is "Motion JPEG" in which each frame is compressed based on a Discrete Cosine Transform (DCT) according to the Joint Image Experts Group (JpEG) compression algorithm. Various different types of compression algorithms can be used, however, these basic principles are still adhered to (for example, wavelet-based compression algorithms such as JpEG_2000). One of the problems with this type of compression is that it reduces each figure. The frame's data rate is high, but it does not take advantage of the similarity between successive frames to reduce the data rate of the total video stream. For example, as illustrated in Figure 5, assuming a frame rate of 640x480x24 bits/pixel = 640*480*24/8/1024=900 kilobytes/frame (KB/frame), A quality image, motion JpE (J may only compress the stream by 1/10, resulting in a 9 〇 KB/frame data stream. In 7 seconds, this will require 90 KB* 8 bits* 60 frames per second = 42.2 Mbps channel bandwidth 'for almost all home Internet connections in the United States today 139849.doc •64- 200949567 will be extremely high bandwidth' and for many office internet In terms of connection, it is too high a bandwidth. In fact, if it is required to blame the data stream in the case of such a high frequency width and it will only feed the device - (4), even if it is in the AN environment, it will It consumes a bandwidth of 1 〇〇MbpS6 too wide LAN LAN and a heavy Ethernet switch that supports LAN. Therefore, it is compared with other compression technologies (such as those described below). When it is used for motion video compression inefficiency. In addition, a single frame compression algorithm using lossy pressure φ reduction algorithm Such as JPEG and JPEG-2000) produce compression artifacts that may be unobtrusive in still images (for example, artifacts in dense foliage in a scene may not appear as artifacts, because the eye does not know exactly how dense foliage should be How to render.) However, once the scene is in motion, the artifact may be prominent, because the eye detects artifacts that change from frame to frame, although the artifact is in the area of the scene (in that area, False shadows may not be noticeable in still images. This leads to the perception of "background noise" in the sequence of frames, the appearance and edge analogy of the "background noise" 「 "snow" visible during TV reception Similarly, this type of compression can still be used in the specific embodiments described herein 'but in general, to avoid background noise in the scene, a high data rate is required for a given perceived quality (ie, Low compression ratio. Other types of shrinkage (such as 'H.264, or Windows Media VC9, MPEG2, and MPEG4) are more efficient in compressed video streams because they take advantage of the similarity between successive frames. The technology relies on the same general techniques for compressing video. Therefore, although the H.264 standard will be described, the same general principles apply to various other compression algorithms. A large number of H.264 compressions 139849.doc -65- 200949567 And decompressor available 'includes the x264 open source software library for compression 264.264 and the FFmpeg open source software library for decompressing H.264. Figures 6a and 6b illustrate an exemplary prior art compression technique in which The compression logic 620 compresses a series of uncompressed video frames 5〇1_5〇3, 559_561 into a series of "I frames" 611, 671; "P frames" 612-613; and "B frames" 670. The vertical axis in Figure 6a generally represents the resulting size of each of the encoded frames (although the frames are not drawn to scale). As described above, those skilled in the art are well aware of the video writing codes using the I frame, the B frame, and the P frame. In short, frame 6 is a DCT-based compression of frame 501 that is completely uncompressed (similar to a compressed JPEG image as described above). The size of P-frames 612-613 is typically significantly smaller than the size of z-frame 611 'because it utilizes data from previous I-frames or ρ-frames; that is, 'which contains an indication between the previous I-frame or frame Changed information. The frame 670 is similar to the frame except that the frame is used in the subsequent reference frame and (possibly) before the reference frame in the frame. For the following discussion, it will be assumed that the desired frame rate is 6 frames/sec, and each I frame is about 160 Kb. The average ρ frame and the β frame are 16 Kb and a new I is generated every second. Frame. Under this set of parameters, the average data rate will be: 160 Kb + 16 Kb * 59 = l.l Mbps. This data rate falls appropriately within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid background noise problems from intra-frame coding only. Because the difference between the P-frame and the B-frame tracking frame, the compression artifact tends to appear and disappear from the frame to the frame. To reduce the background noise problem described above. I39849.doc -66 * 200949567 One of the problems with compression of the above type is that although the average data rate is relatively low (for example, 1.1 Mbps), a single I frame may take several frame times to transmit. For example, using prior art, a 2.2 Mbps network connection (eg, a DSL or cable modem with a 2.2 Mbps peak from the largest available data rate 302 of Figure 3&amp;) would typically be sufficient to stream video at M Mbps ' One 60 Kbps I frame per 60 frames. This will be done by having the decompressor queue the 1 second video before decompressing the video. Within 1 second, 1_1 Mb of data will be transmitted, which will be easily accommodated by the 2 2 Mbps maximum available data rate, even if it is assumed that the available data rate may periodically drop by as much as 50%. The technical method will result in a leap second delay of the video due to the 1 second video buffer at the receiver. This delay is sufficient for many prior art applications (e.g., playback of linear video), but it is an extremely long delay for fast motion video games that cannot tolerate delays greater than 7 milliseconds _ 8 milliseconds. If you try to eliminate the 1 second video buffer, it will still not be used for

frame. (One of the consequences is: The method will be to eliminate the B frame and only use the J frame and the p image. The result is I for a given quality level. The data rate will increase by 139849.doc •67· 200949567 plus, but for this For consistency in the example, it is assumed that the size of each frame is 160 Kb and the average P frame size is 16 Kb, and therefore the data rate is still 1.1 Mbps ^ This method eliminates the introduction of the 3 frame The inevitable delay, because the decoding of each P frame depends only on the previously received frame. The problem with this method is that the frame is much larger than the average p frame, so that it is on the low frequency wide channel (such as In most homes and in many offices, the transmission of the I-frame adds a substantial delay. This is illustrated in Figure 。. The video stream data rate 624 is lower than the maximum data rate 621 available (except for the I frame). , where the peak data rate 623 required by the frame exceeds the maximum data rate 622 (and even exceeds the maximum data rate of 621). » The data rate required for the P frame is less than the maximum data rate available. Even at 2.2 Mbps. maximum The material rate peak is steadily maintained at its peak rate of 2 2 Mbps, and it will take 16 〇Kb/22 Mb=7i milliseconds to transmit the work frame, and if the maximum data rate 622 is reduced by 5〇% (1” Mbps), then It will take 142 milliseconds to transfer the frame. Therefore, the delay in the transmission 1 frame will be somewhere between 71 milliseconds and 142 milliseconds. This delay is added to the delay identified in Figure 4b (the delays are at the most In the case of a bad condition, the total is up to 7 〇 milliseconds, so this will result in a total round trip delay of 141 - 222 milliseconds from the moment the user actuates the input device 421 until the image is seen on the display device 422. If the maximum available data rate drops below 2 2 Mbps, the delay will increase further. Also note that 'the peak data rate 623 far exceeds the available data rate 622. The isp "blocking" usually has serious consequences. The devices in different isps will be different. Performance, but when receiving packets at a data rate much higher than the available data rate 622, I39849.doc 200949567, the following behavior is quite common among DSL and cable modems: (a) by placing packets in 伫And delay the packet (introduction delay (b) discard some or all packets, (4) disable connection duration - time period (most. may be because 1SP is concerned about malicious attacks, such as "denial service" attack) Transmitting packet streams at full data rate (with features such as those shown in Figure 6b) is not a viable option. The host escrow service can rank the peak 623 (four) and below the available maximum data speed. The data rate is transmitted 'and thus introduces the unacceptable delay described in the pre-segment. In addition, the video stream data rate sequence 624 shown in Figure 6b is a very "tamed (four) e)" video stream data rate sequence and will The sequence of data rates expected to occur due to dust reduction from video of the video sequence, the video sequence does not change the report size and has very little motion (eg, as would be common in video teleconferences, in video telephony) During the conference, the camera is in a fixed position with very little motion, and the objects in the scene (eg, the person sitting in the seat Φ show less movement). The video stream data rate sequence (4) shown in Figure 6c is a typical sequence that is expected to be visible from video motions that are much more active (such as may be generated in a movie or video game/in or in an application software). ^, (10) In addition to the frame bee 633, there are also quite large and high-resolution data frame peaks (such as milk and 636) that can be used on many occasions. Although the peak value of such a frame is not as large as the peak value of the I frame, it is still so large that it is not carried by the channel at the full data rate, and like the peak of the old frame, the peak of the ρ frame Must be transmitted slowly (by increasing latency 139849.doc -69- 200949567 on - high-bandwidth channels (for example, 100 Mbps LAN, or high-bandwidth (10) Mbps private connections), the network will be able to tolerate peaks such as I-frames 633 or p frame peak 636 peaks 'but in principle, low latency can be maintained: however, these networks are often shared among many users (for example, in an office environment), and the "peak" data Will affect the performance of the LAN, especially if the network traffic is routed to a private shared connection (for example, from a remote data center to the office). First, remember that this example is relatively low resolution of 640x480 pixels. An example of video streaming. The 1920x1080 HDTV stream at 60 fps is easily handled by modern computers and displays, and the 2560x1440 resolution display at 60 fps is increasingly available (for example, the company's 30&quot; display). Using Η Μ# The 1920x1080 high motion video sequence may require 4 5 Mbps to obtain a reasonable quality level. If the I frame peak is assumed to be 1 times the nominal data rate, it will produce a 45 Mbps peak, and Small but still quite large p-frame peaks. If several users are receiving video streams on the same 1 Mbps network (for example, a private network connection between the office and the data center), it is easy to see How the peak of the video stream of several users can be aligned, which drowns the bandwidth of the network and may overwhelm the bandwidth of the backplane of the switch supporting the user on the network, even in the case of ultra-high speed Ethernet. If enough users have enough peaks for simultaneous alignment, they can flood the network or network switch. In addition, once 256〇xl44〇 resolution video becomes more common, the average video stream data rate may be 9.5. Mbps, which may produce a peak data rate of 95 Mbps. Needless to say, the 100 Mbps connection between the data center and the office (which is now an extraordinarily fast connection) will be completely 139849.doc • 70- 200949567 Peak traffic crash from a single user. Therefore, even if LAN and private network connections are more tolerant to peak streaming video, streaming video with high peaks is not required and possible Special plans and adaptations for the office's Ting department are required. Of course, for standard linear video applications, these issues are not a problem' because the data rate is "smoothed" at the transmission point and the data for each frame is below the maximum available. Data rate 622, and decompress I frame, p

Before the frame and B frame sequence, the buffer in the client side stores the I frame, the p frame and the B frame sequence. Therefore, the data rate on the network remains close to the average data rate of the video stream. Unfortunately, this introduction delay, even if the B-frame is not used, is unacceptable for low-latency applications such as video games and applications that require fast response times. The technique used to mitigate video streams with high peaks is a technique often referred to as "value-located meta-rate" (CBR) encoding. Although the term CBR appears to imply that all frames are compressed to have the same bit rate (ie, size), it is often referred to as the I-shrinking paradigm, which in the compression paradigm 'allows a certain number of frames to be crossed (in In the case of our person, (4) the maximum bit rate of the frame. For example and t, in Figure 6. In the case where a CBR constraint is applied to the code (which limits the bit rate to, for example, the nominal maximum data rate 621), then the compression algorithm will limit the compression of one of the frames: Any frame that will be compressed using more than 70% of the nominal maximum data rate 621 will be compressed with fewer bits. The result of this is that it will usually require more bits to maintain a given level of quality. The frame is "extremely lacking" and the quality of the frame is less than that of the other frames of the 139849.doc •71 · 200949567 maximum data rate of 621. This method produces acceptable results for a particular type of compressed video (where (a) expected eve movement or % scape change and (b) user accepts periodic quality degradation. A good example of a suitable CBR application is a video conference call' because there are fewer peaks and the quality is temporarily degraded (for example, if the camera is swept, resulting in significant scene motion and large peaks), it may not be during the panning. There are enough bits for high-quality image compression, which will result in degraded image quality.) Most users τ accept that: 'CBR is not well suited for scenes with high complexity _ or a lot of motion and/or Many other applications that require a reasonably constant quality level 0', the low-latency compression logic used in the embodiment 4〇4 uses a number of techniques to solve the many problems of streaming low-latency compressed video while maintaining high quality. First, the low-latency compression logic 4〇4 only generates Z-frames and p-frames, thereby alleviating the need to wait for several frame times to decode each frame. . In addition, as illustrated in Figure 7a, in one embodiment, the low latency &amp; reduction logic 404 subdivides each uncompressed frame 7〇1_76() into a series of "shadow 10 blocks (four)" and will each - The image block is coded as a top frame or a p frame. ^ In this paper, the group of compressed frames and ρ frames are called "r frames". 77. Specific examples not shown in Figure 7a In 'replace each uncompressed frame into 16 image blocks of 4χ4 matrix. However, these base = principles are not limited to any particular sub-dividing mechanism. ^ In the C example, 'low-latency I-reduction logic 4〇4 divides the video frame into a number of jersey blocks, and encodes the image block from each frame (ie, 139849.doc •72·200949567) i frame (ie, compressing the image block as if it were 1/16 of the size of the full image, and the compression for this "mini" frame is I frame compression) and The remaining image blocks are encoded as P-frames (that is, compressed by the P-frame for each "逑" type 1/16 frame). Compressed. Image blocks that are 1 frame and image blocks that are compressed into P frames will be referred to as I~blocks and P blocks. The image block to be encoded as an I image block is changed with each successive video frame. Therefore, in a given frame time Φ, only one image block in the image block in the video frame is an I image block, and the rest of the X scenes/image blocks are p image blocks. For example, in Figure h, the image block of uncompressed frame 701 is encoded as image block 1 and the remaining 1-15 image blocks are encoded as p image blocks (p to u to generate r Block 711. In the next uncompressed video frame 7〇2, the uncompressed frame 71 is encoded as an image block and the remaining image blocks and 2 to 15 are encoded. For example, the image block (Ρ〇, and Ρ2 to Ρ5) is used to generate the R frame 712. Therefore, the I image block and the 卩 image block for the image block are successively staggered in time on successive frames. Continue until R image block 77 is generated (in the matrix, the last image block is encoded as a ! image block (ie, Μ. The process then restarts, resulting in, for example, frame 7n (ie, for image block 〇 ' Editing the image block) and other _R frames. Although not illustrated in the figure, in the K example, the first R frame of the video sequence of the R frame contains only the image block ^, so that The p-frame has reference image data (from which motion is calculated). Or, in one embodiment, the start sequence is used ^often the same! Image block type, but does not include a P block for encoding m彡 blocks that have not been coded together with the ι image block. In the case of the d image 139849.doc •73· 200949567, the block does not come together. Any data - encoding a particular image block, thereby avoiding the on-shot peak in the video stream data rate 934 in Figure 9a, which is described in further detail below. Further, various sizes and shapes such as Μ X τ π ir can be used for These image blocks still adhere to these basic principles.

The video decompression logic 412 executing on the client 415 decompresses each image block as if it were a separate video sequence of the small I frame and the p frame, and then reproduces each image block to the map of the drive display device 422. Frame buffer. For example, the image block from the R frame is used to decompress and reproduce the video image. Similarly, use from R frame 7 to 87

To reconstruct the image block Bu et al. As described above, the decompression of the Z frame and the p frame is well known in the art, and the decompression of the old image block and the p image block can be performed by causing multiple execution entities of the video decompressor to be at the user end. Execute on 415 to complete. Although the multiplication process seems to increase the computational burden on the client, it does not actually increase the computational burden on the client side because the image block itself is proportionally smaller (relative to the extra processing: number) Therefore, the number of pixels displayed (four) seems to be a process and uses the conventional full-size frame and p-frame. This R-frame technique significantly mitigates the bandwidth peaks typically associated with the frame (as illustrated in Figures 6b and 6e) because any material frame is primarily composed of P-frames that are typically smaller than the I-frame. For example, suppose that the typical frame is 160 Kb, then the j image block of each of the frames in Figure 7a will be a large 6 or 10 Kbe for this amount, assuming a typical p The frame is Μ Kb, and the p frame for each of the image blocks illustrated in Figure 7a can be approximately 1 Kb. The final result is approximately 1〇Kb+15iM Kb=25 Figure 139849.doc -74- 200949567 box. Therefore, each 60 frame sequence will be 25 Kb * 60 = 1.5 Mbps. Therefore, at 60 frames per second, this would require a channel capable of maintaining a bandwidth of 丨5 Mbps, but because the I-picture block is dispersed throughout the 6-frame interval, it has a much lower peak. - Note that in the previous example, the average data rate was i" Mbps for the same assumed data rate for the I and P frames. This is because in the previous example, only a new 1 frame is introduced every 60 frame time, and in the example of φ, the 16 image blocks constituting the 1 frame are cycled in 16 frame times, and thus , an equalization of one frame is introduced every 16 frame times, resulting in a slightly higher average data rate. Despite this, in practice, the introduction of more frequent I frames does not linearly increase the data rate. This is due to the fact that the P-frame (or p-image block) is mainly encoded from the previous frame to the next frame. So if the previous frame is quite similar to the next frame, then the P frame will be very different if the previous frame is quite different from the next frame, then the p frame will be very large. (d) The p-frame is largely derived from the previous ® box, rather than from the actual frame, so the resulting coded frame may contain more errors than a t-frame with a sufficient number of bits (eg , visual artifacts). In addition, when a P-picture pivot follows another _p frame, an error accumulation can occur (when there is a long P-frame sequence, it gets worse). Now, the visual compression of the tip 2 will measure the fact that the image f of the image is degraded after the sequence p frame and, if necessary, it will compete for one, and assign the more 夕 to the P after P Frame to improve quality, or if it is the most effective # π 乍 private, use 1 frame, frame. Therefore, when using a long frame sequence (for example, a continent box, as in the previous example above), when the scene has a large number of complex 139849.doc -75 - 200949567 noise and / or motion, usually, for P More bits are needed for the frame (because it becomes farther away from the I frame). Or 'from the relative view point to see the P-frame, the P-frame that closely follows the I-frame tends to require fewer bits than the p-frame farther from the I-frame. Thus, in the example shown in Figure 7a, the 'no P frame is farther away from the I frame (1 before the I frame)' while in the previous example, the p frame can be self The I frame is separated by 59 frames. Therefore, in the case of more frequent I frames, the P frame is smaller. Of course, the relative size of the bee cut will change based on the nature of the video stream. However, in the example of the figure, if the I image block is 10 Kb, then The size of the P image block can average only 0.75 kb, resulting in 1〇Kb+15*0.75 Kb=21.25 Kb, or at 60 frames/sec, the data rate will be 21 25 Kb*60=1.3 Mbps, or The data rate of a stream with an i frame followed by 59 p frames at 1 Mbps is still about 16°/. . Again, the relative results between these two methods for video compression will vary depending on the video sequence, but in general, we have empirically found that for a given quality level, using an R frame is more efficient than using an I/p map. The frame sequence requires about 20% more bits. However, of course, the r frame sharply reduces the peak value. This makes the video sequence available at a much slower delay than the I/p frame sequence. The R frame is configured in a number of different ways depending on the nature of the video sequence, the reliability of the channel, and the available data rate. In an alternate embodiment, a number of image blocks other than 16 are used in the 4χ4 configuration. For example, you can use 2 image blocks in a 2χΐ or 1x2 configuration, 4 image blocks in a 2x2, 4xl or ι configuration, and 6 images in a 3x2, 2x3, 6x1 or 1x6 configuration. The block may use 8 image blocks in 4χ2 (as shown in Figure 7b), 2x4, 8x1 or 1x8 groups. Note that the image block does not need to be a square video image. 139849.doc •76· 200949567 The frame does not have to be square, or even rectangular. The image block can be broken down into whatever shape best suits the video stream and application being used. In another embodiment, the loop of the I image block and the p image block is not locked to the number of image blocks. For example, in an 8 image block 4χ2 configuration, a 16-cycle sequence can still be used as illustrated in Figure 7b. The sequential uncompressed frames 721, 7U, 7U are each divided into 8 image blocks 〇_7, and each image block is individually compressed. From R frame 731, only the image block is compressed into an image block, and the remaining image blocks are compressed into p image blocks. For subsequent R frame 732, all 8 image blocks are compressed into p-image blocks, and then for subsequent R frame 733, image blocks! It is compressed into [image block and other image blocks are compressed into p image blocks. In addition, the 16 frames continue to be sorted so that only one image block is generated every other frame, so during the third frame time (Fig. 7b 'not shown) and during the 16th frame time Generated for image block = last! Image block (compress r frame 780 using all p image blocks). The sequence of the receivers is again reduced to the image block by the image block _ &amp; and the video block begins with the (4) P picture block. For example, in the previous implementation, the entire frame will always be 1 image block to provide a reference for the P image block forward from the other point. 1 The loop of the image and P blocks does not need to be an even multiple of the number of image blocks. For example, in the case of a _image block, after using another "know-tail", there is an I image 堍=mother-frame, which can be two frames of all p-image blocks. - More frequent I and German, £ domain have a better movement (requires frequent "blocks", while other areas are more static (for example, scores of play) (requires less frequent search, 'views/images Block) 'Compared with other image blocks 139849.doc •77- 200949567, you can more often combine specific image blocks! Image blocks are sorted together. In addition, although it is illustrated in FIG. 7a and FIG. 7b that each frame has a single-I image ghost, a plurality of i image blocks can be encoded in a single frame (depending on the transmission bandwidth). A particular frame or frame sequence can be transmitted without an I-picture block (ie, only a p-picture block). The reason why the pre-segment method works properly is that although the 1 image block that does not have a spread across each early-frame appears to cause a large peak, the behavior of the system is not so simple. Because each image block is compressed separately from other image blocks, so when the image block becomes smaller, each image block is less effective, because the compressor of a given image block cannot utilize other image blocks. Such as (4) image features and similar sports. Therefore, the disc divides the screen into (4) image blocks, and dividing the screen into 16 image blocks will generally result in less efficient coding. However, if the screen is divided into 8 image blocks and it is induced to be introduced every 8 frames (rather than every 16 frames), then the king I figure t-bee, which leads to overall high More data rates. Therefore, by introducing a full frame every 16 frames (rather than every 8 frames), the total data rate 1 is reduced by using 8 larger image blocks instead of 16 The small image block) reduces the total data rate, which also reduces the peak value of the data caused by the larger image block to some extent. In another embodiment, the low-latency video reduction logic 404 of FIGS. 7a and 7bt is configured by pre-configuration or based on image quality in each image block based on known characteristics of the video sequence to be compressed. The ongoing analysis automatically controls the allocation of the bits of each of the image blocks. For example, in some of the video games, the player's car (which is relatively unmoved in the scene 139849.doc •78·200949567) is completely filled in the upper part of the screen. Most of them, and almost all of them are really close to the roads, buildings and landscapes, used: squatting in sports. If the compression logic 4〇4 assigns an equal number of bit knives to the parent image block, then the image block (image block 4_7) on the lower half of the uncompressed frame 721 in Figure 7b 罄m will usually In the uncompressed frame 721 outside the picture, the image block (image block) in the upper part of the screen 〇h~

If the specific 2 of this particular game or game is known to have this property, then the host generation #„2(10) operator can configure: reduce the logic 404 to allocate more bits to the image block at the top of the screen. (Compared to the bit of the image block at the bottom of the screen assigned to the screen.) Alternatively, the compression logic can estimate the compression quality of the image block after compressing the frame (using many compressions. - Orality - or more I , such as peak signal noise than Lang Qing's and if it is determined that a particular image block consistently produces better quality results over a particular time window, it gradually assigns more bits to image blocks that produce lower quality results. Until the various image blocks reach a similar level of quality. In an alternative embodiment, the compressor logic 4〇4 assigns a bit 乂 to achieve a higher quality in the special image block or image block group. For example, Providing a better overall perceived appearance to have a quality south of the screen than at the edge. In one embodiment, to improve the resolution of a particular region of the video stream, video compression logic 404 encodes the image using a smaller image block. The area of the stream that has relatively large scene complexity and/or motion (compared to the area of the video stream that has relatively little scene complexity and/or motion). For example, as illustrated in Figure 8 R frame 8 11 (may be followed by a moving figure in the area of one of the series R frames (not shown) with the same image block size 139849.doc -79- 200949567), with a smaller shirt block Then, when the character moves to a new area of the image, the dragon of the new area in the S-R frame 812 uses a smaller image, such as the 5 brothers. As described above, various sizes and shapes can be used. The "image block" still adheres to these basic principles. Although the cyclic 1 / P image block described above substantially reduces the peak in the data rate of the video stream, it does not completely eliminate the peak, especially in the fast change or Highly complex video images (such as in the movie 'video game and some application software). For example, during a sudden scene transition, a complex frame may be followed by another complex picture that is completely different. Box. Even a few The I image block can lead the scene transition by only a few frame times, which is not helpful in this case, because the material of the new frame is independent of the previous I image block. In this case (and even if not everything) In other cases where all of the images are changed, the video compressor 404 will determine that many, if not all, P-picture blocks are more efficiently coded as image blocks and result in a data rate of the frame. Very large peaks. As discussed earlier, it is only for most consumer-grade Internet connections (and many office connections). It can only be "blocked" more than the one shown in Figure 6c. Maximum data rate and data for rated maximum data rate 621. Note that 'rated maximum data rate 621 (eg, "6 Mbps DSL") is essentially a sales figure for users considering purchasing an internet connection' but usually it is not guaranteed Performance level. For the purposes of this application, it is irrelevant' because we only focus on the maximum data rate 622 available when streaming video over a connection. Therefore, in Figures 9a and 9c, when the solution to the peak problem is described in 139849.doc • 80· 200949567, the nominal maximum data rate is omitted from the graph and only the maximum data rate 922 is displayed. The video streaming data rate must not exceed the maximum available data rate of 922. To solve this problem, the first thing the video compressor 404 does is to determine the peak data rate 941' which is the data rate at which the channel can be processed steadily. This rate can be determined by a number of techniques. One technique is to gradually increase the data rate test stream from the colocation service 21 to the client 415 (Figs. 4a and 4b), and cause the UE to provide feedback on the level of packet loss and delay. Provide to the hosting service. When the packet is lost and/or the delay begins to show a sharp increase, it is an indication that the maximum data rate 922 is available. Thereafter, the colocation service 21 can gradually reduce the data rate of the test stream until the client 415 reports that the test stream has been received within a reasonable period of time (the packet loss and the acceptable level of the delay are near minimum). This determines the peak maximum data rate 941, which will then be used as the peak data rate for streaming video. Over time, peak 941 will fluctuate (eg, &quot;other in the family-user begins to seriously use the internet connection), and client 415 will need to rate 941 to see if the packet is lost or delayed Increase (indicating that the most available material rate 922 falls below the previously determined peak data rate 941), and right then, the peak data rate is 94, the user-side finds that the packet is lost and the right time is Requesting the video compressor to slowly &lt;Feng, whether the data rate is increased (for example, if the home m sees the use of the Internet connection is severely used), and waits again for 139849.doc 200949567 Loss and/or higher latency The indication has exceeded the maximum data rate 922 available, and the lower level for the peak data rate 941 may be found again, but the lower level may be higher than the level prior to testing the increased data rate. Thus, the peak data rate 941&apos; can be found by using this technique (and other techniques like it) and periodically adjusted as needed. The peak data rate 941 is broken by the video compressor 4G4 to allow the video stream to flow to the user's maximum data rate. The logic for determining the peak data rate can be implemented at the user premises 2H and/or on the colocation service 21〇. At the user premises 211, the user device performs computation to calculate the peak data rate and transmits this information back to the host management service 21; at the host hosting service 210, the hosting device at the hosting service A calculation is performed to determine the peak data rate based on statistics received from the client 415 (eg, peak loss, delay, maximum data rate, etc.). Figure 9a shows an example video stream data rate 934 that does not have substantial scene complexity and/or motion, which is generated using the «(4) block compression technique described previously and illustrated in Figures h, and 8 and Figure 8. . The video compressor 404 is configured to output compressed video below the average data rate of the peak data material 941 and note that most of the time, the video stream data rate remains below the peak data rate 941. A comparison of the data rate 934 with the video stream data rate 634 shown in Figures &amp; (which uses I/p/B or I/p maps to produce a magical comparison shows that the cyclic image block compression produces a much smoother data rate. However, at frame 2 times peak 952 (which is close to 2 times peak data rate 942) and frame 4 times peak 954 (which is close to 4 times peak data rate 944), the data rate still exceeds the front data rate 941, which is not In practice, even for 139849.doc -82- 200949567 high-motion video from fast-changing video games, the peak value exceeding the peak data rate 941 appears in the frame of less than 2%, more than 2 times the peak data. The peak of rate 942 is rare, and peaks above 3 times the peak data rate 943 are difficult to see. See, however, when it does occur (for example, during a scene transition), the required data rate must produce good quality. Video image. One way to solve this problem is to simply configure the video compressor so that

The maximum lean rate output is paid as the peak data rate 941. Unfortunately, the resulting video output quality during the peak frame is poor because the compression algorithm is "extremely lacking" bits. The resulting compression artifacts occur when there is a sudden transition or rapid motion, and in time, the user begins to realize that the artifact always appears suddenly when there is a change in the dog or a rapid movement, and it can become quite annoying. . Although the human visual system is quite sensitive to visual artifacts that occur during sudden changes or rapid motion, it is not very sensitive to the reduction in frame rate detected in such situations. In fact, when these sudden changes occur, it seems that the human visual system is focused on tracking these changes, and if the frame rate temporarily drops from 6〇fps to 30 fps and then immediately returns to 6〇fps, then The visual system will not notice. In addition, in the case of very sharp transitions (such as changes in the canine scene), if the frame rate drops to 2 〇 fps or even 15 fps and then immediately returns to 6 〇 fps, the human visual system will not notice. As long as the frame rate reduction occurs only occasionally, it seems to the human observer that the video system is continuously executed at 60 fps. This feature of the human visual system is utilized by the technique illustrated in Figure 9b. Server 402 (from Figures 4a and 4b) produces an uncompressed video wheel outflow at a stable frame rate (in one embodiment, at 60 fpS). The timetable shows the output of each frame 961-970 for each 1/60 second. Starting from frame 961, each uncompressed video frame is output to the low latency video compressor 404, and the low latency video compressor 404 compresses the frame for less than one frame time, resulting in the first The frame of the frame is compressed 丨 981.麫

The data generated for compressed frame 1 981 may be larger or smaller depending on many of the factors previously described. If the data is small enough to transmit the peak data rate 941 to the client 415 at frame time (1/6 sec) or less than one frame time, then at the transmission time (xmit time) 991 (indicating the transmission time) It is transmitted during the length of the arrow of duration. In the next frame time, the server 402 generates an uncompressed frame 2 962, compresses it into a compressed frame 2 982, and transmits it at a peak data rate of 94 小于 during less than one 992 period. The frame time is transmitted to the client 415.

Next, in the next frame time, the feeder 4〇2 produces an uncompressed frame 3 963. When the uncompressed frame 3 963 is compressed by the video compressor, the resulting compressed frame 3 983 is more data than can be transmitted in the frame time at the peak data rate 941. Therefore, it is transmitted during the transmission time (2 times peak) 993, which occupies part of all frame time and next frame time. Now, during the next frame time, the server 402 generates another uncompressed frame 49M and rotates it out to the video compressor 4G4, but the data is ignored and by 974 for the day/this is because Video compressor 404 is configured to ignore other uncompressed video frames that arrive as they still transmit previously compressed frames. Of course, the video decompressor of the client 139849.doc -84 - 200949567 415 will fail to receive frame 4, but simply continues to display frame 3 on display device 422 for 2 frame times (ie, Temporarily reduce the frame rate from 60 fps to 30 fps). For the next frame 5, the server 402 outputs the uncompressed frame 5 • 965, compresses it into a compressed frame 5 985 and transmits it in a frame 1 during transmission time 995. The user-side video decompressor decompresses frame 5 and displays it on display device 422. Next, the feeder 402 loses the uncompressed frame 6 966, and the video compressor 404 shrinks it to the compressed frame 6 986, but the data obtained at this time is very large. The compressed frame is transmitted at peak data rate 941 during transmission time (4 times, value) 996, but it takes almost 4 @box times to pass the (four) box. During the next 3 frames, video compression! M0Q is slightly from the 3 frames of the word server, and the decompressor of the client 415 stably holds the frame 6 on the display device 422 for 4 frames (that is, temporarily the frame rate). Reduced from 6 〇 to 15fPS). Next, finally, the feeder 402 outputs the frame 10 970, compresses it into a compressed frame 1 () tear according to the reference tfll, and transmits it during the transmission time 997' and the decompressor of the client 415. The frame 1 is decompressed and displayed on the display device 422 and the re-time video is restarted at 60 fps. Note that although the video compressor 4〇4 has a video frame from the video stream generated by the server 4〇2, it does not discard the audio data (regardless of the form of the audio system). ), and when I hold the video frame, the video compressor 404 continues to compress the audio data and transmits the channel to the client 415, and the client 415 continues to decompress the audio data. · For use by the user to play back the sound J39849.doc -85· 200949567 No matter what device. Therefore, during the period in which the frame is discarded, the audio continues without being attenuated. Compressed audio consumes a relatively small percentage of the bandwidth compared to compressed video and therefore does not have a significant impact on the overall data rate. Although not illustrated in any of the data rate maps, there is always a data rate capacity reserved for the compressed audio stream within the peak binary rate 941. 'Select the example just described in Figure 9b to illustrate how the frame rate drops during the peak data rate, but the unspecified system# uses the cyclic Ι/P image block technique described previously. And the consequent discarding of the frames is rarely 'even during high scene complexity/high action sequences (such as those found in video games, movies, and some-application software). Because, 匕' reduced frame rate is rare and temporary, and the human visual system does not detect them. If the frame rate reduction mechanism just described is applied to the video stream data rate as described in _, then the video stream data rate in the figure % is shown in the figure %. In this example, the 2x peak 952 has been reduced to a flattened 2x peak 953 'S 4 times the peak 955 has been reduced to the flattened * times the peak %$, and the entire video stream data 934 is reduced to or Below the peak data rate of 941 °, using the techniques described above, high-motion video streams can be transmitted with low latency via the universal network and consumer-grade Internet connection. In addition, in an office environment on a LAN (for example, 100 Mbs Ethernet or 802.Hg wireless network) or private network (10), data center and office Mbps connection, no Peak transmission 139849.doc • 86 · 200949567 Motion video stream, so that multiple users (for example, 192 〇 x 〇 〇 8 〇 under 6 〇 fPs at 45 Mbps) can use LAN or shared private data connection, Do not flood the network or network switch backplane without overlapping peaks. Data rate adjustment

In one embodiment, the colocation service 210 initially evaluates the channel's available maximum data rate 622 and latency to determine the appropriate data rate for the video stream and then dynamically adjusts the data rate in response thereto. To adjust the data rate, the colocation service 210 can, for example, modify the image resolution and/or the number of frames per second of the video stream to be sent to the client 415. In addition, the colocation service can adjust the quality level of the compressed video. When changing the resolution of the video stream (e.g., from 1280x720 resolution to 64〇x36〇), the video decompression logic 412 on the client 415 can scale the image to maintain the same image size on the display screen. In one embodiment, the colocation service 210 suspends the game in the event that the channel is completely exited. In the case of a multiplayer game, the colocation service reports to other users that the user has quit the game and/or paused the game for other users. The discarded or delayed packet is in a real case. 'If the data is lost due to packet loss between the video squad 4〇4 and the client 415 in Figure 4a or 4b, or cannot be decompressed due to reaching + night. And the video that satisfies the delay requirement of the decompressed frame is received without loss of order, and the video decompression logic 412 can divide the visual artifact. In the streaming I/p frame implementation, if there is a lost/delayed white packet, the entire screen is affected, which may cause the screen to completely freeze. Name 139849.doc -87- 200949567 A time period or display other screen wide visual artifacts . For example, if the lost/delayed packet causes the loss of the I frame, then the decompressor will lack all of the p-frames to follow before receiving the new I-frame. If the P frame is missing, it will affect the following p-frame for the entire screen. 'How long before the I frame appears, this will have a longer or shorter visual impact. Use as shown in Figure 7a and Figure 7b. Demonstrating interleaved I/p blocks, lost/delayed packets are less likely to affect the entire screen because it only affects the image blocks contained in the affected packets. If the data of each image block is sent in an individual packet, if the packet is lost, it affects only one image block. Of course, the duration of the illusion will depend on whether the 丨 image block packet is lost and how many frames will be spent before the image block appears in the event of a p-picture block loss. However, assuming that the different image blocks on the screen are updated very frequently (perhaps with each frame), even if one of the image blocks on the screen is affected, the other image blocks may not be affected. In addition—if the packet is lost at the same time (for example, the power of the adjacent DSL line: the spike of the bedding stream), some image blocks will be more affected than other image blocks, but because some of the image blocks will be replaced by new ones. The image block is updated quickly, so it is only temporarily affected. Moreover, in the case of the implementation of the streaming I/p frame, not only the I frame is the most critical frame, but the old frame is extremely large, so if the ft causes the event of the discarded/delayed packet, then the I is much smaller. The image block t 'I frame x affects the existence of a higher probability (that is, any part of the w frame is lost) is impossible to decompress the I frame). For all of these reasons, when the packet is discarded/extended, the block is caused by a much smaller visual artifact than when the ι/p frame is in the same state. 139849.doc -88 - 200949567 An embodiment attempts to reduce the effects of lost packets by intelligently encapsulating compressed video blocks into TCP (Transmission Control Protocol) packets or UDP (User Datagram Protocol) packets. For example, in one embodiment, as long as .τ can' align the image block with the packet boundary. Figure 1Ga illustrates how an image block can be encapsulated in a series of packets 1001_1005 without implementing this feature. Specifically, in Figure 1A, the image block crosses the packet boundary and is inefficiently packaged so that the loss of the single-packet causes multiple frames to be lost. For example, if the packet 1003 or 1004 is lost, I lose three image blocks, resulting in visual artifacts. In contrast, Figure 10b illustrates image block packaging logic 1010 for intelligently encapsulating image blocks within a packet to reduce the effects of packet loss. First, image block encapsulation logic 1010 aligns the image block with the packet boundary. Therefore, the image blocks ΤΙ, T3, T4, T7, and T2 are aligned with the boundaries of the packets 1001_1 〇〇 5, respectively. Image block encapsulation logic also attempts to combine image blocks into packets in a potentially more efficient manner without crossing the packet boundaries. Based on the size of each Φ in the image block, the image blocks T1 and T6 are combined in one packet 1〇〇1; T3 and T5 are combined in one packet 1002; and the image blocks are combined into one packet 1003. Medium 'Add image block D8 to packet 1〇〇4; and add image block T2 to packet 1005. Therefore, under this scheme, a single packet loss will result in the loss of no more than 2 image blocks (rather than the 3 image blocks as illustrated in Figure 10a). An additional benefit of the embodiment shown in Figure 1 is that the image blocks are transmitted in a different order in which they are displayed within the image. This approach produces less noticeable artifacts on the display if the adjacent packets are lost due to the same event that interferes with the transmission (which would affect areas that are not close to each other on the screen 139849.doc •89-200949567). An embodiment uses forward error correction (FEC) techniques to protect a particular portion of the video stream from channel errors. As is known in the art, FEC techniques such as Reed-Solomon and Viterbi generate error correction information and attach it to the material transmitted via the communication channel. If an error occurs in the basic data (for example, the frame), FEC can be used to correct the error. The FEC code increases the data rate of the transmission, so ideally it is only used when it is most needed. If the material is being sent and it will not result in a very noticeable visual artifact, then the FEC code may preferably not be used to protect the material. For example, the p-image block immediately before the missing I image block will only produce a 1/60 second visual artifact on the screen (ie, the image block on the screen will not be updated). This visual artifact can hardly be detected by the human eye. As the p-image block is further backwards from the I-image block, the loss of the p-image block becomes more and more noticeable. For example, if the image block cycle pattern is that an I image block is followed by 15 P image blocks before the image block is available again, then if the P image block immediately after the z image block is lost, it leads to The image block shows an incorrect image for 15 frame times (at 60 fpS, it will be 250 milliseconds). The human eye will easily detect a 250 millisecond flow interruption. Therefore, the more the p image block is from the new I image block (that is, the closer the P image block follows the image block), the more noticeable ~ is noticed. As discussed earlier, in general, the closer the p-image block follows the Ϊ image block, the smaller the data used for the P-image block. Therefore, the p-image block following the I image block is not only critical for protection from loss, but also smaller. In addition, in general, the smaller the information that needs to be protected, the smaller the FEC code needed to protect it.

Thus, as illustrated in Figure la, in one embodiment, only the j image block has an FEC code due to the importance of the j image block in the video stream. Therefore, FEC '1101 contains an error correction code for 1 video block 1100 and FEC 1104 contains an error correction code for I picture block 1103. In this embodiment, for? The image block does not produce FEC. • In one embodiment illustrated in Figure lib, an FEC code is also generated for P-picture blocks that are most likely to cause visual artifacts when lost. In this embodiment, the coffee 11〇5 provides error correction codes for the first 3 p-image blocks but not for the following p-images. In another embodiment, an FEC code is generated for a P-image block having the smallest data size (which would tend to be the first Ρ image block that appears after the Γ image block, which is most critical for protection). In another embodiment, instead of transmitting the FEC code along with the image block, the image block is transmitted twice, each time in a different packet. If a φ packet is lost/delayed, another packet is used.

415 is reproduced without distortion. For example, ticking or snoring will result in particular undesirableness. The coffee code helps ensure that the audio content is:

In another embodiment, instead, the audio data is detected. I39849.doc -91 - 200949567 If one packet is lost/delayed, another packet is used. In addition, in one embodiment illustrated in the figure, FEC^1121 and 1123 are respectively used for user input commands (eg, button presses) 112 from the user terminal 415 to the host escrow service 21, respectively. 1122. This is important because missing button presses or mouse movements in a video game or application can result in an undesirable user experience. In another embodiment, rather than transmitting the FEC code along with the user input command data, the user input command data is transmitted twice, each time in a different packet. If a packet is lost/delayed, another packet is used. In one embodiment, the colocation service 21 evaluates the quality of the communication channel with the client 415 to determine whether to use the FEC, and if so, determines which portion of the video, audio, and user commands should be applied for the FEc. The "quality" of the evaluation channel may include functions such as estimating packet loss, delay, etc. as described above. If the channel is particularly unreliable, the colocation service 21 can apply FEc to all I video blocks, p video blocks, audio and user commands. In contrast, if the channel is reliable, the colocation service 21 can apply FEC only for audio and user commands, or can apply fec to audio or video, or can not use FEC at all. Various other arrangements of FEC applications can be used while still adhering to these basic principles. In an embodiment, the colocation service 2 10 continuously monitors the status of the channel and changes the fec policy accordingly. In another embodiment, referring to the figure and the servant, when the packet is lost/delayed, resulting in loss of image block data, or if the FEC is unable to correct the lost image block data due to the particularly bad packet loss, Using 139849.doc -92- 200949567, the client 415 evaluates how many frames remain before the new i-picture block will be received and the round-trip delay of the client 415 to the colocation service 210 is less than the new round-trip delay. The old image block should reach the number of previous frames. Then the client 415 sends a message to the colocation service 21, please. Please bundle the new 16 image block. This message is routed to the video squeezer 404, which does not generate a ρ image block for the image block from which the data has been lost, but instead produces a [image block. Assuming that the system shown in Figure 4a and the servant is designed to provide a normal runback delay of 8 〇 milliseconds, this results in the image block being corrected to 80 s and within (at 60 fpS, the frame has 16 67 The duration of milliseconds is therefore in the full frame time, the 8G millisecond delay will result in a corrected image block in (four) milliseconds, and 83.33 milliseconds is 5 frame times, which is an attractive interruption, but far less than (for example) For 15 frames, the 25-millisecond interrupt is noticeable). This occurs when the compressor 4〇4 leaves its usual cycle order! In the image block, if the I image block will cause the bandwidth of the frame to exceed the available bandwidth, the compressor 404 will delay the loop of the other image blocks so that the other image blocks receive the P image block during the frame time ( Even if a block of images will normally be a block of images during the frame, and then the usual loop will continue from the next frame, and usually the image block that has received the image block in the previous frame will Receive an I image block. Although this action temporarily delays the stage of the ruler loop, it will usually be visually unobtrusive. Video and Audio Compressor/Decompressor Implementation Figure 2 illustrates a particular embodiment in which multiple cores and/or multiple processors 1200 are used to compress eight image blocks in parallel. In one embodiment, at 2.66 GHz or more High-performance dual-core processor, quad-core Xe〇n cpu computer 139849.doc • 93· 200949567 system, each core as an independent process to implement open source x264 Η.264 compressor. However, a variety of other hardware/software configurations can be used while still adhering to these basic principles. For example, each of the CPU cores can be replaced by an H.204 compressor implemented in an FPGA. In the example shown in Figure 12, cores 1201-1208 are used to process I and P blocks simultaneously as eight separate threads. As is well known in the art, current multi-core and multi-processor computer systems are inherently integrated with multi-threaded processing operating systems such as Microsoft Windows XP Professional (64-bit or 32-bit) and Linux. Ability to perform multi-thread processing. In the embodiment illustrated in Figure 12, since each of the eight cores is only responsible for one image block, it operates largely independently of the other cores, each performing a separate instantiation of x264. Use a PCI Express xl-based DVI capture card (such as the Sendero Video Imaging IP Development Board from Microtronix of Oosterhout in the Netherlands) to capture uncompressed video at 640x480, 800x600 or 1280x720 resolution, and the FPGA on the card Direct memory access (DMA) is used to transfer the captured video to the system RAM via the DVI bus. The image blocks are configured in a 4x2 configuration 1205 (although this is illustrated as a square image block, in this embodiment it has a 160x240 resolution). Each instantiation of x264 is configured to compress one of the eight 160x240 image blocks and is synchronized such that each core enters a loop after initial I image block compression, each frame and another The frames are out of phase to compress an I image block followed by seven P image blocks, as illustrated in FIG. At each frame time, the resulting compressed 139849.doc -94 - 200949567 thumbnail blocks are combined into a packet stream using the techniques previously described, and the compressed image block is then transmitted to the destination client 41 5 . Although not illustrated in Figure 12, if the data rate of the combined 8 image blocks exceeds the specified peak data rate 941, all 8 x264 processes will temporarily suspend the necessary frame time until the 8 for the combination has been transmitted. The data of the image block. In one embodiment, client 415 is implemented as a software executing on 8 instantiated PCs of FFmpeg. The receiving process receives eight image blocks and routes each image block to FFmpeg for instantiation, and FFmpeg instantiates the decompressed image block and reproduces it to the appropriate image block location on display device 422. The client 41 5 receives the keyboard, mouse or game controller input from the input device driver of the PC and transmits it to the server 402. The server 402 then receives the input device data received by the application and applies it to the game or application executed on the server 402. The server 402 is a PC that executes Windows using an Intel 2.16 GHz dual-core CPU. Server 402 then generates a new frame and exports the new frame to the motherboard based graphics system via its DVI output or via the DVI output of the NVIDIA 8800GTX PCI Express card. At the same time, the server 402 outputs audio generated by the game or application via its digital audio output (eg, S/PDIF), which is coupled to a dual quad-core Xeon-based PC that implements video compression. Digital audio input. The Vorbis open source audio compressor is used to compress audio simultaneously with video, regardless of the core that can be used to process the thread. In one embodiment, the core of the compressed image block is first compressed to perform an audio signal 139849.doc • 95· 200949567. The compressed audio is then transmitted along with the compressed video and the compressed audio is decompressed at the client 415 using a Vorbis audio decompressor. The colocation service server center distributes light traveling through the glass (such as an optical fiber) at a fraction of the speed of the light in the vacuum, and thus the exact propagation speed of the light in the fiber can be determined. However, in practice, considering the routing delay, transmission inefficiency, and other elapsed time, we have observed that the optimal delay on the Internet reflects a transmission speed that is closer to 50% of the speed of light. Therefore, the optimal 1 mile round trip delay is about 22 milliseconds, and the optimal 3000 mile round trip delay is about 64 milliseconds. As a result, a single server on the US coast will be too far away to be able to compromise the client on the other coast (which may be up to 3 miles away) with the desired delay. However, as illustrated in Figure 13a, if the colocation service 21 is located at the center of the United States (eg, Kansas, Nebraska, etc.), the distance to any point in the continental United States is approximately 5 〇. For miles or less than 1,500 miles, the internet latency can be as low as 32 milliseconds. Referring to Figure 4b, note that although the worst case delay allowed by the user ISP 453 is 25 milliseconds, in general, in the case of DSL and cable modem systems, we observe a delay of approximately 10-15 milliseconds. Again, 'Fig. 4b assumes that the maximum distance from the user premises 211 to the colocation center 210 is 1 mile. Therefore, the typical 15 millisecond user isP round trip delay and 32 milliseconds back and forth. In the case of a maximum internet distance of 15 miles for the travel delay, the total round trip delay from the moment the user actuates the input device 421 to the response seen on the display device 422 is 139849.doc • 96· 200949567 1 + 1 + 15+32 + 1 + 16+6+8=80 milliseconds, therefore, typically achieves 8 milliseconds (10) (4) in the Internet distance of 1500 miles. Any use of the user's ISP delay 453 is short enough in the continental United States. The site accesses a single server center located at the center. In another embodiment illustrated in Figure 13b, the colocation service (4) server center HS1-HS6 is strategically located around the United States (or other geographic area), a particular larger colocation service server towel Positioned close to the center of the population (eg 'HS2 and HS5'). In the embodiment, the server center Cong-intestine exchanges information via the network 13〇1, and the network η〇ι can be an internet or a private network or a combination of the two. 4 (4) server center, the case Under the lower delay, it can provide services to users with high user Isp delay 453.

The distance on the B-week network is indeed a factor in the response to the Internet through the Internet, but sometimes other factors that are largely unrelated to the delay also work. Sometimes the packet flow is routed to a remote location via the Internet and returned again, resulting in a delay from the long loop. Sometimes there are routing devices that are not operating properly on the path, resulting in delays in transmission. There are sometimes traffic that overloads the path, which introduces delays. In addition, sometimes, there is a failure to prevent the user's ISP from routing to a given destination. Therefore, while the universal Internet usually provides connections from one point to another with fairly reliable and optimal routing and latency, this fairly reliable and optimal route and delay is largely determined by distance ( Especially in the case of long-distance connections that lead to the outside of the local area of the user) 'but the reliability and delay are not guaranteed and often cannot be used by 139S49.doc -97- 200949567 Achieving β to a given destination on the general Internet In one embodiment, 'when the user client 415 is initially connected to the colocation service 210 to play a video game or use an application, the client is available at startup and available. Each of the colocation service server centers Hs丨_HS6 communicates (eg, using the techniques described above). If the delay is low enough for a particular connection, use the connection. In one embodiment, the client communicates with a subset of all colocation server servers or colocation service servers, selecting the colocation service server center with the lowest latency connection. The client can select the service with the lowest latency connection, or the server can identify the server center with the lowest latency connection and provide this information (for example, in the form of an internet address) to the client. The right-hand host hosting service server center overload and/or user's exhaustion or application can tolerate the delay of another, less loaded colocation service server center, then the client terminal 415 Redirect to another host in the server service server center. In this case, the game or application being executed by the user will be suspended on the feeding device 4 in the user's super uniform and the game or application status data will be transferred to the other: The server service at the server hosting server. Then the % or the game will be reopened. In an embodiment, the colocation service add will wait until the game or the liaison reaches the (four) stop (eg, between levels in the game, or after the user initiates a "save" operation in the application) Only transfer. In yet another embodiment, the colocation service 21 (will wait until the user activity ceases for a specified period of time (eg, ^139849.doc -98.200949567 clock) and then will start transmitting at the time as above As described, in an embodiment, the "hosting service 21" subscribes to the Internet bypass service 4 of Figure 14 as if attempting to provide a guaranteed delay to its client. As used herein, the Internet bypass The service provides a service from a point of St 5 S wi. on the Internet to another point of private network routing with guaranteed features (eg, latency, data rate, etc.). f service 210 is being used in "(10) heart.

AT&amp;T's pro-service receives a large amount of traffic from the system (rather than routing to the central office based on AT&amp;T's San Francisc〇), then the host f service 210 will be in San Franeise. Hiring high-capacity private data from a service provider (possibly AT&amp;T itself or another provider) between the ten central office of the base and the server center for the colocation service 210 connection. Then, ############################################################################################# Although the private data connection is usually more expensive than the route through the universal Internet, as long as it maintains a small percentage of the colocation service to the user, the total cost is low, and (4) (4) is more consistent. Service experience. In the event of a power failure, the server center often has two spare power layers. The first layer is typically backup power from a battery (or from an alternate ready-to-use energy source, such as a flywheel that remains operational and attached to the generator) that provides power immediately in the event of a power mains failure and remains in the server~ Running. If the power failure is temporary and the mains return quickly (eg, 139849.doc -99- 200949567, for example, within one minute), then the battery is required to maintain the server center. However, if the power failure lasts for a long period of time, the generator (e.g., diesel powered) is typically activated to replace the battery and the generator can operate as long as it has fuel. These generators are extremely expensive because they must be capable of generating as much power as the servo center typically draws from the mains. In one embodiment, each of the colocation services HS1_HS5 shares user data with each other so that when there is a power failure in the server center, the game and application in progress are paused, and then the game or The application status data is transferred from each server to the server 402 at the other server towel, and then the user of each user is notified to communicate to the new word processor. 2. Assuming that the situation occasionally appears, the user is transferred to the colocation service center that is unable to provide the best &amp; time (ie, the user will only have to tolerate the duration of the higher delay duration power failure) It may be acceptable that it will allow for the option of transferring a much wider range of users. For example, given the time difference across the United States, users on the East Coast may be sleeping at 11:30 PM, while the makers on the West Coast are starting to peak on the video at 8:30 PM. If there is a power failure in the hosted service server center on the West Coast, there may not be enough West Coast servers 402 to handle all users at the other hosting server center. In this case, some users may be transferred to the hosted service server center with the available word service 402 on the East Coast, and the consequences for the user will be higher. Once the user has transferred the feed from the loss of power, the server center can then start its feeder and set up an orderly cutoff for the battery (or other immediate power backup). ) Cut off all equipment before exhausting. In this way, the cost of the generator for the servo center can be avoided. In one embodiment, during the time of severe loading of the colocation service 210 (either due to peak user loading or because one or more server centers fail), based on the game or application being used by the user The program delay requires the user to be transferred to another server center. Therefore, users of games or applications that require low latency will be given the preference for available low latency server connections with limited supply. Hosted Hosting Service Features Figure 15 illustrates an embodiment of the components of the server center for the hosted hosting service 210 utilized in the following description of the features. As with the colocation service 21 illustrated in the figure, the components of this server center are controlled and coordinated by the colocation service 21〇 control system 4〇1 unless otherwise conditional.指引 Direct the incoming Internet traffic 15〇1 from the user client 415 to the incoming route 15 02. Typically, incoming Internet traffic screenings will enter the server center via high-speed fiber-optic connections to the Internet, but any network connectivity component with sufficient bandwidth, reliability, and low latency will suffice. Inbound routing 1502 is a network (which can be implemented as an Ethernet, fiber channel network, or via any other transport component) switch and a system that supports the routing servers of the switches, which arrives Packets are routed and routed to the appropriate application/game server 1521_1525. In one embodiment, the packet delivered to the particular application/game server represents a subset of the data received by the client 139849.doc -101 - 200949567 and/or may be other components within the data center (eg, Network connection components, such as gateways and routers, are translated/changed. In some cases, the packet is routed to more than one server 1521·1525 each time, for example, if the game or application is executing concurrently on multiple servers. The RAID arrays 1511_1512 are connected to the human routing network 1502 such that the application/game server can read the arrays 1511 - 1512 and the write shaku 11 ) arrays 1511 - 1 512 . In addition, the size array 1515 (which can be implemented as multiple RAID arrays) is also coupled to the incoming routing 1502, and the data from the RAID array 1515 can be read from the application/game servers 1521-1525. Incoming routing 15〇2 can be implemented in a variety of prior art network architectures (including tree-structured switches, inbound Internet traffic 在5〇ι at its root); in a mesh structure interconnecting all of the various devices Implemented, or implemented as a sub-network sequence of interconnects (concentrated traffic among interworking devices and centralized traffic isolation among other devices). One type of network is configured as a SAN, which, although commonly used for storage devices, can also be used for general high-speed data transfer between devices. Also, the application/game servers 1521-1525 can each have multiple network connections to the incoming route 15〇2. For example, server 1521-1525 may have a network connection to a subnet attached to RAID arrays 1511-1512 and another network connection to a subnet attached to other devices. The application/game servers 1521-1525 can be configured identically, somewhat differently, or all differently, as previously described with respect to server 402 in the illustrated embodiment of the fishing. In one embodiment, each user typically has at least one application/game servo when using the colocation service 139849.doc • 102· 200949567

1521-1525. For simplicity of description, it will be assumed that a given user is using the application/game server 1521, but multiple servers can be used by one user, and multiple users can share a single application/game server 1521 -1525. The user's control input sent from the client 41 5 (as previously described) is received as an incoming Internet traffic 1 5 〇1 and routed to the application/game server via the incoming route 1502. 52 1. The application/game server 1521 uses the user's control input as a control input to the game or application executing on the server and calculates the video and the next frame of the audio associated therewith. The application/game word processor 1521 then outputs the uncompressed video/audio 1529 to the shared video compression 15 3 0. The application/game server can output uncompressed video via any component (including one or more super-fast Ethernet connections), but in the real example, the 'holes are connected via D VI, And audio and other compression and communication channel status information is output via a universal serial bus (USB) connection. The shared video compression 1530 compresses uncompressed video and audio from the application/game server 1521 1525. This compression can be implemented entirely in hardware or in hardware that executes the software. There may be a dedicated compressor for each application/game server 1521-1525 or if the compressor is fast enough, the compression/compression may be collapsed from video/audio from more than one application/game server 152Μ525. For example, at 6〇fps, the video frame time is off milliseconds. If a compressor can compress 1 frame in i milliseconds, then the compressor can be used to compress from up to 16 applications/game servers 139849 by taking round-ups from one server. .doc -103- 200949567 1521-1525 Video/audio, which stores the state of each video/audio compression process and switches the background as it cycles through the video/audio stream from the server. This results in substantial cost savings for the compressed hardware. Since different servers will complete the frame at different times, in one embodiment, the compressor resources are in a shared state with shared storage components (eg, RAM, flash memory) for storing the state of each compression process. In the pool 153, and when the server 1 521 -1 525 is complete and ready to be compressed, the control component determines which compression resource is available at that time, provides the state of the compression process of the server and the to be compressed for the compressed resource Uncompressed video/audio frame. Note that one of the states of the compression process for each server includes information about the compression itself, such as the decompressed frame buffer data of the previous frame (which can be used as a reference for the P image block), Resolution of video output; quality of compression; image block structure; allocation of bits per image block, compression quality, audio format (eg stereo, surround sound, Dolby® AC-3). However, the compression process state also includes information about the following communication channel status: peak data rate 941, and whether the previous frame (as illustrated in the figure) is currently being output (and therefore the current frame should be ignored), and potentially There are channel characteristics that should be considered in compression (such as excessive packet loss) that affect compression decisions (eg, in terms of the frequency of I-picture blocks, etc.). Since the peak data rate 941 or other channel characteristics change over time, as determined by the application/game server 1521_1525 that supports each user monitoring the data sent from the client 415, the application/game server 152 1-1525 sends the relevant information to the shared hardware compression 139849.doc -104- 200949567 1530. The shared hardware compression 1530 also encapsulates the compressed video/audio using components such as those previously described, and, where appropriate, applies the fec code, copies the particular material, or takes other steps to fully ensure the view. The 7 data stream is received by the client 415 and decompressed with a viable high quality and reliability. Some applications (such as those described below) require φ the video/audio output of a given application/game server ^21-1525 to be available at multiple resolutions (or in multiple formats) at the same time. . If the application/game server 1521-1525 thus notifies the shared hardware compression 153 resource, the uncompressed video audio 1529 of the application/game word processor 1521_1525 will be in a different format, different resolution and/or Simultaneous compression in different packet/error correction structures. In some cases, some compression resources may be shared among multiple compression processes that compress the same video/audio (eg, in many compression algorithms, there are steps to scale the image to multiple sizes before applying compression). If you need to output images of different sizes, this step can be used to simultaneously servo several compression processes. ^ In other cases, separate compression resources will be required for each format. In any event, the compressed video/audio (10) for all of the various resolutions and formats required for a given application/game server 1521-1525 (- or more) is simultaneously output to the outgoing routing (10). In the implementation of the financial, the output of the video/audio 1539 is in the UDp format, so it is a one-way packet stream. The outgoing routing network 1540 includes a series of routing feeders and switches, and the 139849.doc 200949567 series routing server and switch streams each compressed video/audio stream via the outgoing Internet traffic 1599 interface (which Typically connected to the optical interface of the Internet) to the desired user or other destination and/or back to the delay buffer 1515, and/or back to the incoming route 15〇2, and/or via private It is output by a network (not shown) for video distribution. Note (as described below): Outbound Routing 1540 can simultaneously output a given video/audio stream to multiple destinations. In the only example, this is done using Internet Protocol (JP) multicasting. At the same time, a given UDP stream is streamed to multiple destinations' and the broadcast is repeated by the way (4) server and switch in the outgoing route 154(). The multiple destinations of the broadcast may be via the Internet to the user terminals 41 5 of the plurality of users, to the plurality of application/game servers 1521-1525 via the routing route 55 〇2, and/or to one Or a plurality of delay buffers 1515. Thus, the output of a given server 1521_1522 is compressed into one or more formats and each compressed stream is directed to one or more destinations. Additionally, in another embodiment, if multiple application/game servers 1521-1525 are simultaneously used by a user (eg, in a parallel processing configuration for generating 3D output with complex scenes) and each The server generates what it pays: like it. For the P-knife, the video output of the plurality of feeding devices 1521-1525 can be combined into a combined frame by the common hardware compression 15 3 ,, and the combined frame is processed as described above from the other point as if it came from a single Application/game server 1521-1525. Note that in one embodiment, a copy of all of the video generated by the application/game server 1521_ 1525 (at least at the resolution or resolution of the video viewed by the user) is recorded in the delay buffer 1515 for a duration of 139849.doc 200949567 Less than a certain number of minutes (15 minutes in one embodiment). This allows each user to "rewind" the video from each session to check previous work or performance (in the case of the game). Thus, in an embodiment, each compressed video/audio output 1539 stream routed to user client 415 is also multicast to delay buffer 1515. When the video/audio is stored on the delay buffer 1515, the directory on the delay buffer 1515 provides the network φ address of the application/game server 1521·1525, which is the source of the delayed video/audio. A reference between the locations of the delayed video/audio can be found on the delay buffer 1515. The live, instantly viewable, instantly playable game application/game server 1521-1525 can be used not only to execute a given application or video game of the user, but also to establish support for hosting via the host A user interface application for the hosted service 210 of the navigation and other features of the service. A screen shot of the user interface application is shown in Figure 16 ("Game Viewfinder" screen). This particular player interface screen allows the user to view 15 games played (or delayed) from other uses. Each of the "thumbnail" video windows (e.g., 1600) is a live video window in motion that displays a video from a user's game. The view shown in the thumbnail may be the same view that the user is looking at, or it may be a delayed view (eg, if the user is playing a fighting game, the user may not want other users to see where they are hidden and It can choose to delay any view of its game play for a period of time (eg, 10 minutes). The view can also be a camera view of the game that is different from the view of any user. The user can select the game to be viewed simultaneously based on a variety of criteria via the 139849.doc • 107. 200949567 menu selection (not shown in this description). As a small selection of exemplary choices, the user can select a random selection of games (such as those shown in Figure 16). 'All games in one category (both played by different players), only top players in the game, games A given level of player's or lower level players (eg, if the player is learning the basics), a player who is a "partner" (or a competitor), a game with the largest number of viewers, and the like. Note that in general, each user will determine whether the video from their game or application can be viewed by others, and if so, determine which other people can view and when they can be viewed by others, and decide whether the video can only have View in case of delay. The application/game server 1521-1525 that produces the user interface screen shown in FIG. 16 is applied to each user (the application/game server 1521-1525 is requesting a game from the user) The program/game server 1521-1525 sends a message to retrieve the 15 video/audio feeds. The message is sent via the incoming route 1502 or another network. The message will include the size and format of the requested video/audio and will identify the user viewing the user interface screen. A given user may choose to select the "Piracy" mode and not allow any other user to view the video/audio of their game (from their view point or from another view point), or as described in the previous paragraph, the user may choose Allow viewing of video/audio from its games, but delay the video/audio being viewed. User Application/Game Feeder i52i_ Help (it receives and accepts the video/audio that is allowed to be viewed by the user, so it will be notified to the requester, and it will also notify the shared hardware to compress 153. 139849 is required. Doc •108· 200949567 Generates an additional compressed video stream that is formatted or screen size (assuming the format and screen size are different from the generated format and screen size) and will not be compressed for video purposes. Ground (ie, requesting the feeder). If the requested video/audio is only delayed, the requesting application/game server.1521_1525 will be notified as such, and it will be looked up by the lookup delay buffer 1515. The location of the video/audio in the directory and the network address of the application/game feeder 1521·1525 for the delayed video/audio source, and the delayed video/audio from the delay buffer 15丨5... Once all such requests are generated and processed, up to 15 live broadcast thumbnail-sized video streams are routed from the outgoing route 154 to the incoming route 15〇2, to the application that generates the user interface screen. / game server 1521-1525, and will be decompressed and displayed by the server. The delayed video/audio stream may be at an excessive screen size, and if so, the application/game server ΐ52ι_ 1525 will decompress the The stream is scaled down to the thumbnail size. In one embodiment, the request for audio/video is sent to a central "management" service similar to the hosted service control system of Figure ( (Figure i The central "management" service then redirects the requests to the appropriate application/game server 1 521 -1 525. In addition, in an embodiment , may not require a request, because the thumbnails are "pushed" to the users of their users. The simultaneous mixing of audio from 15 games may produce harsh sounds. Users can choose this way All sounds are mixed together (maybe to get the feeling of "喧嚣" produced by all the actions being viewed), or the user can choose to listen to only one game at a time. Audio. Single Tour 139849.doc -109- 200949567 Second Choice: This is done by moving the yellow selection box 1601 to the given game (:, the color frame can be moved by using the arrow keys on the keyboard, by moving the slide It is done by moving the joystick or by pushing another device 2 direction button such as a mobile phone.) Once the single-game is selected, only the audio from the play is played. Also, the game information 1602 is displayed. In the case of 'for example' published trademarks ("EA") and game logos, "Need for Speed Card Canyon" and orange bars indicate the number of people playing games or viewing games at a particular time under relative conditions (here) In the case of a lot, so the game is "hot". In addition, a "status" is provided indicating that there are 8 different instantiations in which 145 players are actively playing Need for Speed games (ie, the game can be played by individual player games or multiplayer games), and there are _ views (This user is among them _). Note that such statistics (and other statistics) are collected by the hosted service control system 4 and stored on the RAID arrays 1511-1512 for maintaining the operation of the host hosting service. Used to properly bill users and pay fees to publishers who provide content. Some statistics are recorded as a result of actions performed by the service control system 4, and some statistics are reported by the individual application/game servers 1521-1525 to the service control system 4〇1. For example, when the game is being viewed (and when the game is stopped viewing), the application/game server 152 executing the game viewfinder application 152 1525 sends a message to the colocation service control system 401 ' The statistics that enable the colocation service control system 401 to update how many games are in view. Some statistics can be used by user interface applications such as this game viewfinder application. 139849.doc -110· 200949567 If the user clicks the start button on their input device, they will see the thumbnail video in the yellow box and the thumbnail video will be live broadcast to full screen size. This effect is shown in the process in Figure 7. Note that the video 1700 is larger in size. In order to implement this effect, the application/game server 1521-1525 requests the full screen size of the game routed to it from the application/game server 1521-1525 executing the selected game (by the user's display device 422) A resolution of the video stream. After the game is executed, the application/game server 1521-1525 notifies the shared hardware compressor 1530 that the thumbnail size of the game is no longer needed (unless another application/game server 1521-1525 requires such a thumbnail) And then it directs the shared hardware compressor 1530 to send a copy of the full screen size of the video to the application/game server 1521_1525 that amplifies the video. The user playing the game may or may not have a display device 422 having the same resolution as the display device of the user who zooms in on the game. In addition, other viewers of the game may or may not have a display φ 422 (and may have a different audio playback component, such as stereo or surround sound) having the same resolution as the user who zoomed in on the game. Therefore, the shared hardware compressor 153 determines whether a suitable compressed video/audio stream that satisfies the requirements of the user of the 4 video/audio stream has been generated, and if a suitable compressed video/audio stream does exist, then The shared hardware compressor 1530 notifies the outgoing route 154 to route the duplicate of the stream to the application/game server 1521_1525 that amplifies the video, and if the appropriate compressed video/audio stream does not exist, the compressed video is suitable. Another copy of the user is directed to the routing route to send the stream back to the incoming route 1502 and the application/game server 521_1525 that amplifies the video. 139849.doc 111 200949567 This server that now receives the full screen version of the selected video will decompress the full screen version and gradually scale it up to full size. Figure 18 illustrates how the screen looks after the game is fully enlarged to full screen and the game is displayed at full resolution of the user's display device 422 (as indicated by the image pointed to by arrow 1800). The application/game server 1521_1525 executing the game viewfinder application sends a message to other application/game servers 1521-1525 that provide thumbnails to indicate that the thumbnails are no longer needed and host the service control server 4 to the host. 〇1 sends a message to indicate that no other games are being viewed. At this point, the only display that is generated is the overlay 1801 at the top of the screen, which provides information and menu controls to the user. Note that as the game progressed, the audience grew to 2, 5, 3 viewers. In the case of so many viewers, there must be many viewers with display devices 422 that have the same or close resolution (each application/game servo ϋ1521_1525 has scaled video for Ability to adjust the fit). Since the game shown is a multiplayer game, the user can decide to add the game at a certain time. For a variety of reasons, the host may not allow the user to add the game. For example, the user may have to

A game that is played remotely (in fact, so it can be viewed on another continent without delay), but for games played 139849.doc 112 200949567, the delay must be low enough for the user to (a) enjoy the game 'and (b) is in the same position as other players who may have a lower latency connection) ° If the user is not allowed to play, the user/game interface user application/game is provided The server 1521-1525 will request the host agent service control server 401 to initiate (ie, locate and start) suitably

应用 The application/game server 1521-1525 configured to play a particular game to load the game from the RAID arrays 1511-1512, and then the host agent service control server 401 will direct the routing route 15〇2 from The user's control signal is sent to the application/game server of the current hosted game and now the hosted game application/game server will guide the shared hardware compression 1530 self-compression from the hosted game viewfinder application. The video/audio of the application/game server switches to compress the video/audio from the application/game server of the current hosted game. The vertical synchronization of the game viewfinder application/game service with the new application/game server of the colocation game is not synchronized, and therefore there may be time i between the two synchronizations. Since the shared video compression hardware WO will start compressing the video after the application/game server 1521_1525 completes the video frame, the first frame from the new server can be completed earlier than the full frame time of the old server. The first frame from the new server may be different, before the frame is completed. (For example, consider the transmission time outside the picture 992: if not (four) shrinks 3 963 completes early - frame One and a half of the time, it will impact the transmission time 992). In this case, the shared video compression hardware 1530 will ignore the first frame from the new feeder: for example, ignore (974) frame 4 964), and the client 415 will be from the old feeding service 139849.doc • 113· 200949567 The last frame of the device maintains an extra frame time, and the shared video compression hardware 1530 will begin compressing the next frame time video from the new application/game server of the colocation game. For the user, the transition from the application/game server to another application/game server will be visually seamless. The colocation service control server 〇1 then switches the application/game server i52i_i525, which informs the host escrow game viewfinder, to the idle state until it is needed again. The user can then play the game. In addition, the exceptional game will be played instantly on the sensation (because the game has been loaded from the Raid array 1511-512 to the app/game feeder l52i_i525 at a bite/second speed) and will be borrowed The game is loaded by the ideal drive, scratchpad configuration (in the case of Wind〇ws2) along with the operating system that is configured for the game, to the exact server that is suitable for the game, and is impossible Other applications that compete with the operation of the game are executed on the server. Also, as the user progresses in the game, each of the segments of the game will be loaded at a rate of one billion bits per second (i.e., less than one billionth bit) from the RAID array, Jl5U -1512m server towel, and because of the huge storage capacity of the RAID array 1511-512 (because it is a resource for many users, it can be very large, but still cost-effective), so ^ pre-calculated geometry (four) History or other game clip settings are stored on the RAID array 1511_1512 and loaded very quickly, because the hardware and computing power of each application/game server 1521_1525 is known, so it can be pre- Calculate pixels and vertex shading. So, the game can be launched almost instantly, it will be executed in an ideal environment, 139849.doc -114- 200949567 and the subsequent clips will be loaded almost instantly. However, in addition to these advantages, the user will be able to view other people playing the game (via the previously described game viewfinder, and other components), and both • decide if the game is interesting 'and if so' then watch it Learning skills from others. In addition, the user will be able to demonstrate the game instantly without having to wait for a large download and/or installation' and the user will be able to play the game instantly (perhaps on a less expensive trial basis or on a longer term basis) The user will be able to play the game on a Windows PC, Macintosh, on a TV, at home, on the go, and even on a mobile phone with a low-latency wireless connection. Not once the entity has a copy of the game. As previously stated, the user may decide not to allow their game play to be viewed by others, allowing their game to be viewed after a delay, allowing their game to be viewed by the user. Or allow their games to be viewed by all the viewers. In any case, in one embodiment, the video/audio is stored in the delay buffer 〇 1515 for 15 minutes, and the user will be able to "rewind" and view their previous The game plays, and the game is paused, the game is played back slowly, the game is decided, etc., just as it is when watching a TV with a digital video recorder (DVR) Although this is the case, the user is playing the game, but if the user is using the application, the same "DVR" capability is available. This is in the check of previous work and other applications as detailed below. It can be useful. In addition, the right game light is designed to have the ability to fall back based on the use of game state information, so that the camera view can be changed, etc., it will also support this "3D DVR" capability but it will need to design the game. To support the "deduction 139849.doc • 115- 200949567 DVR" force, the "DVR" capability using the delay buffer 1515 will be combined with any game or application (of course, limited to the video generated when using the game or application). It works, but in the case of a 3D DVR capable game, the user can control the 3D "travel" of the previously played clip and cause the delay buffer 1515 to record the video and record the game state of the game segment. The specific "traversal" record is compressed video, but because the game state will also be recorded, different traversals will likely be in the same segment of the game. Later date. As described below, users on the colocation service 21 will each have a user page Φ in which (4) can publish information about themselves and other information. One of the things that can be announced is a video clip from a game that has been saved. For example, if the user has overcome the particular difficulty of the game, the user can just "rewind" until they get it in the game. Before the location of the big result, and then instructing the colocation service 210 to save a video clip of a certain duration (eg, 3 () seconds) on the user's user page for viewing by other users. The only thing that the user is using the application/game server 152ι_ΐ525 is to play the video stored in the delay buffer 〖5〗 5 to the RAID array 1 5 11 -1 5 12 and then edit the video clip. Indexed on the user's user page. If the game has the ability to have a 3D DVR, as described above, the user can also record the game state information required by the 3D DVR and make it available to the user's user page. The game is designed to have "bystanders 139849.doc -116. 200949567 (that is, users who can travel in the 3D world and observe movements without participation) in addition to active players. The game viewfinder application will enable the user to join the game as a bystander and a player. From the point of view of the implementation, there is no difference between the user and the active player for the hosted system 210. Enter the application/game server 15 21 -15 2 5 and the user will control the game (for example, control the virtual camera of the world). The only difference will be the user's gaming experience. Another feature of the tube service 210 is the ability of multiple users to collaborate while viewing a live broadcast of the hole (even with a differently written view). When playing games and when using an application, This is useful. Many PCs and mobile phones are equipped with video cameras and have the ability to perform instant video compression (especially when the image is small). Also, a small camera is available. It can be attached to a TV, and it is not difficult to implement instant compression in software or using one of many hardware compressors for compressing video. Also, many PCs and all mobile phones have microphones and headphones Available with a microphone. These cameras and/or microphones combined with native video/audio compression capabilities (specifically, using the low latency video compression techniques described herein) will enable users to view video and/or video. The audio is transmitted from the user premises 211 to the colocation service 21 along with the input device control data. When using these techniques, the capabilities illustrated in Figure 19 can be achieved: the user can have his video and audio 1900 appear On the screen of another user's game or application. This example is a multiplayer game where teammates collaborate in the car. The video/audio using 139849.doc -117- 200949567 can only be viewed selectively by their teammates. /hear. Also, because there is effectively no delay in using the techniques described above, players will be able to talk to each other or exercise instantly Perceived delay. This video/audio integration is accomplished by having the camera/microphone from the user, the second compressed video and/or audio as the incoming Internet traffic 丨1 arrives. Routing 15G2 routes the video and (9) audio to application/game servers 1521-1525 that are permitted to view/hear video and/or audio. Next, select respective applications/game servers 1521 that use video and/or audio. The user of -1525 decompresses the video and/or audio and integrates it as needed to appear in the game or application, such as illustrated by 1900. The example of Figure 19 shows how to use the collaboration in a game but Collaboration can be an extremely powerful tool for applications. Consider a scenario where a large building is being set up by New York-based real estate developers in Chicago by the architects in Chicago, but the decision involves financial investors who are on the move and happen to be at Miami Airport, and need Decisions are made regarding the specific design elements of the building (how it relates to the buildings in its vicinity) to satisfy both investors and real estate developers. Suppose the construction company has a high-resolution monitor with a camera attached to the PC in Chicago, the real estate developer has a laptop with a camera in New York, and the investor has a camera-initiated phone call in Miami. The construction company can use the colocation service 210. It hosts a powerful architectural design application that is highly realistic 3D reproduction, and it utilizes a large database of New York City buildings and a library of buildings being designed. The architectural design application will be executed on the application/139849.doc 200949567 game ship hi52i_1525 (or on several if it requires a lot of computational power). Each of the three users at the disparate location will be connected to the colocation service 21〇, and each will have a simultaneous view of the video output of the architectural design application, but it will be targeted for each - The user has a given device and network connection Φ Lu compression (10) appropriately size (for example, the construction company can see 25, 6 and 6 button display via 2 Internet connection, every real estate developer can You can see the trace 72 "&quot;PS image" via the 6 coffee connection on your laptop and the investor can see 32〇χΐ8〇6 via the (10) Kbps cellular data connection on their mobile phone. Fang Zhisheng (the conference call will be handled by any of the widely available conference call suite software in the application server, game server (5) Bu 1525), the domain is replaced by the button on the user input device The user will be able to use his local camera to make the video appear. As the meeting progresses, the architect will be able to reproduce the display with a photorealistic sense of reality as it rotates the building and makes it adjacent to the area. Another in the domain—what the building looks like when the building traverses, and all parties will see the same video at the resolution of the display devices of each party. Any of the local devices used by either party will be Being able to handle 3 〇 animations with this realism is not a problem, let alone downloading or even storing a huge database needed to reproduce the surrounding buildings of New York City. From the point of view of each of the users, 'although the distance is far, And although it is a disparate native device, it will simply have a seamless experience at an incredible level of realism. Furthermore, 'when a party wants its face to appear to better convey its emotional state, it can do so. In addition, if the real estate 139849.doc 119· 200949567 developer or investor wants to control the building program and use its own wheeled device (which is a keyboard, mouse, keypad or touch screen), then it can be, and it can No sensible delay to respond (assuming that the network connection does not have an unreasonable delay). For example, in the case of a mobile phone, if the mobile phone is connected to the airport WiFi Road, it will have (4) low latency. However, if it uses the cellular data network available in the United States today, it is likely to suffer a noticeable lag. However, for the majority of the conference = (where the investor is Watch the architects control the building crossing or talking about the video conference call) 'Even the hive delay should be acceptable. Finally, in the cooperative meeting call (4), real estate developers and investors will carry out their own materials and from the host The escrow service is off, and (4) the company will be able to "rewind" the video of the conference that has been recorded on the delay buffer 1515 and check the comments, facial expressions and/or actions applied to the 3D model of the building during the conference. If there are specific segments that it wishes to save, then the segments of the video/audio can be moved from the delay buffer 1515 to the raid array 15 11-1 5 12 for archival storage and later playback. Also, from a cost perspective, if an architect only needs to use New York City's computing power and a large database for a 15-minute conference call, then it only needs to pay for the time when the resources are used, without having to have high capabilities. Workbench does not have to purchase expensive copies of large databases. Video-rich community services Hosting services 210 are an unprecedented opportunity to build video-rich community services on the Internet. Figure 2G shows a game play (4) exemplary user page for the colocation service 21〇. Like the game viewfinder application 139849.doc • 120- 200949567 Programs like the 'user page' is an application executed on one of the application/game servers 1521_ 1525. All thumbnails and video windows on this page show a constantly moving video (if the clip is short, it loops). By using a video camera or by uploading video, the user (whose user name is "KILLHAZARD") can publish his own video 2 (other users can view the video). The video is stored on the rAID array i 5丨丨_丨5丨2. 'When other users come to the user page of KILLHAZARD, if KILLHAZARD is using the colocation service 21〇, it will show the live video of whatever it is doing 2001 (assuming it is allowed to view it) The user of the user page views it). This will be requested by the hosted user page application application/game server 1521_1525 from the service control system 401 whether the KILLHAZARD is active (and if so, the application/game server 1521-1 is being requested) 525) to complete. Then, using the same method used by the game viewfinder application, the compressed video stream of the appropriate resolution and format is sent to the application/game server 1 52 1 -1 525 executing the user page φ surface application and will It shows that if the user selects a window with a live broadcast of KILLHAZARD and then clicks on its input device appropriately, the window will be enlarged (again, the person uses the same method as the game viewfinder application), and the scene The live video will be populated with the resolution of the viewing device 422 (suitable for viewing the user's Internet connection). The key advantage of this prior art approach is that the user viewing the user page is able to see a live broadcast of the game that the user does not own, and may not have a local computer or game console capable of playing the game. It is 139849.doc •121 · 200949567 Users have an excellent opportunity to see the users on the user page displayed as “active”, and it is an opportunity to learn about the games that users may wish to try or are good at. . Video clips recorded or uploaded from the camera of KILLHAZARD's partner 2002 are also displayed on the user's page, and below each video clip is the text that should not be played online or not (for example, six-sh〇t is playing) The game "Eragon" and MrSnuggles99 offline, etc.). By clicking on the menu item (not shown), the partner video clip switches from displaying the recorded or uploaded video to the game partner currently playing the colocation service 21〇 at the time of the game. Live video of the content. Therefore, it becomes a game viewfinder that is a group of partners. If a partner game is selected and the user clicks on the game, the game will be zoomed in to the full screen and the user will be able to watch the game played live on the full screen. Again, the user who views the game of the building does not own the copy of the game and does not have the local computing/game console resources for playing the game. The game view is effective instantaneous. As previously described above, when the user plays the game keeper on the colocation service 21, the user can "rewind" the game and find that he wants to save the video I, and then save the video clip. To its user page. These are called "Brag Clips". Video clips 2〇〇3 are self-reward clips 2〇〇3 saved by KILLHAZARD from previous games played by them. The number of 2004 shows does not show how many times the clip has been viewed, and when the clip is viewed, the user has an opportunity to rate it, and the number of icons of the orange keyhole shape 2005 indicates how high the rating is. When the user views the 139849.doc • 122· 200949567 user page, the self-reward clip 2003 is continuously cycled along with the rest of the video on the page. If the user selects and clicks one of the self-review clips 2〇〇3, it zooms in to present the self-reward clip 2〇〇3' and allows the clip to be played, paused, backed down, fast forwarded, stepped, etc. DVR control. • Self-recruiting clip 2003 playback is performed by the application/game server ι521_ 1525 loading the compressed video clips stored on the RAID arrays 1511-1512 when the user records the self-review clips and decompressing them and playing them back. 0 φ Self-recruiting clip 2003 can also be a "3 DD VR" video clip from a 3D DVR-capable game (ie, a 'game sequence from a game that can be played back and allows the user to change the camera view point) ^ In this case, game state information is also stored in addition to the specific "crossing" compressed video recordings that the user performs while recording the game segment. When the user page is being viewed and all thumbnails and video windows are constantly looping, the 3D DVR self-reward clip 2003 will cause the user to record the self-reward clip 2003 of the compressed video when the user records the "travel" of the game segment. Constantly circulate. However, when the user selects the 3D DVR self-reward clip 2003 and clicks the 3D DVR self-reward clip, the user will be able to click to give it, in addition to allowing the playback of the compressed video clip. A button for the 3D 〇¥ ruler of the game clip. It will be able to independently control the camera "crossing" during the game segment, and if it wishes (and the user with the user page so allows), the basin will be able to The alternative self-reward clip "crossing" is recorded in the form of a compressed video, and the alternative self-reward "crossing" will then be available to other viewers of the user page (immediately, or on the owner of the user page 139849) .doc -123- 200949567 After verifying the opportunity to edit the clip). This 3D DVR Self-Camp Clip 2003 capability is enabled by launching a game that will replay the recorded game state information on another application/game server 1521-1525. Because the game can be launched almost instantaneously (as previously described), so it is activated (its playback is limited to the game state recorded by the self-reward clip) and then the user is allowed to record the compressed video to the delay buffer 1515. It is not difficult to "cross" with a camera. Once the user completes the "crossing", the game is deactivated. From the user's point of view, it is no more difficult to control the dvr control with the 3D DVR Self-Tricks 2003. It does not know the game or even know how to play the game. It is only a virtual camera operator staring at the 3D world during the game segment recorded by another operator. The user will also be able to add their own audio to the self-reward clip (or record or upload from the microphone). In this way, a self-reward clip can be used to generate custom animations using characters and actions from the game. This animation technique is often referred to as "machinima". ❹ As users progress through the game, they will reach different skill levels. The played game reports the results to the service control system 4〇 1, and these skill levels will also be displayed on the user page. Interactive animated ads Online ads have evolved from text to still images, video, and now to interactive clips' are typically implemented using an animated, thin-client user like Adobe Flash. The reason for using the animated streamlined client is that the user 139849.doc -124· 200949567 !often::: The privilege of the product or service is delayed and less resistant: the thin client is very inefficient. Execution, and therefore 妯° has a level of confidence: interactive ads will work properly. Aussie, such as AdGbeF] ash's animated, streamlined client is limited in the degree of interactivity and the duration of the experience (to reduce download time). Figure 21 illustrates an interactive advertisement in which the user will select the exterior and interior colors of the car as the car turns red in the showroom, while the instant ray chasing

❷ No, how the car looks. The user then selects the avatar to drive the car, and the pick-up user can use the car for driving on a race track or through a country other than Monaco. The user can select a larger engine or a better tire and then see how the changed configuration affects the car's ability to accelerate or maintain stability. Of course, the advertisement is effectively a cutting-edge 31) video game. But for such an ad that can be played on a PC or video game console, it will need to be 100 MB downloadable 'and in the case of a PC' it may require an An Zhen special drive, and may lack sufficient cpu or Gpu calculations in the Pc Ability is not enforced at all. Therefore, such advertisements are impractical in prior art configurations. In the colocation service 210, the advertisements are placed almost immediately, and preferably perform 'regardless of the capabilities of the user's client 415. As a result, it delivers more quickly than a streamlined, user-friendly ad, with a richer experience and a high degree of reliability. Streaming geometry RAID arrays 1511-1 512 and incoming routing 1502 during instant animation can provide such fast data rates with such low latency that it is possible to design depending on 139849.doc -125 - 200949567 array 15 called 512 and The video routers and the application program reliably deliver the geometric video games and applications directly in the middle of the game play or in the library application during the mobile game. Under the = technology (four) system (such as the video game system shown in Figure 1), the available mass storage test r + Gan a device (especially in practical home devices) is extremely slow, so that it cannot be played in the game. During the cross-flow geometry (except for the slightly predictable geometry of the geometry, t, in the driving game where there is a designated road, it is reasonable to properly predict the building (4) and the shape of the storage device for entering the field of view. You can search for the location of the upcoming geometry in advance. But in complex scenes with unpredictable changes (for example, in a battle scene with complex characters around), if the RAM on the PC or video game system is completely filled Fully for the geometry of the object currently in view, and then if the user suddenly turns his character to view his character, why if the geometry is not preloaded into the RAM, there may be a delay before the geometry can be displayed. In the colocation service 210, the RAID arrays 1511-1512 can exceed the speed of the ultra-fast Ethernet speed data 'and under the SAN network With Sb achieving a speed of 10 billion bits per second that is better than 10 Tencent Ethernet or better than other network technologies. 10 billion bits per second will load ten in less than one second. Information on billions of bytes. In the 60 fps frame time (16.67 milliseconds), about 170 million bits (21 MB) can be loaded. Of course, even in the RAID configuration, rotating media will still incur A time delay greater than one frame time, but the flash memory based on the flash memory will eventually be larger than the rotating 139849.doc 126-200949567 media RAID array and will not incur this high latency. In one embodiment, Uses a large amount of RAM write cache to provide very low latency access. • Therefore, it can be processed with the CPU and/or GPU at a high enough network speed and a large amount of memory with low latency. The 3D data generally streams the geometry to the application/game server 1521-1525. Thus, in the previously given example, where the user suddenly turns his character and looks backwards, the character can be rotated. All the characters that were previously loaded behind them

A geometry, and therefore, for the user, it would seem that it is in a world that is as realistic as a live photo action. As discussed earlier, one of the last boundaries of a photo-realistic computer is the human face, and the slightest error from a photo-realistic face can result from the human eye being sensitive to incompleteness. Negative reactions from viewers. Figure 22 shows the use of ContourTM Authenticity Capture Technology (the following is the subject of the application: Application No. 10/942,609, "Apparatus and method for capturing the motion of a performer"; 2004 10/942,413 M, "Apparatus and method for capturing the expression of a performer", September 15th, 2005; Application No. 11/066,954, r Apparatus and method for improving marker identification within a motion Capture system"; March 10, 2005, pp. 11/077, 628, "Apparatus and method for performing motion capture using shutter synchronization"; Application No. 11/255,854, October 20, 2005, "Apparatus and 139849.doc - 127-200949567 method for performing motion capture using a random pattern on capture surfaces"; June 07, 2006 曰 Application No. 11/449,131 "System and method for performing motion capture using phosphor application techniques"; June 7, 2006 Application No. 11/449,043 "System and method fo r performing motion capture by strobing a fluorescent lamp"; June 1987, pp. 1/449, 127, "System and method for three dimensional capture of stop-motion animated characters", each of which has been How the performance of the live broadcast captured by the assignee of this CIP application results in a very smooth capture surface that achieves a high polygon count tracking surface (i.e., the polygon motion accurately follows the motion of the face). Finally, photo-realistic results are produced when the live broadcast of the live video is mapped onto the tracking surface to create a textured surface. Although current GPU technology is capable of reproducing many polygons in the tracking surface and texture and illuminating the surface in real time, if the polygon and texture each frame time changes (which will produce the most photorealistic result), it will quickly All available RAM of a modern PC or video game console is consumed. Using the stream geometry technique described above, the geometry is continuously fed into the application/game server 1521-1525 so that it can continually animate a photo-realistic face to allow for almost indistinguishable The video game of the facial face of the live action becomes practical. Integration of linear content and interactive features 139849.doc -128· 200949567 Movies, TV shows and audio materials (collectively "Linear Content") are widely available in many forms for home and office users. Linear content is available on physical media such as CD, DVD, HD-DVD and Blu-ray media. It can also be recorded by a DVR broadcast from satellite and cable Tv. In addition, it can be available via pay-as-you-go (ppv) content on satellite and TV, and on-demand video on demand (VOD).

Increasing linear content is available via the Internet for downloading content and rates (10). Today, there is really no place to experience all the features associated with linear media. For example, DVDs and other video optical media typically have interactive features that are not available at other locations (such as directors, "tact" clips, etc.). Online music sites have cover art and song information that are not normally available on cd, but not all cd lines are available. And websites associated with television shows often have additional features, blogs, and sometimes comments from actors or creators. In addition, in the case of many movies or sports events, there are often transactions that can be closely associated with real-world events (eg, player transactions), either in conjunction with linear media (in the case of a movie) or (in the case of sports). ) video game. The colocation service 210 is well suited for delivering linear content when linking disparate forms of related content together. Indeed, delivering a movie movie is less challenging than delivering a highly interactive video game, and the colocation service 21 can deliver linear content to a variety of devices in a home or office, or to a mobile device. Figure 23 shows an exemplary user interface page for the colocation service 21, which shows the selection of linear content. 139849.doc •129· 200949567 However, unlike most linear content delivery systems, the colocation service 210 is also capable of delivering relevant interactive components (eg, menus and features on DVDs, interactive overlays on HD-DVDs). 'And Adobe Flash animation on the website (as explained below)). Therefore, the client device 415 limits no restrictions on which features are available. In addition, the colocation system 210 is capable of dynamically and instantly linking linear content to video game content. For example, if a user is watching a Quidditch game in a Harry Potter movie and decides that he or she is willing to try to play Quidditch, it can simply click the button and the movie will pause and it will be immediately sent to the Harry Potter video. Quidditch clip of the game. After playing the Quidditch game, another click on the button and the movie will start again. In the case of photo-realistic graphics and production techniques, the video captured by the camera cannot be distinguished from the live action characters, when the user performs a video game from the Quidditch game in the live action movie to the colocation service. In the transition of the Quidditch game (as described in this article), the two scenes can't actually be distinguished. This is a brand new creative option for directors of both linear content and interactive (eg, 'video game') content, as the lines between the two worlds become indistinguishable. Using the colocation service architecture shown in Figure 14, the control of the virtual camera in the 3d movie can be provided to the viewer. For example, in a scene that occurs within a train, it will be possible to allow the viewer to control the virtual camera and look around the train as the story progresses. This assumes that all 3D objects in the train ("assets") and enough 139849.doc • 130-200949567 computing power levels are available to instantly reproduce these scenes as well as the original movie. To non-computer-generated entertainment, there is a very stimulating interactive feature that can be provided. For example, the 2005 film "Pride and Prejudice" has many scenes in the beautifully decorated old British mansion. For a specific building scenario, the user can pause the video and then control the camera to patrol the building, or: the surrounding area can be carried out, can carry the camera with the fisheye lens over the building, when it tracks its position, It is similar to the implementation of the prior art Apple's QuickTime VR. Various frames will then be converted by Φ

This scene/image is not distorted, and then it is stored on the array J 15 12 along with the movie and is played back when the user chooses to continue the virtual tour. In the case of a sporting event, a live event (e.g., a basketball game) can be streamed via the colocation service 21 for viewing by the user (as it would be for a typical TV). After the user views a particular play, the video game of the game (finally the basketball player looks as realistic as the real player's photo) can catch up with the player starting in the same position, and the user (possibly controlling one player each) can replay To see if it can do better than these players. The colocation service 210 described herein is extremely well-suited to support this future world's computing power and large-capacity storage resources that are unrealistic enough to be installed in the home or in most office settings, and its computing resources Always up to date (in the case of the latest computing hardware available), but in the home background, there will always be families with older generations of pCs and video games. In addition, 'in the colocation service 21〇, the use of 139849.doc -131 - 200949567 is concealed of all this computational complexity', so even if the user may be using a very sophisticated system' from the user's point of view, It is generally simple to change the channel on TV. In addition, the user will be able to access all computing power and computing power from any client 415. The multiplayer game to the extent that the game is a multiplayer game, it will be able to communicate not only to the application/game server ΐ52ι_ΐ525 via the network 1502 but also to the host escrow service 21 via the network bridge. The Internet (not shown) of the server or game machine being executed. When playing multiplayer games on a computer on a universal Internetwork, the application/game server 1521-1525 will have the benefit of extremely fast access to the Internet (with the game system running on the feeder in the home) Compared to the situation, but it will be limited by the ability of other computers playing games on slower connections, and potentially by the game servers on the Internet to be designed to accommodate the least commonalities (these game pirates will The factual limit for a relatively slow consumer Internet connection on a home computer). However, when playing multiplayer game© in the server hosting service 21〇 server center completely, a great difference can be achieved. Each application/game server 1521-1525 hosted by the host for the user's game will be connected to other applications/games 5 1521-1525 and by extremely high speed, very low latency and huge, very The Fast Storage Array Host hosts any server interconnects that are centrally controlled by multiplayer games. For example, if a hyper-speed Ethernet is used to route the 1502 network' then the application/game server 1521-1525 will communicate in the field and communicate to the potential at a bite/second speed. Only 1 139849.doc -132- 200949567 milliseconds or less than 1 millisecond delay under the host to host any server for the central control of multiplayer games. In addition, RAID arrays 15 11 - 1512 will be able to respond very quickly and then transfer data at a gigabit per second rate. As an example, if a user customizes a character in appearance and clothing so that the character has a large number of geometric shapes and behaviors that are unique to the character, the game user is limited to playing on a PC or game console in the home. Under the prior art system, if the character enters the field of view of another user, the user_ will have to wait until a long slow download is completed in order to load all geometry and behavioral data into his computer. Within 21 主机 of the colocation service, the same download can be superior to the ultra-high-speed Ethernet from the RAID array ι511_ 15 12 at a rate of one billion bits per second. Even if the home user has a ^ Mbps Internet connection (which is extremely fast based on today's standards), the ultra-fast Ethernet network is 100 times faster. Therefore, it takes less than a second to spend a minute on a fast internet connection on a super-fast Ethernet. Top Player Grouping and Tournaments Φ The Hosting Service 210 is ideal for tournaments. Since no game is executed in the local client, there is no chance for the user to cheat. Again, due to the ability of the output route 1540 to multicast UDP streams, the colocation service 210 can simultaneously broadcast larger tournaments to thousands of people in the audience. In fact, when there are so popular that thousands of users are receiving the same stream of specific video streams (for example, showing a larger tournament view), the video stream is sent to a content delivery network (CDN) (such as from a coffee) Or

Limelight) may be more efficient for distribution to a large number of client devices 415. 139849.doc 133- 200949567 Although CDN is used to display the top viewer grouped game viewfinder page, similar level efficiency can be obtained. For larger tournaments, live celebrity commentators can be used to provide comments during specific competitions. Although a large number of users will be watching the larger championships, a relatively small number will be played in the tournament. The audio from the celebrity commentator can be routed to the application/game server U2M525 that hosts the host in the tournament and hosts any of the bystander mode games of the game in the tournament, and can add the audio Recorded on the game audio. You can use the celebrity to explain the video of the speaker on the game (or just on the bystander view). Accelerated webpage loading. The main delivery protocol and hypertext transfer protocol (Ηττρ) of Bellow.com are conceived and defined in which only commercials have high-speed Internet connections, and consumers on the line use dial-up data machines or ISDN. In the era. At this time, the "gold standard" for fast connection is 71 lines, which symmetrically provides a data rate of ^ 5 Mbps (i.e., equal data rates in both directions). Today, the situation is completely different. In a large number of developed worlds, the average home connection speed via a muscle or electricity modem connection has a much higher downlink data rate than the line. In fact, in some parts of the world, fiber-to-the-road (flber_t〇-the-CUrb) is introducing data rates of 5 to 100 Mbps up to 5 〇 river leaves into the home. Unfortunately, HTTP has not been architected (and not implemented) to effectively utilize these dramatic speed improvements. The website is a collection of buildings on the remote server. Quite simply, the HTTP request is the first file, waiting to be downloaded, and then I39849.doc -134- 200949567 requests the second file 'waiting to download the standard, etc.'. In fact, HTTP allows for more than one "open connection" (that is, each request - more than one case) °, because of the agreed standards (and the desire to prevent the network word server from being overloaded), so that only very few permits Open connection. In addition, because web pages are structured, viewers are often unaware of the multiple simultaneous pages available for immediate download (i.e., 'only after profiling—the page becomes apparent: a new building such as an image needs to be downloaded). Therefore, the powers on the website are essentially one by one.

Load. In addition, due to the request and response protocol used by Ηττρ, there is a delay of approximately 1 〇 milliseconds associated with each file loaded (typical access to the US network feeder). In the case of a relatively low-speed connection, this does not cause a lot of problems, because = the download time of the file itself depends on the waiting time of the web page. However, as the connection speed increases (especially in the case of complex web pages), it starts to cause problems. In the example shown in Figure 24, a typical commercial website (this particular website is from a larger athletic shoe trademark) is shown. There are 54 files on the website. The files include HTML, CSS, JPEG, PHp, 2, and include video content. A total of 15 "bytes" must be loaded before the live web page (that is, the user can hit it early and start using it). There are many reasons for the large number (four). First, it is a complex and sophisticated web page, and secondly a web page that is dynamically combined based on information about users accessing the page (eg, from which country the user came from, what language, whether the user previously made a purchase, etc.), and downloaded differently depending on all of these factors Archive. However, it is still a very typical commercial web page. 139849.doc -135- 200949567 Figure 24 shows the amount of time that elapses before the live web page increases as the connection speed increases. At 1.5 Mbps connection speed 24〇1, use The familiar network ship n with a conventional network browser takes seconds before the live web page. At 12 Mbps connection speed 2402, the load time is reduced to 65 seconds, or approximately doubled. But at 96 Mbps The connection speed is only 24 〇3, and the load time is only reduced to about 5.5 seconds. The reason is because at this high download speed, the time to download the file itself is the smallest, but each file is roughly The delay of 丨〇〇 milliseconds is still maintained, resulting in a delay of 54 files * 1 〇〇 milliseconds = 54 seconds. Therefore, no matter how fast the connection to the home, this website will always take at least 5.4 seconds before the live broadcast. Another factor is that the server side is queued up; each HTTP request is added after the queue, so on a busy server, this will have a significant impact because for each small one to be obtained from the web server Projects, HTTP requests need to wait for them to return. One way to solve these problems is to discard or redefine HTTp. Or 'may make the site owner better merge their files into a single file (for example, in Adobe Flash format). As a practical matter, this company and many others have invested heavily in its website architecture. In addition, ©B, there are 12.100 Mbps connections in the courts' but most families still have slower speeds, and HTTP is slow. The speed does work well. An alternative is to host the web browser host on the application/games server 1521-1525' and the file master will be used for the web server. The machine is hosted on the RAID arrays 1511-1512 (or potentially hosted in the RAM on the application/game server 1521-1525 hosted on the hosted web browser or on the local storage).非常 Route 1502 (or to the local storage 139849.doc -136- 200949567 ❹ ) ) is a very fast interconnection, not using HTTP with a delay of 1 〇〇 per file, but will exist under HTTP Minimum latency per file. Next, instead of allowing a user in the home to access the web page via HTTP, the user can access the web page via the client 415. Then, even under the i 5 Mbps connection (because the web page does not require a lot of bandwidth for its video), the web page will be live broadcast in less than i seconds per line 2400. In essence, there will be no delay before the web browser executing on the application/game server 1521_1525 displays the live broadcast page, and there will be no detectable before the user terminal 415 displays the video output from the web browser. The delay to arrive. When the user uses the mouse to search the webpage and/or type a word on the webpage, 'the user's input information is sent to the web browser executed on the application/game server 1521-1525, and the web browser A response will be responded accordingly. The disadvantage of this method is that if the compressor is constantly transmitting video data, the bandwidth is used, even if the web page becomes static. This can be remedied by the configuration compressor to transmit data only when (and if) the web page changes and then only transfer the data to the portion of the page where the change occurred. #有有有有地地的快闪标语, etc. - Some pages tend to be fascinating. Web pages are usually static unless there is a reason to move something (for example, a video clip). For these web pages, it may be possible to use the host agency service 21G to transfer less data for the following situations (since compared to the network server), since only the actual displayed image will be transmitted, no compact client executable code And there are no large objects (such as moving images) that may never be viewed. 139849.doc • 137- 200949567 Therefore, using the colocation service 210 to host an old web page can reduce the web page load time to the extent that opening the web page is similar to changing the channel on the TV: effectively live broadcasting the live Web page. Promoting the Debugging of Games and Applications As just described, video games and applications with instant graphics are very complex applications and often have defects when they are released into the field. Although the software developer will get feedback on the defect from the user and it may have some way to return the machine state after the crash, it is exactly what is causing the game or instant application to crash or improperly execute. very difficult. Any machine state will be available for collection. The video/audio output of the game or application is constantly recorded in the delay buffer 1 along with the video/audio recorded by the delay buffer 1515 when the game or application is executed in the colocation service 2丨〇. on. In addition, the watchdog process executes each application/game server 1521.1525, which will periodically report to the colocation service control system 401 that the application/game ship 152ι_ΐ 525 is executing smoothly. If the dog walking process fails to report, (4) the server control system tears the attempt to communicate with the application/game server 1521_1525, and if successful, sends any information about the available L/audio to the software colocation service.

When the game or application software developer gets a record of crashing 4 frames from 2 10 . 139849.doc 200949567 It is also noted that when the application/game server 1521-1525 crashes, the server is restarted at the most recent restartable time and the message is provided to the user, apologizing for technical difficulties. Resource sharing and cost savings

The systems shown in Figures 4a and 4b provide multiple benefits to both end users and game and application developers. For example, in general, home and office client systems (e.g., PCs or game consoles) are only in use for a small percentage of the week. According to Nieisen Entertainment "Live rc.c om/cgi- leap player benchmark study" (hup://www.prnewswi bin/stories.pl?ACCT=l〇4&amp;STORY=/www/story/l〇- 〇5-2006/ 〇〇〇4446115&amp;EDATE=) 2nd 6th year 〇 5 5th communication, active players spend an average of 14 hours a week on the video game console and about 17 hours a week Play on handheld devices. The report also states that for all game play activities (including console, handheld devices & PC games, active players average - 13 hours per week. Considering the higher number of console video game play time, there is a week 24* 7=168 hours, he suggested that in the active gamer's home, the video console is only in the hour of 17/168 = 10% of the week. Or, the time of the video game console The high cost of a given video game console, and the fact that the manufacturer subsidizes such devices' is a very inefficient use of expensive resources. PCs in the business are usually only _ ^ 兀 only in - Week-times, hours, especially for southern applications (such as 1 m)

Autodesk Desktop PCs, although some commercial-style PCV-m--operated in all hours and with false alarms, and PCs (for example, portable PCs brought home for a brother-in-the-job work) In 139849.doc • 139· 200949567 All hours and holidays, / 4, C Most commercial activities tend to focus on the week-to-week from 9 AM to 5 PM in a given industry time zone, less False and disconnection times (such as lunch), and because most PCs appear when they are used again and actively use the PC's, so they follow: desktop PCs tend to follow these hours of operation. If the five days of the fake I week are from 5 PM to the use of pc, then he will imply that the % is in a week.

4〇/168=24% of the hours are utilized. High-performance desktop PCs are very expensive investments for business and this reflects very low utilization. Schools that teach on a desktop can use the computer for a week or even less, and although it varies depending on the number of hours taught, most of the teaching occurs during the hours of the week-week. Therefore, in general, a video-like game console is only utilized in the hour of the minute. Notably, because many people are in the business s in the business week during the non-holiday week - to the time of the Friday, these people usually do not play video games during these hours, and therefore when they do play When video games are played, they are usually during other hours (such as evenings, weekends, and fakes).

Given the configuration of the colocation service shown in Figure 4a, the usage patterns described in the two paragraphs above result in the non-f efficient use of resources. Obviously, there is a limit to the number of users that can be served by the colocation service at a given time, 'especially when the user needs instant responsiveness for complex applications = (such as cutting-edge 3D video games) under. However, unlike video game consoles in the court or PCs that are used commercially (which are usually left idle most of the time), server technology can be reused by different users in different ways. For example, a high-performance dual cpu and dual Gpu 139849.d〇c-140. 200949567 and a large amount of RAM high-performance server 4〇2 can be utilized by commercial and school in non-false 9 9 AM to 5 PM, but by Players who play cutting-edge video games use it at night, on weekends and on holidays. Similarly, low-performance applications can be utilized by businesses and schools during business hours on low-performance servers with Celeron CPUs, no Gpu (or non-normally low-end GPUs), and limited RAM. The low performance server 4〇2 can be utilized during non-commercial hours. In addition, in the case of the colocation service configuration described herein, the participating resources are effectively shared among thousands (if not millions) of users. In general, online services only use a small percentage of their total user base to use the service at a given time. If you consider the statistics of the game games listed previously, it is easy to understand why. If active players play console games for only 17 hours a week, and if the peak usage time of the game is assumed to be at night (5-12 AM, 7*5 days = 35 hours/week) and weekends (8 am_12 AM, 16*) During a typical non-working, non-commercial hour of 2 = 32 hours/week, there is a 35+32=65 peak hour for a 17-hour game. It is difficult to estimate the exact peak user load on the system for a number of reasons: some users will play during peak out-of-peak time. There may be a cluster peak of the user at a particular daytime, and the peak time may be affected by the type of game being played ( For example, 'the game of the child will probably play at an earlier time in the evening." However, assuming that when the player is likely to play the game, the average number of hours played by the player is much less than the number of hours between the players, then only the number of users of the colocation service 210 will use the host generation at a given time. Pipe service 2 1 0. For this analysis, the peak load is assumed to be 丨2 $. /. Therefore, only 12.5% of the computing, compression and bandwidth resources are used to support 139849.doc • 141 - 200949567 at a given time, resulting in only 12.5% of the hardware cost due to the reuse of resources to support a given user. One of the performance games is given a level. In addition, assuming that some games and applications require more computing power than others, resources can be dynamically allocated based on the game being played by the user or by the application executed by the user. Therefore, users who choose low-performance games or applications will be assigned low-performance (lower-cost) servers 4〇2, and users who choose high-performance games or applications will be assigned high-performance (more expensive) feeders. 402. In fact, a given game or application may have a lower performance and higher performance area of the game or application, and the user can switch from a server 4〇2 to a game or application area. Another server 402' is executed to maintain the user on the lowest cost server 402 that satisfies the needs of the game or application. Note that raids that are much faster than a single disk. Column 405 will be used even for low performance servers 4〇2, which has the benefit of faster disk transfer rates. Thus, the average cost per server 402 across all played games or applications used is much less than the cost of playing most of the most expensive servers 402 of the highest performance game or application, however, even the low performance server 402 The disk performance benefit © will be obtained from the raid array 405. In addition, the server 402 in the colocation service 210 may be only a pc motherboard without a disk or a peripheral interface (different from the network interface), and just can be integrated down to a fast network just to the SAN 403. A single piece of the interface. Also, the RAID array 405 may be shared among users who are more likely to have a disk, so the cost per disk of each active user will be much less than that of a disk drive. All of this equipment will likely reside in the environment controlled by the bracket t in the environment of the 139849.doc •142· 200949567 two server room. If the feeder fails, it can be easily repaired or replaced at the colocation service 210. Compared to 2, the pc or game console in the home or office must be sturdy and must be able to survive the reasonable wear and tear to prevent being hit or landed. 2. Need to be external, with at least one disk Machines must survive unfavorable environmental conditions (for example, being forced into a heated AV cabinet with other appliances), require service assurance, must be packaged and shipped, and sold by retailers who may receive retail sales. In addition, the PC or game console must be configured to meet the peak performance of the expected game or application that will be used in a future-timed, even low-performance game or application (or game or application). District) may also play most of the time. In addition, if a PC or console fails, it can be a costly and time consuming process (which adversely affects manufacturers, users, and software developers). The system shown in Figure 4a provides the user with the experience of the experience of the computing resources of the local end, so that the user can experience the computing power of the location in the home, office or school. The architecture shown in 4a provides much less computational power. Eliminate the need for upgrades. ‘Users no longer have to worry about upgrading PCs and/or consoles to play new games or handle new applications with better performance. Any game or application on the escrow service (regardless of the type of food server 402 required by their games or applications) can be used by the user and all games and applications are executed in close proximity (ie , quickly loaded from the RAID array 405 or the local storage on the server 402) and suitably with the latest updates and defects 139849.doc -143- 200949567 fix (ie 'software developers will be able to choose to execute The ideal server configuration for the game 402 of the game or application, and then the feed 1 § 402 is configured with the best drive' and then over time, the developer will be able to simultaneously update, fix, etc. All copies of the game or application provided to the host hosting service 210). In fact, after the user starts using the colocation service 210, the user may find that the game and the application continue to provide a better experience (eg, via update and/or defect repair) and may be in the following situation: the user is one year later It is found that new games or application privates can be used to take advantage of the computing technology (eg, higher performance Gpu) (which does not even exist in a year), so it is impossible for the user to purchase A year ago, the technology to play games or execute applications a year later. Because the computing resources for playing games or executing applications are invisible to the user (ie, from the user's point of view, the user only chooses a game or application that is close to being executed immediately. More like the user changes the TV. The channel is on, so the user's hardware will be "upgraded" even if the user is not aware of the upgrade. Eliminate the need for backup, another big problem for users in business, school and home, is that the right disk fails. Or if there is an inadvertent erasure, it is saved on the local side or as a video game. The information in the console (for example, in the case of the console, the user's game results and ratings) may be lost. There are many applications available for heart-to-heart backup, and the game control can be uploaded to the online server for backup, but the local backup is usually stored in a safe and organized manner. Somewhere else - the local disk (or 139849.doc 200949567 other non-volatile storage devices), and because of the slow uplink speed available via a typical low-cost Internet connection, the backup for online services is often limited. . In the colocation service 21G of Figure 4a, the data stored in the RAID array 405 can be configured using prior art RAID configuration techniques well known to those skilled in the art, such that when the disk fails, The bedding will not be lost and the technician at the server center hosting the failed disk will be notified, and then the technician will replace the disk, which will then be automatically

The ground is updated so that the RAID array tolerates the failure again. In addition, because all drives are close to each other with a fast local network via SAN 4〇3, all disk systems are configured in the server center to be periodically backed up to secondary storage (which can be stored in It is not difficult to relocate at the center of the feeding device or easily. From the point of view of the hosted service user, the data is always completely secure and never has to be considered for backup. Access to the presentation Users often want to try out a game or application before purchasing a game or programming. As previously mentioned (4), there is a demonstration (the "verb" form means that the demo version is also called "demo", but as a noun) the prior art components of games and applications, but each of them Subject to restrictions and/or inconvenience. Using the colocation service 21〇, for the user, it is easy and easy to try out the demo. In fact, what the user does is select the presentation and try (4) the presentation via the user interface (such as the user interface described below). The demo will load almost instantly for the demo, and it will be executed entirely by any other game or application. Regardless of whether the presentation requires a very efficient feeding device 139849.doc • 145- 200949567 402 is still a low-performance server 4〇2, and regardless of the user's home or office client type 4 5, from the user's point of view Look, the demo will work. The software publisher of the game demo or application demo will be able to precisely control how long the demo and trials are allowed for the user to try, and of course, the performance may not include providing the user with a full version of the game or application being demonstrated. The user interface element of the access opportunity. Because the presentation may be lower than the cost price or provided free of charge, some users may attempt to use repeated presentations (especially to repeatedly play potentially interesting game presentations). The colocation service 210 can use various techniques to limit the presentation usage for a given user. The most straightforward method is to establish a user ID for each user and limit the number of times a given user m can play the presentation. However, the user can set multiple user IDs, especially if they are free. One technique for addressing this problem is to limit the number of times a given client 415 can play a presentation. If the client is a standalone device, the device will have a sequence number and the colocation service 210 can limit the number of times the presentation can be accessed by the client with the serial number. If the client 415 is executing on a PC or other software on the device, the host escrow service 210 can assign the serial number and store the serial number on the PC and use the serial number to limit the presentation usage, but assume that the PC can be re-used by the user. Stylized, and the sequence number is erased or changed, then another option is that the colocation service 210 maintains the Pc network adapter media access control (MAC) address (and/or other machine specific identifiers, such as hard Record the number of the disc player, etc.) and limit the use of the demo to the MAC address. Assume that the mac address of the network adapter can be changed, however, this is not an extremely simple method. Another method is to limit the number of times a presentation can be played to a given IP address. 139849.doc -146- 200949567 Although the IP address can be reassigned periodically by cable modems and DSL providers, it does not happen very frequently in practice, and if it can be determined (eg 'by contacting the ISP IP is in the block of the IP address used for residential DSL or electric slow data machine access, and it is usually possible to establish a small (4) demo for a given home. Also, there may be multiple devices in the home after sharing the HP address router, but typically in a residential setting, there will be a limited number of such devices. If the IP address is in the φ block of the commercial area, then a larger number of presentations for business can be established. However, in the end, all combinations of the previously described methods are the best way to limit the number of presentations on the pc. There may be no way for a determined and technically skilled user to be limited in the number of replayed presentations, but creating a large number of obstacles can create enough obstacles to make most PC users unworthy The gods go to abuse the demo system, and instead, they use the demo when they want to try out new games and applications. Benefits to Schools, Businesses, and Other Institutions Φ Significant benefits especially occur in commercial, dry, and other institutions that utilize the systems shown in Figure 4a. Businesses and schools have substantial costs associated with installing, maintaining, and upgrading PCs, especially when it comes to implementing the high-performance applications of the Spear-Type PCs. As stated previously, PCs are typically only used in a two-week rate of one week, and as in the home, PCs with the ability to position the quasi-effectiveness in the office-to-school environment are much more expensive than in the word processor. The cost in the central environment. In the case of larger businesses or schools (for example, large universities), the 1T department of these entities sets up the feeder center and maintains the remote connection through the LAN level 139849.doc -147· 200949567 It is actual. There are many solutions for remote access to a computer via a LAN or via a private office to a wide-band connection, σ, by Microsoft's Windows terminal, or via a virtual network. Road computing applications (such as from RealVNC, Inc.) or remote access to PCs or servers by Sun Microsystems' streamlined user component users, with a series of graphical response times and user experience quality. In addition, these self-managed server centers are typically dedicated to a single business or school, and therefore cannot be utilized in disparate applications (eg, entertainment and business applications) to utilize the same computing resources for a week. The overlap of possible uses. Therefore, Xuxiang Commercial and School lacks the scale, resources or expertise to independently set up a server center with a network connection to each user's secondary speed. In fact, a large percentage of schools and businesses have the same Internet as the family. Network connection (for example, DSL, cable modem). However, such organizations may still have the need for very efficient calculations (either periodically or periodically, for example, small construction companies may only have a small number of buildings from the currency' when designing work, I have A relatively modest amount of 60 is needed, but it may periodically require very efficient 3D calculations (for example, when establishing a yang crossing for a new architectural design at the user end). The system shown in Figure 4a is extremely suitable for Such organizations. These organizations only need to provide the same kind of network connection to the home (for example, DSL, cable modem) and are usually very inexpensive. They can use the low % as the client 415, or no pC at all. However, an inexpensive specializer 139849.doc-148-200949567 that implements only control signal logic 413 and low-latency video decompression 412 can be utilized. These features are likely to be theft of #pc or damage to dedicated components within the pc. The problematic school is particularly attractive. This configuration solves many of the problems for these organizations (and many of these/points are also used for home use for general purpose computing) Shared). For example, the cost of 乍 (which must ultimately be passed back to the user in some form to have a viable business) may be much lower, because (4) the computing resource system has a non-interest value in the week. Sharing of other applications, (b) such groups are required to obtain (and incur costs) access to high-performance computing resources, (C) such organizations do not have to provide for backup or otherwise Resources for maintaining high-performance computing resources. Elimination of piracy In addition, games, applications, interactive movies, etc. may no longer be pirated as they are today. Because the game is executed at the server center, the user does not have the basic code. Access, so there is no piracy. Even if the user is going to copy the source code, the user cannot execute the code on standard game control Φ 家庭 or home computer. This opens the world where standard video games are not available (such as ' China's market. Resale of used games is also impossible. For game developers, as in today's situation, there are Less market discontinuity. With the new generation of technology forcing users and developers to upgrade and game developers depending on the current situation of the timely delivery of the hardware platform, the host can be gradually updated as the game requirements change over time Managed Services 210. Streaming Interactive Video 139849.doc • 149· 200949567 The above description provides low-latency streaming interactive video based on the Universal Internet (which also implicitly includes audio along with video, such as The novel basic concepts enabled in this article enable a variety of applications. Prior art systems for streaming video over the Internet have only applications that can be implemented by high latency interactions. For example, The basic playback control for linear video (eg, pause, rewind, fast forward) works properly at high latency, and it is possible to select among the online video feeds. Moreover, as previously stated, the nature of some video games allows them to be played with high latency. However, the high latency (or low compression ratio) of prior art methods for streaming video severely limits the potential application of streaming video or narrows its deployment to a specialized network environment, and even in such environments. The prior art also introduced the substantial burden of the road. The techniques described herein open the door to a variety of applications that are likely to be possible via low-latency streaming interactive video over the Internet, especially those enabled via consumer-grade Internet connections. ~ In a consistent manner, under the user interface of the client 465, which is small enough, it is sufficient to use any amount of computing power, any amount of fast storage, and extremely fast network connection between powerful servers. Providing an enhanced user experience' that enables a new era of computing. As the bandwidth requirements do not increase with the computing power of the system (that is, because the bandwidth requirements are only about the resolution, quality, and frame rate), the broadband Internet connection The problem is that typical users will be ubiquitous (for example, via a widely distributed low-latency wireless coverage), reliable enough, and having a sufficiently high bandwidth to satisfy all users' display devices 422. 139849.doc 200949567 What is necessary for business applications is a thick client (such as performing a partial,

PCs such as Linux OSX or payphones are even thinner clients (such as Adobe Flash or Java). The emergence of streaming interactive video has led to a reconsideration of assumptions about the structure of the computing architecture. This is an example of the colocation service center shown in Figure 15. The video path feedback loop for the delay buffer buffer and/or the group video (10) is used by the application/game server 15 15 1525. The streamed interactive video output of the second day broadcast via the path Mu is instantaneous or via Path 1551 is fed back into application/game server 1521_1525 after a selectable delay. This can be done by prior art servers or local computing (4) as to what is or is not feasible for practical applications (eg, such applications as those illustrated in Figures 16, 17, and 2). The architectural feature 'Feedback Loop 155' provides recursion for the interactive interactive video position because the video can be looped indefinitely when the application requires video. There are many application possibilities that have never been available before. Another key architectural feature is that the video stream is a one-way UDP stream. This effectively enables any degree of multicasting of streaming interactive video (compared to a bi-directional stream such as TCP/IP streams that will increase over the number of users as the number of users increases) The traffic is stagnant). Multicast pedigree. An important capability within it because it allows the system to respond to the growing needs of Internet users (widely the 'world's population) to communicate on a one-to-two-to-many basis. Again, examples of the use of both stringed mobile video recursion and multicast discussed herein (such as Figures 139849.doc - 151 - 200949567 16) are only the tip of a very large iceberg of possibility. In one embodiment, the various functional modules and associated steps described herein may be comprised of specific hardware components (such as special application integrated circuits ("ASICs") that contain the fixed-line logic for performing the steps. )) or by any combination of programmed computer components and custom hardware components. In an embodiment, the modules may be implemented in a programmable digital signal processor ("Dsp") such as a Texas Instruments 320X architecture (eg, TMS32〇c_〇, tms32〇c5〇〇〇, etc.) on. A variety of different DSPs can be used while still adhering to these basic principles. The embodiments may include various steps as set forth above. These steps can be embodied in machine executable instructions that cause a general purpose or special purpose processor to perform particular steps. Various components (such as computer memory, hard disk drive, and input device) that are not related to these basic principles have been omitted from the figure to avoid confusing the elements of the related objects that are not disclosed. Execution instructions are provided by machine readable media. The machine-readable medium can include, without limitation, flash memory, optical disc, cd_r〇m, DVDr〇m, ram, epr(10), EEPR〇m, magnetic or optical card, media, or other type suitable for storing electronic instructions. Machine readable medium. For example, t, the present invention can be downloaded as a computer program that can be accessed from a remote computer via a communication key (eg, a computer, connected) by means of a carrier or other medium of communication (eg, Servo host computer (for example, the user terminal). 'Retrospection to the understanding should also be understood, the elements of the disclosed subject matter can also be provided as computer programs. 39849.doc -152- 200949567 products, the computer program products can be Included in the machine readable medium having instructions stored thereon, the instructions can be used to program a computer (eg, at a location; other electronic devices) to perform-sequence operations. Or, material operations; · by a combination of hardware and software The machine-readable medium can include... not limited to) flexible disk, optical disk, CD_R〇M, and magneto-optical disk, fine, RAM, EPR〇M'EEPR (10), magnetic or optical card, propagation medium or suitable for _ sub-command Other types of media/machine readable media. For example, Φ ❷ and Lu's elements of the subject matter disclosed can be downloaded as a computer program product. (4) Data can be transmitted via a communication link (eg, 'data machine or network connection') by means of a carrier or other communication medium. The signal is transmitted from the remote computer or electronic device to the request process. In addition, the disclosed subject matter has been described in conjunction with the specific embodiments, but the <RTIgt; </ RTI> modifications and variations will be in the scope of the present disclosure. Therefore, the description and drawings are to be regarded as illustrative and not restrictive. Figure 1 illustrates the architecture of a prior art video game system. 2a-2b illustrate a high order system architecture in accordance with an embodiment. Figure 3 illustrates the actual, prior and required data rate for communication between the client and the server. Figure 4a shows a colocation service and a client 0 used in accordance with an embodiment. Figure 4b illustrates an exemplary delay associated with communication between the client and the colocation service. Figure 4e illustrates a client device in accordance with an embodiment. 139849.doc -153- 200949567 Figure 4d illustrates a client device in accordance with another embodiment. Figure 4e illustrates an example block diagram of the client device of Figure 4c. Figure 4f illustrates an example block diagram of the client device of Figure 4d. Figure 5 illustrates an example of video compression that may be used in accordance with an embodiment. Figure 6 &amp; illustrates the video compression that can be used in another embodiment. Figure 6b illustrates the peak in the data rate associated with transmitting a low complexity, low motion video sequence. Figure 6c illustrates the peak in the data rate associated with transmitting a high complexity, high motion video sequence. Figures 7a through 7b illustrate an example video compression technique used in an embodiment. Figure 8 illustrates an additional example video compression technique used in an embodiment. Figures 9a through 9c illustrate example techniques for use in mitigating data rate peaks in an embodiment. Figures 10a through 10b illustrate an embodiment of efficiently encapsulating a video image block within a packet. Figures 11a through 15 illustrate an embodiment using a forward error correction technique. Figure 12 illustrates an embodiment of compression using a multi-core processing unit. Figures 13a-13b illustrate geolocation and communication between colocation services, in accordance with various embodiments. Figure 14 illustrates an exemplary delay associated with communication between the client and the colocation service. 139849.doc -154- 200949567

Figure 15 illustrates an example master M J server hosting server architecture. §. An example of an embodiment of a user interface of a plurality of live video windows is screen shot. Figure 17 illustrates the user interface of Figure 16 after the selection of a particular video window. Figure 18 illustrates the user interface of Figure 17 after a particular video window has been enlarged to full screen size. Figure 19 illustrates example collaborative user video material overlaid on a multiplayer game screen.

Figure 20 illustrates an example user page for a game player on a colocation service. Figure 21 illustrates an example 3D interactive advertisement. Figure 22 illustrates an example sequence of steps for surface capture from a live broadcast performance to produce a photorealistic image with a textured surface. Figure 23 illustrates an example user interface page that allows selection of linear media content. Figure 24 is a graph illustrating the amount of time elapsed and the connection speed before the live web page is live. [Main component symbol description] 100 CPU/GPU 101 Random access memory (RAM) 102 Display device (SDTV/HDTV or computer monitor 103 Hard disk drive 104 Optical media drive/Optical media/CD player 105 Network connection 139849.doc -155- 200949567 106 205 206 210 211 220 221 222 301 302 303 401 402 403 404 405 406 406a 406b 409 410 412 413 Game Controller Client Device / Client / Client Platform Internet Host Hosting Service / Host Generation Pipe system user location game or application software developer input device display device / monitor or TV nominal maximum data rate available data rate required maximum data rate colocation service control system / colocation service control server Server Storage Area Network (SAN) Video Compression Logic/Video Compressor Independent Redundant Disk Array (RAID) Control Signal Control Signal Control Signal Routing Logic Network Low Latency Video Decompression Logic Control Signal Logic 139849.doc -156 - 200949567 Client/Client Device/Customer Computer Display Device Input Device Display Device /display Internet bypass service existence point WAN interface

Firewall/Router/NAT (Network Address Translation) Device WAN Interface One-way transmission time for transmitting control signals Round-trip routing/user field routing via user premises

Arrow/round trip delay/user ISP universal internet delay one-way routing

415 420 421 422 440 441 442 443 444 451 452 453 454 455 456 457 458 462 463 464 465 Maximum one-way delay arrow / compressed video decompressed Ethernet jack / Internet connection HDMI (High Definition Multimedia Interface) , connector display capability, client device / client 466, blackout glasses

468 Display Device / SDTV (Standard Definition TV) or HDTV 139849.doc -157- 200949567 (High Definition TV) / Monitor 469 Input Device 475 Client Device 476 Flash Memory 477 Video Camera 478 Display Device 479 Bluetooth Input Device / Wired USB Input 480 Bus 481 Ethernet Interface 482 WiFi Subsystem 483 Control CPU / Control Processor 484 Bluetooth Interface 485 USB Subsystem 486 Video Decompressor 487 Video Output Subsystem 488 Audio Decompression Subsystem 489 Audio Output Subsystem 490 HDMI Interface 491 DVI-I 492 S - Video 493 Composite Video 494 Digital Interface 495 Stereo Analog Interface 497 Ethernet 139849.doc 158- 200949567 499 Power 501 Uncompressed Video Frame 502 Compressed video frame 503 uncompressed video frame 511 compressed frame 512 compressed frame 513 compressed frame 520 compressed logic 559-561 uncompressed video frame 611 I frame 612- 613 P frame 620 Compression logic 621 Rated maximum data rate 622 623 φ 624 633 Maximum data rate available / available data Rate I frame required peak data rate / peak data rate video stream data rate / video stream data rate sequence I frame peak 634 635 video stream data rate sequence / video stream data rate P frame peak 636 P frame peak 670 Block B 671 I Frame 701-760 Uncompressed Frame 711-770 R Frame 139S49.doc •159- 200949567 721-770 Uncompressed Frame 731-780 R Frame 805 Moving Character 811 R Box 812 R frame 922 Available maximum data rate 934 Video stream data rate 941 Peak data rate / Peak maximum data rate 942 2 times Peak data rate 943 3 times Peak data rate 944 4 times Peak data rate 952 Frame 2 times peak 953 Flat 2x peak 954 frame 4x peak 955 flattened 4x peak 961 uncompressed frame 1 962 uncompressed frame 2 963 uncompressed frame 3 964 uncompressed frame 4 965 Uncompressed frame 5 966 Uncompressed frame 6 967 Uncompressed frame 7 968 Uncompressed frame 8 969 Uncompressed frame 9 139849.doc - 160- 200949567

970 uncompressed frame 10 981 compressed frame 1 982 compressed frame 2 983 compressed frame 3 985 compressed frame 5 986 compressed frame 6 987 compressed frame 10 991 Transmission time (xmit time) 992 Transmission time 993 Transmission time (2 times peak value) 995 Transmission time 996 Transmission time (4 times peak value) 997 Transmission time 1001-1005 Packet 1010 Image block package package logic 1100 I image block 1101 Forward error Correction Code (FEC) 1103 I Image Block 1104 Forward Error Correction Code (FEC) 1105 Forward Error Correction Code (FEC) 1110 Audio Packet 1111 FEC Code 1112 Audio Packet 1113 FEC Code 139849.doc -161 - 200949567 1120 1121 1122 1123 1200 1201- 1205 1300 1301 1501 1502 1511 1512 1515 1521- 1529 1530 1539 1540 1550 1551 1552 1599 User input command FEC code User input command FEC code Multicore and / or multiprocessor 1208 Core 4x2 configuration / uncompressed map Frame Server Center/Host Hosting Service Network Incoming Internet Traffic Incoming Routing/Incoming Routing Network RAID Array RAID Array RAID Array/Delay Buffer 1525 Application/Game Server Uncompressed Video/Audio Shared Video Compression/Common Cluster/Common Hardware Compression/Common Hardware Compressor/Common Video Compression Hardware Compressed Video/Audio Output Routing/ Outbound routing network delay buffer and / or group video / feedback loop path path out of the Internet traffic 139849.doc -162- 200949567

1600 "thumbnail" video window 1601 yellow selection box 1602 game information 1700 video window 1800 arrow 1801 over 1900 video and audio 2000 user's own video 2001 live video 2002 partner 2003 video clip / self-reward clip / 3D DVR from Reward clip 2004 number 2005 icon 2400 line 2401 1.5 Mbps connection speed 2402 12 Mbps connection speed 2403 96 Mbps connection speed HS1-HS6 colocation service server center 139849.doc 163-

Claims (1)

200949567 VII. Patent application scope: 1. A device comprising: means for recording an application stream interactive video output and simultaneously recording an application state. A component of information; a component for playing the recorded interactive video output of the application; And means for executing the application based on the recorded application status information. Reference 139849.doc