JP7258864B2 - Venue Mapping for Virtual Reality Spectator of Electronic Sports - Google Patents

Description

The present disclosure relates to venue mapping for virtual reality viewing of electronic sports.

Electronic sports (esports) generally represent a form of sports in which key aspects of the sport are facilitated by electronic systems, with player and team inputs and esports system outputs mediated by human-computer interfaces. (See, for example, Juho Hamari, Max Sjoblom, (2017) "What is eSports and why do people watch it?", Internet Research, Vol. 27 Issue: 2, pp. 211-232, which is incorporated herein by reference.) see). Substantially, esports includes competitive and professional video game events that are watched. Esports can be, by way of example and without limitation, live watched (eg, at a tournament venue) via online broadcast or online streaming and via television broadcast. Many esports events take the form of organized tournaments and, in particular, feature multiplayer video game competition between teams of players that may include both amateur and professional players. Common video game genres associated with esports include real-time strategy (RTS), fighting, first-person shooter (FPS), and multiplayer online battle arena (MOBA).

Video games are played by computing devices such as personal computers, game consoles, and mobile devices. One example of a gaming platform is the Sony Playstation4® (PS4), which is sold in the form of a game console. As is well known, game consoles are designed to connect to a display (usually a television) and allow user interaction via a handheld controller. A game console is dedicated processing hardware, including a CPU, a graphics synthesizer to handle heavy graphics operations, a vector math unit to perform geometry transformations, and other glue hardware, firmware, and software. Designed. The game console may further be designed with an optical disc reader for receiving discs of games for local play through the game console. Online games are also possible in which users can interactively compete against or play interactively with other users over the Internet. To increase the complexity of games and keep them interesting for players, game and hardware manufacturers have taken new approaches to enable more interactivity.

There is a growing trend in the computer game industry to develop games that enhance the interaction between the user and the game system. One way to achieve a higher quality interactive experience is to track the player's movements with a wireless game controller whose movements are tracked by the game system and use these movements as input for the game. . Generally speaking, gestural input refers to having an electronic device, such as a computing system, video game console, or smart appliance react to some gesture made by a player and captured by the electronic device.

Another way to achieve a more immersive and interactive experience is to use a Head Mounted Display (HMD). A head-mounted display is worn by a user and can be configured to present various graphics, such as views of a virtual space. Graphics presented on a head-mounted display can cover most or all of the user's field of view. Thus, HMDs can provide users with a visually immersive virtual reality experience, as the head-mounted display renders a three-dimensional real-time view of the virtual environment in response to the user's movements. A user wearing the HMD is provided with omnidirectional freedom of movement and thus can obtain an omnidirectional view of the virtual environment through the HMD.

Embodiments of the present disclosure have been made in this context.

Embodiments of the present disclosure include devices, methods, and systems related to venue mapping for virtual reality viewing of electronic sports.

In some implementations, the following operations: receiving a request from a client device over a network to view a live event through a head-mounted display by a virtual reality spectator; receiving multiple video feeds from multiple cameras positioned within the venue; accessing video processing parameters stored in association with the seat; and using the video processing parameters. selecting video feeds using a head, and stitching the selected video feeds together to produce a composite video providing a view of the venue from a vantage point substantially defined by the three-dimensional position of the seats within the venue; and transmitting the composite video over a network to a client device for rendering on a mounted display.

In some implementations, the video processing parameter identifies which video feed to select for stitching, the video processing parameter provides the three-dimensional position of the sheet with which the video processing parameter is associated, and the video feed. is defined based on the 3D position of the camera that

In some implementations, assigning the virtual reality spectators to seats includes identifying an occupancy status of the seats within the venue, wherein the occupancy status of a given seat is determined by whether the given seat actually exists within the venue. The seat to which the virtual reality spectator is assigned is the seat not occupied by a real spectator.

In some implementations, the occupancy status of a given seat further indicates whether the given seat is occupied by another virtual reality spectator, and the seat to which the virtual reality spectator is assigned is occupied by another virtual reality spectator. A seat not occupied by a reality spectator.

In some implementations, assigning a virtual reality spectator to a seat includes accessing the virtual reality spectator's social graph and assigning a seat to another virtual reality spectator who is a member of the social graph. and selecting a seat based on proximity.

In some implementations, the method includes accessing audio processing parameters stored in association with the seat, and using the audio processing parameters to view a viewpoint substantially defined by the three-dimensional position of the seat within the venue. Generating audio data simulating what it sounds like from a device, and transmitting the audio data over the network to the client device.

In some implementations, the audio processing parameter identifies audio captured by one or more microphones in the venue and generates audio data therefrom, and the audio processing parameter is associated with the audio processing parameter. It is defined based on the three-dimensional position of the seat and the three-dimensional position of the microphone.

In some embodiments, when executed on at least one computer, the at least one computer is given the following operations: viewing a live event through a head-mounted display by a virtual reality spectator over a network from a client device; receiving a request; assigning virtual reality spectators to seats in a venue where a live event is being held; receiving multiple video feeds from multiple cameras positioned within the venue; Accessing stored video processing parameters, selecting video feeds using the video processing parameters, and stitching the selected video feeds to a viewpoint substantially defined by the three-dimensional position of the seats within the venue. generating a composite video that provides a view of the venue from the A non-transitory computer-readable medium having program instructions embodied thereon is provided.

Some embodiments include at least one computing device, said at least one computing device having at least one processor and at least one memory, said at least one computing device comprising a client receiving requests over the network from a device to view a live event through a head-mounted display by a virtual reality spectator; assigning the virtual reality spectator to a seat at the venue where the live event is held; receiving multiple video feeds from multiple positioned cameras; accessing video processing parameters stored in association with the sheet; using the video processing parameters to select a video feed; to generate a composite video that provides a view of the venue from a viewpoint substantially defined by the three-dimensional position of the seats within the venue, and over a network for rendering to a head-mounted display. and transmitting composite video to a client device.

Other aspects and advantages of the present disclosure will become apparent from the following summary of the problem, taken in conjunction with the accompanying drawings, which are illustrated as a means of illustrating the principles of the disclosure.

The present disclosure may be better understood by reference to the following description in conjunction with the accompanying drawings.

1 illustrates a view of an electronic sports (esports) venue, according to an embodiment of the present disclosure;

1 is a conceptual overhead view of a venue, according to an embodiment of the present disclosure; FIG.

1 conceptually illustrates a portion of a seat at a venue where a live esports event is being held, according to an embodiment of the present disclosure;

1 conceptually illustrates a process for mapping the three-dimensional space of an esports venue to determine seat-specific parameters for virtual reality viewing, according to an embodiment of the present disclosure;

1 conceptually illustrates a virtual reality spectator's field of view, according to an embodiment of the present disclosure;

1 conceptually illustrates a system for providing virtual reality viewing of an esports event, according to an embodiment of the present disclosure;

4 illustrates a technique for determining whether seats in a venue are occupied by actual spectators, according to an embodiment of the present disclosure;

4 illustrates a method for a virtual reality spectator to see themselves in a real venue situation, according to an embodiment of the present disclosure;

FIG. 10 illustrates seating in a venue with additional functionality for sensing physical spectators to enable virtual reality spectator interactivity, according to an embodiment of the present disclosure; FIG.

1 illustrates a system for interaction with a virtual environment via a head-mounted display (HMD), according to an embodiment of the present disclosure;

1 illustrates a head-mounted display (HMD), according to an embodiment of the present disclosure; 1 illustrates a head-mounted display (HMD), according to an embodiment of the present disclosure;

1 illustrates one example of an HMD user interfacing with a client system and the client system providing content to a second screen display, referred to as the second screen, according to one implementation.

1 conceptually illustrates the functionality of an HMD in conjunction with a running video game, according to an embodiment of the present disclosure;

1 illustrates components of a head-mounted display, according to an embodiment of the present disclosure;

12 is a block diagram of a gaming system 1200, according to various implementations of the present disclosure; FIG.

The following embodiments of the present disclosure provide devices, methods, and systems for venue mapping for virtual reality viewing of electronic sports. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details described herein. In other instances, well-known process operations have not been described in detail so as not to unnecessarily obscure the present disclosure.

In various implementations, the methods, systems, image capture objects, sensors, and associated interface objects (e.g., controllers, gloves, peripherals, etc.) are configured to render data on a display screen in near real time. is configured to handle Broadly speaking, embodiments are described with respect to a display that is a head mounted display (HMD). However, in other implementations, the display is a second screen, a display of a mobile device, a display of a computer, a display panel, one or more people (e.g., viewing content or sharing an interactive experience). It may also be a human remotely connected user display or the like.

FIG. 1A illustrates a view of an electronic sports (esports) venue, according to an embodiment of the present disclosure. Esports generally refers to competitive or professional games, especially multiplayer video games, that are watched by a variety of spectators or users. In recent years, as e-sports have grown in popularity, so has interest in live viewing of e-sports events at physical venues, many of which can seat thousands of people. A suitable venue is any location capable of hosting an eSports event for spectators to watch live, including, but not limited to, arenas, stadiums, theaters, convention centers, gymnasiums, community centers, etc. can be

However, the hosting and production of esports events, such as tournaments at discreet physical venues, means that not everyone who wants to watch in person can do so. Therefore, it is desirable to provide remote spectators with a live experience that allows them to experience an esports event as if they were directly at the venue where the esports event is taking place.

With continued reference to FIG. 1A, a view of a venue 100 hosting an esports event is shown. A typical esports event is a tournament in which teams of players compete against each other in a multiplayer video game. In the illustrated implementation, the first team consists of players 102a, 102b, 102c, and 102d, and the second team consists of players 104a, 104b, 104c, and 104d. A first team and a second team are positioned on stage 106 with announcer/host 110 . A first team and a second team are competing against each other in competitive gameplay of a multiplayer video game at venue 100, and spectators 113 are there to watch the event.

Large displays 108a, 108b, and 108c provide spectator 113 with a view of the gameplay. Displays 108a, 108b, and 108c are capable of presenting gameplay content to spectators, including, by way of example and without limitation, LED displays, liquid crystal displays, DLP, etc., known in the art. It is understood that the display may be any type of display. In some implementations, displays 108a, 108b, and 108c are display screens onto which gameplay video/images are projected by one or more projectors (not shown). Displays 108a, 108b, and 108c may display, by way of example and without limitation, gameplay content, video game player views, game maps, video game spectator views, commentator views, player/team stats and scores, advertising. It should be appreciated that it can be configured to present any type of content, including;

In addition, commentators 112a and 112b provide commentary on gameplay, such as explaining gameplay in real time as gameplay occurs, providing analysis of gameplay, highlighting particular actions, and the like.

FIG. 1B is a conceptual overhead view of venue 100, according to an embodiment of the present disclosure. As previously described, a first team and a second team of players are positioned on a stage and are competing in gameplay of a multiplayer video game. A plurality of seats 114 are shown conceptually, which seats are available when a spectator directly participates in and watches an esports event. As previously mentioned, there are large displays 108a, 108b, and 108c that provide spectator 113 with a view of gameplay and other content. In addition, they may be distributed throughout the venue to provide audio for spectators to hear, including audio associated with or relating to any content rendered on displays 108a, 108b, and 108c. A good number of speakers 118 are present.

Furthermore, a venue 100 configured to capture video of an esports event for processing, distribution, streaming, and/or direct live and/or remote viewing by spectators in accordance with embodiments of the present disclosure. There are any number of cameras 116 distributed throughout the . Some of the cameras 116 may have a fixed position and/or orientation, while some of the cameras 116 may change position and/or orientation, move to new locations and/or move to new orientations. It is understood that changes in the It is understood that camera 116 may have various fields of view. Additionally, some of the cameras may be 360-degree cameras capable of capturing a 360-degree field of view (e.g., combining a 360-degree horizontal field of view with a 180-degree vertical field of view to obtain a full spherical field of view). can be provided). Such 360 degree cameras typically include multiple image capture devices in a single device package. In some implementations, multiple cameras are configured at substantially or nearly the same location, and their feeds stitched together to allow a 360 degree view from that location's point of view.

Embodiments of the present disclosure can provide virtual reality spectators 120 with a “live” viewing experience of an esports event. That is, the virtual reality spectator 120 has a head-mounted display (HMD) (or Alternatively, a view through a virtual reality headset) is provided. Broadly speaking, the three-dimensional (3D) position of the virtual reality spectator's seat 122 can be determined, and the video feed from a particular one of the various cameras 116 is used to capture the seat 122 (or virtual reality spectator's seat). It can be stitched together to provide a virtual reality view of the venue 100 from the point of view of the designated location to which the participants were assigned.

Furthermore, although not shown in detail, each camera may include at least one microphone for capturing audio from venue 100 . There may also be additional microphones distributed throughout the venue 100 . Audio from at least some of these microphones is also based on the three-dimensional position of the virtual reality spectator's seat 122 to provide audio that simulates what is heard from the perspective of a spectator occupying the seat 122 . can be processed by

FIG. 1C conceptually illustrates a portion 124 of seats in a venue 100 where a live esports event is being held, according to an embodiment of the present disclosure. As shown, virtual reality spectators 120 are presented with a view through HMD 150 that simulates occupation of seats 122 within venue 100 . In some implementations, the view of the esports event provided to virtual reality spectators 120 is provided from streaming service 142 over network 144 . That is, the streaming service 142 includes one or more server computers configured to stream video for rendering on the HMD 150, where the rendered video provides a virtual reality spectator's 120 view of the esports event. provide to Although not specifically shown in the illustrated embodiment, streaming service 142 may first transmit the video in the form of data over network 144 to computing devices local to virtual reality spectators 120 . Well, it should be appreciated that a computing device may process data for rendering on HMD 150 .

Streaming service 142 may provide virtual reality spectators 120 with an interface through which virtual reality spectators 120 may select or subscribe to one or more views to be streamed for rendering on HMD 150 . As previously mentioned, these views can be 360-degree views of the event/venue that provide an immersive viewing experience for the virtual reality spectators 120, from the perspective of a particular seat or position within the venue. belongs to.

It should be appreciated that the views provided react in real-time to the movements of the virtual reality spectator 120, so, for example, if the virtual reality spectator 120 turns to the left, the virtual reality spectator 120 (HMD 150 If the virtual reality spectator 120 looks to the right (through the HMD 150 ) at the left view of the seat 122 , the virtual reality spectator 120 sees the right view of the seat 122 (through the HMD 150 ), and so on. In some implementations, virtual reality spectators 120 are provided with possible views of esports venue 100 in all directions, including a 360 degree horizontal field of view. In some implementations, the virtual reality spectator 120 has e A possible view of a sports venue 100 is provided. In some implementations, the areas directly above or below may be excluded from the view provided. In some implementations, areas excluded from view of the esports venue may be provided with other content such as, for example, advertisements, splash screens, logo content, game-related images or videos.

In some implementations, the virtual reality spectator 120 can select a seat through the interface, and the virtual reality spectator may view the esports event from their chosen viewpoint. In some implementations, the selectable seats are seats that are not physically occupied by spectators directly present at the esports event. In other embodiments, both unoccupied and occupied seats are selectable for virtual reality viewing.

In some implementations, the streaming service 142 may automatically assign virtual reality spectators to specific seats. In some implementations, this may be the highest available seat (eg, according to a predetermined order or ranking of available seats).

In some implementations, virtual reality spectators may be assigned seats near other spectators based on various characteristics of the spectators. In some implementations, virtual reality spectators are assigned to seats based at least in part on their membership in the social network/graph. For example, with continued reference to FIG. 1C, another virtual reality spectator 132 may be a friend of virtual reality spectator 120 on social network 130 (eg, defined by membership in a social graph). Streaming service 142 may use this information to assign virtual reality spectators 132 to seats that are close to virtual reality spectators 120, such as seat 134 next to seat 122 to which virtual reality spectators 120 are assigned. good. In the illustrated implementation, this allows virtual reality spectators 120 to see the avatars of virtual reality spectators 132 sitting next to them when they virtually "sit" and turn to the right. be able to.

In some implementations, the interface for seat selection and/or assignment allows a given user to confirm that one or more of their friends on social network 130 are also participating in the esports event virtually. and may provide options to be automatically assigned to seats near one or more of the friends. In this way, friends participating in the same event as the virtual reality spectator can enjoy the event together.

In various implementations, virtual reality spectators may choose to: Can be assigned to sheets near each other.

In some implementations, these concepts can be applied to direct "real" spectators (physically existing as opposed to virtual reality spectators) when knowing information about such real spectators. can be extended to include For example, a real spectator 138 seated on a seat 136 may be determined to be a friend of the virtual reality spectator 120 so that the virtual reality spectator 120 sits next to the real spectator 138's seat 136 . may be assigned (or offered to be assigned) to a seat 122 in the .

It is understood that the virtual reality spectator may use one or more controller devices to provide input while viewing content through the HMD. In the illustrated implementation, virtual reality spectators 120, 132, and 140 operate controller devices 152, 156, and 160, respectively, to provide input, e.g., streaming video for virtual reality viewing. Start and stop, select a seat in the esports venue 100 for watching, etc.

It is understood that in some implementations, spectators may ask each other if they are near each other within the esports venue 100, whether virtual or real. For example, virtual reality spectators 120 and 132 can communicate from their respective local environments (e.g., microphones of HMDs 150 and 154, controllers 152 and 156, or other locations in the local environment of spectators 120 and 132). via) is provided to other audio streams so they can hear each other. In some implementations, virtual reality spectators 120 may hear sounds from real spectators 138 captured by local microphones.

In some implementations, virtual reality spectators may occupy seats that are physically occupied by real spectators. For example, in the illustrated implementation, virtual reality spectators may occupy seats 136 that are physically occupied by real spectators 138 . A virtual reality spectator 140 may be provided with a view that simulates being in the seat 136 position. When the virtual reality spectator 140 turns to his right, he may see an avatar of the spectator 120; You can look.

FIG. 2 conceptually illustrates a process for mapping the three-dimensional space of an esports venue to determine seat-specific parameters for virtual reality viewing, according to an embodiment of the present disclosure. A three-dimensional (3D) space 200 of the venue 100 is shown to the left of the illustrated embodiment. According to embodiments of the present disclosure, a three-dimensional spatial map of venue space 200, as indicated by reference number 210, can be generated to facilitate determination of seat-specific parameters for virtual reality viewing.

Various techniques may be utilized to generate a three-dimensional spatial map of space 200 of venue 100 . In some implementations, data from one or more sensor devices 202 is processed to generate a three-dimensional spatial map. In various implementations, sensor device 202 may include any of an image capture device (eg, camera), depth sensing camera, ultrasonic sensor, IR camera, or the like. A three-dimensional spatial map of the venue space 200 allows the 3D coordinates of each seat 114 within the venue 100 to be determined. In some implementations, each seat in the venue has a unique assigned identifier. For each sheet, the corresponding three-dimensional coordinates can be mapped to the sheet's identifier, as indicated by reference number 212 . Some implementations employ, by way of example and without limitation, Wi-Fi-based positioning, magnetic positioning, visual markers and/or visual features to facilitate generation of a three-dimensional spatial map. Various localization/positioning techniques may be utilized, including visual recognition and the like. Sensors utilizing such localization/positioning techniques may be placed at the seat position to determine the seat position's three-dimensional position in space.

In some implementations, the 3D space map also includes the three-dimensional position and orientation of camera 116 in 3D space 200 of venue 100 . Using this information, video processing parameters for each sheet are then generated at reference numeral 214 . For a given seat, the video processing parameters are based on the view direction in 3D space, from the viewpoint of the given seat (rather, from the viewpoint of the 3D coordinates corresponding to the given seat) in the venue space. It defines which cameras' video feeds are stitched together and how they are stitched together to generate a view of . For cameras that can be oriented (e.g., controlled by an operator), the video processing parameters, when associated with a given seat, are such cameras as possible video feed sources, depending on the orientation of the camera. and how video from such cameras is stitched together (based on the virtual reality spectator's viewing direction), such as parameters for spatial projection, alignment, blending, anchor points, etc. of the video feed. may be further defined. It will be appreciated that the video stitching process may be configured to generate three-dimensional or two-dimensional video in various implementations. Generating a 3D video can involve generating a separate video for each eye.

As indicated by reference number 216 , audio processing parameters are generated for each seat in the venue 100 according to embodiments of the present disclosure. In some implementations, a three-dimensional space is acoustically modeled using a three-dimensional space map to determine acoustic properties of the three-dimensional space. In some implementations, microphones are placed at various locations within the venue to determine acoustic characteristics. For example, sounds may be played by speakers 118 in a venue, and audio recorded from microphones can be analyzed given the known positions of the microphones to determine acoustic characteristics and provide three-dimensional Enables spatial acoustic modeling. This may be used to generate audio processing parameters for each sheet. The audio processing parameters for a given seat are used to match audio from various sources, e.g. gameplay audio, commentator audio, venue You can define how to handle audio from live microphones in 100, and so on.

As indicated by reference numeral 218, the video processing parameters and audio processing parameters for a given sheet are stored in association with that sheet's identifier (ID). In some implementations, the video processing parameters and audio processing parameters for a given seat form at least a portion of the seat profile for the given seat within the venue.

Although the foregoing mapping of venue space was described with respect to seats within the venue, it should be recognized that the concepts are applicable to any specified location within the venue. In some implementations, for example, seats within the venue (which may or may not coincide with specific seats within the venue) to provide a 360-degree view from a specified position viewpoint. One or more designated locations of are identified, and video processing parameters and audio processing parameters are determined above for the designated locations. In some implementations, virtual reality spectators may select from such available locations to watch the live event. That is, a virtual reality spectator may select a view or set of views that stream the live event for viewing through their HMD.

FIG. 3 conceptually illustrates a virtual reality spectator's field of view, according to an embodiment of the present disclosure. As shown, the virtual reality spectator 120 has a 360 degree field of view 300 conceptually represented in the illustrated embodiment. The 360 degree field of view 300 may be, by way of example and without limitation, a horizontal field of view in some implementations, or a vertical field of view in some implementations, or any other plane of view in various implementations. can be expressed as

As indicated above, the view of the esports venue 100 provided to the virtual reality spectators 120 may be stitched together from video feeds from multiple cameras within the esports venue 100 . In the illustrated implementation, camera 304a provides a video feed with field of view 306a, camera 304b provides a video feed with field of view 306b, camera 304c provides a video feed with field of view 306c, and camera 304d provides a video feed with field of view 306c. 306d. Depending on the direction the virtual reality spectator 120 is looking, different ones of the video feeds from different cameras are selected for stitching together to provide the virtual reality spectator 120 with an appropriate view of the HMD. Form a video for rendering (150). In the illustrated implementation, virtual reality spectator 120 is shown looking in a direction to have field of view 302 . Based on this, the rendered video for viewing by virtual reality spectators 120 may be stitched together from the video feeds of cameras 304a, 304b, and 304d. The video feed from camera 304c is not used to provide the current view because it does not cover any portion of the area included in the virtual reality spectator's field of view 302.

It is understood that various cameras have different positions within the three-dimensional venue space and therefore have different points of view. Thus, the use or non-use of video feeds from these cameras (and the method of stitching, if used) to provide a view for a particular virtual reality spectator depends on their orientation and/or It may further depend on the viewpoint position. For example, in some implementations, a video feed covering the area currently viewed by the virtual reality spectator, but in a direction substantially opposite the virtual reality spectator's line-of-sight direction provides video to the virtual reality spectator. not used for

As shown, some cameras may have a fixed orientation, while others may change orientation. Accordingly, in some implementations, the current orientation information of the camera is obtained to determine whether a given camera's video feed should be used and/or, if selected, to use the given camera's video feed. Used to determine how to use.

In some implementations, the orientation of a given camera is adjusted based on the virtual reality spectator's field of view. For example, the orientation of the camera may be adjusted so that its field of view covers at least a portion of the area viewed by the virtual reality spectator. In some implementations, the orientation of a given camera is adjusted based on the views of multiple virtual reality spectators to optimize their views. In some implementations, a density mapping of the virtual reality spectator's field of view is determined, which identifies and/or quantifies the relative amount that different regions of space are being watched. Based on this information, camera orientation can be determined and adjusted, for example, to favor areas of space that are being watched to a greater degree.

It is understood that the available video feeds from the various cameras may not provide coverage of all areas viewed by a given virtual reality spectator, including the required orientation and/or viewpoint. . The coverage area that can be provided based on the available video feed defines the area of the venue space that is available for live view. With continued reference to FIG. 3, the available camera fields of view 306a, 306b, 306c, and 306d provide real-time live views of the venue space as determined by the extent of the venue space provided by the video feeds from the available cameras. Together, we define a live view region 308, which is the region of the 360 degree field of view 300 that can be provided.

In some implementations, the remaining regions not provided with live view can be presented in prerecorded views, defining prerecorded view regions 310 in the illustrated implementation. That is, the video or images of region 310 may be recorded from an earlier time (eg, when one or more of the cameras are aimed to cover at least a portion of region 310) and may have been previously recorded. A video/image can be stitched together and rendered to provide a view of region 310 . In some implementations, the view presented to the virtual reality spectator 120 is a live video view when the virtual reality spectator's 120 field of view includes portions of both the live view area 308 and the pre-recorded view area 310 . and composites generated from pre-recorded video/images.

In other implementations, areas of the virtual reality spectator's field of view 300 that are not available for live view may be presented with other content, such as, for example, advertising/sponsored content, game-related content, and the like.

With continued reference to FIG. 3, video feeds from various cameras, as indicated by reference numeral 320, provide the appropriate view for the virtual reality spectator 120 based on the virtual reality spectator's current field of view 302. May be stitched together to form a video. Furthermore, as indicated by reference numeral 322, the stitched video may undergo further processing such as compression processes to reduce the amount of data to be transmitted. In some implementations, the compression process may include foveated rendering, whereby it is determined that the virtual reality spectator 120 is viewing, for example, as determined from gaze direction information detected by the HMD 150. Regions of the image frame that have been rendered more efficient than others, for example, through the use of increased resolution, update frequency, bitrate, color, dynamic range, sharpness, contrast, or any other parameter that affects image quality. rendered in high quality. The output of the compression process is image frames 324 that define the video for the virtual reality spectator 120 . As shown in the illustrated embodiment, region 326 of the video image frame is rendered at a higher quality than region 328 .

The compressed image frames are then transmitted/streamed for display through the virtual reality spectator's HMD 150 , as indicated by reference number 330 .

FIG. 4 conceptually illustrates a system for providing virtual reality viewing of an esports event, according to an embodiment of the present disclosure. Although not described in particular detail for ease of explanation, the various systems, components, and modules described herein can be implemented in a single processor having one or more processors for executing program instructions. or by a plurality of computers or servers and one or more memory devices for storing data and said program instructions. Any such systems, components, and modules may communicate with any other such systems, components, and modules, and/or where appropriate, according to embodiments of the present disclosure. It should be appreciated that data may be sent/received via one or more networks to facilitate functionality. In various implementations, various portions of the system, components, and modules may be local to each other or distributed over one or more networks.

In the illustrated implementation, virtual reality spectators 120 interface with the system through HMDs 150 and use one or more controller devices 152 for additional interactivity and input. In some implementations, video images displayed to virtual reality spectators 120 via HMDs are transmitted via network 402 (which may include the Internet) to various systems and devices as described herein. is received from a computing device 400 that communicates with.

To initiate access to watch an esports event, the virtual reality spectator 120 may access the event manager 404, which processes requests to watch the esports event. Event manager 404 may include seat assignment logic 405 configured to assign virtual reality spectators 120 to particular seats within the venue of the esports event. Seat assignment logic 405 can utilize a variety of information to determine which seats to assign virtual reality spectators 120 to, including based on spectator user profile data 408 stored in user database 407. . By way of example, such user profile data 408 may include demographic information about the user such as age, geolocation, gender, nationality, primary language, occupation, and interests, preferences, games played/games owned/purchases. It can include other types of information such as games played, gaming experience level, Internet browsing history, and so on.

In some implementations, the seat assignment logic 405 can use information obtained from the social network 410 to assign, for example, spectators who are friends on the social network to seats near or next to each other. . To obtain such information, social network 410 may store social information about the user (including social graph membership information) as social data 414 in social database 412 . In some implementations, the seat allocation logic 405 may access social data (eg, access a given user/spectator's social graph) via the API of the social network 410 .

In some implementations, the seat allocation logic 405 is configured to determine which seats are available, e.g., not occupied by real and/or virtual spectators, and such information includes at least Assign virtual reality spectators based in part. In some implementations, the seat assignment logic 405 is configured to automatically assign virtual reality spectators to the best available seats as determined from a pre-defined ranking of seats within the venue.

It is understood that the seat assignment logic 405 can use any of the factors described herein along with any other factor(s) to determine which seat to assign a given virtual reality spectator. be. In some implementations, available seats are scored based on various factors, and seat assignments are determined based on the scores (e.g., virtual reality spectators are assigned to the highest scoring seats). be done). In some implementations, the seat assignment logic 405 presents a recommended seat for acceptance by the spectator 120, who upon acceptance is assigned to the recommended seat.

In another embodiment, the virtual reality spectator 120 accesses an interface provided by seat selection logic 406 configured to allow the virtual reality spectator to select a given seat from available seats. good too.

Venue database 416 stores data about one or more venues as venue data 418 . Venue data 418 can include any data describing the venue, such as a three-dimensional spatial map, camera, speaker, and microphone locations. Venue data 418 may further include a table 420 that associates seat profiles with unique seat identifiers. In some implementations, each sheet has its own sheet profile. In some implementations, groups of sheets (eg, adjacent to each other) may share the same sheet profile. The exemplary seat profile 422 includes information such as the seat's three-dimensional position 424 , video processing parameters 426 , and audio processing parameters 428 .

The video processor 432 uses the video processing parameters 426 and/or the spectator's assigned seat 3D position 424 to generate a composite video that provides a view to the virtual reality spectator 120 according to the virtual reality spectator's viewing direction. includes a stitch processor 434 that may stitch the video feed 430 from the camera 116 using . In some implementations, the spatial modeling module 438 creates a spatial model of the venue's three-dimensional environment (including, for example, camera positions and spectator seat positions) to facilitate stitching together the video feed 430 . create and access it. The stitching of the video feeds may involve aerial projection of the video feeds to provide the viewer with a point-of-view accurate video. In some implementations, the resulting composite video is 3D video, while in other implementations the composite video is 2D video.

Compression processor 436 is configured to compress the raw composite video, using video compression techniques known in the art, as well as foveated rendering, to reduce the amount of data required for streaming. Reduce. The compressed video data is then streamed by streaming server 448 over network 402 to computing device 400, which processes and/or processes the video to HMD 150 for viewing by virtual reality spectator 120. or render.

In some implementations, the video feed is transmitted from the camera to one or more computing devices local to the camera/venue that also perform video processing. In some implementations, cameras are directly connected to such computing devices. In some implementations, video feeds are transmitted to such computing devices via local networks (eg, including local area networks (LAN), Wi-Fi networks, etc.). In some implementations, the computing device is remotely located and the video feed may be transmitted over one or more networks such as the Internet, LANs, wide area networks (WANs), and the like.

Audio processor 444 is configured to process audio data 442 from audio source 440 that is streamed along with the compressed video data. The processing may use the audio processing parameters 428 and/or the three-dimensional position 424 of the spectator's seat. In some implementations, audio modeling module 446 utilizes an audio model based on the three-dimensional space of the venue to process audio data 442 . Such an audio model may simulate the acoustics of assigned seats within a venue such that the audio is rendered to virtual reality spectators in a realistic manner. By way of example and without limitation, sounds from other virtual reality spectators can be directional with respect to the virtual reality spectator's 120 seat position, as well as acoustic effects (e.g., delay, reverberation, etc.) appropriate to their seat position within the venue. etc.) may also be processed to simulate. As previously mentioned, audio sources can include gameplay audio, commentator(s), house music, audio from microphones in the venue, and the like.

FIG. 5 illustrates a technique for determining whether seats in a venue are occupied by actual spectators, according to an embodiment of the present disclosure. In the illustrated implementation, various seats 500, 502, and 504 within the venue 100 are shown. Seats 502 are occupied by actual spectators 506 . Camera 512 may be configured to capture an image of the sheet. The captured image can be analyzed according to an image recognition process 514 that searches the image for indicators that a given seat is occupied. By way of example and without limitation, image recognition processes may include facial recognition or other forms of person recognition to identify the presence of a person occupying a given seat, motion to detect movement occurring near a given seat. It may also include detection, image recognition of empty seat configurations, and the like.

In some implementations, the sheet may include (or may be attached to) a tag that can be recognized by the image recognition process 514 . In the illustrated embodiment, tag 508 is attached to sheet 500 and tag 510 is attached to sheet 504 . The seat 502 is now occupied by a real spectator 506 and thus the tag attached to the seat 502 is no longer visible and recognizable. In this way, empty sheets can be identified by identifying the tag in the captured image from camera 512 . Tags may, in various embodiments, include, by way of example and without limitation, retroreflectors or retroreflective materials, recognizable images/patterns/graphics, lights (e.g., LEDs), specific color(s), etc. It may take any recognizable form, including

By way of example and without limitation, the tag may, in some embodiments, be along the back or seat cushion, more particularly along the top of the back or seat cushion, or when the seat cushion is folded, the seat. is not occupied and can be seen when the seat cushion is folded, along the front of the seat cushion, or along the armrests, etc. good too. Furthermore, it is understood that a given sheet may have any number of tags attached to the sheet.

Based on the above detection of an empty or occupied seat, as indicated at reference numeral 516, the occupancy status of a given seat is changed to reflect whether the seat is occupied or empty. can be updated. As previously mentioned, in some implementations, empty seats are made available for virtual reality viewing at venue 100 .

FIG. 6 illustrates a method for a virtual reality spectator to see themselves in a real venue situation, according to an embodiment of the present disclosure. In the illustrated implementation, an esports event is held at venue 100, which has many seats. For example, seats 136 are occupied by actual spectators 138 . A virtual reality spectator 120 can “occupy” the seat 122 and thus experience the view through the HMD 150 of the venue 100 as if the virtual reality spectator 120 were physically present in the seat 122 .

Camera 600 may be operated by operator 602 and directed toward seats 122 and 136 , thereby capturing video of actual spectator 138 at seat 136 . This video, or a portion thereof, may be projected on display 108c, which may be one of the larger displays within venue 100 that may be viewed by many spectators at the same time. As shown, the projected video rendered on display 108 c shows an image 606 of the actual spectator 138 .

However, in some implementations, when virtual reality spectator 120 looks toward display 108c, virtual reality spectator 120 also sees himself within the projected video shown on display 108c. That is, when the video projected on the display 108c includes the position of the seat 122 "occupied" (or assigned) by the virtual reality spectator, it is provided to the virtual reality spectator 120, including the display 108c, through the HMD. In the view shown, the rendered image of display 108c includes an image of virtual reality spectator 120 (or avatar 604 associated with virtual reality spectator 120) positioned appropriately to occupy seat 122. is changed to In this manner, the virtual reality spectators 120 can see themselves on the seats 122 within the venue 100, further increasing the realism and interactivity of the virtual reality spectators' 120 experience. As shown in the illustrated embodiment, the rendered video on display 108c viewed by virtual reality spectator 120 is a representation of virtual reality spectator 120 (or virtual reality spectator 120's associated avatar 604). Contains image 608 .

It is understood that in some implementations, the video actually rendered on display 108c in venue 100 is not altered to include images of virtual reality spectators. However, in other implementations, the video actually rendered on display 108c in venue 100 is modified to include images of virtual reality spectators.

Continuing to refer to FIG. 6, a method for performing video modification to include virtual reality spectators as described above is shown. At operation 610, a video is captured with a camera in the venue, the video including the position/seat to which the virtual reality spectator has been assigned, the position/seat to which the virtual reality spectator is provided with a view of the venue through the HMD. defines the position where

At operation 612, the sheet and/or position captured by the camera is determined. This may involve determining the three-dimensional position and/or orientation of cameras within the venue to determine which seats are being captured in the video. For example, using the known position and orientation of the camera, the three-dimensional space within the venue captured in the video can be determined, and using the mapping of the seat to the three-dimensional position within the venue, , can identify specific sheets captured in the video. By way of example and without limitation, camera position/orientation is known in the art, including Wi-Fi based positioning, magnetic positioning, visual markers and/or visual feature recognition, etc. It can be determined using any kind of localization/positioning technique.

At operation 614, for a given identified seat, it is determined whether a virtual reality spectator has been assigned to the seat, and if so, the virtual reality spectator is identified. At operation 616, when the view provided to the virtual reality spectators includes the display on which the captured video is rendered at the venue, the view of the display is transferred to the seats occupied by the virtual reality spectators or to the seats occupied by the virtual reality spectators. Modified to show an alternate avatar for the virtual reality spectator. For example, a virtual reality spectator's view direction can be tracked to determine when the view includes the display. In this case, the view can be modified as described above to include the virtual reality spectator. In some implementations, the image of the virtual reality spectator (or the virtual reality spectator's avatar) is formed as an overlay of the view of the display presented to the virtual reality spectator. In some implementations, the camera video feed including the display is modified to include images of virtual reality spectators (or avatars), and the modified video feeds are stitched together as previously discussed. to form the view for the virtual reality spectator.

FIG. 7 illustrates seating in a venue with additional functionality for sensing physical spectators to enable interactivity for virtual reality spectators, according to an embodiment of the present disclosure. In the illustrated embodiment, seat 700 and seat 702 are shown, with seat 702 in front of seat 700 . Also shown are sensor units 704 and 706 configured for seats 700 and 702, respectively. In some implementations, the sensor unit is a separate attachment that is removable from the seat. In some implementations, the sensor unit and/or any component described herein can be incorporated as part of the seat.

For ease of explanation, sensor unit 706 is described herein, and it is understood that sensor unit 704 is similar. Sensor unit 706 includes one or more occupancy sensors 708 configured to detect whether seat 702 is occupied by an actual spectator. As shown, occupancy sensors 708 can be positioned along the seat and/or backrest. In some implementations, occupancy sensor 708 includes one or more pressure sensors that detect when pressure is applied to detect whether a person is sitting on seat 702 . Additionally, such a pressure sensor can determine/quantify how much pressure (force) is being applied and may be capable of weight discrimination between a person and another object. In some implementations, the occupancy sensor 708 can detect when another object is nearby (eg, within a predetermined distance), and in some cases can detect the distance of the object from the proximity sensor. Contains one or more proximity sensors.

In some implementations, the sensor unit 706 further includes a camera 710 configured to capture video/images from the viewpoint of the seat 702 , the camera 710 being a virtual reality spectator assigned to the seat 702 . can be used to provide a view of the venue to In some implementations, camera 710 is a spherical panoramic camera capable of capturing a 360 degree field of view. In various implementations, camera 710 includes one or more image capture devices. In some implementations, camera 710 is a stereoscopic camera capable of providing stereoscopic images with three-dimensional depth.

In some implementations, the sensor unit 706 further includes one or more microphone arrays 716, each microphone array including multiple microphones. Sound from the microphone array can be processed to identify the location of the sound source based on analysis of timing differences in reception between different microphones. The sound captured by the microphone array can be processed and/or placed in a virtual microphone assigned to seat 702 so that the virtual reality spectator can experience the sound from locations within the venue that are virtually located. It can be provided to reality spectators.

In some implementations, sensor unit 712 includes hardware that allows virtual reality spectators to interact with spectators in adjacent or nearby seats. In the illustrated implementation, sensor unit 712 further includes display 712 and speaker 714 . The display 712 can display an image of the virtual reality spectator (or avatar) assigned to the seat 702, and the actual spectator in a nearby seat is the virtual reality spectator (or avatar of the virtual reality spectator). ) can be seen. In some implementations, the virtual reality spectators can interact through speakers 714, thereby capturing (eg, by a microphone on the virtual reality spectator's HMD or in the virtual reality spectator's local environment) d) sounds from the virtual reality spectator can be transmitted to the sensor unit 706 and output through the speaker 714; Similarly, a nearby real spectator may speak, which is captured by the microphone array 716 to allow the virtual reality spectator to hear the nearby real spectator. transmitted to the person's HMD. In this way, real and virtual reality spectators may interact with each other. It is understood that a virtual reality spectator can see a nearby real spectator through camera 710 in some implementations.

In some implementations, sensor unit 706 includes a rear-facing sensor 718 that may be configured to detect occupancy of seat 700 behind seat 702 . In some implementations, the rear-facing sensor may be a proximity sensor and may use IR, ultrasonic, or other sensing technology.

In some implementations, the sensor unit 706 includes positioning technology that enables determination of the position of the sensor unit. This may include, by way of example and without limitation, positioning over Wi-Fi, magnetic positioning, GPS, and the like.

Sensor unit 706 may also communicate and transmit/receive data over one or more networks, such as cellular, Wi-Fi, or other types of networks, including wireless and/or wired networks. good.

In some implementations, the sensor unit 706 includes a tag that is recognizable by an image recognition process so that the sensor unit can be identified in captured images by cameras within the venue.

While implementations of the present disclosure have generally been described in terms of esports events and esports venues, in other implementations the principles of the present disclosure can be applied to sporting events (e.g., football, basketball, It should be recognized that it is applicable to any other type of spectated live event and corresponding venue, including baseball, soccer, hockey, etc.), concerts, ceremonies, presentations, and the like.

FIG. 8 illustrates a system for interaction with a virtual environment via a head-mounted display (HMD), according to an embodiment of the disclosure. HMDs can also be referred to as virtual reality (VR) headsets. As used herein, the term "virtual reality" (VR) generally refers to viewing a virtual space through an HMD (or VR headset) in a manner that reacts in real time to the movements of the HMD (controlled by a user). Represents the user's interaction with the virtual space/environment, including viewing, and provides the user with the feeling of being in the virtual space. For example, the user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns sideways, thereby similarly pointing the HMD, that side of the virtual space can be seen. is rendered on the HMD. In the illustrated embodiment, a user 120 is shown wearing a head-mounted display (HMD) 150 . HMD 150 is worn in a manner similar to eyeglasses, goggles, or a helmet and is configured to display video games or other content to user 120 . HMD 150 provides a highly immersive experience to the user by providing a display mechanism that is close to the user's eyes. Thus, HMD 150 can provide display areas for each of the user's eyes that occupy most or all of the user's field of view, and may also provide views with three-dimensional depth and perspective.

In the illustrated implementation, HMD 150 is wirelessly connected to computer 806 . In other implementations, HMD 150 is connected to computer 806 via a wired connection. Computer 806 can be any general or special purpose computer known in the art, including game consoles, personal computers, laptops, tablet computers, mobile devices, mobile phones, tablets. , thin clients, set-top boxes, media streaming devices, etc. In some implementations, computer 106 may be configured to run a video game and output video and audio from the video game for rendering by HMD 150 . In some implementations, computer 806 is configured to run any other type of interactive application that provides a virtual space/environment viewable through an HMD. Transceiver 810 is configured to transmit video and audio (via wired or wireless connection) from the video game to HMD 150 for rendering thereon. Transceiver 810 includes a transmitter for data transmission to HMD 150 as well as a receiver for receiving data transmitted by HMD 150 .

In some implementations, HMD 150 may communicate with the computer through alternative mechanisms or channels, such as via network 812 to which HMD 150 and computer 106 are both connected.

User 120 may manipulate interface objects 152 to provide input to the video game. Additionally, camera 808 may be configured to capture an image of the interactive environment in which user 120 is located. These captured images can be analyzed to determine the position and motion of user 120 , HMD 150 and interface objects 152 . In various implementations, the interface object 152 includes trackable lights and/or inertial sensor(s) to enable determination of the position and orientation of the interface object and tracking of movement.

In some implementations, a magnetic source 816 is provided that emits a magnetic field to enable magnetic tracking of HMD 150 and interface object 152 . Magnetic sensors in HMD 150 and interface object 152 can be configured to detect magnetic fields (e.g., strength, direction), and this information identifies the position and/or orientation of HMD 150 and interface object 152, Can be used for tracking.

In some implementations, interface object 152 is tracked relative to HMD 150 . For example, HMD 150 may include an outward-facing camera that captures images containing interface object 152 . The captured image can be analyzed to determine the position/orientation of the interface object 152 relative to the HMD 150, and the known position/orientation of the HMD can be used to determine the position/orientation of the interface object 152 in the local environment. identify.

The manner in which a user interfaces with the virtual reality scene displayed on HMD 150 may vary, and other interface devices in addition to interface object 152 may be used. For example, various types of one-handed as well as two-handed controllers can be used. In some implementations, the controller itself can be tracked by tracking lights contained in the controller, or by tracking geometry, sensors, and inertial data associated with the controller. Interfacing with and controlling the virtual reality environment presented on the HMD 150 using these various types of controllers or even using simple hand gestures captured by one or more cameras , manipulate, interact with, and participate in its environment.

Additionally, HMD 150 may include one or more lights that can be tracked to determine the position and orientation of HMD 150 . Camera 808 may include one or more microphones for capturing sound from the interactive environment. Sound captured by the microphone array may be processed to identify the location of the sound source. Sounds from identified locations can be selectively utilized or processed to the exclusion of other sounds from outside the identified locations. Furthermore, camera 808 can be defined to include multiple image capture devices (eg, stereoscopic pairs of cameras), IR cameras, depth cameras, and combinations thereof.

In some implementations, computer 806 functions as a thin client that communicates with cloud application (eg, gaming, streaming, spectator, etc.) provider 814 over network 812 . In such implementations, generally speaking, cloud application provider 114 maintains and runs the video games being played by users 150 . Computer 806 sends input from HMD 150, interface object 152, and camera 808 to the cloud application provider, which processes the input affecting the game state of the running video game. Output from the running video game, such as video data, audio data, and haptic feedback data, is sent to computer 806 . Computer 806 may further process the data before transmission or may transmit the data directly to the associated device. For example, video and audio streams are provided to HMD 150 while haptic/vibration feedback commands are provided to interface object 152 .

In some implementations, HMD 150, interface object 152, and camera 808 may themselves be networked devices that connect to network 812 to communicate with cloud application provider 814, for example. In some implementations, computer 106 may be a local network device, such as a router, that does not perform video game processing, but facilitates the passage of network traffic. Connections to the network by HMD 150, interface object 152, and camera 108 may be wired or wireless.

Further, while embodiments of the present disclosure may be described in terms of head-mounted displays, in other embodiments, mobile device screens (e.g., tablets, smartphones, laptops, etc.) or video rendering and/or in accordance with the present embodiments or any other type of display that can be configured to provide display of an interactive scene or virtual environment. .

Figures 9A-1 and 9A-2 illustrate head-mounted displays (HMDs), according to embodiments of the present disclosure. FIG. 9A-1 particularly illustrates one example of an HMD, a Playstation® VR headset, according to an embodiment of the present disclosure. As shown, HMD 150 includes multiple lights 900A-H. Each of these lights may be configured to have a particular shape and may be configured to have the same color or different colors. Lights 900A, 900B, 900C, and 900D are positioned in front of HMD 150 . Lights 900</b>E and 900</b>F are arranged on the sides of HMD 150 . Lights 900G and 900H are arranged at the corners of HMD 150 so as to straddle the front and side surfaces of HMD 150 . It is understood that the lights can be identified in the captured image of the interactive environment using the HMD 150 by the user. Based on light identification and tracking, the position and orientation of the HMD 150 in the interactive environment can be determined. It is further understood that some of the lights may or may not be visible depending on the particular orientation of HMD 150 relative to the image capture device. Also, different portions of the lights (eg, lights 900G and 900H) may be exposed to image capture depending on the orientation of HMD 150 relative to the image capture device.

In one implementation, the lights can be configured to indicate the current status of the HMD to others nearby. For example, some or all of the lights may be configured to have a particular color arrangement, intensity arrangement, may be configured to blink, may indicate a particular on/off configuration or current status of the HMD 150. It may be configured to have other arrangements as shown. For example, lights may be used during active gameplay of a video game (generally during active timelines or gameplay that occurs within scenes of the game), such as browsing a menu interface or performing game settings. Other inactive gameplay aspects of the video game (while the game timeline or scene may be inactive or dormant) may be configured to display different configurations. Lights may also be configured to indicate relative intensity levels of gameplay. For example, the intensity of the light or the rate of blinking may increase as the intensity of gameplay increases. In this way, an outsider to the user can see the lights on the HMD 150 and know that the user is actively engaged in intense game play and may not want to be interrupted at the time. It can be understood.

HMD 150 may further include one or more microphones. In the illustrated implementation, HMD 150 includes microphones 904A and 904B defined on the front of HMD 150 and microphone 904C defined on the side of HMD 150. FIG. By utilizing an array of microphones, the sound from each microphone can be processed to localize the sound source. This information can be used in a variety of ways, including eliminating unwanted sound sources, correlating with visual identification of sound sources, and the like.

HMD 150 may also include one or more image capture devices. In the illustrated embodiment, HMD 150 is shown to include image capture devices 902A and 902B. By utilizing a stereoscopic pair of image capture devices, three-dimensional (3D) images and videos of the environment can be captured from the perspective of HMD 150 . Such videos can be shown to the user while wearing the HMD 150 to provide the user with a "video see-through" function. That is, the user cannot strictly see through HMD 150, but nevertheless image capture devices 902A and 902B (or, for example, one or more outward-facing (e.g., , front-facing camera) can provide the functional equivalent of being able to see the environment external to HMD 150 as if viewed through HMD 150 . Such videos can be augmented with virtual elements to provide an augmented reality experience, or may be otherwise composited or blended with virtual elements. In the illustrated implementation, two cameras are shown in front of HMD 150, but it is understood that any number of outward-facing cameras, oriented in any direction, may be installed on HMD 150. . For example, in another implementation there may be a camera mounted on the side of HMD 150 to provide further panoramic image capture of the environment. Additionally, in some implementations, such an outward facing camera can be used to track other peripherals (eg, controllers, etc.). That is, the position/orientation of the peripheral relative to the HMD can be identified and tracked in captured images from such an outward-facing camera on the HMD, and the known position/orientation of the HMD in the local environment. can be used to determine the exact location/orientation of the peripheral.

FIG. 9B illustrates one example of user 120 of HMD 150 interfacing with client system 806 and client system 806 providing content to a second screen display, referred to as second screen 907 . Client system 806 may include integrated electronics for handling content sharing from HMD 150 to second screen 907 . Other implementations may include separate devices, modules, connectors that interface between the client system and HMD 150 and second screen 907, respectively. In this general example, user 120 is wearing HMD 150 and playing a video game using a controller, which may be interface object 152 . Interactive play by user 120 produces video game content (VGC) that is interactively displayed on HMD 150 .

In one implementation, the content displayed on HMD 150 is shared to second screen 907 . In one example, a person viewing second screen 907 can see content being interactively played on HMD 150 by user 120 . In another implementation, another user (eg, Player 2) can interact with client system 806 to generate second screen content (SSC). Second screen content, also generated by a player interacting with the controller 104 (or any kind of user interface, gesture, voice, or input) may be generated as an SSC to the client system 806, which It can be displayed on the second screen 907 along with the VGC received from HMD 150 .

Interactivity by other users, who may be co-located with the HMD user or remote from the HMD user, is therefore social, interactive, and interactive with the HMD user played by the HMD user on the second screen 907 and It can be more immersive for both users who can see the content. As shown, client system 806 can connect to the Internet 910 . The Internet can also provide client systems 806 and access to content from various content sources 920 . Content sources 920 can include any type of content accessible over the Internet.

Such content can include, but is not limited to, video content, movie content, streaming content, social media content, news content, friend content, advertising content, and the like. In one implementation, the client system 806 can be used to concurrently serve content for the HMD user, thereby providing the HMD with multimedia content associated with interactivity during game play. be done. Client system 806 can then provide other content as well. It may be independent of the video game content to the second screen. Client system 806, in one implementation, can receive second screen content from one of content sources 920 or from a local or remote user.

FIG. 10 conceptually illustrates the functionality of HMD 150 in conjunction with a running video game or other application, according to an embodiment of the present disclosure. A running video game/application is defined with a game/application engine 1020 that receives input to update the game/application state of the video game/application. A game state in a video game defines, at least in part, various aspects of current gameplay, such as the presence and location of objects, the state of the virtual environment, triggering events, user profiles, viewpoints of view, etc. It can be defined by the values of various parameters of the game.

In the illustrated implementation, the game engine receives controller input 1014, audio input 1016, and motion input 1018, as examples. Controller input 1014 is defined from the operation of a game controller separate from HMD 150 , such as a handheld game controller (eg, Sony DUALSHOCK® 4 wireless controller, Sony Playstation® Move motion controller) or interface object 152 . may By way of example, controller input 1014 may include directional inputs, button presses, trigger activations, actions, gestures, or other types of input processed from manipulation of a game controller. In some implementations, the movement of the game controller is tracked by an outward facing camera 1011 of HMD 102 that provides the position/orientation of the game controller relative to HMD 102 . Audio input 1016 may be processed from microphone 1002 of HMD 150, or from a microphone included in image capture device 808 or elsewhere in the local environment. Motion input 1018 may be processed from motion sensor 1000 included in HMD 150 or may be processed from image capture device 808 when capturing images for HMD 150 . Game engine 1020 receives input that is processed according to the configuration of the game engine to update the game state of the video game. The game engine 1020 outputs game state data to various rendering modules that process the game state data to define the content presented to the user.

In the illustrated implementation, a video rendering module 1022 is defined for rendering video streams for presentation on HMD 150 . The video stream may be presented on display/projector mechanism 1010 and viewed through optics 1008 by user's eyes 1006 . Audio rendering module 1024 is configured to render an audio stream for listening by a user. In one implementation, the audio stream is output by speaker 1004 associated with HMD 150 . It should be appreciated that the speakers 1004 may take the form of open air speakers, headphones, or any other type of speaker capable of presenting audio.

In one implementation, an eye-tracking camera 1012 is included in HMD 150 to enable tracking of the user's eye gaze. An eye-tracking camera captures an image of the user's eyes, which is analyzed to determine the user's gaze direction. In one implementation, information about the user's viewing direction can be utilized to influence video rendering. For example, if it is determined that the user's eyes are looking in a particular direction, then prioritizing or emphasizing video rendering in that direction, e.g. by providing a more detailed or faster update in the area the user is looking can do. It is recognized that the user's gaze direction can be defined with respect to the head-mounted display, with respect to the real environment in which the user is located, and/or with respect to the virtual environment being rendered with the head-mounted display. should.

Broadly speaking, analysis of the images captured by eye-tracking camera 1012 provides the user's eye-gaze direction relative to HMD 150 when considered alone. However, when considered in combination with the tracked position and orientation of HMD 150, the user's real-world gaze direction can be determined because the position and orientation of HMD 150 is synonymous with the position and orientation of the user's head. That is, the user's viewing direction in the real world can be determined from tracking the movement of the user's eye position and tracking the position and orientation of the HMD 150 . When the view of the virtual environment is rendered on the HMD 150, the user's real world gaze direction can be applied to determine the user's virtual world gaze direction in the virtual environment.

In addition, haptic feedback module 1026 is configured to provide signals to haptic feedback hardware contained within either HMD 150 or another device manipulated by a user, such as interface object 152 . Haptic feedback may take the form of various types of tactile sensations such as vibration feedback, temperature feedback, pressure feedback, and the like. Interface object 152 may include corresponding hardware to render such forms of haptic feedback.

Referring to FIG. 11, a diagram illustrating components of a head-mounted display 150 is shown, according to an embodiment of the present disclosure. Head mounted display 150 includes processor 1100 for executing program instructions. Memory 1102 is provided for storage and may include volatile and nonvolatile memory. A display 1104 is included that provides a visual interface viewable by a user. A battery 1106 is provided as a power source for the head mounted display 150 . Motion detection module 1108 may include any of various types of motion sensitive hardware such as magnetometer 1110 , accelerometer 1112 and gyroscope 1114 .

An accelerometer is a device for measuring acceleration and gravity-induced reaction forces. Uniaxial and multiaxial models are available to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one implementation, three accelerometers 1112 are used to provide the absolute reference direction of gravity for two angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic field near the head-mounted display. In one implementation, three magnetometers 1110 are used in the head-mounted display to define an absolute reference for world space yaw angle. In one embodiment, the magnetometer is designed to cover the earth's magnetic field which is ±80 microTesla. The magnetometer provides a yaw measurement that is influenced by metal and exhibits a monotonic relationship with the actual yaw. The magnetic field can be distorted by the effects of metal present in the environment, which distorts the yaw measurement. If desired, this distortion can be calibrated using information from other sensors such as gyroscopes or cameras. In one implementation, accelerometer 1112 is used in conjunction with magnetometer 1110 to obtain the tilt and azimuth angles of head-mounted display 150 .

In some implementations, the head-mounted display's magnetometer is configured to be read while electromagnets in other nearby devices are inactive.

A gyroscope is a device for measuring and maintaining orientation, based on the principle of angular momentum. In one implementation, three gyroscopes 1114 provide information about movement in each axis (x, y, z) based on sensed inertia. A gyroscope assists in detecting high speed rotation. However, gyroscopes can drift over time without an absolute reference. For this reason, periodic resetting of the gyroscope is required and should be done using other available information such as position/orientation determination based on visual tracking of objects, accelerometers, magnetometers, etc. can be done.

A camera 1116 is provided to capture images and image streams of the real environment. Head-mounted display 150 may include one or more cameras, a rear camera (a camera that moves away from the user when the user is looking at the display of head-mounted display 150) and a front camera (the user (a camera pointed at the user when viewing the display on head-mounted display 150). Additionally, a depth camera 1118 may be included in the head-mounted display 150 to sense depth information for objects in the real environment.

Head-mounted display 150 includes speakers 1120 for providing audio output. A microphone 1122 may also be included to capture audio from the real environment, including sounds from the surrounding environment, voices produced by the user, and the like. Head-mounted display 150 includes haptic feedback module 1124 for providing haptic feedback to the user. In one implementation, haptic feedback module 1124 can cause movement and/or vibration of head-mounted display 150 to provide haptic feedback to the user.

LED 1126 is provided as a visual indicator of the status of head mounted display 150 . For example, LEDs may indicate battery level, power on, and the like. A card reader 1128 is provided to allow the head mounted display 150 to read information from and write information to the memory card. A USB interface 1130 is included as one example of an interface for enabling connection of peripherals or other devices such as other portable devices, computers, and the like. In various implementations of head-mounted display 150, any of various types of interfaces may be included to allow more connections for head-mounted display 150. FIG.

A Wi-Fi module 1132 is included to enable connection to the Internet or local area networks via wireless networking technology. Head-mounted display 150 also includes Bluetooth® module 1134 to enable wireless connectivity to other devices. A communications link 1136 may also be included for connection to other devices. In one embodiment, communication link 1136 utilizes infrared transmission for wireless communication. In other implementations, communication link 1136 may utilize any of a variety of wireless or wired transmission protocols for communication with other devices.

Input buttons/sensors 1138 are included to provide an input interface for the user. Any of various types of input interfaces such as buttons, touchpads, joysticks, trackballs, etc. may be included. An ultrasound communication module 1140 may be included in the head mounted display 150 to facilitate communication with other devices via ultrasound technology.

A biosensor 1142 is included to allow detection of physiological data from the user. In one embodiment, biosensor 1142 includes one or more dry electrodes for detecting the user's bioelectrical signals through the user's skin.

Video input 1144 is configured to receive video signals from a primary processing computer (eg, main game console) for rendering on the HMD. In some implementations, the video input is an HDMI input.

The above components of head-mounted display 150 are described as merely exemplary components that may be included in head-mounted display 150 . In various implementations of the present disclosure, head-mounted display 150 may or may not include some of the various components described above. Embodiments of head-mounted display 150 may include other components not described herein but known in the art to facilitate aspects of the disclosure as described herein. It may contain further.

FIG. 12 is a block diagram of a gaming system 1200, according to various implementations of the disclosure. Gaming system 1200 is configured to provide video streams to one or more clients 1210 over network 1215 . Gaming system 1200 typically includes a video server system 1220 and an optional game server 1225 . Video server system 1220 is configured to provide video streams to one or more clients 1210 with a minimum quality of service. For example, video server system 1220 receives game commands that change the state or location of a view within a video game, and provides client 1210 with an updated video stream that reflects this state change with minimal latency. I have something to do. Video server system 1220 may be configured to provide video streams in a wide variety of alternative video formats, including formats not yet defined. Additionally, a video stream may include video frames configured for presentation to a user at a wide variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, higher or lower frame rates are also included in alternative implementations of the present disclosure.

Clients 1210, 1210A, 1210B, etc., which are referred to individually herein, include head-mounted displays, terminals, personal computers, game consoles, tablet computers, telephones, set-top boxes, kiosks, wireless devices, digital pads, stand-alone devices, handheld game-playing devices, and/or the like. Typically, client 1210 is configured to receive an encoded video stream, decode the video stream, and present the resulting video to a user, eg, a player of a game. The process of receiving an encoded video stream and/or decoding a video stream typically involves saving individual video frames to a client's receive buffer. The video stream may be presented to the user on a display integrated with the client 1210 or on a separate device such as a monitor or television. Client 1210 is optionally configured to support two or more game players. For example, game consoles may be configured to support two, three, four, or more players simultaneously. Each of these players may receive a separate video stream, or a single video stream generated specifically for each player, e.g., based on each player's point of view. may contain regions of the frame that have been Clients 1210 are optionally geographically distributed. The number of clients included in gaming system 1200 may vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "game-playing device" refers to a device used to play a game. used for In some implementations, a game-playing device may refer to multiple computing devices that work together to deliver a gaming experience to a user. For example, game consoles and HMDs may cooperate with a video server system 1220 to deliver games viewed through the HMD. In one implementation, the game console receives a video stream from the video server system 1220, and the game console forwards this video stream or updates to the video stream to the HMD for rendering.

Client 1210 is configured to receive the video stream over network 1215 . Network 1215 may be any type of communication network including telephone networks, the Internet, wireless networks, power line networks, local area networks, wide area networks, private networks, and/or the like. In typical implementations, the video stream is communicated via standard protocols such as TCP/IP or UDP/IP. Alternatively, the video stream is communicated via proprietary standards.

A typical example of client 1210 is a personal computer that includes a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. Decoding logic may include hardware, firmware, and/or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

Client 1210 may further include a system configured to modify received video, but this is not required. For example, the client may be configured to perform further rendering, overlaying one video image onto another, cropping video images, and/or the like. For example, client 1210 receives various types of video frames (such as I-frames, P-frames, and B-frames) and processes these frames into images for display to a user. may be configured to In some implementations, members of client 1210 are configured to perform further rendering, shading, conversion to 3D, or similar operations on the video stream. Members of client 1210 are optionally configured to receive more than one audio or video stream. Input devices of client 1210 may include, for example, one-handed game controllers, two-handed game controllers, gesture recognition systems, gaze recognition systems, speech recognition systems, keyboards, joysticks, pointing devices, force feedback devices, motion and/or position sensing devices, They may include mice, touch screens, neural interfaces, cameras, as yet undeveloped input devices, and/or the like.

Video streams (and, optionally, audio streams) received by client 1210 are generated and provided by video server system 1220 . The video stream includes video frames (and the audio stream includes audio frames), as further described elsewhere herein. A video frame is constructed to contribute significantly to the image displayed to the user (eg, the video frame contains pixel information in a suitable data structure). As used herein, the term "video frame" is used to refer to a frame containing information that primarily contributes to, e.g., influences, the image shown to the user. . Most of the teachings herein regarding "video frames" are also applicable to "audio frames."

Client 1210 is typically configured to receive input from a user. These inputs may include game commands configured to change the state of the video game or otherwise affect gameplay. Game commands may be received using an input device and/or game commands may be automatically generated by computing instructions to execute on client 1210 . Received game commands are communicated from client 1210 over network 1215 to video server system 1220 and/or game server 1225 . For example, in some implementations, game commands are communicated to game server 1225 via video server system 1220 . In some implementations, separate copies of game commands are communicated from client 1210 to game server 1225 and video server system 1220 . Communication of game commands optionally relies on the identification of the command. Game commands are optionally communicated from client 1210A through different routes or communication channels used to provide audio or video streams to client 1210A.

Game server 1225 is optionally operated by a different entity than video server system 1220 . For example, game server 1225 may be operated by a multiplayer game publisher. In this example, video server system 1220 is optionally viewed as a client by game server 1225 and is optionally a prior art client executing a prior art game engine from the perspective of game server 1225. is configured to represent Communication between video server system 1220 and game server 1225 optionally occurs over network 1215 . As such, game server 1225 may be a prior art multiplayer game server that sends game state information to multiple clients, one of which is game server system 1220 . Video server system 1220 may be configured to communicate with multiple instances of game server 1225 simultaneously. For example, video server system 1220 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by different game servers 1225 and/or published by different entities. In some implementations, several geographically distributed instances of video server system 1220 are configured to serve game videos to multiple different users. Each of these instances of video server system 1220 may communicate with the same instance of game server 1225 . Communication between the video server system 1220 and one or more game servers 1225 optionally occurs via dedicated communication channels. For example, video server system 1220 may be connected to game server 1225 via a high bandwidth channel dedicated to communication between these two systems.

Video server system 1220 includes at least video sources 1230 , I/O devices 1245 , processor 1250 , and non-transitory storage 1255 . Video server system 1220 may include a single computing device or be distributed among multiple computing devices. These computing devices may optionally be connected via a communication system such as a local area network.

Video source 1230 is configured to provide a video stream, eg, streaming video, or a series of video frames that form a motion picture. In some implementations, video source 1230 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from the player and maintain a copy of the state of the video game based on the received commands. This game state includes the positions of objects within the game environment, as well as typically the viewpoint. A game state may also include object properties, images, colors, and/or textures. Game states are typically maintained based on game rules and game commands such as move, rotate, attack, focus, interact, use, and/or the like. A portion of the game engine is optionally located within the game server 1225 . The game server 1225 may maintain a copy of the game state based on game commands received from multiple players using geographically distributed clients. In these cases, the game state is provided by the game server 1225 to the video source 1230 and a copy of the game state is saved and rendered. Game server 1225 may receive game commands directly from client 1210 via network 1215 and/or may receive game commands via video server system 1220 .

Video source 1230 typically includes rendering logic, eg, hardware, firmware, and/or software stored on a computer-readable medium such as storage 1255 . The rendering logic is configured to produce video frames of the video stream based on the game state. All or part of the rendering logic is optionally located within a graphics processing unit (GPU). Rendering logic typically configures processing steps to determine 3D spatial relationships between objects, apply appropriate textures, etc., based on game state and viewpoint. include. Rendering logic typically produces raw video that is then encoded before being communicated to client 1210 . For example, raw video can be stored in Adobe Flash® standard, . wav, H. 264, H. 263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid. May be encoded by FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding process optionally produces a packaged video stream for delivery to decoders on remote devices. A video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800x600, 1280x720 (eg, 720p), 1024x768, although any other frame size may be used. Frame rate is the number of video frames per second. A video stream may contain different types of video frames. For example, H. The H.264 standard includes 'P' frames and 'I' frames. I-frames contain information to refresh all macroblocks/pixels on the display device, while P-frames contain information to refresh a subset thereof. P-frames are typically smaller in data size than I-frames. As used herein, the term "frame size" means the number of pixels in one frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

In alternative implementations, video source 1230 includes a video recording device, such as a camera. This camera may be used to produce delayed or live video included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still or video camera. Video source 1230 may also include a storage device configured to store previously recorded video for inclusion in the video stream. Video source 1230 includes a motion or position sensing device configured to detect motion or position of an object, e.g., a person, and determine game state or output video based on detected motion and/or position. It may also include logic configured to generate.

Video source 1230 is optionally configured to provide overlays configured to be placed over other videos. For example, these overlays may include command interfaces, login instructions, messages to game players, images of other game players, video feeds of other game players (eg, webcam video). In implementations of client 1210A that include a touch screen interface or eye-gaze detection interface, overlays may include virtual keyboards, joysticks, touchpads, and/or the like. In one example of an overlay, the player's voice is superimposed over the audio stream. Video source 1230 optionally further includes one or more audio sources.

In implementations in which the video server system 1220 is configured to maintain game state based on input from more than one player, each player may have a different perspective, including position and orientation of the view. Video source 1230 is optionally configured to provide each player with a separate video stream based on their point of view. Additionally, video source 1230 may be configured to provide different frame sizes, frame data sizes, and/or encodings to each of clients 1210 . Video source 1230 is optionally configured to provide three-dimensional video.

I/O devices 1245 provide video, commands, information requests, game state, line of sight information, device movement, device location, user movement, client identification information, player identification information, game commands for the video server system 1220 . , security information, audio, and/or the like. I/O devices 1245 typically include communication hardware such as network cards or modems. I/O devices 1245 are configured to communicate with game server 1225 , network 1215 , and/or client 1210 .

Processor 1250 is configured to execute logic, eg, software, contained within the various components of video server system 1220 discussed herein. For example, processor 1250 may be programmed with software instructions to perform the functions of video source 1230, game server 1225, and/or client qualifier 1260. Video server system 1220 optionally includes more than one instance of processor 1250 . Processor 1250 may also be programmed with software instructions to execute commands received by video server system 1220 or to coordinate the operation of the various elements of game system 1200 discussed herein. good. Processor 1250 may include one or more hardware devices. Processor 1250 is an electronic processor.

Storage 1255 includes non-transitory analog and/or digital storage devices. For example, storage 1255 may include an analog storage device configured to store video frames. Storage 1255 may include computer-readable digital storage such as hard drives, optical drives, or solid state storage. Storage 1215 is configured to store video frames, artificial frames, video streams containing both video frames and artificial frames, audio frames, audio streams, and/or the like (e.g., suitable data structures, or by the file system). Storage 1255 is optionally distributed among multiple devices. In some implementations, storage 1255 is configured to store the software components of video source 1230 discussed elsewhere herein. These components may be stored in a form ready to be supplied as needed.

Video server system 1220 optionally further comprises client qualifier 1260 . Client qualifier 1260 is configured to remotely determine capabilities of a client, such as client 1210A or 1210B. These capabilities can include both the capabilities of client 1210A itself, as well as the capabilities of one or more communication channels between client 1210A and video server system 1220. FIG. For example, client qualifier 1260 may be configured to test communication channels through network 1215 .

Client qualifier 1260 can manually or automatically determine (eg, discover) the capabilities of client 1210A. Manual determination involves communicating with the user of client 1210A and asking the user to provide capabilities. For example, in some implementations, client qualifier 1260 is configured to display images, text, and/or the like within the browser of client 1210A. In one implementation, client 1210A is an HMD that includes a browser. In another implementation, client 1210A is a game console with a browser that may be displayed on an HMD. The displayed objects request information from the user such as client 1210A's operating system, processor, video decoder type, network connection type, display resolution, and the like. The information entered by the user is communicated back to the client qualifier 1260 .

Automatic determination may occur, for example, by running an agent on client 1210A and/or by sending a test video to client 1210A. Agents may include computing instructions such as java scripts embedded in web pages or installed as add-ons. Agents are optionally provided by client qualifiers 1260 . In various embodiments, the agent controls the processing capabilities of client 1210A, the decoding and display capabilities of client 1210A, the latency reliability and bandwidth of the communication channel between client 1210A and video server system 1220, the display of client 1210A. , firewalls present on the client 1210A, hardware of the client 1210A, software running on the client 1210A, registry entries within the client 1210A, and/or the like.

Client qualifier 1260 includes hardware, firmware, and/or software stored on a computer-readable medium. Client qualifier 1260 is optionally located on a computing device separate from one or more other elements of video server system 1220 . For example, in some implementations, client qualifier 1260 is configured to determine characteristics of a communication channel between client 1210 and two or more instances of video server system 1220 . In these embodiments, the information discovered by the Client Qualifier is used to determine which instance of the Video Server System 1220 is most suitable for delivery of streaming video to one of the Clients 1210. can be used.

Embodiments of the present disclosure may be practiced with various computer system configurations, such as handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

With the above embodiments in mind, it should be understood that the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations include those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the disclosure are useful machine operations. The present disclosure also relates to devices or apparatus for performing these operations. An apparatus may be specially constructed for the required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus to perform the required operations. There is good news.

The present disclosure can also be embodied as computer readable code on a computer readable medium. A computer-readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer-readable media include hard disks, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. and optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the method operations are described in a particular order, other housekeeping operations may be performed between the operations, or the operations may be coordinated so that they occur at slightly different times, or the operations may be performed by the system. It should be understood that this allows the processing operations to occur at various intervals associated with the processing, so long as the processing of the overlay operations is performed in the manner desired. .

Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered illustrative and not restrictive and the present disclosure is not limited to the details given herein, but rather within the scope of the appended claims and their equivalents. may be modified within

Claims (21)

receiving a request to watch a live event through a head-mounted display by a virtual reality spectator over a network from a client device;
assigning the virtual reality spectators to seats at a venue where the live event is held;
When the seat of the venue is captured and projected onto a display in the venue, an image of the virtual reality spectator assigned to the seat or an avatar associated with the virtual reality spectator occupies the seat. modifying a video rendered on the display within the venue to be included in the location;
receiving multiple video feeds from multiple cameras positioned within the venue;
accessing video processing parameters stored in association with the sheet;
a live view area capable of providing a real-time live view of a venue space, wherein the virtual reality spectator's field of view is defined by a coverage area capable of providing a real-time live view of a venue space based on the plurality of video feeds from the plurality of cameras; for the live view region, selecting the video feed using the video processing parameters and stitching the selected video feeds together to generate live video; and for said remaining viewing area, stitching pre-recorded videos by said plurality of cameras to generate a non-live video, combining said live video and said non-live video to produce said seat in said venue. generating a composite video providing a view of the venue from a viewpoint substantially defined by the three-dimensional position of
transmitting the composite video over the network to the client device for rendering on the head-mounted display. the video processing parameters identify which of the video feeds to select for stitching;
2. The method of claim 1, wherein the video processing parameters are defined based on the three-dimensional position of the seat with which the video processing parameters are associated and the three-dimensional position of the camera providing the video feed. . assigning the virtual reality spectators to the seats includes identifying an occupancy status of seats within the venue;
the occupancy status of a given seat indicates whether the given seat is occupied by an actual spectator within the venue;
2. The method of claim 1, wherein the seats to which the virtual reality spectators are assigned are seats not occupied by real spectators. the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality spectator;
4. The method of claim 3, wherein the seat to which the virtual reality spectator is assigned is a seat not occupied by another virtual reality spectator. assigning the virtual reality spectators to the seats;
accessing the virtual reality spectator's social graph;
and selecting the seat based on proximity to a seat assigned to another virtual reality spectator who is a member of the social graph. accessing audio processing parameters stored in association with the sheet;
using the audio processing parameters to generate audio data that simulates hearing from a viewpoint substantially defined by the three-dimensional position of the seat within the venue;
2. The method of claim 1, further comprising transmitting said audio data over said network to said client device. wherein the audio processing parameters identify audio captured by one or more microphones within the venue to generate the audio data therefrom;
7. The method of claim 6, wherein the audio processing parameter is defined based on the three-dimensional position of the seat with which the audio processing parameter is associated and the three-dimensional position of the microphone. A non-transitory computer-readable medium which, when executed on at least one computer, causes said at least one computer to perform the following operations:
receiving a request to watch a live event through a head-mounted display by a virtual reality spectator over a network from a client device;
assigning the virtual reality spectators to seats at a venue where the live event is held;
When the seat of the venue is captured and projected onto a display in the venue, an image of the virtual reality spectator assigned to the seat or an avatar associated with the virtual reality spectator occupies the seat. modifying a video rendered on the display within the venue to be included in the location;
receiving multiple video feeds from multiple cameras positioned within the venue;
accessing video processing parameters stored in association with the sheet;
a live view area capable of providing a real-time live view of a venue space, wherein the virtual reality spectator's field of view is defined by a coverage area capable of providing a real-time live view of a venue space based on the plurality of video feeds from the plurality of cameras; for the live view region, selecting the video feed using the video processing parameters and stitching the selected video feeds together to generate live video; and for said remaining viewing area, stitching pre-recorded videos by said plurality of cameras to generate a non-live video, combining said live video and said non-live video to produce said seat in said venue. generating a composite video providing a view of the venue from a viewpoint substantially defined by the three-dimensional position of
sending the composite video over the network to the client device for rendering on the head-mounted display. A non-transitory computer-readable medium, comprising: the video processing parameters identify which of the video feeds to select for stitching;
9. The apparatus of claim 8, wherein the video processing parameters are defined based on the three-dimensional position of the seat with which the video processing parameters are associated and the three-dimensional position of the camera providing the video feed. Temporary computer-readable medium. assigning the virtual reality spectators to the seats includes identifying an occupancy status of seats within the venue;
the occupancy status of a given seat indicates whether the given seat is occupied by an actual spectator within the venue;
9. The non-transitory computer readable medium of claim 8, wherein the seats to which the virtual reality spectators are assigned are seats not occupied by real spectators. the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality spectator;
11. The non-transitory computer readable medium of claim 10, wherein the seat to which the virtual reality spectator is assigned is a seat not occupied by another virtual reality spectator. assigning the virtual reality spectators to the seats;
accessing the virtual reality spectator's social graph;
9. The non-transitory computer-readable medium of claim 8, comprising selecting the seat based on proximity to a seat assigned to another virtual reality spectator who is a member of the social graph. accessing audio processing parameters stored in association with the sheet;
using the audio processing parameters to generate audio data that simulates hearing from a viewpoint substantially defined by the three-dimensional position of the seat within the venue;
9. The non-transitory computer-readable medium of claim 8, further comprising transmitting the audio data over the network to the client device. wherein the audio processing parameters identify audio captured by one or more microphones within the venue to generate the audio data therefrom;
14. The non-transitory computer-readable medium of claim 13, wherein the audio processing parameter is defined based on the three-dimensional position of the sheet with which the audio processing parameter is associated and the three-dimensional position of the microphone. comprising at least one computing device;
said at least one computing device having at least one processor and at least one memory;
the at least one computing device;
receiving a request to watch a live event through a head-mounted display by a virtual reality spectator over a network from a client device;
assigning the virtual reality spectators to seats at a venue where the live event is held;
When the seat of the venue is captured and projected onto a display in the venue, an image of the virtual reality spectator assigned to the seat or an avatar associated with the virtual reality spectator occupies the seat. modifying a video rendered on the display within the venue to be included in the location;
receiving multiple video feeds from multiple cameras positioned within the venue;
accessing video processing parameters stored in association with the sheet;
a live view area capable of providing a real-time live view of a venue space, wherein the virtual reality spectator's field of view is defined by a coverage area capable of providing a real-time live view of a venue space based on the plurality of video feeds from the plurality of cameras; for the live view region, selecting the video feed using the video processing parameters and stitching the selected video feeds together to generate live video; and for said remaining viewing area, stitching pre-recorded videos by said plurality of cameras to generate a non-live video, combining said live video and said non-live video to produce said seat in said venue. generating a composite video providing a view of the venue from a viewpoint substantially defined by the three-dimensional position of
transmitting the composite video over the network to the client device for rendering on the head-mounted display. the video processing parameters identify which of the video feeds to select for stitching;
16. The system of claim 15, wherein the video processing parameters are defined based on the three-dimensional position of the seat with which the video processing parameters are associated and the three-dimensional position of the camera providing the video feed. . assigning the virtual reality spectators to the seats includes identifying an occupancy status of seats within the venue;
the occupancy status of a given seat indicates whether the given seat is occupied by an actual spectator within the venue;
16. The system of claim 15, wherein the seats to which the virtual reality spectators are assigned are seats not occupied by real spectators. the occupancy status of the given seat further indicates whether the given seat is occupied by another virtual reality spectator;
18. The system of claim 17, wherein the seat to which the virtual reality spectator is assigned is a seat not occupied by another virtual reality spectator. assigning the virtual reality spectators to the seats;
accessing the virtual reality spectator's social graph;
16. The system of claim 15, comprising selecting the seat based on proximity to seats assigned to other virtual reality spectators who are members of the social graph. accessing audio processing parameters stored in association with the sheet;
using the audio processing parameters to generate audio data that simulates hearing from a viewpoint substantially defined by the three-dimensional position of the seat within the venue;
16. The system of claim 15, further comprising transmitting said audio data over said network to said client device. wherein the audio processing parameters identify audio captured by one or more microphones within the venue to generate the audio data therefrom;
21. The system of claim 20, wherein the audio processing parameter is defined based on the three-dimensional position of the seat with which the audio processing parameter is associated and the three-dimensional position of the microphone.

Images