JP2024502045A - Method and system for dynamic summary queue generation and provision - Google Patents

Abstract

A stream of media items is sent from a server computing system to one or more client computing systems. A media item has a timeline for playing the media item. Participant-generated content is received at a server computing system from one or more participants at any one or more client computing systems. Participant-generated content is time-indexed relative to the media item's timeline. The participant-generated content is processed to identify key portions of the media item based on representations present within the participant-generated content. Generate a dynamic summary queue containing key parts of the media item. Dynamic summary queues are assembled for streaming. A request to view a dynamic summary queue is received from a requester. In response to the request, a stream of dynamic summary queues is sent to the requester's computing device. [Selection diagram] Figure 2

Description

1. FIELD OF THE DISCLOSURE The present disclosure relates to methods and systems for providing video game and/or media content over a network between a server computing system and a client computing system.

2. Description of Related Technology The video game industry has undergone many changes over the years. With the increase in computing power, video game developers have also developed game software that takes advantage of this increased computing power. To that end, video game developers have coded games that incorporate advanced arithmetic and mathematics to create highly realistic gaming experiences.

Exemplary gaming platforms are Sony PlayStation®, Sony PlayStation2® (PS2), Sony PlayStation3® (PS3), Sony PlayStation4® (PS4), and Sony PlayStation® n5 (registered trademark) (PS5), each of which is sold in the form of a game console.
As is well known, gaming consoles are designed to connect to a display (typically a television) and allow interaction with a user by means of a handheld controller. Game consoles are equipped with dedicated processing hardware, including a CPU, a graphics synthesizer to handle intensive graphics operations, a vector unit to perform geometry transformations, and other glue hardware, firmware, and software. It is designed to. Furthermore, for local play by means of a gaming machine, the gaming machine may be designed to be provided with an optical disc reader for receiving game discs. Online gaming is also possible, allowing users to play interactively against or with other users over the Internet.
For example, the PlayStation® Network provides online playback of video games, streaming of video content (e.g., television, movies, sporting events, gameplay videos, etc.), and streaming of music and other audio content. It is a digital media entertainment service. Video game developers are also developing online games that not only accommodate the expansion of computing power, but also enhance the interaction between users and gaming systems. For example, interactive online games provide the ability for users to interactively play against or with other users over the Internet and obtain such interactions. It is in this context that the present invention arises.

In an exemplary embodiment, a method for generating summaries of media items is disclosed. The method includes transmitting a stream of media items from a server computing system to one or more client computing systems. A media item has a timeline for playing the media item. The method also includes receiving participant-generated content at the server computing system from one or more participants at any of the one or more client computing systems.
Participant-generated content is time-indexed relative to the media item's timeline. The method also includes processing the participant-generated content to identify key portions of the media item based on representations present in the participant-generated content. The method also includes generating a dynamic summary cue that includes a major portion of the media item. Dynamic summary queues are assembled for streaming. The method also includes receiving a request from a requester to view the dynamic summary queue. The method also includes transmitting the stream of the dynamic summary queue to the requester's computing device in response to the request.

In an exemplary embodiment, a system for generating video summaries is disclosed. The system includes program instructions stored in computer memory for transmitting a stream of media items from a server computing system to one or more client computing systems. A media item has a timeline for playing the media item. The system also includes program instructions stored in the computer memory for receiving participant-generated content at the server computing system from one or more participants on any of the one or more client computing systems.
Participant-generated content is time-indexed relative to the media item's timeline. The system also includes program instructions stored in the computer memory for processing the participant-generated content to identify key portions of the media item based on representations present in the participant-generated content. The system also includes program instructions stored in computer memory for generating a dynamic summary queue that includes a major portion of the media item. Dynamic summary queues are assembled for streaming. The system also includes program instructions stored in the computer memory for receiving requests from requesters to view the dynamic summary queue. The system also includes program instructions stored in the computer memory for transmitting the stream of dynamic summary queues to a requester's computing device in response to a request.

1 illustrates a cloud gaming system according to some embodiments. 2 illustrates an example user interface displayed on a display device to a user/participant during play of an online video game provided by a cloud gaming system, according to some embodiments. 2 illustrates an example user interface displayed on a display device to a user/participant upon selection of a summary request control, according to some embodiments. 1 illustrates a diagram of a system that generates and provides a dynamic summary queue of media items, according to some embodiments. 2 illustrates an example user interface that provides various controls to enable a requester of a dynamic summary queue to customize how a dynamic summary queue is generated, in accordance with some embodiments. 5A illustrates an example user interface that is displayed when a "change" control is selected among the participant-based controls of the user interface of FIG. 5A, according to some embodiments. FIG. 5A illustrates an example user interface that is displayed when a “change” control is selected among the time frame controls of the user interface of FIG. 5A, according to some embodiments. FIG. 5A illustrates an example user interface that is displayed when a "change" control is selected among the summary decomposition controls of the user interface of FIG. 5A, according to some embodiments; FIG. 5A illustrates an example user interface that is displayed when a “change” control is selected among the event type controls of the user interface of FIG. 5A, according to some embodiments. FIG. 5A illustrates an example user interface that is displayed when a “change” control is selected among the display type controls of the user interface of FIG. 5A, according to some embodiments. FIG. 2 illustrates a flowchart of a method for generating summaries of media items (dynamic summarization queues), according to some embodiments. 1 illustrates a diagram of components within an example computing device that can be used to perform aspects of various embodiments of this disclosure, according to some embodiments. FIG.

In the following detailed description, certain specific details are set forth to provide a thorough understanding of the disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure this disclosure.

FIG. 1 illustrates a cloud gaming system 101 according to some embodiments. Cloud gaming system 101 provides online video game play by one or more users/participants A, B, C, D (A-D). For example, FIG. 1 shows users/participants AD playing an online video game provided by cloud gaming system 101 via network 123.
Each of the users/participants A-D corresponds to a person 121A, 121B, 121C, and 121D (121A-121D), respectively, operating a corresponding local computing system 120A, 120B, 120C, and 120D (120A-120D), respectively. do. Thus, FIG. 1 shows an example of group play with a plurality of users/participants A to D. Although the example of FIG. 1 includes four users/participants A to D for purposes of illustration, users/participants who can participate in group play and/or watching online video games provided by the cloud gaming system 101 It is understood that there is no limit to the number of participants.
As used herein, the term online video game refers to a video game provided by cloud gaming system 101 to a client computing device for play and/or viewing by a user/participant at the client computing device, e.g. refers to Online video games referred to herein may be single-player or multi-player video games. In some embodiments, each of users/participants A-D is engaged in playing an online video game. In some embodiments, some of the users/participants A-D are engaged in playing online video games, and some of the users/participants A-D are watching others play online video games. They are the audience watching.

In some embodiments, cloud gaming system 101 is equipped to provide other media items in addition to providing online video game play. For example, in some embodiments, cloud gaming system 101 provides audiovisual presentations as media items. In some embodiments, the audiovisual presentations provided by cloud gaming system 101 include movies, television shows, real-world sporting events, electronic sporting events, online shows, online videos, and uploaded videos, among others. One or more.
Cloud gaming system 101 is capable of providing essentially any data stream that can be transmitted as a stream of digital data over a network for presentation and consumption (viewing) by users/participants A-D at respective local computing systems 120A-120D. It should be understood that the computer is operable to provide media items of type.
In various embodiments, users/participants A-D view media items provided by cloud gaming system 101 as a group, such that users/participants A-D participate in a group event. During a group event, the cloud gaming system 101 provides user-generated content to allow users/participants A-D to interact with each other and express their feelings regarding the media item being consumed. to be able to provide Again, as shown by way of example in FIG. 1, the four users/participants AD do not represent any limitation on the number of users/participants who can participate in a group event. It should be understood that there is no limit to the number of users/participants that can participate in a group event provided by cloud gaming system 101.

In some embodiments, cloud gaming system 101 includes multiple (N) cloud gaming servers 103-1 through 103-N. In some embodiments, the number (N) of cloud game servers 103-1 to 103-N may be up to 100 or more. In some embodiments, cloud gaming system 101 also includes multiple (X) management servers 105-1 through 105-X and one or more storage servers 107. Storage server 107 includes a number of data storage devices 107A, such as hard disk drives (HDDs) and/or solid state drives (SSDs). Storage server 107 stores video games that can be played on cloud game system 101.
In some embodiments, storage server 107 stores various media items available for provision by cloud gaming system 101. In some embodiments, management servers 105-1 through 105-X may also have access to other cloud-based storage systems separate from storage server 107, where the other cloud-based storage systems may store video game data and /or store media item data.
Each management server 105-1 to 105-X is responsible for managing cloud gaming sessions for users/participants A to D. Also, in some embodiments, a given management server 105-1-105-X is responsible for managing multiple cloud gaming sessions simultaneously.
In some embodiments, each management server 105-1 through 105-X powers on one or more cloud game servers 103-1 through 103-N and plays the requested video game when necessary. from the storage server 107 or other cloud-based storage system to a predetermined one of the cloud game servers 103-1 to 103-N; , initiates execution of the requested video game, logs the user/participant A-D into a given one of the cloud game servers 103-1-103-N, and logs the user/participant A-D into the storage server 107 or other, among other operations. The user data 111 of the logged-in users/participants A to D is operated to mount the user data 111 of the logging-in users/participants A to D from the cloud-based storage system of .

In a cloud game (or online game), users/participants A to D operate local computing systems 120A to 120D to log into cloud game system 101 on network 123, such as the Internet, and connect to cloud game server 103-1. ~103-N to play remotely executed online video games on one or more of the following. In some embodiments, users/participants A-D log into cloud gaming system 101 via one of management servers 105-1 through 105-X. However, it should be understood that there are various cloud services associated with user/participant AD login. For example, authenticating users/participants A-D, ensuring that users/participants A-D have an internet connection of sufficient quality, and each user/participant A-D can log in to. /There is a cloud service involved in finding available cloud game servers 103-1 to 103-N near participants A to D.

In some embodiments, the local computing systems 120A-120D include corresponding game controllers 125A, 125B, 125C, 125D (125A-125D) and corresponding display devices 126A, 126B, 126C, 126D (125A-125D), such as televisions or monitors. 126A to 126D). In some embodiments, local computing systems 120A-120D include corresponding dongles 128A, 128B, 128C, 128D (128A-128D) connected to corresponding display devices 126A-126D, where dongles 128A-128D include: A corresponding game controller 125A-125D and a corresponding display device 126A such that the network interface controller (NIC) of the display device 126A-126D can be used to transmit signals from the game controller 125A-125D to the network 123. 126D.

In some embodiments, local computing systems 120A-120D also include corresponding local gaming consoles 127A, 127B, 127C, 127D (127A-127D). In these embodiments, the signals generated by the game controllers 125A-125D may be transmitted to the network 123 through the corresponding local gaming consoles 127A-127D using the NICs of the corresponding local gaming consoles 127A-127D. I can do it. Also, in some embodiments, game controllers 125A-125D may communicate signals directly to network 123 through a local area network, such as a WIFI network. In some embodiments, the local computing systems 120A-120D also capture still images, video, and/or audio of the users/participants A-D and/or the environment in which the users/participants A-D are present. It includes corresponding cameras 122A, 122B, 122C, 122D (122A-122D) for recording. Content captured by the cameras 122A-122D may be transmitted to the network 123 via the corresponding local gaming consoles 127A-127D using the NICs of the corresponding local gaming consoles 127A-127D.

Also, in some embodiments, users/participants AD may play online video games on local computing devices other than local computing systems 120A-120D. For example, in some embodiments, any of users/participants A-D may play online video games on a mobile computing device such as a cell phone, tablet, laptop, smart car system, among others. Also, in some embodiments, users/participants A-D may play online video games on desktop computing systems. For ease of explanation, references herein to local computing systems 120A-120D also convey references to any other computing device, in which users/participants A-D are online Video games may be played and/or media items provided by cloud gaming system 101 may be consumed.

Further, in various embodiments, data (such as video stream data or audio data or haptic feedback data) is communicated from cloud gaming system 101 to local computing systems 120A-120D via network 123. In some embodiments, data is communicated from cloud gaming system 101 to display devices 126A-126D via network 123. In some embodiments, data is communicated via network 123 from cloud gaming system 101 to local gaming consoles 127A-127D, and from local gaming consoles 127A-127D to corresponding display devices 126A-126D. Also, in some embodiments, data is communicated from cloud gaming system 101 to game controllers 125A-125D via network 123.

During game play, users/participants A-D generate game commands that are transmitted via network 123 to cloud game servers 103-1-103-N. In some embodiments, users/participants A-D generate game commands using game controllers 125A-125D, and the game commands are transmitted from game controllers 125A-125D to network 123 and to cloud gaming server 103. -1 to 103-N in signal format.
In some embodiments, game commands are transmitted in the form of signals from game controllers 125A-125D to corresponding local gaming consoles 127A-127D or corresponding display devices 126A-126D, from where game commands are transmitted over network 123. It is transmitted to cloud game servers 103-1 to 103-N.
In various embodiments, transmission of game commands in the form of signals from game controllers 125A-125D to network 123 may occur via wired or wireless network connections, or a combination thereof. Cloud game servers 103-1 to 103-N execute video games according to game commands received from users/participants A to D, and generate video stream(s) reflecting video game play by users/participants A to D. possible). Video stream(s) are transmitted via network 123 from cloud gaming servers 103-1 to 103-N to users/participants A to D's local computing systems 120A to 126D for display on corresponding display devices 126A to 126D. 120D.

Cloud game servers 103-1 through 103-N load game data 113 and game executable files 115 from storage server 107 or other cloud-based storage system. Also, in some embodiments, cloud gaming servers 103-1 through 103-N load user data 111 from storage server 107 or other cloud-based storage system to management servers 105-1 through 105-X.
Modern video games generate large amounts of user data 111. Depending on the video game and video game platform (video game console), there may be various types of user data 111 generated for a given user/participant AD's video game.
For example, in some embodiments, user data 111 includes saved data for users/participants A-D, among other types of data for users/participants A-D. Downloaded data for users/participants A-D, recorded data 111A for users/participants A-D (records of users/participants A-D's game play for length of time in buffer or recently played), for users/participants A-D; Trophy data of participants A to D, profile data of users/participants A to D, video data of users/participants A to D, audio data of users/participants A to D, user/participants A to D Contains pause and resume data (captures the overall state of the online gaming system at a particular time).
Save data 111A is part of user data 111 and is typically a data image of a video game that includes all data regarding the state of the game and the state of users/participants within the game at a particular save time. As users/participants A-D play the video game, there may be many save data 111A images created for users/participants A-D. For example, in some embodiments, when transitioning from one level to another, or from one scene to another, among many other times/examples where the generation of save data 111A is triggered. , save data 111A of users/participants A to D is generated at a specific point during game play. Also, in some embodiments, the generation of save data 111A is triggered by a menu selection by users/participants AD.

FIG. 2 illustrates an example user interface 200 displayed on display device 126A to user/participant A during play of an online video game provided by cloud gaming system 101, according to some embodiments. User interface 200 includes a player identification section 201 that indicates which users are currently participating in an online video game as players or spectators.
Player identification section 201 indicates that users/participants A-D are currently participating in an online video game and uniquely marks (e.g., with shading) an identifier for user/participant A in the user interface. 200 is currently being provided to user/participant A based on user/participant A's game play or viewing status. User interface 200 includes a video display area 203 in which a video of a game corresponding to user/participant A's current playing/watching of an online game is displayed. The user interface 200 also includes a time display 205 that provides a display of time marks 205A corresponding to the game video currently displayed in the video display area 203, the time marks 205A spanning the expected duration of the online game event. This is indicated by a time bar 205B representing the scale of time.

User interface 200 also includes a communication display area 207 in which various communications provided by users/participants AD are displayed. In various embodiments, the communications display area 207 displays text data, voice data, audio data, pictographic data, image data, camera-captured video data, camera-captured image data, among other types of communication data. , tag data, and controller device input data, essentially any type of user communication (participant generated content).
In some embodiments, each item presented in communication display area 207 is presented with an identifier by the user/participant AD who contributed to the communication. In some embodiments, links to audio, video, and image content can be activated from within communications display area 207. For example, in some embodiments, selecting an audio link 209 plays the corresponding audio file. Additionally, in some embodiments, the audio communication is automatically transcribed into a text message by cloud gaming system 101, and the automatically transcribed text message is displayed as the communication, as illustrated by text message 209A corresponding to audio link 209. It is displayed in area 207.
Scroll control 211 is provided to move content displayed within communication display area 207. In some embodiments, the items presented within communication display area 207 are time indexed relative to the video of the game displayed in video display area 203. In some embodiments, user interface 200 also includes user communications input controls 213 that allow users/participants A-D to provide content for display within communications display area 207; The specification of one or more of text, audio, video, pictograms, and image data, among other types of content.

User interface 200 also includes tagging controls 215 that allow users/participants AD to place tags on the video displayed in video display area 203. In some embodiments, selection of tag control 215 initiates a process in which users/participants AD select a location within video display area 203 to place a tag. In various embodiments, the tag has some associated content that is one or more of text, audio, video, emoticons, and image data, among others.
The tag is placed at a user-selected pixel location within the video displayed in video display area 203. The tag is associated with a user-selected pixel location. The tag is also time indexed to the video displayed in the video display area 203 when the tag is placed. In some embodiments, the tag is time indexed relative to the set of video frames such that the tag is displayed while displaying a video frame within the set of video frames to which the tag is time indexed. is added.
In some embodiments, selecting tagging control 215 freezes the video currently displayed in video display area 203 and allows users/participants AD to specify the content of the tag. However, in some embodiments, when a user/participant AD specifies content for a tag, selecting the tagging control 215 does not freeze the video currently displayed in the video display area 203. In some embodiments, the time display 205 is also configured such that the user/participant A-D selects the time mark 205A and the time bar 205B to display the desired frame of the video within the video display area 203. It will be configured so that it can be moved along, and it will be tagged. In some embodiments, the insertion of tags by users/participants AD is shown within communication display area 207.
For example, FIG. 2 shows tag T1 entered by user/participant A into communications display area 207. Tag T1 is associated with a pixel location 217 within the video currently displayed in video display area 203, and tag T1 includes audio content. In some embodiments, selection of a tag in video display area 203 and/or communications display area 207 triggers display/playback of the content of the tag. For example, when tag T1 in the video display area 203 is selected, the audio content of tag T1 is played.

In some embodiments, the user interface 200 allows users/participants A-D to invite another person to participate in an ongoing group play event (multiplayer online game) and/or group viewing event. Also included is an invite control 219 that allows invitations to be made within interface 200. In some embodiments, selection of the invitation control 219 allows users/participants A-D to specify the identifier of the person to be invited to participate and/or send an email inviting that person to participate; It becomes possible to send text messages, or other types of messages.
In some embodiments, when a new user/participant joins a group play event (multiplayer online game) and/or group viewing event currently in progress within user interface 200, the new user/participant , group play events (multiplayer online games), and/or group viewing events to quickly learn about what happened or is happening. In this regard, the user interface 200 includes a summary request control 221 that, when selected, provides an audiovisual summary to the summary requester from a dynamic summary queue generated at the cloud gaming system 101. Trigger the process.

FIG. 3 illustrates an example user interface 300 displayed on display device 126D to users/participants AD upon selection of summary request control 221, according to some embodiments. In some embodiments, the user interface 300 includes a video display in which game video is displayed corresponding to current play of the online game by one or more of the other users/participants A-C who did not request the summary. Including area 301. The user interface 300 also includes a time display 305 that provides a display of time marks 305A corresponding to the game video currently being displayed in the video display area 301, where the time marks 305A are time marks over the expected duration of the online game event. is indicated by a time bar 305B representing the scale of.

User interface 300 includes a video display area in which an audiovisual summary of a group play event (multiplayer online game) and/or group viewing event is presented to a summary requester from a dynamic summary queue generated in cloud gaming system 101. 303 included.
The user interface 300 also includes a time display 307 that provides the display of a time mark 307A that corresponds to the summary content currently being displayed in the video display area 303, the time mark 307A indicating the current content of the dynamic summary queue of the cloud gaming system 101. The time bar 307B represents the scale of time for the content. In some embodiments, time mark 307A is draggable along time bar 307B to enable navigation by the user through the contents of the dynamic summary queue.
In some embodiments, time bar 307B is divided into segments S1-S(L), where L is any integer greater than zero. Each of the segments S1-S(L) corresponds to a continuous portion of the timeline of historical content of a group play event (multiplayer online game) and/or group viewing event. In some embodiments, user interface 300 includes video display area 303 and time display 307, but does not include video display area 301 and video display area 305.

User interface 300 also includes a communications display area 309 for displaying various past user/participant provided content related to dynamic summary cue (summary) content currently or recently displayed in video display area 303 . In various embodiments, the communications display area 309 may display text data, voice data, audio data, pictogram data, image data, camera-captured video data, camera-captured image data, among other types of communication data. , tag data, and controller device input data.
In some embodiments, each item of content presented in communication display area 309 is presented with an identifier by the user/participant AD who contributed the content. In some embodiments, links to audio, video, and image content can be activated from within communications display area 309. For example, in some embodiments, selecting audio link 310A plays the corresponding audio file. Additionally, in some embodiments, the audio communication is automatically transcribed into a text message by cloud gaming system 101, and the automatically transcribed text message is displayed as the communication, as illustrated by text message 310B corresponding to audio link 310A. It is displayed in area 309. Scroll controls 311 are provided to navigate user/participant provided content within communication display area 309. In some embodiments, the user/participant-provided content items presented within the communications display area 309 are time-indexed relative to the content of the dynamic summary cue (summary) displayed in the video display area 303. Can be attached.

Also, in some embodiments, the dynamic summary queue can be shared with another user/participant by selecting a share summary control 315 provided within the user interface 300. For example, in some embodiments, selection of the sharing summary control 315 triggers an interface that can select the identifier of the person with whom the dynamic summary queue is to be shared. In some embodiments, selection of the sharing summary control 315 triggers an interface in which an email and/or message is sent to the person with whom the dynamic summary queue is shared. In some embodiments, the sharing of dynamic summary queues is private and/or done in an anonymous manner. In some embodiments, user interface 300 also includes an exit summary control 317 that, when selected, closes user interface 300 and returns the user/participant to another user interface, such as user interface 200 of FIG.

By generating a dynamic summary queue in the cloud gaming system 101 and providing the dynamic summary queue to the requester, a requester who joins an online group event (such as the presentation of an online video game or media item) late can participate in the online group event. You can quickly find out about what happened during an event. It should also be appreciated that the content of the dynamic summary queue is selected/generated based on historical input content provided by other users/participants in the group during the group event.
In this way, the content of the dynamic summary cue reflects the past thoughts and actions of other users/participants in the group during the group event. Additionally, although the examples of FIGS. 1-3 are described primarily in the context of group play of online video games, the principles of dynamic summary cue generation and provision disclosed herein are applicable to essentially any type of dynamic summary cue. It should be understood that the same applies to multi-user/participant online consumption (viewing) of media items. Accordingly, in some embodiments, the media items for which dynamic summary cues are generated are streams of game video and audio of a video game. Also, in some embodiments, the media item for which the dynamic summary cue is generated is an audiovisual presentation, and the audiovisual presentation includes a movie, a television show, a real-world sporting event, an electronic sporting event, an online show, One or more of online videos, and/or uploaded videos, and essentially any other type of audiovisual presentation. In describing the generation and provision of dynamic summary cues below, the term "media item" is used to refer to the content for which a dynamic summary cue is generated.

FIG. 4 depicts a diagram of a system 400 that generates and provides a dynamic summary queue 425 of media items 415, according to some embodiments. When a media item 415 is consumed (viewed) by a user/participant A-D in an online group event, different users/participants A-D can access their respective input channels 401A, 401B, 401C, 401D (401A-D). 401D) as a function of time. Input channels 401A to 401D are collectively referred to as input channel 401. Input channel 401 includes various types of user/participant generated content that are provided when users/participants A-D consume media items 415. In various embodiments, the type of user/participant-generated content within input channel 401 is unique to a given user/participant A-D when a given user/participant A-D consumes media item 415. text data, voice data, audio data, emoticon data, image data, video data captured by a camera, image data captured by a camera, among essentially any other types of data that can be associated with One or more of tag data and controller device input data is included.
In various embodiments, users/participants A-D within a group may provide essentially any type of input and associate that input with media item 415 and/or a particular feature within media item 415; A media item 415 can be associated with a particular point in time in its timeline. For example, in some embodiments, input channel 401 may include a chat stream in which users/participants A-D provide input as a function of time as media item 415 is consumed; Participant-generated input is time-indexed to the timeline of media items 415. Input channel 401 is generated by users/participants A-D as they consume media item 415, so that each item of content in input channel 401 and the time of media item 415 There is a temporal correlation between the lines. In some embodiments, input channel 401 includes images and/or videos captured with users/participants AD's own cameras, which are analyzed within system 400 to provide other types of input information. Among other things, it can capture input information such as reactions, facial expressions, and gestures.

The system 400 is operative to analyze the content of the input channel 401 as a function of time and obtain/determine information about which portions of the media items 415 are included in the dynamic summary queue 425 as a function of time. do. The input channels 401 provided by users/participants A-D in the group are analyzed within the context of the media item 415 to determine which parts of the media item 415 are more relevant than other parts. , these more relevant portions of the media item 415 are identified for inclusion in the dynamic summary queue 425. The system 400 also analyzes the content of the media item 415 itself as a function of time to obtain/determine information about which portions of the media item 415 are included in the dynamic summary queue 425 as a function of time. It works like this. In some embodiments, state data and/or metadata (state/metadata 417) for media item 415 also determines which portions of media item 415 should be included in dynamic summary queue 425 as a function of time. analyzed as a function of time to obtain/determine information about

The dynamic summary queue 425 determines which users/participants A-D in the group are considered to be the most relevant based on an analysis of the content of the input channels 401 provided by the users/participants A-D in the group. includes a portion of the media item 415 that was Accordingly, the dynamic summary queue 425 of the media item 415 is provided by other users/participants A-D in the group that participated in the consumption of the media item 415 before the new user/participant A-D arrived in the group. By indicating portions of media items 415 that are considered relevant based on an analysis of the input channels 401 that have been input, it allows new users/participants A-D in the group to quickly get up to speed. Dynamic summary queue 425 is always dynamically generated. The content of the dynamic summary queue 425 is based on the processing of the content of the input channel 401 provided by users/participants A-D in the group as a function of time, and the content of the input channel 401 changes as a function of time. The composition of users/participants A-D within a group is subject to change as a function of time.
Therefore, if the content in the input channel 401 provided by users/participants AD in the group changes, the content of the dynamic summary queue 425 may change (is dynamic). For example, if the content of input channel 401 includes changes in the actions, statements, expressions, reactions, emotions, personalities, etc. of users/participants A-D in the group that occur later in the timeline of media item 415, then The changes may be used to update the content of the dynamic summary queue 425 that was previously determined based on the content of the input channel 401 that was previously received in the timeline of the media item 415. Accordingly, in some embodiments, the dynamic summary queue 425 provides the possibility that content already placed in the dynamic summary queue 425 may be modified based on information later received in the input channel 401. It is dynamic in the sense that there is.

In some embodiments, input channel 401 is processed through feature selector 403 to select features from input channel 401. In some embodiments, feature selector 403 includes a computer that automatically analyzes input channel 401 in real time to select features from input channel 401 once data defining input channel 401 is received by system 400. It is an algorithm/program that is executed on a processing system. The features selected from the input channels 401 are various types of input that convey information about how the users/participants A-D respond to the media items 415 that they consume. .
In various embodiments, the features include user/participant-generated content within input channel 401, e.g., when a given user/participant A-D consumes media item 415. Text data, voice data, audio data, emoticon data, image data, camera-captured video data, camera-captured image data, among essentially any other types of data that can be associated with D. , tag data, and controller device input data. In some embodiments, the feature selector 403 is redundant in determining how the user/participant A-D responded to the media item 415 that the user/participant A-D consumed; and and/or operate to delete input items in input channel 401 that are irrelevant.

The features selected from the input channel 401 by the feature selector 403 are provided as input features to a feature extractor 405 which operates to extract/derive new features from the input features, and these new features are used as the extracted features. It is called 407. In some embodiments, feature extractor 405 is an algorithm/program running on a computing system that automatically extracts/derives features in real time from the input features received from feature selector 403. In some embodiments, the extracted features may be information indicative of how the users/participants A-D reacted to the media items 415 consumed by the users/participants A-D. is defined as Also, in some embodiments, the extracted features are defined so as not to be redundant. In some embodiments, extracted features 407 collectively represent a real-time dynamic integration of information received on input channel 401 from users/participants A-D while consuming media item 415. represent.

In some embodiments, extracted features 407 are provided as input to a feature classifier 409 that operates to determine which of a plurality of categories the extracted features 407 belong to. In some embodiments, information about the categorical attributes of extracted features 407 is appended to extracted features 407 to generate a set of classified features 411 that is provided as input to machine learning model 413. . Machine learning model 413 is an algorithm executed on a computing system that automatically determines which portions of media items 415 and/or which portions of information within input channel 401 are included in dynamic summary queue 425. / program. In some embodiments, machine learning model 413 uses content-based filtering algorithms and/or collaborative filtering algorithms to determine which portions of media items 415 and/or which portions of information within input channel 401 A requestor system implementing other approaches for determining which is deemed more appropriate by users/participants A-D when consuming a . In some embodiments, media item 415 and state/metadata 417 of media item 415 are provided as input to machine learning model 413.

In some embodiments, the system 400 also includes information about the media item 415, the state/metadata 417 of the media item 415, the features selected from the input channel 401 by the feature selector 403, the extracted information output by the feature extractor 405, implemented as an algorithm/program executed on a computing system that automatically determines/identifies the context associated with one or more of the classified features 407 and the set of classified features 411 output by the feature classifier 409; The content analysis engine 418 includes a context analysis engine 418. In various embodiments, the context determined/identified for a given piece or set of information by context analysis engine 418 is provided as an input to machine learning model 413. For example, in some embodiments, the context determined/identified by context analysis engine 418 for a given piece of information or set of information may include a specific scene within media item 415, such as a particular scene within media item 415. is a description or characterization of a particular part of

It should be appreciated that feature selector 403, feature extractor 405, feature classifier 409, context analysis engine 418, and machine learning model 413 represent example embodiments of system 400. In other embodiments, the system 400 determines which of the media items 415 based on user/participant-generated input provided to the input channel 401 during consumption of the media items 415 by the users/participants A-D. Essentially any type of machine learning approach can be implemented that has the ability to identify which parts are considered to be highly relevant to users/participants AD of the group. Regardless of the particular type of machine learning approach implemented by system 400, in some embodiments machine learning model 413 is queried to generate recommendations for content for inclusion in dynamic summary queue 425. be able to. In some embodiments, machine learning model 413 generates dynamic summaries based on user/participant-generated information provided on input channel 401 during consumption of media items 415 by users/participants A-D. A summary request 419 is received that triggers a query of machine learning model 413 to generate recommendations for content to include in queue 425 . In some embodiments, summary request 419 is triggered in response to selection of summary request control 221 of FIG.

System 400 operates to collect/compile dynamic summary queue data 421 corresponding to recommendations of content for inclusion in dynamic summary queue 425 generated by machine learning model 413 . In some embodiments, dynamic summary cue data 421 includes video and/or audio data for various portions of media item 415. Additionally, in some embodiments, dynamic summary queue data 421 further includes a portion of the information received on input channel 401 that corresponds to the portion of media item 415 that is included within dynamic summary queue 425. . In some embodiments, system 400 is also configured to create computer-generated natural language text and/or audio/verbal summaries of one or more portions of media items 415 that are included within dynamic summary queue 425. a natural language generator 420 that operates on the natural language generator 420; In some embodiments, natural language generator 420 receives input data from machine learning model 413 that specifies information to be summarized and generates a natural language textual and/or audio summary of the received input data. It is an algorithm/program that runs on the system.

Dynamic summary queue data 421 is provided to a dynamic summary queue assembler 423 that operates to assemble dynamic summary queue data 421 into a dynamic summary queue 425 . The system 400 is also configured to send the contents of the dynamic summary queue 425 as a requested summary stream 427 to the requester's computer system that received the summary request 419. In some embodiments, requested summary stream 427 is a video stream that includes video and audio data. In some embodiments, system 400 is configured to encode requested summary stream 427 before transmitting it to the requester's computer system over the Internet. In various embodiments, the system 400 may implement essentially any technique for transmitting data over the Internet to transmit the requested summary stream 427 to the requester's computer system. . In some embodiments, requested summary stream 427 is shown in video display area 303 of user interface 300 in FIG. 4.

The content of dynamic summary queue 425 at a given point in time is a function of the input provided to machine learning model 413. In some embodiments, a requester of dynamic summary queue 425 is provided with the ability to customize the generation of dynamic summary queue 425. For example, referring to FIG. 3, in some embodiments, user interface 300 allows a requester of dynamic summary queue 425 to customize various options for the generation of dynamic summary queue 425. It includes an overview options control 313 that triggers the display of a user interface that allows the user to perform the following actions.

FIG. 5A illustrates an example user interface 500 that provides various controls to enable a requester of a dynamic summary queue 425 to customize how the dynamic summary queue 425 is generated, according to some embodiments. . In some embodiments, when a user/participant AD selects summary options control 313, user interface 500 is displayed. In some embodiments, summary options user interface 500 provides one or more of participant-based controls 501, time frame controls 503, summary decomposition controls 505, event type controls 507, and display type controls 509. , these are collectively shown as summary options. In some embodiments, user interface 500 provides a display of current settings for each of the summary options.
In some embodiments, the user interface 500 also provides a respective "Change" control for each of the summary options that, when selected, changes the current setting of the corresponding summary option. Trigger the display of another user interface for. In some embodiments, the user interface 500 also provides a respective "Reset" control for each summary option, which, when selected, resets the corresponding summary option to its default settings. .

FIG. 5B shows an example of a user interface 511 that is displayed when a "Change" control is selected within the participant-based controls 501 of the user interface 500 of FIG. 5A, according to some embodiments. User interface 511 allows a requester of dynamic recap queue 425 (summary: recap) to select which users/participants A-D in the group to use as input sources for generating dynamic recap queue 425. It can be so. In some embodiments, the user interface 511 includes a change control 513 for each user/participant A-D that, when selected, changes the current participant-based settings for the user/participant A-D. Toggle between "include" and "exclude".
If a given user/participant A-D has a current participant-based setting of "Include," then the input channel 401 associated with a given user/participant A-D will not include the dynamic summary queue 425. It will be used by system 400 in generating. Conversely, if a given user/participant A-D has a current participant-based setting of "exclude", then the input channel 401 associated with the given user/participant A-D is It will not be used by system 400 in generating summary queue 425. Therefore, by using participant-based control 501, a requester of dynamic summary queue 425 can determine which users/participants A-D input channels 401 are used as the basis for generating dynamic summary queue 425. or can be customized.

For example, a requester of dynamic summary queue 425 knows that users/participants A and C have similar interests and/or preferences, but the requester does not know users/participants B and D. If not, as shown in the example of FIG. User-based control 501 can be used. In this way, a requester of dynamic summary queue 425 can be confident that the content of dynamic summary queue 425 is more consistent with his or her interests and/or preferences. In another example, if the requester of the dynamic summary queue 425 does not know the users/participants A-D in the group, the requester can use participant-based controls 501 to D can be "included" so that when the requester views the dynamic summary queue 425, the requester not only has the most up-to-date information about the media item 415, but also the users/participants A-D in the group. In addition to being able to determine whether the interests and/or preferences of the dynamic summary queue 425 match those of the user/participant A-D, the requester of the dynamic summary queue 425 may also be able to determine whether the interests and/or preferences of You can make an informed decision about whether you want to. In some embodiments, the default setting of participant-based control 501 is to connect all users/participants A to 401 in a group such that all input channels 401A to 401D are used to generate dynamic summary queue 425. It “includes” D.

FIG. 5C shows an example of a user interface 515 that is displayed when a "change" control is selected within the time frame control 503 of the user interface 500 of FIG. 5A, according to some embodiments. Timeframe control 503 allows a requester of dynamic summary queue 425 to select a timeframe along the timeline of media item 415 in which dynamic summary queue 425 is generated. In some embodiments, the time frame control 503 allows the user to monitor the progress of the media item 415 for groups A-D of users/participants from the start of the media item 415 timeline (time 0:00). A time window 516 for generation of the dynamic summary queue 425 may be selected that spans the entire time span of the current time of the live playback. In some embodiments, the time window control 503 provides a time bar 517 that represents the overall time range within which a time window 516 for the generation of the dynamic summary queue 425 can be set. In some embodiments, the time frame control 503 also includes a first slider control 519 that is slidable along the time bar 517 by the requester and is used to generate the dynamic summary queue 425. The start time is set along the timeline of the media item 415 for the time frame 516 to be used. In some embodiments, the time frame control 503 also includes a second slider control 521 that is slidable along the time bar 517 by the requester and used to generate the dynamic summary queue 425. The end time is set along the timeline of the media item 415 for the time frame 516 to be used. In this way, the dynamic summarization queue 425 is configured to follow a timeline of media items 415 that extends from a start time corresponding to the setting of the first slider control 519 to an end time corresponding to the setting of the second slider control 521. It will be generated over a time frame 516.

In some embodiments, the time frame control 503 may also be used to determine which users/participants A-D are currently included in the generation of the dynamic summary queue 425, as currently set by the participant-based control 501. Based on the associated input channel 401, an indication of the temporal position where an indication of interest (I) and/or an event of interest (E) may exist along the timeline of the media item 415 is provided. The temporal location at which an indication of interest (I) and/or an event of interest (E) may exist along the timeline of a media item 415 is determined in order to generate a dynamic summary cue 425. may be used by the requester as an aid in identifying which timeframes 516 are more desirable for inclusion.
In some embodiments, if the second slider control 521 is set to the current time of live playback of the media item 415, the system 400 dynamically adjusts the dynamic summary queue 425 as it is streamed to the requester. It operates to continue adding content to the summary queue 425 so that when the requester reaches the end of the dynamic summary queue 425, the requester becomes the current time of live playback of the media item 415. Also, in some embodiments, when a requester reaches the end of the dynamic summary queue 425, the requester is immediately able to watch a live playback of the media item 415 along with other users/participants A-D in the group. enter. For example, when a requester reaches the end of dynamic summary queue 425, the user interface currently displayed to the requester automatically changes from user interface 300 of FIG. 3 to user interface 200 of FIG. 2.

FIG. 5D shows an example of a user interface 523 that is displayed when the "Change" control is selected within the summary decomposition control 505 of the user interface 500 of FIG. 5A, according to some embodiments. Summary disaggregation controls 505 allow a requester to configure the content disaggregation of dynamic summary queue 425 to control how much content is included within dynamic summary queue 425 . In some embodiments, user interface 523 includes a disassembly bar 525 and a slider control 527 that is slidable along disassembly bar 525 by the requester.
Placing the slider control 527 toward the "Less Detail" end of the decomposition bar 525 will cause the dynamic summary queue 425 to be generated with less content (shorter). Conversely, placing slider control 527 toward the "More Detail" end of decomposition bar 525 causes dynamic summary queue 425 to be generated with more content (longer). cause. In some embodiments, the user interface 523 also includes a field 529 in which the requester can enter an integer value indicating the number of segments (or maximum number of segments) to be included in the dynamic summary queue 425; corresponds to a continuous portion of the playback timeline of media item 415. In some embodiments, user interface 523 also includes a field 531 in which a requester can enter the amount of time (or maximum amount of time) to be used for each segment in dynamic summary queue 425.

FIG. 5E shows an example of a user interface 533 that is displayed when a "change" control is selected among the event type controls 507 of the user interface 500 of FIG. 5A, according to some embodiments. Event type control 507 allows a requester of dynamic summary queue 425 to specify particular types of events within media item 415 that are of interest and should be considered for inclusion in dynamic summary queue 425. In some embodiments, the user interface 533 includes a custom event settings chart 534 that specifies various events that can occur within the media item 415 and are currently available for selection by the requester for inclusion within the dynamic summary queue 425. including.
In some embodiments, custom event settings chart 534 includes an event descriptor column 543 that includes brief descriptions of various events that may occur within media item 415. The types of events that may occur within media item 415, and their corresponding brief descriptions, depend on the subject matter of media item 415. For example, if media item 415 is a video of a car race (occurring either in the real world or in a computer game), some example events that may occur within media item 415 include a green flag. , yellow flags, checkered flags, crashes, spinouts, passes, blocks, pits, etc., such as those shown in the example custom event settings chart 534 of FIG. 5E.

In some embodiments, the custom event settings chart 534 determines whether the events listed in the corresponding rows of the event descriptor column 543 are active for inclusion in the basis for generating the dynamic summary queue 425. It includes an event status column 545 indicating whether it is inactive. In some embodiments, a setting of "Yes" in the event status column 545 for a given event indicates that the given event is a media item during the time frame 516 during which the dynamic summary queue 425 is generated. 415 is included as part of the basis for generating the dynamic summary queue 425.
Conversely, a setting of "No" in the event status column 545 for a given event indicates that the given event occurred within the media item 415 during the time frame 516 during which the dynamic summary queue 425 was generated. If it occurs, indicates that it is not included as part of the basis for generating the dynamic summary queue 425. In some embodiments, the custom event settings chart 534 has a "change" control that, when selected, toggles the settings in the corresponding row of the event status column 545 between "yes" and "no" and vice versa. An adjustment column 547 that includes.
In some embodiments, the custom event settings chart 534 also includes a global "No" control 549 that, when selected, sets the event status column setting for all listed events to "No". . In some embodiments, the custom event settings chart 534 also includes a global "Yes" control 551 that, when selected, sets the event status column setting for all listed events to "Yes". . In some embodiments, the default setting for the event status column for all listed events is "Yes."

In some embodiments, custom event settings chart 534 includes an event strength column 541 that allows event weighting to be specified for the event, as described in the corresponding row of event descriptor column 543. . In some embodiments, the event weighting for a given event is specified on a scale of 1 to 10, with 1 being the lowest weighting and 10 being the highest weighting. The event weights listed in event strength column 541 are used by system 400 to determine which events should be prioritized higher than other events for inclusion in dynamic summary queue 425. Particularly when there is more content available for inclusion with dynamic summary queue 425 than the current settings of summary decomposition control 505 can accommodate. In various embodiments, the scale used for event weighting is defined in essentially any manner that provides standardization of the event weighting input so that it can be determined which events are more important than other events. be able to.

In some embodiments, user interface 533 includes an add custom event control 553 that, when selected, allows the requester to add a new event (new row) to custom event settings chart 534. Additionally, in some embodiments, user interface 533, when selected, configures all settings in custom event settings chart 534 for each of the settings that existed when user interface 533 was displayed in response to event type control 507 selection. includes a reset event control 555 that resets the value to the value of .
In some embodiments, user interface 533 also includes automatic enablement controls 535 and custom enablement controls 537. Selection of automatic enable control 535 relies on machine learning models within system 400 in determining which events occurring within media item 415 should be included in the basis for generating dynamic summary queue 425. Involved in the use of 413. Selection of custom enable control 537 allows system 400 to use custom event settings chart 534 in determining which of the events that occur within media item 415 should be included in the basis for generating dynamic summary queue 425. Involve the use of. In some embodiments, either or both automatic enablement control 535 and custom enablement control 537 may be selected at a given time. In some embodiments, selection of automatic enable control 535 is implemented by default when custom enable control 537 is deselected.

FIG. 5F shows an example of a user interface 561 that is displayed when a "change" control is selected among the display type controls 509 of the user interface 500 of FIG. 5A, according to some embodiments. Display type control 509 allows a requester of dynamic summary queue 425 to specify a particular type of user/participant display of interest while consuming media items 415 by users/participants AD. do it like this. It will be considered in determining which portions of media items 415 should be included in the basis for generating dynamic summary queue 425.
In some embodiments, the user interface 561 includes a custom display settings chart 563 that specifies the display of various user/participant interests that may occur during consumption of the media item 415 by the user/participant. include. In some embodiments, custom display settings chart 563 includes a display descriptor column 565 containing brief descriptions of various user/participant displays of interest. Various user/participant indications may include various types of reactions, facial expressions, gestures, actions, speech, sounds, and/or that users/participants A-D may make as they consume media item 415. Represents transmission. For example, in some embodiments, various user/participant displays specified in custom display settings chart 563 may be performed by any of users/participants AD during consumption of media item 415. , even if it includes likes, dislikes, excitement, happiness, anger, screaming, nervousness, laughter, among other essentially any other type of reaction, facial expression, gesture, action, speech, sound, and/or communication. good.

In some embodiments, the custom display settings chart 563 determines which content is selected for inclusion within the dynamic summary queue 425 by the user/participant display listed in the corresponding row of the display descriptor column 565. It includes a display status column 567 that indicates whether it is active or inactive for inclusion based on whether it is active or inactive for inclusion. In some embodiments, a setting of “Yes” in the display status column 567 for a given user/participant representation indicates that the given user/participant representation is to be included in the dynamic summary queue 425. This means that the criteria include which content is to be identified/selected. Conversely, a “No” setting in the display status column 567 for a given user/participant display indicates which content the given user/participant display identifies for inclusion in the dynamic summary queue 425. / This means that whether the item is selected is not included in the criteria.
In some embodiments, the custom display settings chart 563 has a "change" control that, when selected, toggles the setting in the corresponding row of the display status column 567 between "yes" and "no" and vice versa. and an adjustment column 569 containing. In some embodiments, the custom display settings chart 563 also includes a global "no" control 571 that, when selected, controls the display status column 567 for all listed user/participant displays. The setting will be set to "No". Additionally, in some embodiments, the custom display settings chart 563 includes a blanket "yes" control 573 that, when selected, controls the display status column 567 for all listed user/participant displays. The setting will be set to "Yes". In some embodiments, the default setting of the display status column 567 for displaying all listed users/participants is "Yes".

In some embodiments, the custom display settings chart 563 provides a user/participant display that can specify display weights for the user/participant display, as described in the corresponding row of the display descriptor column 565. Includes participant display strength column 575. In some embodiments, the display weighting of a given user/participant display is specified on a scale of 1 to 10, with 1 being the lowest weighting and 10 being the highest weighting. The display weights listed in the display strength column 575 determine which portions of the media item 415 to include in the criteria used to identify/select which portions should be included within the dynamic summary queue 425. Used by system 400 to determine whether user/participant displays should be prioritized higher than others. In various embodiments, the scale used to weight the user/participant representations may be configured such that the scale used to weight the user/participant representations is configured such that the user/participant representation weighting input allows determining which user/participant representations are more important than others. can be defined in essentially any way that provides standardization.

In some embodiments, user interface 561 includes an add custom display control 577 that, when selected, allows the requester to add a new user/participant display (new row) to custom display settings chart 563. Additionally, in some embodiments, user interface 561, when selected, configures all settings in custom display settings chart 563 that were present when user interface 561 was displayed in response to the selection of display type control 509. It includes a reset display control 579 that resets to each value.
In some embodiments, user interface 561 also includes automatic enablement controls 581 and custom enablement controls 583. Selection of automatic enable control 581 determines which user/participant representations that occur during consumption of media item 415 by users/participants A-D and which portions of media item 415 are included in dynamic summary queue 425. involves the use of the machine learning model 413 of the system 400 in determining what should be included in the criteria for determining whether the Selection of custom enable control 583 determines which user/participant views that occur during consumption of media item 415 by users/participants A-D and which portions of media item 415 are included in dynamic summary queue 425. involves the use of a custom display settings chart 563 by the system 400 in determining what should be included in the criteria for determining whether the In some embodiments, either or both automatic enablement control 581 and custom enablement control 583 may be selected at a given time. In some embodiments, selection of automatic enable control 581 is implemented by default when custom enable control 583 is deselected.

FIG. 6 depicts a flowchart of a method for generating a summary (dynamic summary queue 425) of a media item (media item 415), according to some embodiments. The method includes transmitting a stream of media items 415 from a server computing system (e.g., cloud gaming system 101 of FIG. 1) to one or more client computing systems (e.g., local computing systems 120A-120D of FIG. 1, etc.). ). Media item 415 has a timeline for playing media item 415. In some embodiments, media item 415 is an audiovisual presentation.
In various embodiments, audiovisual presentations include movies, television shows, real-world sporting events, electronic sporting events, online shows, online videos, and uploads, among essentially any other types of audiovisual presentations. one or more of the videos. In some embodiments, media item 415 is a stream of game video and audio for a video game. In some embodiments, the server computing system controls each controller device 125A of one or more users/participants (e.g., game players/spectators A-D) of one or more client computing systems 120A-120D. A cloud gaming system 101 configured to execute a video game according to input received from ~125D.
Cloud gaming system 101 is configured to direct the transmission of streams of game video and audio that reflect the play of a video game by one or more users/participants. In these embodiments, one or more users/participants provide user/participant-generated content that is received via input channel 401 during play of a video game. In some embodiments, the video game is an electronic sports video game. Also, in some embodiments, one or more of the users/participants are spectators of the video game.

The method also includes an act of receiving 603 user/participant generated content at the server computing system from one or more users/participants of any of the one or more client computing systems. User/participant generated content corresponds to any information received via any of the input channels 401, as described with respect to FIG. User/participant generated content is time indexed relative to the timeline of media items 415. It should also be understood that user/participant generated content is provided by any user/participant consuming (viewing) media item 415. In various embodiments, users/participants may be video game players, video game spectators, and/or viewers depending on the nature of the media item 415. In some embodiments, user/participant generated content may include text data, audio data, audio, among essentially any other types of data that may be provided by the user/participant via input channel 401. data, glyph data, image data, camera-captured video data, camera-captured image data, tag data, and controller device input data.

The method also includes an act 605 of processing the user/participant-generated content to identify key portions of the media item 415 based on representations present within the user/participant-generated content; portions are identified for inclusion in dynamic summary queue 425. In some embodiments, the identification of user/participant representations present in user/participant-generated content may occur during consumption of media item 415 by the user/participant, as described with respect to FIG. 5F. /Based on the participant's indication of interest. In some embodiments, operation 605 also processes the media item 415 and/or state/metadata 417 associated with the media item 415 to determine the primary content of the media item 415 for inclusion within the dynamic summary queue 425. This includes identifying the parts that are In some embodiments, identification of key portions of media item 415 is based on specifying events that may occur within media item 415, as described with respect to FIG. 5E.

In some embodiments, the act of processing user/participant-generated content 605 includes extracting features from the user/participant-generated content and processing the extracted features as described with respect to feature extractor 405 of FIG. 407 sets. In some embodiments, the act of processing 605 the user/participant-generated content also includes classifying the extracted features 407 into one or more classifications, as described with respect to the feature classifier 409 of FIG. , including obtaining a set of classified features 411. In some embodiments, the act of processing user/participant-generated content 605 also processes the set of features 411 classified through the machine learning model 413 to summarize the media contained within the dynamic summary queue 425. This includes identifying the main parts of item 415.
In some embodiments, the operation 605 also includes processing the state/metadata 417 of the media item 415 via the machine learning model 413 in conjunction with processing the classified feature set 411. 425 to identify key portions of the media items 415 included in the dynamic summary queue 425. The state/metadata 417 of the media item 415 is related to the set of classified features 411 by time index. In some embodiments, the state/metadata 417 of the media item 415 is a video that is performed according to input received from the respective controller devices 125A-125D of one or more users/participants (e.g., A-D). Media items 415 are generated as streams of game video and audio, including game state data.

The method also includes an act 607 for generating a dynamic summary queue (dynamic summary queue 425) that includes the major portions of the media items 415 identified in act 605. The method also includes assembling the dynamic summary queue 425 for streaming to a requester of the dynamic summary queue 425. In some embodiments, operation 607 includes creating an automatically generated verbal summary and/or an automatically generated written summary of the major portions of media item 415 for inclusion in dynamic summary queue 425. In some embodiments, the dynamic summary queue 425 summarizes at least a portion of the user/participant-generated content associated with the main portion of the media item 415 received from the user/participant via the input channel 401. include.

The method also includes an act 609 of receiving a request to view dynamic summary queue 425 from a requester. In some embodiments, the requester is any one or more users/participants consuming media item 415. In some embodiments, a requester is a person who is invited to join one or more users/participants in consuming a media item. In some embodiments, the requestor receives data from the respective controller devices 125A-125D of one or more users/participants (e.g., A-D) to generate media items 415 as streams of game video and audio. A person who is invited to join one or more users/participants in playing a video game that is performed according to the input provided.

In various embodiments, act 609 can occur at any point during the method of FIG. For example, in some embodiments, a request to view dynamic summary queue 425 may be received in operation 609 prior to processing user/participant generated content in operation 605. Alternatively, in another exemplary embodiment, a request to view dynamic summary queue 425 may be received at operation 609 after dynamic summary queue 425 has already been generated at operation 607. Additionally, in some embodiments, the request to view dynamic summary queue 425 received in operation 609 may be performed when generating dynamic summary queue 425, such as through the various controls described with respect to FIGS. 5A-5F. It can include specifications of various controls to be applied.
The method also includes an act 611 for transmitting the stream of the dynamic summary queue 425 to the requester's computing device in response to the request received in act 609. In some embodiments, the method includes: when user/participant generated content is received at the server computing system (via input channel 401) and processed to identify key portions of the media item 415; This includes updating the dynamic summary queue 425. In some embodiments, updating the dynamic summary queue 425 includes adding new content to the dynamic summary queue 425 and/or changing some content that already exists within the dynamic summary queue 425. including doing.

In some embodiments, the method includes receiving control parameters for generating the dynamic summary queue 425, such as the control parameters described with respect to FIG. 5A. In some embodiments, the control parameters include specifying one or more types of display present within the user/participant-generated content to indicate the major portions of the media item 415, as described with respect to FIG. 5F. . In some embodiments, types of displays include reactions, facial expressions, gestures, actions, utterances, sounds, communications, and/or essences that may be emitted by the user/participant that are related to the subject matter within the media item 415. may also include one or more of any other type of representation.

In some embodiments, the control parameters for generating dynamic summary queue 425 include a specification of one or more types of events that may occur within media item 415, as described with respect to FIG. 5E. including. In some embodiments, the control parameters for generating dynamic summary queue 425 include processing user/participant generated content (as received via input channel 401) as described with respect to FIG. 5B. When identifying the essential portions of the media item 415, the user/participant generated content includes a designation to which one or more users/participants should be restricted. In some embodiments, the control parameters for generating dynamic summary cue 425 are within the timeline for playback of media item 415 for which dynamic summary cue 425 is generated, as described with respect to FIG. 5C. Includes a time frame 516 designation. In some embodiments, the control parameters for generating dynamic summary queue 425 include specifying a summary decomposition of dynamic summary queue 425, as described with respect to FIG. 5D.

Further, in some embodiments, the method for generating dynamic summary queue 425 described with respect to FIG. 6 is implemented as a system (eg, system 400) for generating summaries of videos. In these embodiments, the system includes program instructions stored in computer memory for transmitting a stream of media items 415 from a server computing system to one or more client computing systems, wherein the media items 415 are Contains a timeline for playback of media items 415. The system also includes a program stored in computer memory for receiving user/participant generated content at the server computing system from one or more users/participants of any of the one or more client computing systems. The user/participant generated content, including instructions, is time indexed relative to the timeline of media items 415.
The system also includes program instructions stored in computer memory for processing the user/participant generated content to identify key portions of the media item 415 based on representations present within the user/participant generated content. Also included. The system also includes program instructions stored in computer memory to generate a dynamic summary queue 425 that includes a major portion of the media items 415, and the dynamic summary queue 425 is assembled for streaming. The system also includes program instructions stored in computer memory for receiving requests from requesters to view dynamic summary queue 425. The system also includes program instructions stored in computer memory for transmitting the stream of dynamic summarization queues 425 to the requestor's computing device upon request.

FIG. 7 depicts a diagram of components within an example computing device 700 that can be used to implement aspects of various embodiments of this disclosure, according to some embodiments. In various embodiments, computing device 700 may incorporate or incorporate a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. It can be. Computing device 700 includes a central processing unit (CPU) 702 for running software applications and optionally an operating system. CPU 702 may include one or more homogeneous or heterogeneous processing cores.

In some embodiments, the video game is executed locally on a gaming console, personal computer, or server. In some cases, video games are executed by one or more servers in a data center. When a video game is run, some instances of the video game may be a simulation of the video game. For example, a video game may be executed by an environment or server that generates a simulation of the video game. A simulation, in some embodiments, is an instance of a video game. In other embodiments, the simulation may be generated by an emulator. In any case, if a video game is expressed as a simulation, the simulation may be executed to render interactive content (e.g., video frames) that can be interactively streamed, executed, and/or controlled by user input. I can do it.

CPU 702 is one or more general purpose microprocessors that include one or more processing cores. In some embodiments, the computing device 700 performs highly parallel and computationally intensive operations such as interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering the contextually relevant resources within a video game. The CPU 702 is implemented using one or more CPUs 702 having a microprocessor architecture specifically adapted for commercial applications. In various embodiments, the computing device 700 is localized to a player playing a game segment (e.g., a gaming console), or remote from the player (e.g., a backend server processor), or remote to a client. It is one of many servers that use virtualization in the Game Cloud system for remote streaming of gameplay.

Computing device 700 includes computer memory 704 that stores applications and data used by CPU 702 . Computing system 700 also includes a data storage device 706 that provides non-volatile storage of data and computer-readable media for applications. In various embodiments, computing device 700 includes a fixed disk drive, a removable disk drive, a flash memory device, and/or a CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other May include optical storage devices and signal transmission and storage media. Computing device 700 also includes one or more user input devices 708 that communicate user input from one or more users to computing device 700. For example, user input device(s) may include keyboard(s), mouse(s), joystick(s), touch pad(s), touch screen(s), still image or video recorder(s), etc. )/camera(s), tracking device(s) that recognize gestures, and/or microphone(s).
Computing device 700 is connected to a network that enables computing device 700 to communicate with other computer systems via electronic communications networks, which may include wired or wireless communications across local area networks and wide area networks, such as the Internet. Also includes an interface 714. Computing device 700 also includes an audio processor 712 that generates analog or digital audio output from instructions and/or data provided by CPU 702 , computer memory 704 , and/or data storage device 706 . Computing device 700 includes a CPU 702, computer memory 704, data storage device(s) 706, user input device(s) 708, network interface 714, and one or more data buses for connecting audio processor 712. (Multiple possible) 722 is also included.

Graphics subsystem 720 is further coupled to a data bus 722 and various components of computing device 700. Graphics subsystem 720 includes a graphics processing unit (GPU) 716 and graphics memory 718. Graphics memory 718 includes display memory (eg, a frame buffer) used to store pixel data for each pixel of the output image. Graphics memory 718 may be integrated into the same device as GPU 716, may be connected as a separate device from GPU 716, and/or may be implemented within computer memory 704.
Pixel data may be provided directly from CPU 702 to graphics memory 718. Instead, CPU 702 provides data and/or instructions defining a desired output image to GPU 716 from which GPU 716 generates pixel data for one or more output images. Data and/or instructions defining the desired output image may be stored in computer memory 704 and/or graphics memory 718. In some embodiments, the GPU 716 generates the pixel data of the output image from instructions and data that define the geometry, lighting, shading, texturing, motion, and/or camera parameters for the scene. Contains rendering functions. GPU 716 may further include one or more programmable execution units capable of executing shader programs.

Graphics subsystem 720 periodically outputs image pixel data from graphics memory 718 for display on display device 710. Display device 710 is capable of displaying visual information in response to signals from computing device 700, including CRT, LCD, plasma, and OLED displays, among essentially any other types of computer display devices. It may be any device capable of In various embodiments, computing device 700 provides analog and/or digital signals to display device 710.

In various embodiments, the methods and systems disclosed herein are implemented in cloud computing systems that provide access services, such as providing access to video games and/or other media content over a wide geographic area. Implemented. Cloud computing is a computing modality in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the "cloud" technical infrastructure that supports them. Cloud computing can be classified into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide applications such as video games online that are accessed through a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet, based on the way the Internet is depicted in computer network diagrams, and is an abstract concept that hides complex infrastructure.

In some embodiments, a game server is used to perform operations of a persistent information platform for video game players. Most video games played over the Internet operate through a connection to a game server. In some embodiments, online video games use dedicated server applications that collect data from players and distribute it to other players. In other embodiments, online video games are executed by a distributed game engine. In these embodiments, the distributed game engine may be executed in multiple processing entities (PEs) such that each PE performs a functional segment of the given game engine on which the online video game is executed. do. In some embodiments, each PE is recognized by the game engine as a compute node.

Game engines typically perform a functionally diverse set of operations to provide additional services for the user to experience and to run the video game application. For example, a game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, gameplay/playback functionality, help functionality, and the like.
In some embodiments, the game engine executes in an operating system that is virtualized by a hypervisor of a particular server. In some embodiments, segments of the game engine are distributed across multiple PEs, and each PE may reside on a different server unit of the data center. In some embodiments, the various PEs used to run game engine segments may be implemented as server units, virtual machines, or containers, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformations, it will be doing a lot of relatively simple mathematical operations (e.g., matrix transformations), so that particular game engine segment, along with the virtual machine associated with the GPU, May be provisioned. Other game engine segments requiring fewer but more complex operations may be provisioned with processing entities associated with one or more higher-powered CPUs.

By distributing the game engine, the game engine has resilient computational properties that are not tied to the capabilities of physical server units. Instead, the game engine is provisioned with more or fewer compute nodes when needed to meet the demands of the video game. From a video game and video game player perspective, a game engine that is distributed across multiple compute nodes allows a game engine manager or supervisor to distribute the workload and seamlessly integrate the results to provide video game output to the end user. indistinguishable from non-distributed game engines running on a single processing entity.

A user accesses remote services using a client device that includes at least a CPU, a display, and an input/output (I/O) device. The client device can be a personal computer (PC), mobile phone, tablet, netbook, PDA, or any other type of computing device. In some embodiments, the network running on the game server recognizes the type of computing device used by the client and adjusts the data communication method employed. In another case, a client device accesses a game server's application over the Internet using standard data communication methods such as HTML (Hypertext Markup Language).

It should be understood that a given video game or game application may be developed for a particular platform and particular associated controller device. However, when making such games available through a game cloud system, users can access video games with different controller devices. For example, a game may be developed for a gaming console and its associated controller, but a user can access a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration is acceptable for running the video game from inputs that can be generated by the user's available controller devices (in this case, a keyboard and mouse). It is possible to define mappings to inputs.
In other examples, a user may access a cloud gaming system via a tablet computing device, touchscreen smartphone, or other touchscreen-driven device. In this case, the client device and controller device are integrated together within the same device, with input provided as detected touch screen inputs/gestures. For such devices, the input parameter configuration may define specific touch screen inputs that correspond to game inputs for the video game. For example, buttons, directional pads, or other types of input elements are visible or overlaid during the execution of a video game and are touchscreen locations that the user can touch to generate game input. can be shown. Gestures such as swipes in specific orientations, or specific touch motions can also be detected as game inputs. In one embodiment, for example, before beginning to play a video game, a user may be provided with a tutorial that shows how to provide input through a gameplay touch screen.

In some embodiments, a client device acts as an attachment point for a controller device. For example, a controller device communicates with a client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device may then process these inputs and then send the input data to the cloud gaming server via a network (eg, a network accessed through a local network device such as a router).
However, in other embodiments, the controller itself has the ability to communicate inputs directly to the cloud gaming server over the network, without having to first communicate these inputs through the client device, and without having to communicate these inputs through the client device first. It is possible that the device is For example, a controller can connect to a local network device (such as a router) to send data to and receive data from a cloud gaming server. Thus, while the client device can still receive video output from a cloud-based video game and render it to its local display, the input latency is reduced when the controller connects to the cloud game server over the network. This can be reduced by allowing input to be sent directly via the client device, bypassing the client device.

In some embodiments, the networked controller and client device are configured to send certain types of input directly from the controller to the cloud gaming server and other types of input through the client device. It is possible to For example, apart from the controller itself, detection inputs that do not rely on any additional hardware or processing can be sent directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion sensing inputs (e.g., accelerometers, magnetometers, gyroscopes), among any other types of inputs that may be provided by the controller device. . However, inputs that utilize additional hardware or require processing by the client device may be sent by the client device to the cloud gaming server.
These inputs may include video and/or audio obtained from the gaming environment that may be processed by the client device before being sent to the cloud gaming server. In addition, input from the controller motion detection hardware can be processed by the client device in conjunction with the captured video to detect the controller's position and motion, and then communicated by the client device to the cloud gaming server. Ru. In some embodiments, the controller device may also receive data (eg, feedback data, haptic data, etc.) from the client device or directly from the cloud gaming server.

It will be appreciated that the various embodiments defined herein may be combined or assembled into specific implementations using various features disclosed herein. Accordingly, the examples provided are only some of the possible examples and are not intended to be limiting to the various embodiments, which can define many more embodiments by combining various elements. In certain examples, certain embodiments may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments. Embodiments of the present disclosure may be practiced by a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired-based or wireless networks.

Additionally, although some method operations are described herein in a particular order/sequence, other housekeeping operations may be performed between the various method operations and/or Actions may be adjusted to occur at slightly different times and/or the processing of game telemetry and/or game state data to produce a changed game state may be performed in a temporally sufficient manner. processing of the method operations may be performed at various intervals, for example, to make game telemetry and/or game state available as needed by subsequent game executable operations. It should be understood that the operations of the method may be distributed across systems that support the system.

In some embodiments, the various methods and systems disclosed herein are implemented (manufactured) as computer readable code on a computer readable medium. A computer-readable medium is any data storage device that can store data and that can then be read by a computer system. Examples of computer readable media are hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and/or non-optical data. Including storage devices. Computer-readable media can also include computer-readable tangible media that is distributed over a network-coupled computer system so that computer-readable code is stored and executed in a distributed manner.

The foregoing descriptions of embodiments are provided for purposes of illustration and description and are not intended to be exhaustive or restrictive. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but are interchangeable where applicable and may be used in the selected embodiment, even if not specifically shown or described. can be used. Thus, one or more features from one or more embodiments disclosed herein can be combined with one or more features from one or more other embodiments disclosed herein. may be combined with features not explicitly disclosed herein to form further embodiments that are implicitly disclosed herein. Other embodiments may also be modified in many ways. Variations in such embodiments should not be considered a departure from the disclosure herein, and all such variations and modifications in embodiments are included within the scope of the disclosure provided herein. It is intended that

Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be obvious that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the embodiments disclosed herein are to be regarded as illustrative rather than limiting, and the embodiments should not be limited to the details set forth herein, but rather in accordance with the appended claims. Variations may be made within the scope and equivalents.

Claims (24)

A method of generating a summary of a media item, the method comprising:
transmitting a stream of media items from a server computing system to one or more client computing systems, the media items having a timeline for playback of the media items;
receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems, the participant-generated content being time-indexed to the timeline of the media item; is attached,
processing the participant-generated content to identify key portions of the media item based on representations present in the participant-generated content;
generating a dynamic summary queue including the main portion of the media item, the dynamic summary queue being assembled for streaming;
receiving a request to display the dynamic summary queue from a requester;
In response to the request, the method sends the stream of the dynamic summary queue to the requester computing device. 2. The method of claim 1, wherein the media item is an audiovisual presentation. 3. The method of claim 2, wherein the audiovisual presentation is one or more of a movie, a television show, a real-world sporting event, an electronic sporting event, an online show, an online video, and an uploaded video. 2. The method of claim 1, wherein the media item is a stream of game video and audio of a video game. The server computing system is a cloud gaming system configured to execute the video game according to input received from a respective controller device of the one or more participants at the one or more client computing systems. , the cloud gaming system is configured to direct the transmission of the stream of game video and audio reflecting play of the video game by the one or more participants, and the cloud gaming system is configured to direct the transmission of the stream of game video and audio reflecting play of the video game by the one or more participants, 5. The method of claim 4, providing the participant-generated content during game play. 5. The method of claim 4, wherein the video game is an electronic sports video game. The participant-generated content may include one of text data, voice data, audio data, pictographic data, image data, video data captured by a camera, image data captured by a camera, tag data, and controller device input data. 2. The method of claim 1, comprising more than one. Processing the participant-generated content includes extracting features from the participant-generated content to obtain a set of extracted features, and classifying the extracted features into one or more classifications. 2. The method of claim 1, comprising obtaining a set of classified features and processing the classified set of features through a machine learning model to identify key parts of the media item. further processing state data of the media item through the machine learning model and processing the set of classified features through the machine learning model to identify the main portions of the media item; 9. The method of claim 8, correlating the state data of the media item to the set of classified features by a time index. The state data of the media item is state data of a video game that is executed according to input received from a respective controller device of the one or more participants to generate the media item as a stream of game video and audio. 10. The method of claim 9, comprising: 2. The method of claim 1, wherein the dynamic summary cue includes an automatically generated verbal summary and/or an automatically generated written summary of the main portion of the media item. 2. The method of claim 1, wherein the dynamic summary cue includes at least a portion of the participant-generated content associated with the major portion of the media item. updating the dynamic summary queue when the participant-generated content is received at the server computing system and processed to identify key portions of the media item, the updating of the dynamic summary queue comprising: The method of claim 1, further comprising adding new content to the dynamic summary queue and/or modifying some content already present in the dynamic summary queue. 2. The method of claim 1, wherein the requester is any of the one or more participants. 2. The method of claim 1, wherein the requester is a person invited to join the one or more participants in viewing the media item. The requestor may cause the one or more participants to play a video game in accordance with inputs received from respective controller devices of the one or more participants to produce the media items as video and audio streams of the game. 2. The method of claim 1, wherein the person is a person invited to join the above participants. 2. The method of claim 1, further comprising receiving control parameters for generating the dynamic summary queue. 18. The method of claim 17, wherein the control parameters include specifying one or more types of representations present in the participant-generated content that indicate major portions of the media item. The type of display may include one or more of reactions, facial expressions, gestures, actions, utterances, sounds, and/or communications made by any of the one or more participants regarding the subject matter in the media item. 19. The method of claim 18, comprising: 18. The method of claim 17, wherein the control parameters include specifying one or more types of events that occur within the media item. The control parameter determines which of the one or more participants the participant-generated content is limited to when processing the participant-generated content to identify key portions of the media item. 18. The method of claim 17, comprising specifying. 18. The method of claim 17, wherein the control parameters include specifying a time slot within a timeline for playback of the media item for which the dynamic summary cue is generated. 18. The method of claim 17, wherein the control parameters include specifying a summary decomposition of the dynamic summary queue. A system for generating a video summary, the system comprising:
program instructions stored in computer memory for transmitting a stream of media items from a server computing system to one or more client computing systems, the media items having a timeline for playback of the media items; death,
program instructions stored in the computer memory for receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems; the generated content is time-indexed to the timeline of the media item;
program instructions stored in the computer memory for processing the participant-generated content to identify key portions of the media item based on representations present in the participant-generated content;
program instructions stored in the computer memory for generating a dynamic summary queue including the main portion of the media item, the dynamic summary queue being assembled for streaming;
program instructions stored in the computer memory for receiving a request from a requester to view the dynamic summary queue;
A system having program instructions stored in the computer memory for transmitting the stream of the dynamic summary queue to the requester computing device in response to the request.

Images