WO2018053257A1 - Procédés et systèmes de reconnaissance de motif spatiotemporel pour un développement de contenu vidéo - Google Patents
Procédés et systèmes de reconnaissance de motif spatiotemporel pour un développement de contenu vidéo Download PDFInfo
- Publication number
- WO2018053257A1 WO2018053257A1 PCT/US2017/051768 US2017051768W WO2018053257A1 WO 2018053257 A1 WO2018053257 A1 WO 2018053257A1 US 2017051768 W US2017051768 W US 2017051768W WO 2018053257 A1 WO2018053257 A1 WO 2018053257A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video
- event
- feed
- understanding
- events
- Prior art date
Links
- 238000003909 pattern recognition Methods 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims description 696
- 238000011161 development Methods 0.000 title description 2
- 238000010801 machine learning Methods 0.000 claims abstract description 380
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 186
- 238000012545 processing Methods 0.000 claims abstract description 55
- 230000033001 locomotion Effects 0.000 claims description 129
- 230000008569 process Effects 0.000 claims description 81
- 230000003993 interaction Effects 0.000 claims description 70
- 230000003190 augmentative effect Effects 0.000 claims description 45
- 238000004458 analytical method Methods 0.000 claims description 42
- 230000003416 augmentation Effects 0.000 claims description 42
- 230000008859 change Effects 0.000 claims description 34
- 230000009471 action Effects 0.000 claims description 29
- 230000000694 effects Effects 0.000 claims description 28
- 230000002123 temporal effect Effects 0.000 claims description 28
- 238000000275 quality assurance Methods 0.000 claims description 20
- 238000010200 validation analysis Methods 0.000 claims description 20
- 230000001133 acceleration Effects 0.000 claims description 16
- 238000001914 filtration Methods 0.000 claims description 14
- 238000000926 separation method Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 4
- 230000003139 buffering effect Effects 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 230000003466 anti-cipated effect Effects 0.000 claims description 2
- 230000000670 limiting effect Effects 0.000 description 66
- 238000001514 detection method Methods 0.000 description 45
- 238000012800 visualization Methods 0.000 description 40
- 230000007123 defense Effects 0.000 description 30
- 238000003860 storage Methods 0.000 description 25
- 230000002452 interceptive effect Effects 0.000 description 24
- 230000008901 benefit Effects 0.000 description 17
- 230000013016 learning Effects 0.000 description 17
- 230000007704 transition Effects 0.000 description 17
- 238000009826 distribution Methods 0.000 description 16
- 238000005516 engineering process Methods 0.000 description 16
- 238000004519 manufacturing process Methods 0.000 description 16
- 230000000007 visual effect Effects 0.000 description 15
- 230000006870 function Effects 0.000 description 13
- 230000015654 memory Effects 0.000 description 13
- 238000005192 partition Methods 0.000 description 13
- 238000012549 training Methods 0.000 description 13
- 238000000605 extraction Methods 0.000 description 12
- 238000006243 chemical reaction Methods 0.000 description 10
- 238000005520 cutting process Methods 0.000 description 10
- 230000037406 food intake Effects 0.000 description 10
- 239000003973 paint Substances 0.000 description 10
- 238000004088 simulation Methods 0.000 description 10
- 238000012706 support-vector machine Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 230000014509 gene expression Effects 0.000 description 9
- 208000027418 Wounds and injury Diseases 0.000 description 8
- 230000002776 aggregation Effects 0.000 description 8
- 238000004220 aggregation Methods 0.000 description 8
- 230000015556 catabolic process Effects 0.000 description 8
- 230000006378 damage Effects 0.000 description 8
- 208000014674 injury Diseases 0.000 description 8
- 239000000203 mixture Substances 0.000 description 8
- 238000012360 testing method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000012015 optical character recognition Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 7
- 238000013473 artificial intelligence Methods 0.000 description 6
- 238000010276 construction Methods 0.000 description 6
- 238000002955 isolation Methods 0.000 description 6
- 230000004807 localization Effects 0.000 description 6
- 238000009877 rendering Methods 0.000 description 6
- 238000007619 statistical method Methods 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 6
- 230000007306 turnover Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000002372 labelling Methods 0.000 description 5
- 238000012552 review Methods 0.000 description 5
- 238000000638 solvent extraction Methods 0.000 description 5
- 238000012384 transportation and delivery Methods 0.000 description 5
- 239000013598 vector Substances 0.000 description 5
- 238000004140 cleaning Methods 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 4
- 238000002156 mixing Methods 0.000 description 4
- 230000006855 networking Effects 0.000 description 4
- 238000012805 post-processing Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000001413 cellular effect Effects 0.000 description 3
- 230000001143 conditioned effect Effects 0.000 description 3
- 230000001934 delay Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 230000004886 head movement Effects 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000003058 natural language processing Methods 0.000 description 3
- 238000007670 refining Methods 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000010923 batch production Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 244000309464 bull Species 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000002996 emotional effect Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000009191 jumping Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000010422 painting Methods 0.000 description 2
- 238000004091 panning Methods 0.000 description 2
- 239000002243 precursor Substances 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 241001503987 Clematis vitalba Species 0.000 description 1
- 208000027534 Emotional disease Diseases 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 235000012364 Peperomia pellucida Nutrition 0.000 description 1
- 240000007711 Peperomia pellucida Species 0.000 description 1
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000011049 filling Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000013386 optimize process Methods 0.000 description 1
- 238000013488 ordinary least square regression Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007921 spray Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
- H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/10—Machine learning using kernel methods, e.g. support vector machines [SVM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/251—Learning process for intelligent management, e.g. learning user preferences for recommending movies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4722—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content
- H04N21/4725—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content using interactive regions of the image, e.g. hot spots
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8126—Monomedia components thereof involving additional data, e.g. news, sports, stocks, weather forecasts
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/44—Event detection
Definitions
- the present application generally relates to a system and method for performing analysis of events that appear in live and recorded video feeds, such as sporting events.
- the present application relates to a system and methods for enabling spatiotemporal analysis of component attributes and elements that make up events within a video feed, such as of a sporting event, systems for discovering, learning, extracting, and analyzing such events, metrics and analytic results relating to such events, and methods and systems for display, visualization, and interaction with outputs from such methods and systems.
- methods and systems disclosed herein enable the exploration of event data captured from video feeds, the discovery of relevant events (such as within a video feed of a sporting event), and the presentation of novel insights, analytic results, and visual displays that enhance decision-making, provide improved entertainment, and provide other benefits.
- Embodiments include taking data from a video feed and enabling an automated machine understanding of a game, aligning video sources to the understanding and utilizing the video sources to automatically deliver highlights to an end-user.
- using machine learning to develop an understanding of an event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based tracking system and is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop an understanding of an event further comprises aligning multiple unsynchronized input feeds related to an event (e.g., tracking video, broadcast video/audio, play-by-play data) using at least one of a hierarchy of algorithms and a hierarchy of users, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include a variety of feeds of varying types, such as three or more feeds of two or more types related to the event.
- Embodiments may further include validating, confirming, refining, or modifying the understanding or the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds.
- content displaying an event is automatically extracted from a video feed, such as based on a machine understanding of the event.
- extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of machine understanding of events and a machine understanding of another input feed, such as a broadcast video feed, an audio feed, and/or a closed caption feed.
- the machine understanding of the other input feed includes understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- Embodiments may further include using a combination of machine understanding of events from the video feed and a machine understanding of another input feed related to the events, such as a broadcast video feed, an audio feed and a closed caption feed to at least one of edit the video cut and combine the video cut with other content.
- the machine understanding of the other input feed may include understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- Embodiments may further include automatically developing a semantic index of a video feed based on the machine understanding of at least one event in the video feed to indicate the time of the event in the video feed and the location of the display of the event in the video feed, such as a pixel location, a voxel location, a raster image location, or the like that indicates where the event can be found on a display screen.
- the semantic index of the video feed may be provided to enable augmentation of the video feed, such as for adding content based on to the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- a method comprises receiving a sport playing field configuration and at least one image and determining a camera pose based, at least in part, upon the sport playing field configuration and at least one image.
- a method comprises performing automatic recognition of a camera pose based, at least in part, on video input comprising a scene and augmenting the video input with at least one of additional imagery and graphics rendered within the reconstructed 3D space of the scene.
- Methods and systems described herein may include taking a video feed of an event; using machine learning to develop an understanding of the event; automatically, under computer control, aligning the video feed with the understanding; and producing a transformed video feed that includes at least one highlight that may be extracted from the machine learning of the event.
- the event may be a sporting event.
- the event may be an entertainment event.
- the event may be at least one of a television event and a movie event.
- the event may be a playground pickup game or other amateur sports game.
- the event may be any human activity or motion in a home or commercial establishment.
- the transformed video feed creates a highlight video feed of video for a defined set of players.
- the defined set of players may be a set of players from a fantasy team.
- Embodiments may include delivering the video feed to at least one of an inbox, a mobile device, a table, an application, a scoreboard, a Jumbotron board, a video board, and a television network.
- Methods and systems described herein may include taking a source data feed relating to an event; using machine learning to develop an understanding of the event; automatically, under computer control, aligning the source feed with the understanding; and producing a transformed feed that includes at least one highlight that may be extracted from the machine learning of the event.
- the event may be a sporting event.
- the event may be an entertainment event.
- the event may be at least one of a television event and a movie event.
- the source feed may be at least one of an audio feed, a text feed, a statistics feed, and a speech feed.
- Methods and systems described herein may include: taking a data set associated with a video feed of a live event; taking spatiotemporal features of the live event; applying machine learning to determine at least one spatiotemporal pattern of the event; and using a human validation process to at least one of validate and teach the machine learning of the spatiotemporal pattern.
- the event may be a sporting event.
- Methods and systems described herein may include taking at least one of a video feed and an image feed; taking data relating to a known configuration of a venue; and automatically, under computer control, recognizing a camera pose based on the video feed and the known configuration.
- the venue may be a sporting event venue.
- Methods and systems described herein may include taking at least one feed, selected from the group consisting of a video feed and an image feed of a scene; taking data relating to a known configuration of a venue; automatically, under computer control, recognizing a camera pose based on the video feed and the known configuration; and automatically, under computer control, augmenting the at least one feed with at least one of an image and a graphic within the space of the scene.
- the methods and systems may include using human input to at least one of validate and assist the automatic recognition of the camera pose.
- the methods and system may include presenting at least one metric in the augmented feed.
- the methods and systems may include enabling a user to interact with at least one of the video feed and a frame of the video feed in a 3D user interface.
- the methods and systems may include augmenting the at least one feed to create a transformed feed.
- the transformed video feed may create a highlight video feed of video for a defined set of players.
- Methods and systems described herein may include taking a data set associated with a video feed of a live event; taking spatiotemporal features of the live event; applying machine learning to determine at least one spatiotemporal pattern of the event; and calculating a metric based on the determined pattern.
- the metric may be at least one of a shot quality (SEFG) metric, an EFG+ metric, a rebound positioning metric, a rebounding attack metric, a rebounding conversion metric, an event-count per playing time metric, and an efficiency per event-count metric.
- SEFG shot quality
- Methods and systems described herein may include providing an interactive, graphical user interface for exploration of data extracted by machine learning from the video capture of live events.
- the graphical user interface enables exploration and analysis of events.
- the graphical user interface is at least one of a mobile device interface, a laptop interface, a tablet interface, a large-format touchscreen interface, and a personal computer interface.
- the data may be organized to present at least one of a breakdown, a ranking, a field-based comparison, and a statistical comparison.
- the exploration enables at least one of a touch interaction, a gesture interaction, a voice interaction, and a motion- based interaction.
- Methods and systems described herein may include taking a data set associated with a video feed of a live event; automatically, under computer control, recognizing a camera pose for the video; tracking at least one of a player and an object in the video feed; and placing the tracked items in a spatial location corresponding to spatial coordinates.
- Methods and systems described herein may include taking a data set associated with a video feed of a live event; taking spatiotemporal features of the live event; applying machine learning to determine at least one spatiotemporal pattern of the event; and delivering contextualized information during the event.
- the contextualized information includes at least one of a statistic, a replay, a visualization, a highlight, a compilation of highlights, and a replay.
- the information may be delivered to at least one of a mobile device, a laptop, a tablet, and a broadcast video feed.
- the methods and systems may include providing a touch screen interaction with a visual representation of at least one item of the contextualized information.
- the methods and systems provided herein may include taking a video feed of an event in which there is a participant; identifying a point of view of the participant; and automatically, under computer control, selecting a plurality of video frames from the video feed showing at least one view from the point of view of the participant.
- the methods and systems may further include rendering a 3D video using the selected plurality of video frames.
- the methods and systems may further include providing an interface wherein a user may select the participant from a plurality of participants.
- the event may be a sporting event and the participant may be a player.
- the event may be basketball and the video feed may be associated with 3D motion capture data from cameras that capture the video feed.
- a method for providing enhanced video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events and to determine at least one event type for each of the plurality of events within the at least one video feed.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes extracting a plurality of video cuts from the at least one video feed.
- the method includes indexing the extracted plurality of video cuts based on the at least one event type determined by the understanding developed by the machine learning that corresponds to an event in the plurality of events detectable in the plurality of video cuts.
- the method further includes automatically, under computer control, generating an enhanced video content data structure using the extracted plurality of video cuts based on the indexing of the extracted plurality of video cuts.
- the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern selected from the group consisting of relative motion of two visible features toward each other for at least a duration threshold, acceleration of motion of at least two visible features with respect to each other for acceleration of at least an acceleration threshold, rate of motion of two visible features toward each other, projected point of intersection of the two visible features, and separation distance between the two visible features being less than a separation threshold.
- automatically generating the enhanced video content data structure includes combining the understanding of the plurality of events with an understanding of at least one of a broadcast video event or a broadcast audio event developed with the machine learning.
- generating the enhanced video content data structure is based at least in part on at least one of a user preference and a user profile for a user for which the enhanced video content data structure is generated.
- the method includes providing a user interface for display on a mobile device.
- the user interface includes at least one of a search option and a filtering option to allow a user to at least one of specify and select a description of a type of event.
- the enhanced video content data structure is generated to match the description.
- using the machine learning to develop the understanding of the event further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed by the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two event types related to the event.
- the method includes further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating the understanding and modifying the understanding developed by the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes automatically developing a semantic index of the at least one video feed based on the understanding developed with the machine learning of at least one event of the plurality of events in the video feed to indicate a game time of the at least one event in the video feed and a location of a display of the at least one event in the video feed.
- the location of the display of the at least one event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed configured to enable semantic-based augmentation of the video feed.
- augmentation of the video feed includes adding content based on the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- extracting the plurality of video cuts includes automatically extracting a cut from the video feed using a combination of the machine learning developed understanding of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the method includes the understanding developed with the machine learning of the another input feed includes understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the another input feed.
- the method includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the video feed.
- the method includes extracting a plurality of video cuts from the at least one video feed.
- the method includes indexing the plurality of video cuts based on the at least one event type determined by the understanding developed by machine learning.
- the method further includes providing a mobile application having a user interface configured to permit a user to find the extracted plurality of video cuts based on the indexing of the extracted plurality of video cuts with the mobile application.
- the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern selected from the group consisting of relative motion of two visible features toward each other for at least a duration threshold, acceleration of motion of at least two visible features with respect to each other for acceleration of at least an acceleration threshold, rate of motion of two visible features toward each other, projected point of intersection of the two visible features, and separation distance between the two visible features being less than a separation threshold.
- the method includes the understanding developed by the machine learning includes generating at least one metric associated with at least one event of the plurality of events.
- the user interface of the mobile application is configured to permit the user to enhance a video cut from the plurality of video cuts by selecting the metric to be included in the video cut.
- the method includes the user interface of the mobile application is configured to permit the user to share an edited video via the mobile application.
- the method includes using the machine learning to develop the understanding of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the method includes using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event of the plurality of events.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- the method includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validating is based on a nature of the input feed.
- extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding of the another input feed includes an understanding developed with the machine learning of at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for providing enhanced video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events.
- the method includes extracting a plurality of video cuts from the at least one video feed.
- the method includes indexing the extracted plurality of video cuts based on the at least one event type determined by the understanding developed by machine learning and determining at least one pattern relating to the extracted plurality of video cuts.
- the method further includes indexing at least a portion of the plurality of the extracted video cuts with an indicator of the pattern.
- the at least one pattern is developed by applying machine learning.
- the understanding developed by the machine learning includes identifying at least one player involved in an event. Indexing of the extracted plurality of video cuts includes identifying at least one player represented in at least one of the video cuts from the plurality of the video cuts.
- the at least one pattern relates to a series of same event types involving a same player over time.
- the plurality of video cuts includes a player during multiple, identical event types over time.
- the method includes providing an enhanced video feed that shows a player during the plurality of events over time.
- the enhanced video feed is at least one of a simultaneous, superimposed video of the player involved in multiple, identical event types and a sequential video of the player involved in the same event type.
- determining at least one pattern includes identifying sequences of events that predict a given action that is likely to follow.
- the method includes determining the at least one pattern includes identifying similar sequences of events across the plurality of video feeds.
- the method includes providing a user interface that enables a user to at least one of view and interact with the at least one pattern.
- the at least one pattern and the interaction options are personalized based on at least one of a user preference and a user profile.
- the at least one pattern relates to an anticipated outcome of at least one of a game and an event within a game.
- the method includes providing a user with at least one of a statistic, trend information and a prediction based on the at least one pattern.
- the at least one the statistic, the trend information or the prediction is based on at least one of a user preference and a user profile.
- the method includes the at least one pattern relates to play of an athlete.
- the method includes providing a comparison of the play of the athlete with another athlete based on a similarity of at least one of the extracted plurality of video cuts and the at least one pattern.
- the comparison is between a professional athlete and a non-professional user.
- the comparison is based on a similarity of a playing style of the professional athlete, as determined by the understanding developed with the machine learning of at least one event of the plurality of events and the at least one pattern, with at least one feature of the playing style of the non-professional user.
- the understanding developed with the machine learning of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the understanding developed with the machine learning of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating the understanding and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding of the other input feed includes an understanding developed with the machine learning of at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for providing enhanced video content including processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events.
- the method includes extracting a plurality of video cuts from the at least one video feed.
- the method includes indexing the extracted plurality of video cuts based on at least one event type determined by the understanding developed by machine learning.
- the method further includes automatically, under computer control, delivering the extracted plurality of video cuts to at least one user based on at least one of a user profile and a user preference.
- the at least one of the user preference and the user profile is continuously updated based on a user's indication of at least one of liking and disliking at least one of video cuts of the plurality of extracted video cuts.
- the understanding developed with the machine learning is based on human-identified video alignment labels for semantic events.
- the at least one spatiotemporal pattern recognition algorithm uses time-aligned content from multiple input sources to develop the understanding with the machine learning.
- the method includes the at least one spatiotemporal pattern recognition algorithm uses a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds to handle the multiple input sources.
- using machine learning to develop the understanding of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating the understanding and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the method includes the understanding developed with the machine learning of the other input feed includes an understanding developed with machine learning of at least one video cut of the plurality of video cuts of a broadcast commentary and a change in camera view in the input feed.
- a method for enabling a user to express preferences relating to display of video content includes processing at least one video feed through at least one spatio temporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure.
- the method includes providing a user interface configured to permit a user to indicate a preference for at least one event type.
- the method includes upon receiving an indication of the preference by the user, retrieving the at least one video content data structure that was determined by the machine learning to be associated with the at least one event type indicated by the user.
- the method further includes providing the user with a video feed containing the video content including the at least one video content data structure.
- using machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the at least one event, one or more feeds of tracking video for the at least one event, and one or more play-by-play data feeds of the at least one event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds.
- the method includes at least one of validating the understanding and modifying the understanding developed by the machine learning using a hierarchy involving at least two of at least one algorithm, at least one human operator, and at least one input feed where at least one item in the hierarchy used for validating the understanding is based on a nature of the one or more input feeds.
- the method includes the user interface is of at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, a touch screen device, a virtual reality or augmented reality headset, and a smart phone.
- the method includes the user interface further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.
- the method includes the understanding developed by the machine learning includes a context for the at least one of event and the context is stored with the at least one video content data structure.
- the user interface further comprises an element configured to permit a user to indicate a preference for the context.
- the method includes retrieving portion of the video content corresponding to the context and displaying the portion of the video content corresponding to the context to the user upon receiving an indication of the preference for the context.
- the context comprises at least one of a presence of a preferred player in the at least one video feed, a preferred matchup of players in the at least one video feed, a preferred team in the at least one video feed, and a preferred matchup of teams in the at least one video feed.
- the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed. The at least one metric is based at least in part on the understanding developed by the machine learning.
- extracting the content displaying the at least one event includes automatically extracting a cut from the video feed using a combination of the understanding of events developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for enabling a mobile application that allows user interaction with video content includes taking a video feed and processing the video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed.
- the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed.
- the method includes automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information.
- the method includes producing a video content data structure that includes the context information.
- the method includes automatically, under computer control, producing a story that includes the video content data structure. A portion of the story is based on a user preference, the context information, and the video content data structure.
- extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of the understanding of the event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- the method includes using a combination of the understanding developed with the machine learning of the event from the video feed and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed to at least one of edit the video cut and combine the video cut with other content.
- the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- the method includes automatically developing a semantic index of a video feed based on the understanding developed with the machine learning of at least one event in the video feed indicative of a time of the event in the video feed and a location of a display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed in combination to enable augmentation of the video feed.
- the augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- the method includes using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system.
- the understanding developed with machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the method includes using machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the method includes the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on a nature of the input feeds.
- the method includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- the user preference for a type of content is based on at least one of a user expressed preference and a preference that is inferred based on user interaction with an item of content.
- a system for enabling a user to express preferences relating to display of video content includes a machine learning facility that uses at least one spatiotemporal pattern recognition algorithm for developing an understanding of at least one event within at least one video feed to determine at least one event type.
- the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the system includes a video production facility for automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in at least one video content data structure.
- the system includes a server for serving data to a user interface that is configured to permit a user to indicate a preference for at least one event type, that retrieves the at least one video content data structure that was determined by the machine learning to have an event type preferred by the user, and that provides the user with a video feed containing the event type preferred by the user.
- the method includes the user interface that is configured to permit the user to indicate the preference for at least one event type is at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, and a smart phone.
- the user interface that is configured to permit the user to indicate the preference for at least one event type further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.
- the understanding developed by the machine learning facility includes a context for the at least one event and the context is stored with the at least one video content data structure.
- the user interface further comprises an element configured to permit a user to indicate a preference for at least one context.
- the server for serving data to the user interface retrieves video content corresponding to the preference for at least one context and displays the video content to the user upon receiving an indication of the preference for the at least one context.
- the method includes the at least one context comprises at least one of a presence of a preferred player in the video feed, a preferred matchup of players in the video feed, a preferred team in the video feed, and a preferred matchup of teams in the video feed.
- the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed.
- the metric is based at least in part on the understanding developed with the machine learning facility.
- the machine learning facility to develop the understanding of the at least one event uses events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the machine learning facility to develop the understanding of the at least one event aligns multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.
- the video production facility at least one of validates and modifies the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the video production facility at least one of validates the understanding and modifies the understanding developed by the machine learning facility using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- the video production facility for automatically, under computer control, extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning facility and an understanding developed with the machine learning facility of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding developed with the machine learning facility of the another input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the at least one video feed.
- a method for delivering personalized video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure.
- the method includes developing a personal profile for a user based on at least one of expressed preferences of the user, information about the user, and information collected about actions taken by the user with respect to at least one event type.
- the method further includes upon receiving an indication of the user profile, retrieving at least one video content data structure that was determined by the machine learning to have an event type likely to be preferred by the user based on the personal profile for the user.
- Using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed by the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the method includes using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the method includes the understanding of the other input feed developed with the machine learning includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for delivering personalized video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed of a professional game.
- the understanding developed by the machine learning includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes developing an understanding using the machine learning of at least one event within a data feed relating to motion of a nonprofessional player.
- the method includes automatically, under computer control, providing an enhanced video feed that represents the non-professional player playing within a context of the professional game based on the understanding of the at least one event within the at least one video feed of the professional game and a data feed relating to the motion of the non-professional player.
- the method includes providing a facility having cameras for capturing 3D motion data, and capturing video of a non-professional player to provide the data feed for the non-professional player.
- the method includes the non-professional player is represented by mixing video of the non-professional player with at least one video of the professional game.
- the method includes the non-professional player is represented as an animation having attributes based on the data feed relating to motion of the nonprofessional player.
- the method includes using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the at least one event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. In embodiments, the method includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- a method includes taking a video feed and processing the video feed through a spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed.
- the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed.
- the method includes automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information.
- the method further includes producing a video content data structure that includes the context information.
- the method includes determining a plurality of semantic categories for the context information, and filtering a plurality of video content data structures based on the plurality of semantic categories.
- Each of the video content data structures of the plurality of video content data structures includes context information relating to the event.
- the method includes matching events that occur in a first video feed to events that occur in a second video feed, the second video feed separate from the first video feed, and using a semantic understanding captured in the first video feed to at least one of filter and cut a separate second video feed based on the matching events that occur in the first video feed and the second video feed.
- the method includes determining a pattern relating to a plurality of events and providing a content data structure based on the pattern.
- the pattern comprises a plurality of important plays in a sports event that are identified based on comparison to similar plays from previous sports events. In embodiments, the pattern comprises a plurality of plays in a sports event that is determined to be unusual based on comparison to video feeds from other sports events.
- the method includes extracting semantic events over time to draw a comparison of at least one of a player and a team over time.
- the method includes superimposing video of events extracted from video feeds from at least two different time periods to illustrate the comparison.
- the method includes allowing a user to interact with the video content data structure to produce an edited video data stream that includes the video data structure.
- the interaction includes at least one of editing, mixing, cutting, and sharing a video clip that includes the video content data structure.
- the method includes enabling users to interact with the video content data structure through a user interface to enhance the video content data structure with at least one graphic element selected from a menu of options. In embodiments, the method includes enabling a user to share the enhanced video content. In embodiments, the method includes enabling a user to find similar video clips based on a semantic context identified in the video clips. In embodiments, the method includes using the video data structure and the context information to construct modified video content for a second screen that includes the video content data structure. In embodiments, the modified video content for the second screen correlates to the timing of an event displayed on a first screen. In embodiments, the modified video content for the second screen includes a metric determined based on the machine understanding.
- using the machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes automatically developing a semantic index of the video feed based on the understanding developed with the machine learning of the event in the video feed to indicate a time of the event in the video feed and a location of a display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- the augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- extracting content displaying the event includes automatically extracting a cut from the video feed using a combination of the understanding of events developed from the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding of the other input feed developed by the machine learning includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a system includes an ingestion facility for ingesting a plurality of video feeds.
- the system includes a machine learning system for processing the video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on a series of events in the plurality of video feeds in order to develop an understanding of the series of events within the plurality of video feeds.
- the understanding includes identifying context information relating to the series of events and an entry in a relationship library at least detailing a relationship between two visible features of the plurality of video feeds.
- the system includes an extraction facility for automatically, under computer control, extracting content displaying the series of events and associating the extracted content with the context information.
- the system includes a video publishing facility for producing a video content data structure that includes the context information.
- the system includes an analytic facility for determining a plurality of semantic categories for the context information, and filtering the plurality of video content data structures based on the semantic categories
- the system includes a matching engine for matching a series of events that occur in a first video feed to those that occur in a second video feed, the first video feed being separate from the second video feed, and using a semantic understanding captured in the first video feed to at least one of filter and cut the second video feed based on the matching of the series of events that occur in the first video feed with those that occur in the second video feed.
- the system includes a pattern recognition facility that determines a pattern relating to the series of events and provides a content data structure based on the pattern.
- using machine learning to develop the understanding of the series of events further comprises using events in position tracking data over time obtained from the plurality of video feeds and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop the understanding of the series of events further comprises aligning multiple unsynchronized input feeds related to the series of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-byplay data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds and at least one algorithm in the hierarchy for validation of the alignment of the unsynchronized input feeds is based on a nature of the input feeds.
- the system includes at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds. The one or more algorithms are based on the nature of the input feed.
- the system includes automatically developing a semantic index of a video feed from the plurality of video feeds based on the understanding developed by the machine learning of at least one event from the series of events in the video feed.
- the semantic index of the video feed indicates a time of the event in the video feed and a location of a display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- a system for enabling interaction with a broadcast video content stream includes a machine learning facility for processing at least one video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on at least one event in the at least one video feed in order to develop an understanding of the at least one event with the at least one video feed.
- the at least one video feed is for a video broadcast.
- the understanding includes identifying context information relating to the at least one event and an entry in a relationship library at least detailing a relationship between two visible features of the plurality of video feeds.
- the system includes a touch screen user interface configured to permit at least one broadcaster to interact with the at least one video feed.
- the touch screen user interface includes options for interaction with the at least one broadcaster that are based on the context information.
- the touch screen user interface is configured to control a portion of content of the at least one video feed for the video broadcast.
- the method includes an interface configured to permit remote viewers to control the portion of the content of the at least one video feed for the video broadcast.
- the touch screen user interface includes options that are based on the context information for controlling a portion of content of the at least one video feed.
- the touch screen interface is a relatively large screen configured to be seen by viewers of the video broadcast as the broadcaster uses the touch screen interface.
- the touch screen interface is configured to permit the broadcaster to select from a plurality of context-relevant metrics to be displayed on the large screen.
- the touch screen interface is configured to permit the broadcaster to display a plurality of video feeds that have similar contexts as determined by the machine learning facility.
- the similarity of contexts is determined by comparing events within the plurality of video feeds.
- the touch screen interface is configured to permit the broadcaster to display a superimposed view of at least two video feeds to facilitate a comparison of events from the plurality of video feeds.
- the comparison of the events the from the plurality of video feeds is of similar players from different time periods of the video broadcast.
- the understanding developed by the machine learning includes detail regarding similarity of players based on characteristics of the players identified during the different time periods.
- the touch screen interface is configured to permit the broadcaster to display a plurality of highlights that is automatically determined by an understanding developed with the machine learning of a live sports event that is a subject of the at least one video feed.
- the plurality of highlights is determined to be based on similarity to highlights that have been identified for other events.
- using the machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- the system includes at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. The one or more algorithms in the hierarchy for validation is based on a nature of an input feed.
- the system includes automatically developing a semantic index of the at least one video feed based on the understanding developed with the machine learning of the at least one event in the at least one video feed to indicate at least one time of the at least one event in the at least one video feed and a location of a display of the at least one event in the at least one video feed.
- the location of the display of the at least one event in the at least one video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the at least one video feed with the at least one video feed to enable augmentation of the at least one video feed.
- the enabled augmentation of the at least one video feed includes adding content based on the location of the display.
- the touch screen user interface is configured to enable a touch screen feature and a mouse interface feature based on the identified location.
- a method of enabling interaction with a broadcast video content stream includes processing a video feed through a spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed for a video broadcast.
- the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed.
- the method includes providing a touch screen user interface configured to permit a broadcaster to interact with the video feed and configured to permit the broadcaster to select options in the touch screen user interface based on the context information.
- the options in the touch screen user interface control content of a portion of the video broadcast.
- the touch screen interface is a relatively large screen adapted to be seen by viewers of the video broadcast as the broadcaster uses the touch screen interface.
- the touch screen interface is configured to permit the broadcaster to select from a plurality of metrics relevant to the context information relating to the event and to display the plurality of metrics on the large screen.
- the touch screen interface is configured to permit the broadcaster to display a plurality of video feeds for the video broadcast.
- the understanding developed with the machine learning includes similar identifying context information in the plurality of video feeds for the video broadcast.
- the system includes the similar context information in the plurality of video feeds for the video broadcast is determined by comparing events within the plurality of video feeds.
- the touch screen interface is configured to permit the broadcaster to display a superimposed view of at least two video feeds to facilitate a comparison of events from a plurality of video feeds.
- the comparison of the events from the plurality of video feeds is of similar players from different time periods.
- the understanding developed by the machine learning includes similarity of players based on characteristics of the similar players from the different time periods.
- the understanding developed by the machine learning includes a plurality of highlights based on a live sports event that is a subject of the video feed.
- the touch screen interface is configured to permit the broadcaster to display the plurality of highlights.
- the system includes the plurality of highlights is determined based on similarity to highlights identified for other events.
- using the machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the system includes using the machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the at least one event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the aligning of the multiple unsynchronized input feeds related to the event using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. In embodiments, the method includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. In embodiments, the method includes automatically developing a semantic index of the video feed based on the understanding developed with the machine learning of the event in the video feed to indicate a time of the event in the video feed and a location of a display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- the method includes augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- a system for enabling user interaction with video content includes an ingestion facility, that may be executing on at least one processor and may be adapted or configured to access at least one video feed.
- the system includes a machine learning system that may be adapted or configured to process the at least one video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on an event in the at least one feed in order to develop an understanding of the event within the at least one video feeds.
- the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the system includes an extraction facility that may be adapted or configured to automatically, under computer control, extract content displaying the event and associate the extracted content with the context information.
- the system includes a video production facility that may be adapted or configured to produce a video content data structure that includes the context information.
- the system further includes an application having a user interface configured to permit a user to interact with the video content data structure.
- the user interface may further be configured with options for user interaction that are based on the context information.
- the application is a mobile application.
- the application is at least one of a smart television application, a virtual reality headset application and an augmented reality application.
- the user interface is a touch screen interface.
- the user interface is configured to permit a user to enhance the video feed by selecting a content element to be added to the video feed.
- the content element is at least one of a metric and a graphic element that is based on the understanding developed with the machine learning.
- the user interface is configured to permit the user to select content for a particular player of a sports event.
- the user interface is configured to permit the user to select content relating to a context involving a matchup of two particular players in a sports event.
- the system takes at least two video feeds from different time periods, the machine learning facility determines a context that includes a similarity between at least one of a plurality of players and a plurality of plays in the two feeds.
- the user interface is configured to permit the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the at least one video feed comprises 3D motion camera data captured from a live sports venue.
- the machine learning facility increases its ability to develop the understanding by ingesting a plurality of events for which context has already been identified.
- the method includes using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes automatically developing a semantic index of a video feed based on the machine understanding of the event in the video feed to indicate a time of the event in the video feed and a location of a display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- extracting the content displaying the event includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the system includes the understanding developed with machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for enabling a mobile application allowing user interaction with video content includes taking at least one video feed and processing the at least one video feed through a spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the at least one video feed.
- the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information.
- the method includes producing a video content data structure that includes the context information.
- the method further includes providing a mobile application having a user interface configured to permit a user to interact with the video content data structure.
- the user interface is configured to include options for user interaction based on the context information.
- the user interface is a touch screen interface.
- the user interface is configured to permit a user to enhance the video feed by selecting a content element to be added to the video feed.
- the content element is at least one of a metric and a graphic element that is based on the machine understanding.
- the user interface is configured to permit the user to select content for a particular player of a sports event.
- the system includes the user interface is configured to permit the user to select content relating to a context involving the matchup of two particular players in a sports event.
- the method includes taking at least two video feeds from different time periods, the machine learning facility determines a context the includes a similarity between at least one of a plurality of players and a plurality of plays in the at least two feeds and the user interface is configured to permit the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the video feed comprises 3D motion camera data captured from a live sports venue.
- the method includes the machine learning facility increases its ability to develop the understanding by ingesting a plurality of events for which context has already been identified.
- the method includes the machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- the method includes using the machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes automatically developing a semantic index of a video feed based on the understanding developed with the machine learning of at least one event in the video feed to indicate a time of the event in the video feed and a location of a display of the event in the video feed.
- the method includes the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- a method for providing enhanced video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events and to determine at least one event type for each of the plurality of events within the at least one video feed.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes extracting a plurality of video cuts from the at least one video feed and indexing the plurality of video cuts based on at least one event type determined by the understanding developed by the machine learning.
- the method includes making the indexed and extracted video cuts available to a user.
- the method includes enabling the user to at least one of edit, cut, and mix the plurality of video cuts to provide an enhanced video containing at least one video cut from the plurality of the video cuts.
- the user is enabled to share the enhanced video.
- the method includes indexing at least one shared, enhanced video with a semantic understanding of the event type determined by the machine learning.
- the indexing of the at least one shared, enhanced video with the semantic understanding of the event type determined by the machine learning is used to determine similarity between the shared, enhanced video and at least one other video content item.
- the similarity between the shared, enhanced video and at least one other video content item is used to identify additional extracted, indexed video cuts of interest to the user.
- the similarity between the shared, enhanced video and at least one other video content item is used to identify other users who have shared similar enhanced video.
- the method includes the similarity between the shared, enhanced video and at least one other video content item is used to identify other users who are likely to have an interest in the shared, enhanced video.
- the method includes recommending at least one of the shared, enhanced video and one of the video cuts based on an understanding of the preferences of the other users.
- the similarity between the shared, enhanced video and at least one other video content item is based at least in part on user profile information for users who have indicated an interest in the video cut and the other video content item.
- using the machine learning to develop the understanding of the plurality of events further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- extracting the plurality of video cuts from the at least one video feed includes using a combination of the understanding of the plurality of events developed with machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the method includes the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in an input feed.
- a method for providing enhanced video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events and to determine at least one event type for each of the plurality of events within the at least one video feed.
- the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes extracting a plurality of video cuts from the at least one video feed and indexing the plurality of video cuts to form an indexed set of extracted video cuts. The indexing is based on at least one event type determined by the understanding developed with the machine learning.
- the method includes determining at least one pattern relating to the plurality of events in the at least one video feed.
- the method includes adding the determined the at least one pattern to the indexed set of video cuts and making the indexed and extracted video cuts available to a user.
- the user is enabled to at least one of edit, cut, and mix the plurality of video cuts to provide an enhanced video containing the plurality of video cuts.
- the user is enabled to share the enhanced video.
- the plurality of video cuts is clustered based on the patterns that exist within the plurality of video cuts.
- the pattern is determined automatically using an understanding developed with the machine learning and is based on the understanding developed with the machine learning of the of the plurality of events in the at least one video feed.
- the method includes the pattern being a highlight event within the at least one video feed.
- a highlight event is presented to the user when the indexed and extracted plurality of video cuts is made available to the user.
- the user is prompted to watch a longer video feed upon viewing the indexed and extracted plurality of video cuts.
- using the machine learning to develop the understanding of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- extracting the plurality of video cuts from the at least one video includes automatically extracting a cut from the at least video feed using a combination of the understanding of events developed with the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for embedding video content in an application includes taking a video feed and using machine learning to develop an understanding of an event within the video feed. The understanding including identifying context information relating to the event.
- the method includes automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information.
- the method includes producing a video content data structure that includes the associated context information and taking an application that displays video content.
- the method also includes embedding the video content data structure in the application.
- the user interface of the application offers the user the option to control the presentation of the video content from the video content data structure in the application.
- the control of the presentation is based on at least one of a user preference and a user profile.
- the application is a mobile application that provides a story about an event.
- the video content data structure comprises at least one of a content card and a digital still image.
- the method includes using machine learning to develop an understanding of an event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop an understanding of an event further comprises aligning multiple unsynchronized input feeds related to an event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the method includes the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event. In embodiments, the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes automatically developing a semantic index of a video feed based on the machine understanding of at least one event in the video feed to indicate the time of the event in the video feed and the location of the display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the method includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- a system for enabling a mobile application that allows user interaction with video content includes a video ingestion facility for taking a video feed and a machine learning facility for developing an understanding of an event within the video feed.
- the understanding includes identifying context information relating to the event.
- the system includes a video production facility for automatically, under computer control, extracting the content displaying the event, associating the extracted content with the context information and producing a video content data structure that includes the associated context information.
- the system also includes using the context information for a plurality of such video content data structures to generate, automatically under computer control, a story that includes a sequence of the video content data structures.
- the content of the story is based on a user profile that is based on at least one of an expressed user preference, information about a user interaction with video content, and demographic information about the user.
- the method includes determining a pattern relating to a plurality of events in the video feed and associating the determined pattern with the video content data structure as additional context information.
- the pattern relates to a highlight event within the video feed.
- the method includes the highlight event is associated with at least one of a player and a team.
- the embedded application allows a user to indicate at least one of a player and a team for which the user wishes to obtain video feeds containing the highlight events.
- the pattern relates to a comparison of events occurring at least one of within the video feed or within a plurality of video feeds. In embodiments, the comparison is between events occurring over time.
- the embedded application allows a user to select at least one player to obtain a video providing a comparison between the player and at least one of a past representation of the same player and a representation of another player.
- the pattern is a cause-and-effect pattern related to the occurrence of a following type of event after the occurrence of a pre-cursor type of event.
- the user to review video cuts in a sequence that demonstrate the cause-and-effect pattern.
- the application provides a user interface for allowing a user to enter at least one of text and audio input to provide a narrative for a sequence of events within the video feed.
- the user may select a sequence of video events from within the feed for display in the application.
- the method includes upon accepting the user narrative, the system automatically generates an electronic story containing the events from the video feed and the narrative.
- using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system and. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop an understanding of the event further comprises aligning multiple unsynchronized input feeds related to an event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the event.
- the method includes the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.
- the system includes automatically developing a semantic index of a video feed based on the machine understanding of at least one event in the video feed to indicate the time of the event in the video feed and the location of the display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- the system includes extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of machine understanding of events and a machine understanding of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the machine understanding of the other input feed includes understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method for enabling a mobile application that allows user interaction with video content includes taking a video feed and using a machine learning facility to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event.
- the method includes automatically, under computer control, extracting the content displaying the event, associating the extracted content with the context information and producing a video content data structure that includes the associated context information.
- the system includes using the context information for a plurality of such video content data structures to generate, automatically under computer control, a story that includes a sequence of the video content data structures.
- using machine learning to develop an understanding of an event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to an event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the event.
- the system includes the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event. In embodiments, the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. In embodiments, the system includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds. In embodiments, the system includes automatically developing a semantic index of a video feed based on the machine understanding of at least one event in the video feed to indicate the time of the event in the video feed and the location of the display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- the system includes extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of machine understanding of events and a machine understanding of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.
- the system includes the machine understanding of the other input feed includes understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.
- a method includes taking a video feed and using machine learning to develop an understanding of a semantically relevant event within the video feed.
- the method includes indexing video segments of the video feed with information indicating the semantically relevant events identified within the feed by the machine learning.
- the method includes applying machine learning to a plurality of the semantically relevant events to determine a pattern of events.
- the pattern is within a video feed.
- the pattern is across a plurality of video feeds.
- the pattern corresponds to a narrative structure.
- the narrative structure corresponds to a recurring pattern of events.
- the system includes the narrative structure relates to a sporting event.
- the pattern relates to at least one of a blow-out victory pattern, a comeback win pattern, a near comeback pattern, a back-and-forth game pattern, an individual achievement pattern, an injury pattern, a turning point moment pattern, a close game pattern, and a team achievement pattern.
- the indexed video segments are arranged to support the narrative structure.
- the arranged segments are provided in an interface for developing a story using the segments that follow the narrative structure.
- a user may at least one of edit and enter additional content for the story.
- the system includes summary content for the narrative structure is automatically generated, under computer control, to provide a story that includes the video sequences.
- the system includes delivering a plurality of the automatically generated stories at least one of from a defined time period and of a defined type, allowing a user to indicate whether they like or dislike the delivered stories, and using the indications to inform later delivery of at least one additional story.
- the pattern is relevant to a prediction.
- the prediction is related to a wager, and the pattern corresponds to similar patterns that were used to make predictions that resulted in successful wagers in other situations.
- using machine learning to develop an understanding of an event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system and the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to an event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- a method includes machine-extracting semantically relevant events from a video content stream, determining a pattern relating to the events, and providing a content stream based on the pattern.
- the content stream is used to provide coaching information based on the pattern.
- the content stream is used to assist prediction of an outcome of a fantasy sports contest.
- the pattern is used to provide content for a viewer of a sporting event.
- a method includes machine-extracting semantically relevant events from a video content stream and determining a pattern relating to the events.
- the method includes storing the pattern information with the extracted events and providing a user with the option to view and interact with the patterns. At least one of the patterns and the interaction options are personalized based on a profile of the user.
- the profile is based on at least one of user indication of a preference, information about actions of the user, and demographic information about the user.
- the pattern comprises at least one of a trend and a statistic that is curated to correspond with the user profile.
- the pattern relates to a comparison of a professional athlete to another athlete.
- the other athlete is the user and the comparison is based on a playing style of the user as determined by at least one of information indicated by the user and a video feed of the user.
- the system includes the pattern relates to an occurrence of an injury.
- the pattern information is used to provide coaching to prevent an injury.
- the system includes automatically generating, under computer control, an injury prevention regimen based on the pattern and based on information about the user.
- the system includes automatically developing a semantic index of a video feed based on the machine understanding of at least one event in the video feed to indicate the time of the event in the video feed and the location of the display of the event in the video feed.
- the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.
- the system includes providing the semantic index of the video feed with the video feed to enable augmentation of the video feed.
- augmentation of the video feed includes adding content based on to the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.
- an analytic system includes a video ingestion facility for ingesting at least one video feed and a machine learning system for processing the at least one video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on at least one event in the at least one video feed in order to develop an understanding of the at least one event within the at least one video feed.
- the understanding includes identifying at least one event type and a time of the at least one event in an event data structure.
- the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes a computing architecture enabling a model that takes an event data structures as input and applies at least one calculation to transform the event data structures into an output data structure.
- the method further includes a data transport layer of the computing architecture for populating the model with the event data structure as input to the model.
- the output data structure includes at least one prediction.
- the prediction is of an outcome of at least one of a sporting event and at least one second event occurring within a sporting event.
- the at least one video feed is of a live sporting event. The prediction is made during the live sporting event. The prediction relates to the same sporting event.
- the prediction is based on the event data structure from the at least one video feed. In embodiments, the prediction is used for at least one of placing a wager, setting a line for a wager, interacting with a fantasy program, setting a parameter of a fantasy program, providing insight to a coach and providing information to a fan.
- the system includes the model takes inputs from a plurality of data sources in addition to the event data structure obtained from the at least one video feed.
- the system includes a pattern analysis facility that takes a plurality of event data structures and enables analysis of patterns among the plurality of event data structures.
- the pattern analysis facility uses distributions over observed outcomes from the plurality of event data structures that are characterized by similar motion data.
- the user can specify use of probability distributions conditioned on multiple types of motion/position data and other data to generate metrics, pattern analysis or predictions.
- the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the system includes a second machine learning facility for refining the model based on outcomes of a plurality of predictions made using the model.
- using the machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play- by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- an analytic method includes ingesting at least one video feed in a computing platform capable of handling video data.
- the method includes processing the at least one video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on at least one event in the at least one video feed in order to develop an understanding of the at least one event within the at least one video feed.
- the understanding includes identifying at least one event type and a time of the at least one event in an event data structure.
- the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed.
- the method includes providing a computing architecture that enables a model that takes the event data structure as input and applies at least one calculation to transform the event data structure into an output data structure.
- the method further includes populating the model with the event data structure as input to the model.
- the output data structure includes at least one prediction.
- the prediction is of an outcome of at least one of a sporting event and at least one second event occurring within a sporting event.
- the video feed is of a live sporting event. The prediction is made during the live sporting event. The prediction relates to the same sporting event.
- the prediction is based on the event data structure from the plurality of video feeds.
- the prediction is used for at least one of placing a wager, setting a line for a wager, interacting with a fantasy program, setting a parameter of a fantasy program, providing insight to a coach and providing information to a fan.
- the model takes inputs from a plurality of data sources in addition to the event data structure obtained from the at least one video feed.
- the system includes, further comprising providing a pattern analysis facility that takes a plurality of event data structures and enabling analysis of patterns among the plurality of event data structures.
- the system includes the pattern analysis facility uses distributions over observed outcomes from the plurality of event data structures that are characterized by similar motion data.
- the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the system includes at least one of providing and using a second machine learning facility to refine the model based on outcomes of a plurality of predictions made using the model.
- the system includes using the machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system.
- the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the system includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the system includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- a method includes taking a video feed of a recreational event at a computing facility capable of handling video data.
- the method includes processing the video feed through a spatiotemporal pattern recognition algorithm that applies machine learning on at least one event in the at least one video feed in order to develop an understanding of the at least one event within the at least one video feed.
- the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the video feed of the recreational event.
- the method includes providing an output data structure including information about a player in the recreational event based on the understanding developed with the machine learning.
- the method includes using the understanding developed with the machine learning of the at least one event to develop at least one characteristic of the player based on the output data structure.
- the at least one characteristic is selected from the group consisting of a size of the player, a height of the player, a weight of the player, a style of the player, a pattern of play of the player, a strength of the player and a weakness of the player.
- the system includes providing a comparison of the player to at least one professional player based on a similarity with respect to the at least one characteristic.
- the recreational event is at least one of a practice, a warm-up session, and a game.
- the understanding includes identifying at least one event type.
- the method includes extracting at least one video cut from the video feed and associating the at least one event type for at least one event within the at least one video cut.
- the method includes automatically, under computer control, assembling a plurality of video cuts into a highlight reel for the player based on the event type contained in the plurality of video cuts.
- the method includes developing at least one metric based on the understanding developed with the machine learning of the at least one event and associating the metric with the at least one video cut. In embodiments, the method includes displaying the at least one metric within a video cut. In embodiments, the method includes providing a player with a personalized video cut of the recreational event that displays at least one metric associated with the player. [0160] In embodiments, the method includes providing a sports fantasy game that uses the at least one event for a player in the recreational event as input to determining an outcome of the sports fantasy game. In embodiments, the method includes using the events from the recreational event to compare the events to similar events from a professional event. In embodiments, the comparison is associated with at least one item of content for coaching the recreational player based on the comparison.
- the comparison includes displaying at least one video cut from the at least one video feed of the recreational feed and at least one video cut from a video feed of a professional event.
- the method includes the displaying of the video cuts is at least one of simultaneous and sequential.
- the method includes using the events understood from the recreational event and similar events understood from a video feed of a professional event to produce a highlight reel that features both recreational and professional video cuts of similar events.
- the method includes enabling a user to at least one of edit, cut, mix, and share at least one video clip of the recreational event that is identified based on the output data structure.
- the method includes providing a pattern analysis facility that takes a plurality of the output data structures and enables analysis of patterns among the output data structures.
- the method includes the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the method includes the pattern analysis facility supports the ability to recognize patterns in recreational player activity and draw comparisons to other players.
- the method includes providing physical feedback to a player based on real time recognition of a pattern.
- the physical feedback is at least one of a lighting feedback, an audio feedback, a haptic feedback, and a visual feedback.
- the method includes developing at least one metric for the recreational player based on recognizing at least one pattern of events.
- the at least one metric is used to compare the player to another player.
- the method includes using the machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system. The understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using the machine learning to develop the understanding of an event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators.
- the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-byplay data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method includes at least one of validating and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- a method includes taking a video feed of a recreational event at a computing facility capable of handling video data and using machine learning to develop an understanding of at least one event within the video feed.
- the method includes providing an output data structure including information about a player in the recreational event based on the machine understanding.
- the method further includes using the machine learning understanding of the at least one event to develop at least one characteristic of the player based on the output data structure.
- the characteristic is selected from the group consisting of a size of the player, a height of the player, a weight of the player, a style of the player, a pattern of play of the player, a strength of the player and a weakness of the player.
- the method further includes providing a comparison of the player to at least one professional player based on a similarity with respect to the at least one characteristic.
- the recreational event is at least one of a practice, a warm-up session, and a game.
- the machine learning understanding provides at least one type for the event.
- the method further includes extracting at least one video cut from the video feed and associating the at least one type for at least one event within the at least one video cut.
- the method further includes automatically, under computer control, assembling a plurality of video cuts into a highlight reel for the player based on the type of events contained in the video cuts.
- the method further includes developing at least one metric based on the machine understanding of the at least one event and associating the metric with the at least one video cut.
- the method further includes displaying the at least one metric within a video cut.
- the method further includes providing a player with a personalized video cut of the recreational event that displays at least one metric associated with the player.
- the method further includes providing a sports fantasy game that uses the events for a player in the recreational event as input to determining an outcome of the sports fantasy game.
- the method further includes using the events from the recreational event to compare the events to similar events from a professional event.
- the comparison is associated with at least one item of content for coaching the recreational player based on the comparison.
- the comparison includes displaying at least one video cut from the at least one video feed of the recreational feed and at least one video cut from a video feed of a professional event.
- the displaying of the video cuts is at least one of simultaneous and sequential.
- the method further includes using the events understood from the recreational event and similar events understood from a video feed of a professional event to produce a highlight reel that features both recreational and professional video cuts of similar events.
- the method further includes enabling a user to at least one of edit, cut, mix, and share at least one video clip of the recreational event that is identified based on the output data structure.
- the method further includes providing a pattern analysis facility that takes a plurality of the output data structures and enables analysis of patterns among the output data structures.
- the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the pattern analysis facility supports the ability to recognize patterns in recreational player activity and draw comparisons to other players.
- the method further includes providing physical feedback to a player based on real time recognition of a pattern.
- the physical feedback is at least one of a lighting feedback, an audio feedback, a haptic feedback, and a visual feedback.
- the method further includes developing at least one metric for the recreational player based on recognizing at least one pattern of events.
- the method further includes the at least one metric is used to compare the player to another player.
- using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.
- using machine learning to develop an understanding of an event further comprises aligning multiple unsynchronized input feeds related to the events using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.
- the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.
- the method further includes at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- the method further includes at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.
- a method includes taking a video feed of an event in which there is a participant and identifying a point of view of the participant.
- the method includes automatically, under computer control, selecting a plurality of video frames from the video feed showing at least one view from the point of view of the participant.
- the methods and systems provided herein may include encoding a plurality of video feeds with a video segment generation facility that produces a plurality of encoded video segment files, each encoded video segment file corresponding to a different time interval of at least one of the plurality of video feeds.
- the methods and systems include grouping encoding video segment files from different video feeds (e.g., feeds from different angles, feeds from with different lens, depth of field, and the like, feeds are different frame rates, or the like) into temporal groups that share a common time interval and generating at least one augmented video segment file for each of the video segment files in at least one of the temporal groups by processing each of the video segment files in the at least one of the temporal groups in parallel.
- the methods and systems further include presenting availability and access information that permits switching between at least two of the encoded video segment files, augmented video segment files, or combinations thereof for each time interval to a video playback client.
- the methods and systems further include presenting availability and access information to a video playback client that permits switching between one of at least two encoded video segment files, at least two augmented video segment files, and at least one of the encoded video segment files and one of the augmented video segment files for each time interval.
- grouping the video segment files includes grouping at least one of player tracking information, physiological monitoring information, activity type, play-by-play information, statistical information, and event information for a common time interval with the grouped video segment feeds from different video feeds for the common time interval.
- generating the at least one augmented video segment file includes adding at least one of graphics, audio, and player tracking data based on semantic analysis of the at least one augmented video segment file.
- the video playback client is configured to select among the availability and access information associated with at least one of the encoded video segment file and the augmented video segment file based on at least the availability and access information, an amount of video playback buffering available, and a semantic understanding of events in the video feed.
- the video playback client is configured to switch without temporal interruption among the at least one augmented video segment file for each of the video segment files in the at least one of the temporal groups.
- the semantic understanding of events includes at least one of a change in possession, a timeout, a change in camera angle, and a change in point-of-view.
- the methods and systems further include presenting multiple aligned content channels that are indexed spatially and temporally to the video playback client.
- the multiple aligned content channels include at least three content types selected from the list consisting of video feed content, video segment file content, augmented video segment file content, audio content, temporal group content, and player tracking data.
- the methods and systems provided herein may include a method of active quality assurance of a machine learning including producing live and automatic machine learning classification of live spatiotemporal data by applying a machine learning algorithm that receives as input human responses in a quality assurance process regarding a semantic understanding of at least one event of a sequence of events in a video feed and that produces a time-delayed classification output of neighboring events.
- the methods and systems include aligning a plurality of data sources related to the sequence of events based on generalized distance metrics derived from at least two sequences of event attributes.
- the event attributes are selected from a group consisting of an event duration, an event order, an event addition, an event deletion, and an event spatial distance.
- the time-delayed classification output includes a time-delay that corresponds to a length of time of the quality assurance process from presentation of a quality assurance question to receipt of at least one human response.
- the method includes rendering a 3D video using the selected plurality of video frames.
- the method further includes providing an interface wherein a user may select the participant from a plurality of participants.
- the event is a sporting event and the participant is a player.
- the method further includes the event is basketball and wherein the video feed is associated with 3D motion capture data from cameras that capture the video feed.
- Fig. 1 illustrates a technology stack according to an exemplary and non- limiting embodiment.
- Fig. 2 illustrates a stack flow according to an exemplary and non- limiting embodiment.
- FIG. 3 illustrates an exploration loop according to an exemplary and non- limiting embodiment.
- Fig. 4 illustrates ranking user interface according to an exemplary and non- limiting embodiment.
- Figs. 5A and 5B illustrate a ranking user interface according to an exemplary and non- limiting embodiment.
- Figs. 6A and 6B illustrate a filters user interface according to an exemplary and non- limiting embodiment.
- Fig. 7 illustrates a breakdown user interface according to an exemplary and non- limiting embodiment.
- Fig. 8 illustrates a breakdown user interface according to an exemplary and non- limiting embodiment.
- Fig. 9 illustrates a personalized user interface according to an exemplary and non- limiting embodiment.
- Fig. 10 illustrates an alternative video user interface according to an exemplary and non-limiting embodiment.
- Fig. 11 illustrates an alternative report according to an exemplary and non-limiting embodiment.
- Fig. 12 illustrates a court comparison view according to an exemplary and non-limiting embodiment.
- Fig. 13 illustrates a court view according to an exemplary and non- limiting embodiment.
- Fig. 14 illustrates a report according to an exemplary and non-limiting embodiment.
- FIG. 15 illustrates a detailed depiction of a game according to an exemplary and non- limiting embodiment.
- Fig. 16 illustrates querying and aggregation according to an exemplary and non- limiting embodiment.
- Fig. 17 illustrates a hybrid classification process flow according to an exemplary and non-limiting embodiment.
- Fig. 18 illustrates test inputs according to an exemplary and non-limiting embodiment.
- Fig. 19 illustrates test inputs according to an exemplary and non-limiting embodiment.
- Fig. 20 illustrates player detection according to an exemplary and non-limiting embodiment.
- Fig. 21 illustrates a process flow according to an exemplary and non- limiting embodiment.
- Fig. 22 illustrates rebounding according to an exemplary and non-limiting embodiment.
- Fig. 23 illustrates scatter rank according to an exemplary and non-limiting embodiment.
- Figs. 24A and 24B illustrate reports according to an exemplary and non- limiting embodiment.
- Fig. 25 illustrates a quality assurance user interface according to an exemplary and non-limiting embodiment.
- Fig. 26 illustrates a quality assurance user interface according to an exemplary and non-limiting embodiment.
- Fig. 27 illustrates camera pose detection according to an exemplary and non- limiting embodiment.
- Fig. 28 illustrates camera pose detection according to an exemplary and non-limiting embodiment.
- Fig. 29 illustrates auto-rotoscoping according to an exemplary and non-limiting embodiment.
- Figs. 30A, 30B, and 30C illustrate scripted storytelling with assets according to an exemplary and non-limiting embodiment.
- Fig. 31 illustrates an example according to an exemplary and non- limiting embodiment.
- Fig. 32 illustrates an example according to an exemplary and non- limiting embodiment.
- Fig. 33 illustrates an example according to an exemplary and non- limiting embodiment.
- Fig. 34 illustrates an example according to an exemplary and non- limiting embodiment.
- Fig. 35 illustrates an example according to an exemplary and non-limiting embodiment.
- Fig. 36 illustrates an example according to an exemplary and non-limiting embodiment.
- Fig. 37 illustrates an example according to an exemplary and non-limiting embodiment.
- Fig. 38 illustrates a screen shot according to an exemplary and non- limiting embodiment.
- Figs. 39A, 39B, 39C, 39D, and 39E illustrate a screen shot according to an exemplary and non-limiting embodiment.
- Fig. 39F illustrates a first person process according to an exemplary and non-limiting embodiment.
- Fig. 40 illustrates a screen shot according to an exemplary and non-limiting embodiment.
- Figs. 41A and 41B illustrate a screen shot according to an exemplary and non-limiting embodiment.
- Figs. 42A, 42B, and 42C illustrate a screen shot according to an exemplary and non- limiting embodiment.
- Fig. 43 illustrates a screen shot according to an exemplary and non-limiting embodiment.
- Fig. 44 illustrates a process flow according to an exemplary and non-limiting embodiment.
- Fig. 45 illustrates systems and methods for parallel processing, synchronization, and failover across multiple streams of a live event according to an exemplary and non-limiting embodiment.
- Fig. 46 illustrates systems and methods for player-specific information in three- dimensional position identification and reconstruction in accordance with the various embodiments.
- Fig. 47 illustrates systems and methods for a machine learning model including active learning and active quality assurance in accordance with the various embodiments.
- Fig. 1 illustrates a technology stack 100 indicative of technology layers configured to execute a set of capabilities, in accordance with an embodiment of the present invention.
- the technology stack 100 may include a customization layer 102, an interaction layer 104, a visualizations layer 108, an analytics layer 110, a patterns layer 112, an events layer 114, and a data layer 118, without limitations.
- the different technology layers or the technology stack 100 may be referred to as an "Eagle" Stack 100, which should be understood to encompass the various layers allow precise monitoring, analytics, and understanding of spatiotemporal data associated with an event, such as a sports event and the like.
- the technology stack may provide an analytic platform that may take spatiotemporal data (e.g., 3D motion capture "XYZ” data) from National Basketball Association (NBA) arenas or other sports arenas and, after cleansing, may perform spatiotemporal pattern recognition to extract certain "events".
- the extracted events may be for example (among many other possibilities) events that correspond to particular understandings of events within the overall sporting event, such as "pick and roll” or "blitz.”
- Such events may correspond to real events in a game, and may, in turn, be subject to various metrics, analytic tools, and visualizations around the events.
- Event recognition may be based on pattern recognition by machine learning, such as spatiotemporal pattern recognition, and in some cases, may be augmented, confirmed, or aided by human feedback.
- the customization layer 102 may allow performing custom analytics and interpretation using analytics, visualization, and other tools, as well as optional crowd-sourced feedback for developing team-specific analytics, models, exports, and related insights. For example, among many other possibilities, the customization layer 102 may facilitate in generating visualizations for different spatiotemporal movements of a football player, or group of players and counter movements associated with other players or groups of players during a football event.
- the interaction layer 104 may facilitate generating real-time interactive tasks, visual representations, interfaces, videos clips, images, screens, and other such vehicles for allowing viewing of an event with enhanced features or allowing interaction of a user with a virtual event derived from an actual real-time event.
- the interaction layer 104 may allow a user to access features or metrics such as a shot matrix, a screens breakdown, possession detection, and many others using real-time interactive tools that may slice, dice, and analyze data obtained from the real-time event such as a sports event.
- the visualizations layer 108 may allow dynamic visualizations of patterns and analytics developed from the data obtained from the real-time event.
- the visualizations may be presented in the form of a scatter rank, shot comparisons, a clip view, and many others.
- the visualizations layer 108 may use various types of visualizations and graphical tools for creating visual depictions.
- the visuals may include various types of interactive charts, graphs, diagrams, comparative analytical graphs, and the like.
- the visualizations layer 108 may be linked with the interaction layer so that the visual depictions may be presented in an interactive fashion for a user interaction with real-time events produced on a virtual platform such as the analytic platform of the present invention.
- the analytics layer 110 may involve various analytics and Artificial Intelligence (AI) tools to perform analysis and interpretation of data retrieved from the real-time event such as a sports event so that the analyzed data results in insights that make sense out of the pulled big data from the real-time event.
- AI Artificial Intelligence
- the analytics and AI tools may comprise such as search and optimization tools, inference rules engines, algorithms, learning algorithms, logic modules, probabilistic tools and methods, decision analytics tools, machine learning algorithms, semantic tools, expert systems, and the like without limitations.
- Output from the analytics 110 and patterns layers 112 is exportable by the user as a database that enables the customer to configure their own machines to read and access the events and metrics stored in the system.
- patterns and metrics are structured and stored in an intuitive way.
- the database utilized for storing the events and metric data is designed to facilitate easy export and to enable integration with a team's internal workflow.
- types of events that may be recorded for a basketball game include, but are not limited to, isos, handoffs, posts, screens, transitions, shots, closeouts, and chances.
- table 1 is an exemplary listing of the data structure for storing information related to each occurrence of a screen. As illustrated, each data type is comprised of a plurality of component variable definitions each comprised of a data type and a description of the variable. screens
- Frame ID denoting frame number from the start of the current period. Currently, this marks the frame at which the screener and ballhandler are closest.
- Time stamp provided in Sport VU data for a frame measured in milliseconds in the current epoch (i.e., from 00:00:00 UTC on 1 January 1970).
- ID of team on offense matches IDs in Sport VU data.
- ID of team on defense matches IDs in Sport VU data.
- STRING String representing the observed actions of the ballhandler's defender.
- Actions by the ballhandler taken from the outcomes described at the end of the document, such as FGX or FGM.
- Actions by the screener taken from the outcomes described at the end of the document, such as FGX or FGM.
- the patterns layer 112 may provide a technology infrastructure for rapid discovery of new patterns arising out of the retrieved data from the real-time event such as a sports event.
- the patterns may comprise many different patterns that corresponding to an understanding of the event, such as a defensive pattern (e.g., blitz, switch, over, under, up to touch, contain-trap, zone, man-to-man, or face-up pattern), various offensive patterns (e.g., pick-and-roll, pick-and-pop, horns, dribble-drive, off-ball screens, cuts, post-up, and the like), patterns reflecting plays (scoring plays, three-point plays, "red zone” plays, pass plays, running plays, fast break plays, etc.) and various other patterns associated with a player in the game or sports, in each case corresponding to distinct spatiotemporal events.
- a defensive pattern e.g., blitz, switch, over, under, up to touch, contain-trap, zone, man-to-man, or face-up pattern
- the events layer 114 may allow creating new events or editing or correcting current events.
- the events layer may allow for the analyzing of the accuracy of markings or other game definitions and may comment on whether they meet standards and sports guidelines. For example, specific boundary markings in an actual real-time event may not be compliant with the guidelines and there may exist some errors, which may be identified by the events layers through analysis and virtual interactions possible with the platform of the present invention.
- Events may correspond to various understandings of a game, including offensive and defensive plays, matchups among players or groups of players, scoring events, penalty or foul events, and many others.
- the data layer 118 facilitates management of the big data retrieved from the real-time event such as a sports event.
- the data layer 118 may allow creating libraries that may store raw data, catalogs, corrected data, analyzed data, insights, and the like.
- the data layer 118 may manage online warehousing in a cloud storage setup or in any other manner in various embodiments.
- Fig. 2 illustrates a process flow diagram 200, in accordance with an embodiment of the present invention.
- the process 200 may include retrieving spatiotemporal data associated with a sports or game and storing in a data library at step 202.
- the spatiotemporal data may relate to a video feed that was captured by a 3D camera, such as one positioned in a sports arena or other venue, or it may come from another source.
- the process 200 may further include cleaning of the rough spatiotemporal data at step 204 through analytical and machine learning tools and utilizing various technology layers as discussed in conjunction with Fig. 1 so as to generate meaningful insights from the cleansed data.
- the process 200 may further include recognizing spatiotemporal patterns through analysis of the cleansed data at step 208.
- Spatiotemporal patterns may comprise a wide range of patterns that are associated with types of events. For example, a particular pattern in space, such as the ball bouncing off the rim, then falling below it, may contribute toward recognizing a "rebound" event in basketball. Patterns in space and time may lead to recognition of single events or multiple events that comprise a defined sequence of recognized events (such as in types of plays that have multiple steps).
- the recognized patterns may define a series of events associated with the sports that may be stored in an event datastore at step 210. These events may be organized according to the recognized spatiotemporal patterns; for example, a series of events may have been recognized as "pick,” “rebound,” “shot,” or like events in basketball, and they may be stored as such in the event datastore 210.
- the event datastore 210 may store a wide range of such events, including individual patterns recognized by spatiotemporal pattern recognitions and aggregated patterns, such as when one pattern follows another in an extended, multi-step event (such as in plays where one event occurs and then another occurs, such as "pick and roll” or "pick and pop” events in basketball, football events that involve setting an initial block, then springing out for a pass, and many others).
- the process 200 may further include querying or aggregation or pattern detection at step 212.
- the querying of data or aggregation may be performed with the use of search tools that may be operably and communicatively connected with the data library or the events datastore for analyzing, searching, aggregating the rough data, cleansed, or analyzed data, or events data or the events patterns.
- metrics and actionable intelligence may be used for developing insights from the searched or aggregated data through artificial intelligence and machine learning tools.
- the metrics and actionable intelligence may convert the data into interactive visualization portals or interfaces for use by a user in an interactive manner.
- an interactive visualization portal or interface may produce a 3D reconstruction of an event, such as a game.
- a 3D reconstruction of a game may be produced using a process that presents the reconstruction from a point of view, such as a first person point of view of a participant in an event, such as a player in a game.
- Raw input XYZ data obtained from various data sources is frequently noisy, missing, or wrong.
- XYZ data is sometimes delivered with attached basic events already identified in it, such as possession, pass, dribble, and shot events; however, these associations are frequently incorrect. This is important because event identification further down the process (in Spatiotemporal Pattern Recognition) sometimes depends on the correctness of these basic events. For example, if two players' XY positions are switched, then "over” vs "under” defense would be incorrectly characterized, since the players' relative positioning is used as a critical feature for the classification. Even player-by-player data sources are occasionally incorrect, such as associating identified events with the wrong player.
- Possession / Non-possession models may use a Hidden Markov Model to best fit the data to these states. Shots and rebounds may use the possession model outputs, combined with 1) projected destination of the ball, and 2) player by player information (PBP) information. Dribbles may be identified using a trained ML algorithm and also using the output of the possession model. These algorithms may decrease the basic event labeling error rate by approximately 50% or more.
- the system has a library of anomaly detection algorithms to identify potential problems in the data including, but not limited to, temporal discontinuities (intervals of missing data are flagged), spatial discontinuities (objects traveling is a non-smooth motion, "jumping") and interpolation detection (data that is too smooth, indicating that post-processing was done by the data supplier to interpolate between known data points in order to fill in missing data).
- This problem data is flagged for human review so that events detected during these periods are subject to further scrutiny.
- Spatiotemporal Pattern Recognition 208 is used to automatically identify relationships between physical and temporal patterns and various types of events.
- one challenge is how to turn x, y, z positions of ten players and one ball at twenty-five frames per second into usable input for machine learning and pattern recognition algorithms.
- the raw inputs may not suffice.
- the instances within each pattern category can look very different from each other.
- Features that relate multiple actors in time are key components to the input.
- Examples include, but are not limited to, the motion of player one (PI) towards player two (P2), for at least T seconds, a rate of motion of at least V m/s for at least T seconds and at the projected point of intersection of paths A and B, and a separation distance less than D.
- an algorithm for spatiotemporal pattern recognition can use relative motion of visible features within a feed, duration of relative motion of such features, rate of motion of such features with respect to each other, rate of acceleration of such features with respect to each other, a projected point of intersection of such features, the separation distance of such features, and the like to identify or recognize a pattern with respect to visible features in a feed, which in turn can be used for various other purposes disclosed herein, such as recognition of a semantically relevant event or feature that relates to the pattern.
- these factors may be based on a pre-existing model or understanding of the relevance of such features, such as where values or thresholds may be applied within the pattern recognition algorithm to aid pattern recognition.
- thresholds or values may be applied to rates of motion, durations of motion, and the like to assist in pattern recognition.
- pattern recognition may occur by adjusting weights or values of various input features within a machine learning system, without a pre-existing model or understanding of the significance of particular values and without applying thresholds or the like.
- the spatiotemporal pattern recognition algorithm may be based on at least one pattern recognized by adjusting at least one of an input type and a weight within a machine learning system. This recognition may occur independently of any a priori model or understanding of the significance of particular input types, features, or characteristics.
- an input type may be selected from the group consisting of relative direction of motion of at least two visible features, duration of relative motion of visible features with respect to each other, rate of motion of at least two visible features with respect to each other, acceleration of motion of at least two visible feature with respect to each other, projected point of intersection of at least two visible features with respect to each other and separation distance between at least two visible features with respect to each other, and the like.
- a library of such features involving multiple actors over space and time there is provided a library of such features involving multiple actors over space and time.
- the library may include relationships between actors (e.g., players one through ten in basketball), relationships between the actors and other objects such as the ball, and relationships to other markers, such as designated points and lines on the court or field, and to projected locations based on predicted motion.
- Another key challenge is there has not been a labeled dataset for training the ML algorithms.
- a labeled dataset may be used in connection with various embodiments disclosed herein. For example, there has previously been no XYZ player-tracking dataset that already has higher level events, such as pick and roll (P&R) events) labeled at each time frame they occur. Labeling such events, for many different types of events and sub-types, is a laborious process. Also, the number of training examples required to adequately train the classifier may be unknown. One may use a variation of active learning to solve this challenge.
- P&R pick and roll
- the machine finds an unlabeled example that is closest to the boundary between As and Bs in the feature space. The machine then queries a human operator/labeler for the label for this example. It uses this labeled example to refine its classifier and then repeats.
- the system also incorporates human input in the form of new features. These features are either completely devised by the human operator (and inputted as code snippets in the active learning framework), or they are suggested in template form by the framework.
- the templates use the spatiotemporal pattern library to suggest types of features that may be fruitful to test. The operator can choose a pattern, and test a particular instantiation of it, or request that the machine test a range of instantiations of that pattern.
- Some features are based on outputs of the machine learning process itself. Thus, multiple iterations of training are used to capture this feedback and allow the process to converge. For example, a first iteration of the ML process may suggest that the Bulls tend to ice the P&R. This fact is then fed into the next iteration of ML training as a feature, which biases the algorithm to label Bulls' P&R defense as ices. The process converges after multiple iterations. In practice, two iterations have typically been sufficient to yield good results.
- a canonical event datastore 210 may contain a definitive list of events that the system knows occurred during a game. This includes events extracted from the XYZ data, as well as those specified by third-party sources, such as PBP data from various vendors. The events in the canonical event datastore 210 may have game clock times specified for each event. The datastore 210 may be fairly large. To maintain efficient processing, it is shared and stored in-memory across many machines in the cloud.
- data is divided into small enough shards that each worker shard has a low latency response time.
- Each distributed machine may have multiple workers corresponding to the number of processes the machine can support concurrently.
- Query results never rely on more than one shard, since we enforce that events never cross quarter/period boundaries.
- Aggregation functions all run incrementally rather than in batch process so that as workers return results, these are incorporated into the final answer immediately.
- the aggregator uses hashes to keep track of the separate rows and incrementally updates them.
- an exploration loop may be enabled by the methods and systems disclosed herein, where questioning and exploration can occur, such as using visualizations (e.g., data effects, referred to as DataFX in this disclosure), processing can occur, such as to identify new events and metrics, and understanding emerges, leading to additional questions, processing and understanding.
- visualizations e.g., data effects, referred to as DataFX in this disclosure
- processing can occur, such as to identify new events and metrics, and understanding emerges, leading to additional questions, processing and understanding.
- the present disclosure provides an instant player rankings feature as depicted in the illustrated user interface.
- a user can select among various types of available rankings 402, as indicated in the drop down list 410, such as rankings relating to shooting, rebounding, rebound ratings, isolations (Isos), picks, postups, handoffs, lineups, matchups, possessions (including metrics and actions), transitions, plays and chances.
- Rankings can be selected in a menu element 404 for players, teams, or other entities.
- Rankings can be selected for different types of play in the menu element 408, such as for offense, defense, transition, special situations, and the like.
- the ranking interface allows a user to quickly query the system to answer a particular question instead of thumbing through pages of reports.
- the user interface lets a user locate essential factors and evaluate talent of a player to make more informed decisions.
- Figs. 5A and 5B shows certain basic, yet quite in-depth, pages in the systems described herein, referred to in some cases as the "Eagle system.”
- This user interface may allow the user to rank players and teams by a wide variety of metrics. This may include identified actions, metrics derived from these actions, and other continuous metrics. Metrics may relate to different kinds of events, different entities (players and teams), different situations (offense and defense) and any other patterns identified in the spatiotemporal pattern recognition system.
- Examples of items on which various entities can be ranked in the case of basketball include chances, charges, closeouts, drives, frequencies, handoffs, isolations, lineups, matches, picks, plays, possessions, postups, primary defenders, rebounding (main and raw), off ball screens, shooting, speed/load and transitions.
- the Rankings UI makes it easy for a user to understand relative quality of one row item versus other row items, along any metric.
- Each metric may be displayed in a column, and that row's ranking within the distribution of values for that metrics may be displayed for the user.
- Color coding makes it easy for the user to understand relative goodness.
- Figs. 6A and 6B show a set of filters in the UI, which can be used to filter particular items to obtain greater levels of detail or selected sets of results. Filters may exist for seasons, games, home teams, away teams, earliest and latest date, postseason/regular season, wins/losses, offense home/away, offensive team, defensive team, layers on the court for offense/defense, players off court for offense/defense, locations, offensive or defensive statistics, score differential, periods, time remaining, after timeout play start, transition/no transition, and various other features.
- the filters 602 for offense may include selections for the ballhandler, the ballhandler position, the screener, the screener position, the ballhandler outcome, the screener outcome, the direction, the type of pick, the type of pop/roll, the direction of the pop/roll, and presence of the play (e.g., on the wing or in the middle).
- Many other examples of filters are possible, as a filter can exist for any type of parameter that is tracked with respect to an event that is extracted by the system or that is in the spatiotemporal data set used to extract events.
- the present disclosure also allows situational comparisons.
- the user interface allows a user to search for a specific player that may fit into the offense.
- the highly accurate dataset and easy to use interface allow the user to compare similar players in similar situations.
- the user interface may allow the user to explore player tendencies.
- the user interface may allow locating shot locations and also may provide advanced search capabilities.
- Filters enable users to subset the data in a large number of ways and immediately receive metrics calculated on the subset. Using multiple loops for convergence in machine learning enables the system to return the newly filtered data and metrics in real-time, whereas existing methods would require minutes to re-compute the metrics given the filters, leading to inefficient exploration loops (Fig. 3). Given that the data exploration and investigation process often requires many loops, these inefficiencies can otherwise add up quickly.
- filters that may enable a user to select specific situations of interest to analyze. These filters may be categorized into logical groups, including, but not limited to, Game, Team, Location, Offense, Defense, and Other. The possible filters may automatically change depending on the type of event being analyzed, for example, Shooting, Rebounding, Picks, Handoffs, Isolations, Postups, Transitions, Closeouts, Charges, Drives, Lineups, Matchups, Play Types, Possessions.
- filters may include Season, specific Games, Earliest Date, Latest Date, Home Team, Away Team, where the game is being played Home/ Away, whether the outcome was Wins/Losses, whether the game was a Playoff game, and recency of the game.
- filters may include Offensive Team, Defensive Team, Offensive Players on Court, Defenders Players on Court, Offensive Players Off Court, Defenders Off Court.
- the user may be given a clickable court map that is segmented into logical partitions of the court. The user may then select any number of these partitions in order to filter only events that occurred in those partitions.
- the filters may include Score Differential, Play Start Type (Multi-Select: Field Goal ORB, Field Goal DRB, Free Throw ORB, Free Throw DRB, Jump Ball, Live Ball Turnover, Defensive Out of Bounds, Sideline Out of Bounds), Periods, Seconds Remaining, Chance After Timeout (T/F/ALL), Transition (T/F/ALL).
- Play Start Type Multi-Select: Field Goal ORB, Field Goal DRB, Free Throw ORB, Free Throw DRB, Jump Ball, Live Ball Turnover, Defensive Out of Bounds, Sideline Out of Bounds
- Periods Seconds Remaining, Chance After Timeout (T/F/ALL), Transition (T/F/ALL).
- the filters may include Shooter, Position, Outcome (Made/Missed/ All), Shot Value, Catch and Shoot (T/F/ALL), Shot Distance, Simple Shot Type (Multi-Select: Heave, Angle Layup, Driving Layup, Jumper, Post), Complex Shot Type (Multi-Select: Heave, Lob, Tip, Standstill Layup, Cut Layup, Driving Layup, Floater, Catch and Shoot), Assisted (T/F/ALL), Pass From (Player), Blocked (T/F/ALL), Dunk (T/F/ALL), Bank (T/F/ALL), Goaltending (T/F/ALL), Shot Attempt Type (Multi-select: FGA No Foul, FGM Foul, FGX Foul), Shot SEFG (Value Range), Shot Clock (Range), Previous Event (Multi-Select: Transition, Pick, Isolation, Handoff, Post, None).
- Simple Shot Type Multi-Select: Heave, Angle Layup, Driving Layup, Jumper, Post
- the filters may include Defender Position (Multi-Select: PG, SG, SF, PF, CTR), Closest Defender, Closest Defender Distance, Blocked By, Shooter Height Advantage.
- the filters may include Ballhandler, Ballhandler Position, Screener, Screener Position, Ballhandler Outcome (Pass, Shot, Foul, Turnover), Screener Outcome (Pass, Shot, Foul, Turnover), Direct or Indirect Outcome, Pick Type (Reject, Slip, Pick), Pop/Roll, Direction, Wing/Middle, Middle/Wing/Step-Up.
- the filters may include Ballhandler Defender, Ballhandler Defender Position, Screener Defender, Screener Defender Position, Ballhandler Defense Type (Over, Under, Blitz, Switch, Ice), Screener Defense Type (Soft, Show, Ice, Blitz, Switch), Ballhandler Defense (Complex) (Over, Under, Blitz, Switch, Ice, Contain Trap, Weak), Screener Defense (Complex) (Over, Under, Blitz, Switch, Ice, Contain Trap, Weak, Up to Touch).
- the filters may include Ballhandler, Ballhandler Position, Ballhandler Outcome, Direct or Indirect, Drive Category (Handoff, Iso, Pick, Closeout, Misc.), Drive End (Shot Near Basket, Pullup, Interior Pass, Kickout, Pullout, Turnover, Stoppage, Other), Direction, Blowby (T/F).
- the filters may include Ballhandler Defender, Ballhandler Defender Position, Help Defender Present (T/F), Help Defenders.
- the filters may include Ballhandler, Ballhandler Position, Ballhandler Outcome, Direct or Indirect.
- the filters may include Ballhandler Defender, Ballhandler Defender Position.
- the filters may additionally include Area (Left, Right, Middle).
- the filters may additionally include Double Team (T/F).
- the present disclosure provides detailed analysis capabilities, such as through the depicted user interface embodiment of Fig. 7.
- the user interface may be used to know if a player should try and ice the pick and roll or not between two players. Filters can go from all picks, to picks involving a selected player as ballhandler, to picks involving that ballhandler with a certain screener, to the type of defense played by that screener. By filtering down to particular matchups (by player combinations and actions taken), the system allows rapid exploration of the different options for coaches and players, and selection of preferred actions that had the best outcomes in the past.
- the system may give a detailed breakdown of a player' s opponent and a better idea of what to expect during a game.
- the user interface may be used to know and highlight opponent capabilities.
- a breakdowns UI may make it easy for a user to drill down to a specific situation, all while gaining insight regarding frequency and efficacy of relevant slices through the data.
- the events captured by the present system may be capable of being manipulated using the UI.
- Fig. 8 shows a visualization, where a drop-down feature 802 allows a user to select various parameters related to the ballhandler, such as to break down to particular types of situations involving that ballhandler. These types of "breakdowns" facilitate improved interactivity with video data, including enhanced video data created with the methods and systems disclosed herein.
- each column represents a variable for partitioning the dataset. It is easy for a user to add, remove, and re-arrange columns by clicking and dragging. This makes it easy to experiment with different visualizations.
- a video icon takes a user to video clips of the set of plays that correspond to a given partition. Watching the video gives the user ideas for other variables to use for partitioning.
- Various interactive visualizations may be created to allow users to better understand insights that arise from the classification and filtering of events, such as ones that emphasize color coding for easy visual inspection and detection of anomalies (e.g., a generally good player with lots of orange but is bad/blue in one specific dimension).
- anomalies e.g., a generally good player with lots of orange but is bad/blue in one specific dimension.
- most standard visualizations are static images.
- interactivity enables the user to explore the data, ask new questions, get new answers.
- a breakdown view may be color coded good (orange) to bad (blue) for easy understanding without reading the numbers. Sizes of partitions may denote the frequency of events. Again, one can comprehend from a glance the events that occur most frequently.
- Each column of a visualization may represent a variable for partitioning the dataset. It may be easy to add, remove, and re-arrange columns by clicking and dragging. This makes it easy to experiment with possible visualizations.
- a video icon may take a user to video clips, such as of the set of plays that correspond to that partition. Watching the video gives the user ideas for other variables to use for partitioning.
- a ranking view is provided. Upon moussing over each row of a ranking view, histograms above each column may give the user a clear contextual understanding that row' s performance for each column variable. The shape of a distribution is often informative. Color-coded bars within each cell may also provide a view of each cell's performance that is always available, without moussing over. Alternatively, the cells themselves may be color-coded.
- the system may provide a personalized video in embodiments of the methods and systems described herein. For example, with little time to scout the opposition, the system can provide a user relevant information to quickly prepare the team. The team may rapidly retrieve the most meaningful plays, cut, and compiled to specific needs of players. The system may provide immediate video cut-ups.
- the present disclosure provides a video that is synchronized with identified actions. For example, if spatiotemporal machine learning identifies a segment of a video as showing a pick and roll involving two players, then that video segment may be tagged, so that when that event is found (either by browsing or by filtering to that situation), the video can be displayed.
- a user-customizable segment of video can be created. For example, the user can retrieve video corresponding to x seconds before, and y seconds after, each event occurrence. Thus, the video may be tagged and associated with events.
- the present disclosure may provide a video that may allow customization by numerous filters of the type disclosed above, relating to finding a video that satisfies various parameters, that displays various events, or combinations thereof.
- an interactive interface provided by the present disclosure allows watching videos clips for specific game situations or actions.
- Reports may provide a user with easy access to printable pages summarizing pre-game information about an opponent, scouting report for a particular player, or a post-game summary.
- the reports may collect actionable useful information in one to two easy-to-digest pages. These pages may be automatically scheduled to be sent to other staff members, e.g., post- game reports sent to coaches after each game.
- a report may include statistics for a given player, as well as visual representations, such as of locations 1102 where shots were taken, including shots of a particular type (such as catch and shoot shots).
- the UI as illustrated in Fig. 12 provides a court comparison view 1202 among several parts of a sports court (and can be provided among different courts as well).
- filters 1204 may be used to select the type of statistic to show for a court.
- the statistics can be filtered to show results filtered by left side 1208 or right side 1214. Where the statistics indicate an advantage, the advantages can be shown, such as of left side advantages 1210 and right side advantages 1212.
- a four court comparison view 1202 is a novel way to compare two players, two teams, or other entities, to gain an overview view of each player/team (Leftmost and Rightmost figures) 1208, 1214 and understand each one's strengths/weaknesses (Left and Right Center figures 1210, 1212).
- the court view UI 1302 as illustrated in Fig. 13 provides a court view 1304 of a sport arena 1304, in accordance with an embodiment of the present disclosure. Statistics for very specific court locations can be presented on a portion 1308 of the court view.
- the UI may provide a view of custom markings, in accordance with an embodiment of the present invention.
- filters may enable users to subset the data in a large number of ways, and immediately receive metrics calculated on the subset. Descriptions of particular events may be captured and made available to users.
- Various events may be labeled in a game, as reflected in Fig. 15, which provides a detailed view of a timeline 1502 of a game, broken down by possession 1504, by chances 1508, and by specific events 1510 that occurred along the timeline 1502, such as determined by spatiotemporal pattern recognition, by human analysis, or by a combination of the two.
- Filter categories available by a user interface of the present disclosure may include ones based on seasons, games, home teams, away teams, earliest date, latest date, postseason/regular season, wins/losses, offense home/away, offensive team, defensive team, players on the court for offense/defense, players off court for offense/defense, location, score differential, periods, time remaining, play type (e.g., after timeout play) and transition/no transition.
- Events may include ones based on primitive markings, such as shots, shots with a corrected shot clock, rebounds, passes, possessions, dribbles, and steals, and various novel event types, such as SEFG (shot quality), EFG+, player adjusted SEFG, and various rebounding metrics, such as positioning, opportunity percentage, attack, conversion percentage, rebounding above position (RAP), attack+, conversion+ and RAP+.
- SEFG shot quality
- EFG+ EFG+
- player adjusted SEFG various rebounding metrics, such as positioning, opportunity percentage, attack, conversion percentage, rebounding above position (RAP), attack+, conversion+ and RAP+.
- Offensive markings may include simple shot types (e.g., angled layup, driving layup, heave, post shot, jumper), complex shot types (e.g., post shot, heave, cut layup, standstill layup, lob, tip, floater, driving layup, catch and shoot stationary, catch and shoot on the move, shake & raise, over screen, pullup and stepback), and other information relating to shots (e.g., catch and shoot, shot clock, 2/3S, assisted shots, shooting foul/not shooting foul, made/missed, blocked/not blocked, shooter/defender, position/defender position, defender distance and shot distance).
- simple shot types e.g., angled layup, driving layup, heave, post shot, jumper
- complex shot types e.g., post shot, heave, cut layup, standstill layup, lob, tip, floater, driving layup, catch and shoot stationary, catch and shoot on the move, shake & raise, over screen,
- Other events that may be recognized, such as through the spatiotemporal learning system may include ones related to picks (ballhandler/screener, ballhandler/screener defender, pop/roll, wing/middle, step-up screens, reject/slip/take, direction (right/left/none), double screen types (e.g., double, horns, L, and handoffs into pick), and defense types (ice, blitz, switch, show, soft, over, under, weak, contain trap, and up to touch), ones related to handoffs (e.g., receive/setter, receiver/setter defender, handoff defense (ice, blitz, switch, show, soft, over, or under), handback/dribble handoff, and wing/step-up/middle), ones related to isolations (e.g., ballhandler/defender and double team), and ones related to post-ups (e.g., ballhandler/defender, right/middle/left and double teams).
- picks ballhandler/screener
- Defensive markings are also available, such as ones relating to closeouts (e.g., ballhandler/defender), rebounds (e.g., players going for rebounds (defense/offense)), pick/handoff defense, post double teams, drive blow-bys and help defender on drives), ones relating to off ball screens (e.g., screener/cutter and screener/cutter defender), ones relating to transitions (e.g., when transitions/fast breaks occur, players involved on offense and defense, and putback/no putback), ones relating to how plays start (e.g., after timeout/not after timeout, sideline out of bounds, baseline out of bounds, field goal offensive rebound/defensive rebound, free throw offensive rebound/defensive rebound and live ball turnovers), and ones relating to drives, such as ballhandler/defender, right/left, blowby/no blowby, help defender presence, identity of help defender, drive starts (e.g., handoff, pick, isolation or closeout)
- Markings may relate to off ball screens (screener/cutter), screener/cutter defender, screen types (down, pro cut, UCLA, wedge, wide pin, back, flex, clip, zipper, flare, cross, and pin in).
- Fig. 16 shows a system 1602 for querying and aggregation.
- data is divided into small enough shards that each worker has low latency response time.
- Each distributed machine may have multiple workers corresponding to the number of processes the machine can support concurrently.
- Query results never rely on more than one shard, since we enforce that events never cross quarter/period boundaries.
- Aggregation functions all run incrementally rather than in batch process, so that as workers return results, these are incorporated into the final answer immediately.
- the aggregator uses hashes to keep track of the separate rows and incrementally updates them.
- Fig. 17 shows a process flow for a hybrid classification process that uses human labelers together with machine learning algorithms to achieve high accuracy. This is similar to the flow described above in connection with Fig. 2, except with the explicit inclusion of the human-machine validation process.
- aligned video By taking advantage of aligned video as described herein, one may provide an optimized process for human validation of machine labeled data.
- Most of the components are similar to those described in connection with Fig. 2 and in connection with the description of aligned video, such as the XYZ data source 1702, cleaning process 1704, spatiotemporal pattern recognition module 1712, event processing system 1714, video source 1708, alignment facility 1710 and video snippets facility 1718.
- Additional components include a validation and quality assurance process 1720 and an event-labeling component 1722.
- Machine learning algorithms are designed to output a measure of confidence. For the most part, this corresponds to the distance from a separating hyperplane in the feature space.
- one may define a threshold for confidence. If an example is labeled by the machine and has confidence above the threshold, the event goes into the canonical event datastore 210 and nothing further is done. If an example has a confidence score below the threshold, then the system may retrieve the video corresponding to this candidate event, and ask a human operator to provide a judgment. The system asks two separate human operators for labels. If the given labels agree, the event goes into the canonical event datastore 210.
- the canonical event datastore 210 may contain both human marked and completely automated markings. The system may use both types of marking to further train the pattern recognition algorithms. Event labeling is similar to the canonical event datastore 210, except that sometimes one may either 1) develop the initial gold standard set entirely by hand, potentially with outside experts, or 2) limit the gold standard to events in the canonical event datastore 210 that were labeled by hand, since biases may exist in the machine labeled data.
- Fig. 18 shows test video input for use in the methods and systems disclosed herein, including views of a basketball court from simulated cameras, both simulated broadcast camera views 1802, as well as purpose-mounted camera views 1804.
- Fig. 19 shows additional test video input for use in the methods and systems disclosed herein, including input from broadcast video 1902 and from purpose-mounted cameras 1904 in a venue.
- probability maps 2004 may be computed based on likelihood there is a person standing at each x, y location.
- Fig. 21 shows a process flow of an embodiment of the methods and systems described herein.
- machine vision techniques are used to automatically locate the "score bug" and determine the location of the game clock, score, and quarter information. This information is read and recognized by OCR algorithms.
- Post-processing algorithms using various filtering techniques are used to resolve issues in the OCR.
- Kalman filtering / HMMs used to detect errors and correct them.
- Probabilistic outputs (which measure degree of confidence) assist in this error detection/correction.
- a score bug is nonexistent or cannot be detected automatically (e.g., sometimes during PIP or split screens).
- the Canonical Datastore 2110 contains a definitive list of events that the system knows occurred during a game. This includes events extracted from the XYZ data 2102, such as after cleansing 2104 and spatiotemporal pattern recognition 2108, as well as those specified by third-party sources such as player-by-player data sets 2106, such as available from various vendors.
- Differences among the data sources can be resolved, such as by a resolver process.
- the events in the canonical datastore 2110 may have game clock times specified for each event. Depending on the type of event, the system knows that the user will be most likely to be interested in a certain interval of game play tape before and after that game clock. The system can thus retrieve the appropriate interval of video for the user to watch.
- the methods and systems disclosed herein include numerous novel heuristics to enable computation of the correct video frame that shows the desired event, which has a specified game clock, and which could be before or after the dead ball since those frames have the same game clock.
- the game clock is typically specified only at the one-second level of granularity, except in the final minute of each quarter.
- Another advance is to use machine vision techniques to verify some of the events. For example, video of a made shot will typically show the score being increased, or will show a ball going through a hoop. Either kind of automatic observation serves to help the alignment process result in the correct video frames being shown to the end user.
- the UI enables a user to quickly and intuitively request all video clips associated with a set of characteristics: player, team, play type, ballhandler, ballhandler velocity, time remaining, quarter, defender, etc.
- the user can request all events that are similar to whatever just occurred in the video.
- the system uses a series of cartoon-like illustration to depict possible patterns that represent "all events that are similar.” This enables the user to choose the intended pattern, and quickly search for other results that match that pattern.
- the methods and systems may enable delivery of enhanced video, or video snips 2124, which may include rapid transmission of clips from stored data in the cloud.
- the system may store video as chunks (e.g., one-minute chunks), such as in AWS S3, with each subsequent file overlapping with a previous file, such as by 30 seconds.
- each video frame may be stored twice.
- Other instantiations of the system may store the video as different sized segments, with different amounts of overlap, depending on the domain of use.
- each video file is thus kept at a small size.
- the 30-second duration of overlap may be important because most basketball possessions (or chances in our terminology) do not last more than 24 seconds.
- each chance can be found fully contained in one video file, and in order to deliver that chance, the system does not need to merge content from multiple video files. Rather, the system simply finds the appropriate file that contains the entire chance (which in turn contains the event that is in the query result), and returns that entire file, which is small. With the previously computed alignment index, the system is also able to inform the UI to skip ahead to the appropriate frame of the video file in order to show the user the query result as it occurs in that video file. This delivery may occur using AWS S3 as the file system, the Internet as transport, and a browser-based interface as the UI. It may find other instantiations with other storage, transport, and UI components.
- Fig. 22 shows certain metrics that can be extracted using the methods and systems described herein, relating to rebounding in basketball. These metrics include positioning metrics, attack metrics, and conversion metrics.
- the methods and systems described herein first address how to value the initial position of the players when the shot is taken. This is a difficult metric to establish.
- the methods and systems disclosed herein may give a value to the real estate that each player owns at the time of the shot. This breaks down into two questions: (1) what is the real estate for each player? (2) what is it worth? To address the first question, one may apply the technique of using Voronoi (or Dirichlet) tessellations. Voronoi tessellations are often applied to problems involving spatial allocation.
- Voronoi or Dirichlet
- Players can add value by crashing the boards, i.e., moving closer to the basket towards places where the rebound is likely to go, or by blocking out, i.e., preventing other players by taking valuable real estate that is already established.
- a useful, novel metric for the crash phase is generated by subtracting the rebound probability at the shot from the rebound probability at the rim. The issue is that the ability to add probability is not independent of the probability at the shot.
- a defensive player who plays close to the basket. The player is occupying high- value real estate, and once the shot is taken, other players are going to start coming into this real estate. It is difficult for players with high initial positioning value to have positive crash deltas. Now consider a player out by the three-point line.
- a player has an opportunity to rebound the ball if they are the closest player to the ball once the ball gets below ten feet (or if they possess the ball while it is above ten feet).
- the player with the first opportunity may not get the rebound so multiple opportunities could be created after a single field goal miss.
- One may tally the number of field goal misses for which a player generated an opportunity for themselves and divided by the number of field goals to create an opportunity percentage metric. This indicates the percentage of field goal misses for which that player ended up being closest to the ball at some point.
- the ability for a player to generate opportunities beyond his initial position is the second dimension of rebounding: Hustle. Again, one may then apply the same normalization process as described earlier for Crash.
- the reason that there are often multiple opportunities for rebounds for every missed shot is that being closest to the ball does not mean that a player will convert it into a rebound.
- the raw conversion metric for players is calculated simply by dividing the number of rebounds obtained by the number of opportunities generated.
- This may be accomplished by first discretizing the court into, for example, 156 bins, created by separating the court into 13 equally spaced columns, and 12 equally spaced rows. Then, given some set S of shots from a particular bin, the rebounds from S will be distributed in the bins of the court according to a multinomial distribution. One may then apply maximum likelihood estimation to determine the probability of a rebound in each of the bins of the court, given the training set S. This process may be performed for bins that shots may fall in, giving 156 distributions for the court.
- the preceding section describes a method for determining the players rebounding probability, assuming that the players are stationary. However, players often move in order to get into better positions for the rebound, especially when they begin in poor positions. One may account for these phenomena. Let the player' s raw rebound probability be denoted r p and let d be an indicator variable denoting whether the player is on defense.
- Novel shooting metrics can also be created using this system.
- One is able to determine the probability of a shot being made given various features of the shot s, denoted as F.
- each shot can be characterized by a feature vector of the following form.
- the hoop represents the basket the shooter is shooting at
- defendero refers to the closest defender to the shooter
- defenderi refers to the second closest defender
- hoop 0 ther refers to the hoop on the other end of the court.
- the angle function refers to the angle between three points, with the middle point serving as the vertex.
- Fo through F 5 denote the feature values for the particular shot.
- the target for the regression is 0 when the shot is missed and 1 when the shot is made.
- By performing two regressions one is able to find appropriate values for the coefficients, for both shots within 10 feet, and longer shots outside 10 feet.
- three or four dimensions can be dynamically displayed on a 2- D graph scatter rank view 2302, including the x, y, size of the icon, and changes over time.
- Each dimension may be selected by the user to represent a variable of the user's choice.
- related icons may highlight, e.g., moussing over one player may highlight all players on the same team.
- reports 2402 can be customized by the user so that a team can create a report that is specifically tailored to that team's process and workflow. Another feature is that the report may visually display not only the advantages and disadvantages for each category shown, but also the size of that advantage or disadvantage, along with the value and rank of each side being compared. This visual language enables a user to quickly scan the report and understanding the most important points.
- a quality assurance UI 2502 presents the human operator with both an animated 2D overhead view 2510 of the play, as well as a video clip 2508 of the play.
- a key feature is that only the few seconds relevant to that play are shown to the operator, instead of an entire possession, which might be over 20 seconds long, or even worse, requiring the human operator to fast forward in the game tape to find the event herself. Keyboard shortcuts are used for all operations, to maximize efficiency. Referring to Fig.
- the operator' s task is simplified to its core, so that we lighten the cognitive load as much as possible: if the operator is verifying a category of plays X, the operator has to simply choose, in an interface element 2604 of the embodiment of the QA UI 2602 whether the play shown in the view 2608 is valid (Yes or No), or (Maybe). She can also deem the play to be a (Duplicate), a (Compound) play that means it is just one type-X action in a consecutive sequence of type-X actions, or choose to (Flag) the play for supervisor review for any reason.
- Features of the UI 2602 include the ability to fast word, rewind, submit and the like, as reflected in the menu element 2612.
- a table 2610 can allow a user to indicate the validity of plays occurring at designated times.
- Fig. 27 shows a method of camera pose detection, also known as "court solving.”
- Fig. 27 also shows the result of automatic detection of the "paint,” and use of the boundary lines to solve for the camera pose.
- the court lines and hoop location, given the solved camera pose, are then shown projected back onto the original image 2702. This projection is from the first iteration of the solving process, and one can see that the projected court and the actual court do not yet align perfectly.
- One may use machine vision techniques to find the hoop and to find the court lines (e.g., paint boundaries), then use found lines to solve for the camera pose. Multiple techniques may be used to determine court lines, including detecting the paint area. Paint area detection can be done automatically.
- One method involves automatically removing the non-paint area of the court by automatically executing a series of "flood fill" type actions across the image, selecting for court-colored pixels. This leaves the paint area in the image, and it is then straightforward to find the lines/points.
- One may also detect all lines on the court that are visible, e.g., background or 3-point arc. In either case, intersections provide points for camera solving.
- a human interface 2702 may be provided for providing points or lines to assist algorithms, to fine- tune the automatic solver.
- the camera pose solver is essentially a randomized hill climber that uses the mathematical models as a guide (since it may be under- constrained). It may use multiple random initializations.
- Fig. 46 shows the result of automatic detection of the "paint", and use of the boundary lines to solve for the camera pose.
- the court lines and hoop location, given the solved camera pose, are then shown projected back onto the original image. This projection is from the first iteration of the solving process, and one can see that the projected court and the actual court do not yet align perfectly.
- Fig. 28 relates to camera pose detection.
- the second step 2802 shown in the Figure shows how the human can use this GUI to manually refine camera solutions that remain slightly off.
- Fig. 29 relates to auto-rotoscoping.
- Rotoscoping 2902 is required in order to paint graphics around players without overlapping the players' bodies.
- Rotoscoping is partially automated by selecting out the parts of the image with similar color as the court. Masses of color left in the image can be detected to be human silhouettes.
- the patch of color can be "vectorized" by finding a small number of vectors that surround the patch, but without capturing too many pixels that might not represent a player' s body.
- Figs. 30A, 30B, and 30C relate to scripted storytelling with an asset library 3002.
- a company may either learn heavily on a team of artists, or a company may determine how best to handle scripting based on a library of assets. For example, instead of manually tracing a player's trajectory and increasing the shot probability in each frame as the player gets closer to the ball, a scripting language allows the methods and systems described herein to specify this augmentation in a few lines of code.
- the Voronoi partition and the associated rebound positioning percentages can be difficult to compute for every frame.
- a library of story element effects may list each of these current and future effects.
- scripted story element effects may be best suited for certain types of clips. For example, a rebound and put-back will likely make use of the original shot probability, the rebound probabilities including Voronoi partitioning, and then go back to the shot probability of the player going for the rebound. This entire script can be learned as being well-associated with the event type in the video. Over time, the system can automatically infer the best, or at least retrieve an appropriate, story line to match up with a selected video clip containing certain events. This enables augmented video clips, referred to herein as DataFX clips, to be auto-generated and delivered throughout a game.
- DataFX clips augmented video clips
- Figs. 31-38 show examples of DataFX visualizations.
- the visualization of Fig. 31 requires court position to be solved in order to lay down grid, player "puddles". Shot arc also requires backboard/hoop solution.
- Fig. 32 Voronoi tessellation, heat map, shot and rebound arcs all require the camera pose solution.
- the highlight of the player uses rotoscoping.
- Fig. 33 in addition to the above, players are rotoscoped for highlighting.
- Figs. 34-38 show additional visualizations that are based on the use of the methods and systems disclosed herein.
- DataFX video augmented with data-driven special effects
- DataFX may be provided for pre-, during, or post- game viewing, for analytic and entertainment purposes.
- DataFX may combine advanced data with Hollywood-style special effects. Pure numbers can be boring, while pure special effects can be silly, but the combination of the two and the results can be very powerful.
- Example features used alone or in combination in DataFX can include use of a Voronoi overlay on court, a Grid overlay on court, a Heat map overlay on court, a Waterfall effect showing likely trajectories of the ball after a missed field goal attempt, a Spray effect on a shot, showing likely trajectories of the shot to the hoop, Circles and glows around highlighted players, Statistics and visual cues over or around players, Arrows and other markings denoting play actions, Calculation overlays on court, and effects showing each variable taken into account.
- Figs. 39A through 41B show a product referred to as "Clippertron.”
- Clippertron Provided is a method and system whereby fans can use their distributed mobile devices to control individually and/or collectively what is shown on the Jumbotron or video board(s).
- An embodiment enables the fan to go through mobile application dialogs in order to choose the player, shot type, and shot location to be shown on the video board.
- the fan can also enter in his or her own name so that it is displayed alongside the highlight clip. Clips are shown on the Video Board in real time or queued up for display. Variations include getting information about the fan's seat number. This could be used to show a live video feed of the fan while their selected highlight is being shown on the video board.
- FanMix is a web-based mobile application that enables in- stadium fans to control the Jumbotron and choose highlight clips to push to the Jumbotron.
- An embodiment of FanMix enables fans to choose their favorite player, shot type, and shot location using a mobile device web interface.
- a highlight showing this particular shot is sent to the Jumbotron and displayed according to placement order in a queue. Enabling this capability is that video is lined up to each shot within a fraction of a second. This allows many clips to be shown in quick succession, each showing video from the moment of release to the ball going through the hoop. In some cases, the video may start from the beginning of a play, instead of when a play begins.
- the methods and systems disclosed herein may include methods and systems for allowing a user or group of users to control presentation of a large scale display in an event venue, where the options for control are based on a context of the content as determined by machine extraction of semantically relevant events from the content
- the methods and systems disclosed herein may include methods and systems for enabling interaction with a large scale display system and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing an application by which at least one user can interact with the video content data structure, wherein the options for user interaction are based on the context information, wherein the interaction with the video content data structure controls the presentation of the content on a large scale display.
- one or more users may interact with menus on an application, such as a smart phone application, in an arena or other location that has a large-scale display.
- the users may express preferences, such as by voting, for what content should be displayed, including selecting preferred types of events and/or contexts (which may be organized as noted above based on semantically relevant filters), selecting what metrics should be displayed (options for which may be offered based on context information for particular extracted video events), and the like.
- a large scale display in a venue where a live event is taking place may offer games, quizzes, or the like, where users may respond by text, SMS, or the like.
- the content of such games or quizzes may be constructed at least in part based on a machine semantic understanding of the live event, such as asking users which player has the most rebounds in the first quarter, or the like.
- the methods and systems disclosed herein may include methods and systems for a user to control Jumbotron clips based on contextualized content filters.
- the methods and systems disclosed herein may include methods and systems for a Jumbotron fan quiz based on machine semantic understanding of a live game
- the methods and systems disclosed herein may include methods and systems wherein the application comprises a quiz for a user, wherein the quiz is constructed based at least in part on a machine semantic understanding of a live game that is taking place in a venue where the large-scale display is located.
- a fan quiz may ask questions based on proprietary machine learned metrics such as "which player took the hardest shots in this quarter.”
- the methods and systems disclosed herein may include methods and systems for embedding a machine extracted video cut in an application, where the selection of the embedded cut for the application is based on the context of the video cut.
- interactive visualization 218, as illustrated in Fig. 2 may include producing a reconstruction of an event, such as a game, such as a 3D reconstruction or rendering.
- a 3D reconstruction or rendering of an event may be produced using a process that presents the event from a defined point of view, such as the first person point of view of a participant in the event, such as a player.
- Fig. 39F illustrates an embodiment of such as process, referred to herein in some cases as a first person POV process, or simply a first person process.
- a first person process may allow the user to select a player's view to follow.
- a first person process may automatically pin a user' s view to the head of the selected player.
- the end result of a first person process may be dynamically rendered from the view of the selected player as a play occurs.
- a first person process may be an automated first person process.
- An automated first person process may produce a 3D reconstruction or rendering of a game and render each frame from the view of a player selected by a user.
- a first person process may be a virtual reality-based first person process.
- a virtual reality-based first person process may produce a 3D reconstruction or rendering of a game that allows a user to control the orientation of a view from the head movements of a user.
- the point of view may be controlled by, for example, player head tracking.
- users may choose a player whose point of view will be presented.
- Location of a view may be controlled automatically via head tracking data.
- View orientation may be controlled by the head movements of a user.
- the head movements of a user may be recorded by virtual reality (VR) technology.
- VR technology may be Oculus RiftTM technology and the like.
- a first person process may include constructing a point cloud that provides a 3D model of a real world scene.
- Point cloud construction may begin by producing binary, background-subtracted images for each time- synchronized frame on each camera. Using these binary images and the calibrations of each camera, a 3D convex hull may be produced by discretizing the scene into voxels and filling each voxel, if the voxel is contained within the ray projected from the camera through the image visual hull.
- the image visual hull may be the silhouette of the scene, for example.
- the silhouette of the scene may be a shape-form silhouette.
- the resulting convex hull may contain voxels that may not actually be present in the world, due to reconstructing only of the visual hull.
- the 3D convex hull may be carved using photo consistency methods.
- Photo consistency methods may back-project the surface of a 3D reconstructed visual hull onto each visible camera. Photo consistency methods may also check to ensure the color of the pixels is consistent with the same pixel from another camera, or with nearby pixels, such as to avoid unrealistic discontinuities. If the colors from each visible camera do not agree, the voxel may be carved. This process may be repeated for the entire convex hull, producing the final carved point cloud.
- Point cloud construction may estimate the skeletal pose of all participants in a real world scene.
- Point cloud construction may fit a hand-made participant model to the estimated pose of each participant in a real world scene.
- the real world scene could be a sports court and the participants could be all the players on the sports court.
- point cloud construction could fit a hand-made player model to the estimated pose of each player on the sports court.
- Point cloud construction may include meshing techniques, which may be used to improve the quality of a final visualization for a user. Meshing techniques may be used to mesh multiple point counts. Meshing techniques may be used to provide a view that may be very close to a point cloud, for example.
- a first person process may use player identification to enable the user to select from which player' s view to render the 3D reconstruction.
- Player identification may involve multiple steps in order to produce reliable results.
- Player identification may start by performing jersey number detection, as illustrated in Fig. 39F. Jersey numbers may be mapped to player names. Jersey numbers may then be mapped to player names using official rosters and the like.
- Jersey number detection may be performed frame-by-frame.
- Frame-by-frame jersey number detection may be performed by scanning and classifying each window as a number or as nothing, such as using a support vector machine (SVM), a supervised machine learning model used for classification.
- SVM support vector machine
- the SVM may be trained, such as using training sets of manually marked jersey numbers from the game video, for example.
- Results from individual frame-by-frame detection may be stitched together to form temporal tracks.
- Individual frame by frame detection may be stitched together to form temporal tracks using a k-shortest paths algorithm, as illustrated in Fig. Y.
- Jersey number tracks may be associated with existing, more continuous player tracking data. Associating jersey number tracks with existing, more continuous player tracking data may produce robust tracks of identifiable players.
- a first person process may use head tracking in order to control the location of the view within a 3D reconstruction, as illustrated in Fig. 39F.
- Head tracking may involve multiple steps in order to produce reliable results.
- the first step of head tracking may be the same as for player identification.
- the first step of head tracking may include head detection.
- Head detection may create a model on heads instead of on jersey numbers. Head detection may be performed frame by frame.
- Head detection may include frame by frame head detection.
- Frame-by-frame head detection may be performed by scanning each image.
- Frame -by-frame head detection may be performed by scanning each image and classifying each window as a head or not.
- Classifying each window as a head or not may be performed using an SVM.
- An SVM may be trained.
- An SVM may be trained using manually marked head samples from previously recorded games.
- An SVM maybe be a team-dk-SVM.
- the results of the detection may then be used in 2D tracking to produce temporal 2D tracklets of each head within a camera's frame.
- 2D tracklets may then be triangulated using the results of all cameras to produce a 3D estimation of the location of all heads on the court.
- a 3D estimation of the location of all heads on the court may be 3D tracklets.
- 3D tracklets may then be stitched together. 3D tracklets may then be stitched together using an algorithm.
- An algorithm may be a k-shortest paths (KSP) algorithm. 3D tracklets may be stitched together to produce potential final head tracking results. Linear programming may be used to choose optimal head paths.
- KSP k-shortest paths
- Gaze Estimation As illustrated in Fig. 39F, a first person process may use gaze estimation. Gaze estimation may be used to control the orientation of a view mounted on the player's head within the 3D reconstruction. Gaze estimation may be computed by assuming a player is always looking in the direction opposite the numbers on the back of the player.
- Jersey number detection may be performed frame by frame.
- Frame by frame jersey number detection may be performed by scanning and classifying each window as a number or nothing using an SVM.
- the SVM may be trained using manually marked jersey numbers from an existing game video.
- An assumption may be made to determine the angle of a jersey number located on the back or front of a player's jersey.
- An assumption may be that a jersey number is only visible when the jersey number is perfectly aligned with a camera that made the detection.
- Cameras may have a known location in space. Because the cameras have a known location in space, the vector between the jersey and the camera may be computed using the known location of the camera in space.
- Frame -by-frame estimation may be performed after a vector is calculated.
- the results of the frame-by-frame estimation may be filtered to provide a smoothed experience for a first person process.
- Fig. 41 relates to an offering referred to as "inSight.” This offering allows pushing of relevant stats to fans' mobile devices 4104. For example, if player X just made a three-point shot from the wing, this would show statistics about how often he made those types of shots 4108, versus other types of shots, and what types of play actions he typically made these shots off of. inSight does for hardcore fans what Eagle (the system described above) does for team analysts and coaches. Information, insights, and intelligence may be delivered to fans' mobile devices while they are seated in the arena. This data is not only beautiful and entertaining, but is also tuned into the action on the court.
- the fan is immediately pushed information that shows the shot's frequency, difficulty, and likelihood of being made.
- the platform features described above as "Eagle,” or a subset thereof may be provided, such as in a mobile phone form factor for the fan.
- An embodiment may include a storyboard stripped down, such as from a format for an 82" touch screen to a small 4" screen. Content may be pushed to a device that corresponds to the real time events happening in the game.
- Fans may be provided access to various effects (e.g., DataFX features described herein) and to the other features of the methods and systems disclosed herein.
- Figs. 42 and 43 show touchscreen product interface elements 4202, 4204, 4208, 4302 and 4304. These are essentially many different skins and designs on the same basic functionality described throughout this disclosure. Advanced stats are shown in an intuitive large-format touch screen interface.
- a touchscreen may act as a storyboard for showing various visualizations, metric and effects that conform to an understanding of a game or element thereof.
- Embodiments include a large format touch screen for commentators to use during a broadcast. While InSight serves up content to a fan, the Storyboard enables commentators on TV to access content in a way that helps them tell the most compelling story to audiences.
- Features include providing a court view, a hexagonal Frequency+Efficiency View, a "City/Matrix" View with grids of events, a Face/Histogram View, Animated intro sequences that communicate to a viewer that each head's position means that player's relative ranking, an Animated face shuttle that shows re-ranking when metric is switched, a ScatterRank View, a ranking using two variables (one on each axis), a Trends View, integration of metrics with on- demand video and the ability to r-skin or simplify for varying levels of commentator ability.
- new metrics can be used for other activities, such as driving new types of fantasy games, e.g., point scoring in fantasy leagues could be based on new metrics.
- DataFX can show the player how his points were scored, e.g., overlay that runs a counter over an RB's head showing yards rushed while the video shows RB going down the field.
- a social game can be made so that much of the game play occurs in real time while the fan is watching the game.
- a social game can be managed so that game play occurs in real time while a fan is watching the game, experiencing various DataFX effects and seeing fantasy scoring-relevant metrics on screen during the game.
- the methods and systems may include a fantasy advice or drafting tool for fans, presenting rankings and other metrics that aid in player selection.
- DataFX can also be used for instant replays with DataFX optimized so that it can produce "instant replays" with DataFX overlays. This relies on a completely automated solution for court detection, camera pose solving, player tracking, and player roto-scoping.
- Interactive DataFX may also be adapted for display on a second screen, such as a tablet, while a user watches a main screen.
- Real time or instant replay viewing and interaction may be used to enable such effects.
- the fan could interactively toggle on and off various elements of DataFX. This enables the fan to customize the experience and to explore many different metrics.
- the system could be further optimized so that DataFX is overlaid in true real time, enabling the user to toggle between a live video feed and a live video feed that is overlaid with DataFX. The user would then also be able to choose the type of DataFX to overlay, or which player(s) to overlay it on.
- a touch screen UI may be established for interaction with DataFX.
- Many of the above embodiments may be used for basketball, as well as for other sports and for other items that are captured in video, such as TV shows, movies, or live video (e.g., news feeds).
- video such as TV shows, movies, or live video (e.g., news feeds).
- the computer For non-sports domains, such as TV shows or movies, there is no player tracking data layer that assists the computer in understanding the event. Rather, in this case, the computer must derive, in some other way, an understanding of each scene in a TV show or movie.
- the computer might use speech recognition to extract the dialogue throughout a show. Or it could use computer vision to recognize objects in each scene, such as robots in the Transformer movie. Or is could use a combination of these inputs and others to recognize things like explosions. The sound track could also provide clues.
- the resulting system would use this understanding to deliver the same kind of personalized interactive augmented experience as we have described for the sports domain.
- a user could request to see the Transformer movie series, but only a compilation of the scenes where there are robots fighting and no human dialogue.
- This enables "short form binge watching," where users can watch content created by chopping up and recombining bits of content from original video.
- the original video could be sporting events, other events TV shows, movies, and other sources. Users can thus gorge on video compilations that target their individual preferences.
- This also enables a summary form of watching, suitable for catching up with current events or currently trending video, without having to watch entire episodes or movies.
- Fig. 44 provides a flow under which the platform may ingest and align the content of one or more broadcast video feeds and one or more tracking camera video feeds.
- a broadcast video feed may be ingested, which may consist of an un-calibrated and un- synchronized video feed.
- the ingested broadcast video feed may be processed by performing optical character recognition at a step 4414, such as to extract information from the broadcast video feed that may assist with aligning events within the feed with events identified in other sources of video for the same event. This may include recognizing text and numerical elements in the broadcast video feed, such as game scores, the game clock, player numbers, player names, text feeds displayed on the video, and the like.
- the time on the game clock, or the score of a game may assist with time-alignment of a broadcast feed with another video feed.
- objects may be detected within the broadcast video feed 4404, such as using machine- based object-recognition technologies. Objects may include players (including based on recognizing player numbers), equipment (such as the ball in a basketball game), and many others.
- objects may be tracked over time in a step 4418, such as in progressive frames of the broadcast video feed. Tracked objects may be used to assist in calibrating the broadcast video position by associating them with the same objects as identified in another source, such as a tracking camera video feed.
- a step 4402 in parallel with the steps involved in ingesting and processing a broadcast video feed, video feeds from tracking cameras, such as tracking cameras for capturing 3D motion in a venue (like a sports arena), may be ingested.
- the tracking camera video feeds may be calibrated and synchronized to a frame of reference, such as one defined by the locations of a set of cameras that are disposed at known locations within the venue where the tracking camera system is positioned.
- one or more objects may be detected within the tracking camera video feed, including various objects of the types noted above, such as players, numbers, items of equipment, and the like.
- spatiotemporal coordinates of the objects may be determined by processing the information from the tracking camera video feed, the coordinates being determined for the recognized objects based on the frame of reference defined by the camera positions of the tracking system.
- the coordinates being determined for the recognized objects can be based on the court or the field on which the game is played.
- the coordinates being determined for the recognized objects are based on the boundaries, lines, markers, indications, and the like associated with the court or the field on which the game is played.
- the video feed from the tracking camera system and the information about spatiotemporal object positions may be used to generate a point cloud at a step 4416, within which voxel locations of the objects detected at the step 4406 may be identified at a step 4418.
- the tracking camera video feed that was processed to detect and track objects may be further processed at a step 4410 by using spatiotemporal pattern recognition (such as machine -based spatiotemporal pattern recognition as described throughout this disclosure) to identify one or more events, which may be a wide range of events as described throughout this disclosure, such as events that correspond to patterns in a game or sport.
- spatiotemporal pattern recognition such as machine -based spatiotemporal pattern recognition as described throughout this disclosure
- other feeds may be available that may contain additional information about events that are contained in the tracking camera video feed.
- a data feed such as a play-by-play feed, for a game may be ingested at a step 4422.
- the information from multiple sources may be aligned, such as aligning the play-by-play data feed from the step 4422 with events recognized at the step 4410.
- the recognized event data in the tracking camera video feed at the step 4410 may be aligned with events recognized in the broadcast video feed at the step 4414, resulting in time-aligned broadcast video, tracking camera, and other (e.g., play-by-play) feeds.
- objects detected at the step 4404 in the broadcast video feed and tracked at the step 4418 may be used at a step 4428 to calibrate the broadcast video camera position, such as by identifying the broadcast video camera position within the frame of reference of the tracking camera system used to capture the tracking camera video feed. This may include comparing sizes and orientations of the same object as it was detected at the step 4404 in the broadcast video feed and at the step 4406 in the tracking camera system video feed.
- calibration parameters of the broadcast camera can be determined by, among other things, comparing positions of detected objects in the video with detected three-dimensional positions of the corresponding objects that can be obtained using the calibrated tracking system.
- heads of the players in the game can be suitable objects because the heads of the players can be precisely located relative to other portions of the bodies of the players.
- the broadcast video camera information can be processed as another source just like any of the tracking cameras. This may include re-calibrating the broadcast video camera position for each of a series of subsequent events, as the broadcast video camera may move or change zoom between events.
- pixel locations in the broadcast video feed may be identified, corresponding to objects in the broadcast video feed, which may include using information about voxel locations of objects in the point cloud generated from the motion tracking camera feed at the step 4418 and/or using image segmentation techniques.
- time-aligned broadcast video feeds tracking camera event feeds, and play-by-play feeds, where within each feed pixel locations or voxel locations of objects and backgrounds are known, so that various activities can be undertaken to process the feeds, such as for augmenting the feeds, performing pattern recognition on objects and events within them (such as to find plays following particular patterns), automatically clipping or cutting them to produce content (such as capturing a reaction in broadcast video to an event displayed in or detected by the tracking camera feeds based on a time sequence of time-aligned events), and many others as described throughout this disclosure.
- embodiments of the methods and systems disclosed herein may involve handling multiple video feeds 4502, information from one or more tracking systems 4512 (such as player tracking systems that may provide time-stamped location data and other information, such as physiological monitoring information, activity type information, etc.), and one or more other input sources 4510 (such as sources of audio information, play-by-play information, statistical information, event information, etc.).
- live video input feeds 4502 are encoded by one or more encoding systems 4504 to produce a series of video segment files 4508, each consisting of a video chunk, optionally of short duration, e.g., four seconds.
- Video segment files 4508 from different input feeds corresponding to the same time interval are considered as part of a temporal group 4522 associated with that time interval.
- the temporal group 4522 may also include information and other content from tracking systems 4512 and other input sources 4510.
- each video segment file 4508 may independently and in parallel undergo various processing operations in one or more processing systems 4518, such as transcoding to various file formats, streaming protocols, and the like.
- the derived output files 4520 of processing 4518 may be associated with the same temporal group 4522.
- Temporal grouping 4522 enables time synchronization among the original and derived files without having to further maintain or track timing or synchronization information.
- processing operations 4518 may include, without limitation, standard video on demand (VOD) transcoding, such as into lower bit rate video files.
- Processing operations 4518 may also include augmentation, such as with graphics, audio overlays, or data, producing derived, augmented video files 4518.
- Other data derived from the video streams or obtained from other sources 4510 may also be synchronized to the video files 4508 in a temporal group 4522, such as by adding them as metadata files to the corresponding temporal group or by binding them to the video files 4514.
- a manifest file 4524 based on these temporal groups 4522 may be created to enable streaming of the original video feed 4502, the video segment files 4514 and/or derived files 4520 as a live, delayed or on-demand stream. Synchronization among the output streams may enable combining and/or switching 4528 seamlessly among alternative video feeds (e.g., different angles, encoding, augmentations or the like) and data feeds of a live streamed event.
- synchronization across original video feeds 4502, video segment files 4508, derived video feeds 4520 with encoded, augmented or otherwise processed content, and backup video feeds, described by a manifest file 4524 may allow client-side failover from one stream to another without time discontinuity in the viewing of the event. For instance, if an augmented video stream resulting from processing 4518 is temporarily unavailable within the time offset at which the live stream is being viewed or falls below a specified buffering amount, a client application 4530 consuming the video feed may temporarily fail over to an un- augmented stream 4502 or encoded video segment file 4508.
- the granularity with which the client application 4530 switches back to the augmented stream 4518 when available may depend on semantically defined boundaries in the video feed, which in embodiments may be based on a semantic understanding of events within the video feed, such as achieved by the various methods and systems described in connection with the technology stack 100 and the processes 200 described throughout this disclosure.
- a switch back to derived file 4520 with various augmentations added in processing 4518 may be timed to occur after a change of possession, a timeout, a change in camera angle, a change in point-of-view, or other appropriate points in the action, so that the switching occurs while minimizing disruption of the viewing experience.
- Switching may also be controlled by semantic understanding 4532 of the content of different video feeds 4502 at each time instant; for example, if a camera is not pointing at the current action on the court, an alternative video feed 4502, video segment file 4514 or derived file 4520 may be selected.
- a "smart pipe” may be provided consisting of multiple aligned content channels (e.g., audio, video, or data channels) that are indexed both temporally and spatially.
- Spatial indexing and alignment 4534 may include indexing of pixels in 2D streams, voxels in 3D streams, and other objects, such as polygonal meshes used for animation, 3D representation, or the like.
- a wide variety of elements may be indexed, such as, without limitation, events, and locations of objects (including players, game objects, and objects in the environment, such as a court or arena) involved in those events.
- a further variety of elements may be indexed including information and statistics related to events and locations.
- a further variety of elements may be indexed including locations of areas corresponding to floor areas, background areas, signage areas, or the like where information, augmentations, graphics, animations, advertising, or the like may be displayed over a content frame.
- a further variety of elements may be indexed including indices or indicators of what information, augmentation elements or the like that are available to augment a video feed in a content channel such as ones that may be selected individually or in combination.
- a further variety of elements may be indexed including predefined combinations of content (e.g., particular combinations of audio, video, information, augmentation elements, replays, or other content elements), such as constituting channels or variations from which end-users may choose ones that they prefer.
- a spatial indexing and alignment system 4534 may provide spatial indexing and alignment information to the processing system 4518 (or may be included therein), such that the derived files 4520 (and optionally various objects therein) that are indexed both temporally and spatially.
- the "smart pipe" for synchronized, switchable and combinable content streams 4528 may contain sufficient indexed and aligned content to allow the creation of derived content, the creation of interactive applications, and the like, each optionally tied to live and recorded events (such as sporting events).
- the tracking systems 4512, the spatial indexing and alignment 4534 and the semantic understanding 4532 may be part of the larger alignment, tracking, and semantic system included in the systems and methods disclosure herein that may take various inputs including original video feeds and play-by-play feeds, and may produce X, Y, Z tracking data and semantic labels.
- the X, Y, Z tracking data and semantic labels may be stored as separate metadata files in the temporal group 4522 or used to produce derived video files 4520 in the temporal group 4522.
- any combination of inputs such as from a tracking camera system, a 3D camera array, broadcast video, a smartphone video, lidar, and the like may be used to automatically obtain a 3D understanding of a game.
- the automatically obtained 3D understanding of the game may be used to index voxels of 3D representations (e.g., AR/VR video) or pixels of any 2D video footage (e.g., from tracking cameras, broadcast, smartphones, reconstructed video from any point of view such as first person point of view of players in the game) or alternatively to voxels/pixels, other graphics representations such as polygonal meshes.
- a "smart pipe” may consist of multiple aligned content channels (e.g., audio, video, or data channels) that are indexed both temporally and spatially (e.g., indexing of pixels/voxels/polygonal meshes) with events and locations of players/objects involved in those events.
- the indexing both temporally and spatially with events and locations of players/objects involved in those events may also include information and statistics related to events and locations.
- the indexing both temporally and spatially with events and locations of players/objects involved in those events may also include locations of areas corresponding to floor or background areas where information, augmentations (e.g., filters that manipulate the look of the ball/players) or advertising may be displayed over each video frame.
- information, augmentations e.g., filters that manipulate the look of the ball/players
- available pieces of information and augmentation elements may be selected individually or in combination.
- combinations of audio, video, information, augmentation, replays, and the like may constitute channels for end-users to choose from.
- the smart pipe may contain sufficient indexed and aligned content to create derived content and interactive apps tied to live and recorded games.
- composition of video via frames, layers and/or tracks may be generated interactively by distributed sources, e.g., base video of the sporting event, augmentation/information layers/frames from different providers, audio tracks from alternative providers, advertising layers/frames from other providers, leveraging indexing and synchronization concepts, and the like.
- distributed sources e.g., base video of the sporting event, augmentation/information layers/frames from different providers, audio tracks from alternative providers, advertising layers/frames from other providers, leveraging indexing and synchronization concepts, and the like.
- the base layers and/or tracks may be streamed to the various providers as well as to the clients.
- additional layers and/or tracks may be streamed directly from the providers to the clients and combined at the client.
- the composition of video via frames, layers and/or tracks and combinations thereof may be generated interactively by distributed sources and may be based on user personalizations.
- the systems and methods disclosed herein may include joint compression of channel streams such as successive refinement source coding to reduce streaming bandwidth and/or reduce channel switching time, and the like.
- the systems and methods disclosed herein may include event analytics and/or location-based games including meta-games, quizzes, fantasy league and sport, betting, and other gaming options that may be interactive with many of the users at and connected to the event such as identity-based user input, e.g., touching or clicking a player predicted to score next.
- the event analytics and/or location-based games may include location- based user input such as touching or clicking a location where a rebound or other play or activity is expected to be caught, to be executed, and the like.
- the event analytics and/or location-based games may include timing-based user input such clicking or pressing a key to indicate when a user thinks a shot should be taken, a defensive play should be initiated, a timeout should be requested, and the like.
- the event analytics and/or location-based games may include prediction-based scoring including generating or contributing to a user score based on the accuracy of an outcome prediction associated with the user.
- the outcome prediction may be associated with outcomes of individual offensive and defensive plays in the games and/or may be associated with scoring and/or individual player statistics at predetermined time intervals (e.g., quarters, halves, whole games, portions of seasons, and the like).
- the event analytics and/or location-based games may include game state- based scoring including generating or contributing to a user score based on expected value of user decision calculated using analysis of instantaneous game state and/or comparison with evolution of game state such as maximum value or realized value of the game state in a given chance or possession.
- the systems and methods disclosed herein may include interactive and immersive reality games based on actual game replays.
- the interactive and immersive reality games may include the use of one or more simulations to diverge from actual game events (partially or in their entirety) based on user input or a collection of user input.
- the interactive and immersive reality games may include an action-time resolution engine that may be configured to determine a plausible sequence of events to rejoin the actual game timeline relative to, in some examples, the one or more simulations to diverge from actual game events (partially or in their entirety) based on user input or a collection of user input.
- the interactive and immersive reality games may include augmented reality simulations that may integrate game event sequences, using cameras on located on one or more backboards and/or along locations adjacent to the playing court.
- the systems and methods disclosed herein may include simulated sports games that may be based on detailed player behavior models.
- the detailed player behavior models may include tendencies to take different actions and associated probabilities of success of different actions under different scenarios including teammate/opponent identities, locations, score differential, period number, game clock, shot clock, and the like.
- the systems and methods disclosed herein may include social chat functions and social comment functions that may be inserted into a three-dimensional scene of a live event.
- the social chat and comment functions that may be inserted into the three-dimensional scene of the live event may include avatars inserted into the crowd that may display comments within speech bubbles above the avatars.
- the social chat and comment functions may be inserted into a three-dimensional scene of the live event as a running commentary adjacent to other graphics or legends associated with the event.
- the systems and methods disclosed herein may include the automating of elements of broadcast production such as automatic control of camera pan, tilt, and zoom.
- the automating of elements of broadcast production may also include automatic switching between camera views.
- the automating of elements of broadcast production may include automatic live and color commentary generation and automatic placement and content from synthetic commentators in the form of audio or in the form of one or more audio and video avatars with audio content that may be mixed with semantic and contextual based reactions from the live event and/or from other users.
- the automated elements of broadcast production may include automated generation of commentary in audio only or audio and video form including AR augmentation and associated content by, for example, combining semantic machine understanding of events in the game and semantic machine understanding of camera views, camera cuts, and camera close-ups in broadcast or another video.
- the automated generation of commentary may also be based on semantic machine understanding of broadcaster/game audio, statistics from semantic machine understanding of past games, information/statistics from other sources, and combinations thereof.
- a ranking of potential content items may be based on at least one of the rarity of events, comparison against the rest of the league, diversity with respect to previously shown content, personalization based on channel characteristics, explicit user preferences, inferred user preferences, the like, or combinations thereof.
- the automated generation of commentary may include the automatic selection of top-ranked content items or a short list of top- ranked content items shown to a human operator for selection.
- the automating of elements of broadcast production may include automatic live commentary generation that may be used to assist referees for in situ evaluation or post-mortem evaluation.
- the automatic live commentary generation that may be used to assist referees may also be used to train referees in unusual situations that may be seen infrequently in actual games but may be reproduced or formed from AR content based on or purposefully deviated from live game events.
- the referee assistance, evaluation, training, and the like associated with the improvement of referee decisions may be based on semantic machine understanding of game events.
- the systems and methods disclosed herein may include the use of player- specific information in three-dimensional position identification and reconstruction to improve trade-offs among camera requirements. Toward that end, fewer or lower resolution cameras may be used, computational complexity/delay may be reduced and output quality/accuracy may be increased when compared to typical methods. With reference to Fig.
- the player- specific information in three-dimensional position identification and reconstruction 4600 may be shown to improve the balance in trade-offs of camera requirements including improved localization of keypoints 4602 such as a head, joints, and the like, by using player models 4604 of specific players in conjunction with player identification 4608 such as identifying a jersey number or automatically recognizing a face and remote sensing technology to capture the players such as one or more video cameras, lidar, ultrasound, Wi-Fi visualization, and the like.
- the improved localization of keypoints may include optimizing over constraints on distances between keypoints from player models combined with triangulation measurements from multiple cameras.
- the improved localization of keypoints may also include using the player models 4604 to enable 3D localization with a single camera.
- the system and methods disclosed herein may also include the use of the player models 4604 fitted to detected keypoints to create 3D reconstructions 4620 or to improve 3D reconstructions in combination with point cloud techniques.
- Point cloud techniques may include a hybrid system including the player models 4604 that may be used to replace areas where the point cloud reconstruction does not conform adequately to the model.
- the point cloud techniques may include supplementing the point cloud in scenarios where the point cloud may have a low density of points.
- the improved localization of keypoints may include the use of player height information combined with face detection, gaze detection, posture detection, or the like to locate the point of view of players.
- the improved localization of keypoints may also include the use of camera calibration 4630 receiving one or more video feeds 4632, the 3D reconstruction 4610 and projection onto video in order to improve player segmentation for broadcast video 4640.
- the systems and methods disclosed herein may include using a state- based machine learning model with hierarchical states.
- the state-based machine learning model with hierarchical states may include input training state labels at the finest granularity.
- the machine learning model may be trained at the finest level of granularity as well as at intermediate levels of aggregated states.
- the output and cost function optimization may be at the highest level of state aggregation.
- the machine learning model may be trained using an ensemble of active learning methods for multiclass classification including weighting of methods based on a confusion matrix and a cost function that may be used to optimize the distribution of qualitatively varied instances for active learning.
- the machine learning model may include active learning and active quality assurance on a live spatiotemporal machine learning workflow 4700 in accordance with the various embodiments.
- the machine learning workflow 4700 includes a machine learning (ML) algorithm 4702 that may produce live and automatic machine learning (ML) classification output 4704 (with minimum delay) as well as selected events for human quality assurance (QA) 4708 based on live spatiotemporal data 4710.
- ML machine learning
- QA human quality assurance
- the live spatiotemporal machine learning workflow 4700 includes the data from the human question and answer sessions that may then be fed back into a machine learning (ML) algorithm 4720 (which may be the same as the ML algorithm 4702), which may be rerun on the corresponding segments of data, to produce a time- delayed classification output 4724 with improved classification accuracy of neighboring events, where the time delay corresponds to the QA process.
- ML machine learning
- the machine learning workflow 4700 includes data from the QA process 4708 being fed into ML training data 4722 to improve the ML algorithm models for subsequent segments such as improving on the ML algorithm 4702 and/or the ML algorithm 4702.
- Live spatiotemporal data 4730 may be aligned with other imperfect sources of data related to a sequence of spatial-temporal events.
- the alignment across imperfect sources of data related to a sequence of spatial-temporal events may include alignment using novel generalized distance metrics for spatiotemporal sequences combining event durations, ordering of events, additions/deletions of events, a spatial distance of events, and the like.
- the systems and methods disclosed herein may include modeling and dynamically interacting with an n-dimensional point-cloud.
- each point may be represented as an n-sphere whose radius may be determined by letting each n-sphere grow until it comes into contact with a neighboring n-sphere from a specified subset of the given point- cloud.
- This method may be similar to a Voronoi diagram in that may allocate a single n- dimensional cell for every point in the given cloud, with two distinct advantages.
- the first advantage includes that the generative kernel of each cell may also be its centroid.
- the second advantage includes continuously changing shifts in the resulting model when points are relocated in a continuous fashion (e.g., as a function of time in an animation, or the like).
- ten basketball players may be represented as ten nodes that are divided into two subsets of five teammates. At any given moment, each player's cell may be included in a circle extending in radius until it comes to be mutually tangent with an opponent's cell. By way of this example, players on the same team will have cells that overlap.
- the systems and methods disclosed herein may include a method for modeling locale as a function of time, some other specified or predetermined variable, or the like.
- coordinates of a given point or plurality of points are repeatedly sampled over a given window of time.
- the sampled coordinates may then be used to generate a convex hull, and this procedure may be repeated as desired and may yield a plurality of hulls that may be stacked for a discretized view of spatial variability over time.
- a single soccer player might have their location on a pitch sampled every second over the course of two minutes leading to a point cloud of location data and an associated convex hull.
- the process may begin anew with each two-minute window and the full assemblage of generated hulls may be, for example, rendered in a translucent fashion and may be layered so as to yield a map of the given player's region of activity.
- the systems and methods disclosed herein may include a method for sampling and modeling data by applying the recursive logic of a quadtree to a topologically deformed input or output space.
- the location of shots in a basketball game may be sampled in arc-shaped bins, which may be partitioned by angle-of-incidence to the basket and the natural logarithm of distance from the basket, and, in turn, yielding bins which may be subdivided and visualized according to the same rules governing a rectilinear quadtree.
- the systems and methods disclosed herein may include a method for modeling multivariate point-cloud data such that location coordinates map to the location, while velocity (or some other relevant vector) may be represented as a contour map of potential displacements at various time intervals.
- a soccer player running down a pitch may be represented by a node surrounded by nested ellipses each indicating a horizon of displacement for a given window of time.
- the systems and methods disclosed herein may include a method for modeling and dynamically interacting with a directed acyclic graph such that every node may be rendered along a single line, while the edges connecting nodes may be rendered as curves deviating from this line in accordance with a specified variable.
- these edges may be visualized as parabolic curves wherein the height of each may correspond to the flow, duration, latency, or the like of the process represented by the given edge.
- the methods and systems disclosed herein may include methods and systems for enabling a user to express preferences relating to display of video content and may include using machine learning to develop an understanding of at least one event, one metric related to the event, or relationships between events, metrics, venue, or the like within at least one video feed to determine at least one type for the event; automatically, under computer control, extracting the video content displaying the event and associating the machine learning understanding of the type for the event with the video content in a video content data structure; providing a user interface by which a user can indicate a preference for at least one type of content; and upon receiving an indication of the preference by the user, retrieving at least one video content data structure that was determined by the machine learning to have content of the type preferred by the user and providing the user with a video feed containing the content of the preferred type.
- the user interface is of at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, a touch screen device, a virtual reality or augmented reality headset, and a smart phone.
- the user interface further comprises an element for allowing a user to indicate a preference as to how content will be presented to the user.
- the machine learning further comprises determining an understanding of a context for the event and the context is stored with the video content data structure.
- the user interface further comprises an element for allowing a user to indicate a preference for at least one context.
- video content corresponding to the context preference is retrieved and displayed to the user.
- the context comprises at least one of the presence of a preferred player in the video feed, a preferred matchup of players in the video feed, a preferred team in the video feed, and a preferred matchup of teams in the video feed.
- the user interface allows a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein at least one of the metric and the graphic is based at least in part on the machine understanding.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interacting with video content method and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and using the context information for a plurality of such video content data structures to generate, automatically under computer control, producing a story or video clip that includes the video content data structure, wherein the content of the story is based on a user preference.
- the user preference for a type of content is based on at least one of a user expressed preference and a preference that is inferred based on user interaction with an item of content.
- the methods and systems disclosed herein may include methods and systems for enabling a user to express preferences relating to display of video content and may include a machine learning facility for developing an understanding of at least one event within at least one video feed to determine at least one type for the event; a video production facility for automatically, under computer control, extracting the video content displaying the event and associating the machine learning understanding of the type for the event with the video content in a video content data structure; a server for serving data to a user interface by which a user can indicate a preference for at least one type of content; and upon receiving at the server an indication of the preference by the user, retrieving at least one video content data structure that was determined by the machine learning to have content of the type preferred by the user and providing the user with a video feed containing the content of the preferred type.
- the user interface is of at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, and a smart phone.
- the user interface further comprises an element for allowing a user to indicate a preference as to how content will be presented to the user.
- the machine learning further comprises determining an understanding of a context for the event and the context is stored with the video content data structure.
- the user interface further comprises an element for allowing a user to indicate a preference for at least one context.
- video content corresponding to the context preference is retrieved and displayed to the user.
- the context comprises at least one of the presence of a preferred player in the video feed, a preferred matchup of players in the video feed, a preferred team in the video feed, and a preferred matchup of teams in the video feed.
- the user interface allows a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein the metric is based at least in part on the machine understanding.
- the methods and systems disclosed herein may include methods and systems delivering personalized video content and may include using machine learning to develop an understanding of at least one event within at least one video feed to determine at least one type for the event; automatically, under computer control, extracting the video content displaying the event and associating the machine learning understanding of the type for the event with the video content in a video content data structure; developing a personal profile for a user based on at least one of expressed preferences of the user, information about the user, and information collected about actions taken by the user with respect to at least one type of video content; and upon receiving an indication of the user profile, retrieving at least one video content data structure that was determined by the machine learning to have content of the type likely to be preferred by the user based on the user profile.
- the methods and systems disclosed herein may include methods and systems for delivering personalized video content and may include using machine learning to develop an understanding of at least one event within at least one video feed to determine at least one type for the event, wherein the video feed is a video feed for a professional game; using machine learning to develop an understanding of at least one event within a data feed relating to the motion of a non-professional player; based on the machine learning understanding of the video feed for the professional game and the data feed of the motion of the non-professional player, automatically, under computer control, providing an enhanced video feed that represents the nonprofessional player playing within the context of the professional game.
- the methods and systems may further include providing a facility having cameras for capturing 3D motion data and capturing video of a non-professional player to provide the data feed for the nonprofessional player.
- the non-professional player is represented by mixing video of the non-professional player with video of the professional game.
- the nonprofessional player is represented as an animation having attributes based on the data feed about the non-professional player.
- the methods and systems disclosed herein may also include one or more of the following features and capabilities: spatiotemporal pattern recognition (including active learning of complex patterns and learning of actions such as P&R, postups, play calls); hybrid methods for producing high quality labels, combining automated candidate generation from XYZ data, and manual refinement; indexing of video by automated recognition of game clock; presentation of aligned optical and video; new markings using combined display, both manual and automated (via pose detection etc.); metrics: shot quality, rebounding, defense and the like; visualizations such as Voronoi, heatmap distribution, etc.; embodiment on various devices; video enhancement with metrics & visualizations; interactive display using both animations and video; gesture and touch interactions for sports coaching and commentator displays; and cleaning of XYZ data using, for example, HMM, PBP, video, hybrid validation.
- spatiotemporal pattern recognition including active learning of complex patterns and learning of actions such as P&R, postups, play calls
- hybrid methods for producing high quality labels combining automated candidate generation from XYZ data, and manual refine
- Raw input XYZ is frequently noisy, missing, or wrong.
- XYZ data is also delivered with attached basic events such as possession, pass, dribble, shot. These are frequently incorrect. This is important because event identification further down the process (Spatiotemporal Pattern Recognition) sometimes depends on the correctness of these basic events. As noted above, for example, if two players' XY positions are switched, then "over” vs. "under” defense would be incorrectly switched, since the players' relative positioning is used as a critical feature for the classification. Also, PBP data sources are occasionally incorrect.
- Possession / Non-possession may use a Hidden Markov Model to best fit the data to these states. Shots and rebounds may use the possession model outputs, combined with 1) projected destination of the ball, and 2) PBP information. Dribbles may be identified using a trained ML algorithm and also using the output of the possession model.
- dribbles may be identified with a hidden Markov model.
- the hidden Markov model consists of three states:
- a player starts in State 1 when he gains possession of the ball. At all times players are allowed to transition to either their current state, or the state with a number one higher than their current state, if such a state exists.
- the players' likelihood of staying in their current state or transitioning to another state may be determined by the transition probabilities of the model as well as the observations.
- the transition probabilities may be learned empirically from the training data.
- the observations of the model consist of the player's speed, which is placed into two categories, one for fast movement, and one for slow movement, as well as the ball's height, which is placed into categories for low and high height.
- the cross product of these two observations represents the observation space for the model.
- the observation probabilities, given a particular state may be learned empirically from the training data. Once these probabilities are known, the model is fully characterized and may be used to classify when the player is dribbling on unknown data.
- the system has a library of anomaly detection algorithms to identify potential problems in the data. These include temporal discontinuities (intervals of missing data are flagged); spatial discontinuities (objects traveling is a non-smooth motion, "jumping"); interpolation detection (data that is too smooth, indicating that post-processing was done by the data supplier to interpolate between known data points in order to fill in missing data). This problem data is flagged for human review so that events detected during these periods are subject to further scrutiny.
- Spatio-player tracking may be undertaken in at least two types, as well as in a hybrid combined type.
- the broadcast video is obtained from multiple broadcast video feeds. Typically, this will include a standard "from the stands view” from the center stands midway-up, a backboard view, a stands view from a lower angle from each corner, and potentially other views.
- PTZ pan tilt zoom
- PTZ pan tilt zoom
- An alternative is a Special Camera Setup method. Instead of broadcast feeds, this uses feeds from cameras that are mounted specifically for the purposes of player tracking. The cameras are typically fixed in terms of their location, pan, tilt, zoom.
- a Hybrid/Combined System may be used. This system would use both broadcast feeds and feeds from the purpose-mounted cameras. By combining both input systems, accuracy is improved. Also, the outputs are ready to be passed on to the DataFX pipeline for immediate processing, since the DataFX will be painting graphics on top of the already- processed broadcast feeds. Where broadcast video is used, the camera pose must be solved in each frame, since the PTZ may change from frame to frame. Optionally, cameras that have PTZ sensors may return this info to the system, and the PTZ inputs are used as initial solutions for the camera pose solver.
- players may be identified by patches of color on the court. The corresponding positions are known since the camera pose is known, and we can perform the proper projections between 3D space and pixel space. [0438] Where purpose mounted cameras are used, multiple levels of resolution may be involved. Certain areas of the court or field require more sensitivity, e.g., on some courts, the color of the "paint" area makes it difficult to track players when they are in the paint. Extra cameras with higher dynamic range and higher zoom are focused on these areas. The extra sensitivity enables the computer vision techniques to train separate algorithms for different portions of the court, tuning each algorithm to its type of inputs and the difficulty of that task.
- the outputs of a player tracking system can feed directly into the DataFX production, enabling near- real-time DataFX.
- Broadcast video may also produce high-definition samples that can be used to increase accuracy.
- the methods and systems disclosed herein may include methods and systems for enabling interaction with a broadcast video content stream and may include a machine learning facility for developing an understanding of at least one event within a video feed for a video broadcast, the understanding including identifying context information relating to the event; and a touch screen user interface by which a broadcaster can interact with the video feed, wherein the options for broadcaster interaction are based on the context information, wherein the interaction with the touch screen controls the content of the broadcast video event.
- the touch screen interface is a large screen adapted to be seen by viewers of the video broadcast as the broadcaster uses the touch screen.
- a smaller touch screen is used by a commentator on air to control the information content being displayed, and the images/video on the touch screen is simultaneously displayed on a larger screen that is filmed and broadcast or is simultaneously displayed directly in the broadcast feed.
- the broadcaster can select from a plurality of context-relevant metrics, graphics, or combinations thereof to be displayed on the screen.
- the broadcaster can display a plurality of video feeds that have similar contexts as determined by the machine learning facility.
- the similarity of contexts is determined by comparing events within the video feeds.
- the broadcaster can display a superimposed view of at least two video feeds to facilitate a comparison of events from a plurality of video feeds.
- the comparison is of similar players from different, similar, or identical time periods.
- a similarity of players is determined by machine understanding of the characteristics of the players from the different time periods.
- the broadcaster can display a plurality of highlights that are automatically determined by a machine understanding of a live sports event that is the subject of the video feed.
- the highlights are determined based on similarity to highlights that have been identified for other events.
- the methods and systems disclosed herein may include methods and systems for enabling interaction with a broadcast video content stream and may include developing a machine learning understanding of at least one event within a video feed for a video broadcast, the understanding including identifying context information relating to the event; and providing a touch screen user interface by which a broadcaster can interact with the video feed, wherein the options for broadcaster interaction are based on the context information, wherein the interaction with the touch screen controls the content of the broadcast video event.
- the touch screen interface is a large screen adapted to be seen by viewers of the video broadcast as the broadcaster uses the touch screen.
- the broadcaster can select from a plurality of context-relevant metrics to be displayed on the screen.
- the broadcaster can display a plurality of video feeds that have similar contexts as determined by the machine learning facility.
- the similarity of contexts is determined by comparing events within the video feeds.
- the broadcaster can display a superimposed view of at least two video feeds to facilitate a comparison of events from a plurality of video feeds.
- the comparison is of similar players from different time periods.
- a similarity of players is determined by machine understanding of the characteristics of the players from the different time periods.
- the broadcaster can display a plurality of highlights that are automatically determined by a machine understanding of a live sports event that is the subject of the video feed.
- the highlights are determined based on similarity to highlights that have been identified for other events.
- the methods and systems disclosed herein may include methods and systems for enabling interaction with a broadcast video content stream and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing an application by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information, wherein the interaction with the video content data structure controls the presentation of a broadcast video event on a display screen.
- Methods and systems disclosed herein may include tracklet stitching.
- Optical player tracking results in short to medium length tracklets, which typically end when the system loses track of a player or the player collides (or passes close to) with another player.
- algorithms can stitch these tracklets together.
- systems may be designed for rapid interaction and for disambiguation and error handling. Such a system is designed to optimize human interaction with the system. Novel interfaces may be provided to specify the motion of multiple moving actors simultaneously, without having to match up movements frame by frame.
- custom clipping is used for content creation, such as involving OCR.
- Machine vision techniques may be used to automatically locate the "score bug" and determine the location of the game clock, score, and quarter information. This information is read and recognized by OCR algorithms.
- Post-processing algorithms using various filtering techniques are used to resolve issues in the OCR.
- Kalman filtering / HMMs may be used to detect errors and correct them. Probabilistic outputs (which measure the degree of confidence) assist in this error detection/correction.
- augmented or enhanced video with extracted semantics-based experience is provided based, at least in part, on 3D position/motion data.
- embeddable app content for augmented video with an extracted semantics-based experience there is provided the ability to automatically detect the court/field, and relative positioning of the camera, in (near) real time using computer vision techniques. This may be combined with automatic rotoscoping of the players in order to produce dynamic augmented video content.
- the methods and systems disclosed herein may include methods and systems for embedding video content in an application and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; taking an application that displays video content; and embedding the video content data structure in the application.
- the user interface of the application offers the user the option to control the presentation of the video content from the video content data structure in the application.
- the control of the presentation is based on at least one of a user preference and a user profile.
- the application is a mobile application that provides a story about an event and wherein the video content data structure comprises at least one of a content card and a digital still image.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application that allows user interaction with video content and may include a video ingestion facility for taking a video feed; a machine learning facility for developing an understanding of an event within the video feed, the understanding including identifying context information relating to the event; and a video production facility for automatically, under computer control, extracting the content displaying the event, associating the extracted content with the context information and producing a video content data structure that includes the associated context information; and using the context information for a plurality of such video content data structures to generate, automatically under computer control, a story that includes a sequence of the video content data structures.
- the content of the story is based on a user profile that is based on at least one of an expressed user preference, information about a user interaction with video content, and demographic information about the user.
- the methods and systems may further include determining a pattern relating to a plurality of events in the video feed and associating the determined pattern with the video content data structure as additional context information.
- the pattern relates to a highlight event within the video feed.
- the highlight event is associated with at least one of a player and a team.
- the embedded application allows a user to indicate at least one of a player and a team for which the user wishes to obtain video feeds containing the highlight events.
- the pattern relates to a comparison of events occurring at least one of within the video feed or within a plurality of video feeds. In embodiments, the comparison is between events occurring over time.
- the embedded application allows a user to select at least one player to obtain a video providing a comparison between the player and at least one of a past representation of the same player and a representation of another player.
- the pattern is a cause-and-effect pattern related to the occurrence of a following type of event after the occurrence of a pre-cursor type of event.
- the embedded application allows the user to review video cuts in a sequence that demonstrate the cause-and- effect pattern.
- the application provides a user interface for allowing a user to enter at least one of text and audio input to provide a narrative for a sequence of events within the video feed.
- the user may select a sequence of video events from within the feed for display in the application.
- the system upon accepting the user narrative, the system automatically generates an electronic story containing the events from the video feed and the narrative.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application that allows user interaction with video content and may include taking a video feed; using a machine learning facility to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; and automatically, under computer control, extracting the content displaying the event, associating the extracted content with the context information and producing a video content data structure that includes the associated context information.
- the methods and systems may further include using the context information for a plurality of such video content data structures to generate, automatically under computer control, a story that includes a sequence of the video content data structures.
- the user may interact with an application, such as on a phone, laptop, or desktop, or with a remote control, to control the display of broadcast video.
- an application such as on a phone, laptop, or desktop, or with a remote control
- options for user interaction may be customized based on the context of an event, such as by offering options to display context- relevant metrics for the event.
- selections may be used to control the display of broadcast video by the user, such as by selecting preferred, context-relevant metrics that appear as overlays, sidebars, scrolling information, or the like on the video display as various types of events take place in the video stream.
- a user may select settings for a context like a three point shot attempt, so that when the video displays three point shot attempts, particular metrics (e.g., the average success percentage of the shooter) are shown as overlays above the head of the shooter in the video.
- particular metrics e.g., the average success percentage of the shooter
- the methods and systems disclosed herein may include methods and systems for personalizing content for each type of user based on determining the context of the content through machine analysis of the content and based on an indication by the user of a preference for a type of presentation of the content.
- the methods and systems disclosed herein may include methods and systems for enabling a user to express preferences relating to display of video content and may include: taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing a user interface by which a user can indicate a preference for how content that is associated with a particular type of context will be presented to the user.
- a user may be presented with an interface element for a mobile application, browser, desktop application, remote control, tablet, smart phone, or the like, for indicating a preference as to how content will be presented to the user.
- the preference may be indicated for a particular context, such a context determined by a machine understanding of an event.
- a user may select to see certain metrics, graphics or additional information overlaid on top of the existing broadcast for certain types of semantic events such as players expected field goal percentage when they possess the ball or the type and effectiveness of defense being played on a pick and roll.
- the methods and systems disclosed herein may include methods and systems for automatically generating stories/content based on the personal profile of a viewer and their preferences or selections of contextualized content.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interacting with video content method and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and using the context information for a plurality of such video content data structures to generate, automatically under computer control, a story that includes the video content data structures, wherein the content of the story is based on a user preference.
- the user preference for a type of content is based on at least one of a user expressed preference and a preference that is inferred based on user interaction with an item of content.
- items of content that are associated, based on machine understanding, with particular events in particular contexts can be linked together, or linked with other content, to produce modified content such as stories.
- a game summary such as extracted from an online report about an event
- These stories can be customized for a user, such as linking a story about a game played by the user's favorite team with video cuts of the user's favorite player that were taken during the game.
- the methods and systems disclosed herein may include methods and systems for using machine learning to extract context information and semantically relevant events and situations from a video content stream, such that the events and situations may be presented according to the context of the content.
- the methods and systems disclosed herein may include methods and systems for embedding video content in an application and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; taking an application that displays video content; and embedding the video content data structure in the application, wherein the location of the embedded video content in the application is based on the context information.
- context-identified video cuts can be used to enrich or enhance applications, such as by embedding the cuts in relevant locations in the applications.
- a mobile application displaying entertainment content may be automatically populated with video cuts of events that are machine-extracted and determined to be of the appropriate type (based on context), for the application.
- a video game application can be enhanced, such as by including real video cuts of plays that fit a particular context (e.g., showing a pick-and-roll play where players A and B are matched up against players C and D in a real game, and the same matchup is determined to occur in the video game).
- a set of protocols such as APIs, may be defined, by which available categories (such as semantic categories, types of contexts, types of events, and the like) are specified, such that an application may call for particular types of events, which can, in turn, be embedded in the application.
- an application may be constructed with appropriate pointers, calls, objects, or the like, that allow a designer to specify, and call for, particular types of events, which may be automatically extracted from a library of machine- extracted, context-identified events and then embedded where appropriate into the application code.
- an application may provide stories about events, such as sporting events, and the machine-extracted content may include content cards or digital stills that are tagged by context so that they can be placed in appropriate locations in a story.
- the application can provide automatically generated content and stories, enhanced by content from a live game.
- an application may recommend video clips based on the use of keywords that match machine learned semantics that enable users to post or share video clips automatically tailored to text that they are writing. For example, clips may be recommended that include the presence of a particular player, that include a particular type of play (e.g., "dunks") and/or that are from a particular time period (e.g., "last night,” etc.).
- semantic events may be translated and cataloged into data and patterns.
- the methods and systems disclosed herein may include methods and systems for embedding content cards or digital stills with contextualized content stories/visualizations into a mobile application. They may include automatically generated content, such as stories, extracted from a live game delivered to users via an application, such as a mobile application, an augmented reality glasses application, a virtual reality glasses application, or the like.
- the application is a mobile application that provides a story about an event and wherein the video content data structure comprises at least one of a content card and a digital still image.
- the methods and systems disclosed herein may include methods and systems for applying contextualized content from actual sporting events to video games to improve the reality of the game play.
- the methods and systems disclosed herein may include methods and systems for improving a video game and may include taking a video feed; using machine learning to develop an understanding of at least one first real event within the video feed, the understanding including identifying context information relating to the first real event; taking a game event coded for display within a video game; matching the context information for the real event with the context of the game event in the video game; comparing the display of the game event to the video for the real event; and modifying the coding of the game event based on the comparison.
- context information can be used to identify video cuts that can be used to improve video games, such as by matching the context of a real event with a similar context in a coded video game event, comparing the video for the real event with the video game display of a similar event, and modifying the video event to provide a more faithful simulation of the real event.
- the methods and systems disclosed herein may include methods and systems for taking the characteristics of a user either from a video capture of their recreational play or through user generated features and importing the user' s avatar into a video game.
- the methods and systems disclosed herein may include methods and systems for interactive contextualized content that can be filtered and adjusted via a touch screen interface.
- the user interface is a touch screen interface.
- the methods and systems disclosed herein may include methods and systems for real time display of relevant fantasy and betting metrics overlaid on a live game feed.
- the methods and systems disclosed herein may include methods and systems for real time adjustment of betting lines and/or additional betting option creation based on in-game contextual content.
- the methods and systems disclosed herein may include methods and systems for taking a video feed and using machine learning to develop an understanding of at least one first event within the video feed. The understanding includes identifying context information relating to the first event.
- the methods and systems also include determining a metric based on the machine understanding.
- the metric is relevant to at least one of a wager and a fantasy sports outcome.
- the methods and systems include presenting the metric as an overlay for an enhanced video feed.
- the metrics described throughout this disclosure may be placed as overlays on video feeds. For example, metrics calculated based on machine-extracted events that are relevant to betting lines, fantasy sports outcomes, or the like, can be presented as overlays, scrolling elements, or the like on a video feed.
- the metrics to be presented can be selected based on context information, such as showing fantasy metrics for players who are on screen at the time or showing the betting line where a scoring play impacts the outcome of a bet.
- the displays may be customized and personalized for a user, such as based on that user' s fantasy team for a given week or that user's wagers for the week.
- the methods and systems disclosed herein may include methods and systems for taking a video feed of a recreational event; using machine learning to develop an understanding of at least one event within the video feed, the understanding including identifying context information relating to the event; and based on the machine understanding, providing content including information about a player in the recreational event based on the machine understanding and the context.
- the methods and systems may further include providing a comparison of the player to at least one professional player according to at least one metric that is based on the machine understanding.
- machine understanding can be applied to recreational venues, such as for capturing video feeds of recreational games, practices, and the like. Based on machine understanding, highlight clips, metrics, and the like, as disclosed throughout this disclosure, may be extracted by processing the video feeds, including machine understanding of the context of various events within the video.
- metrics, video, and the like can be used to provide players with personalized content, such as a highlight reel of good plays, or a comparison to one or more professional players (in video cuts, or with semantically relevant metrics).
- Context information can allow identification of similar contexts between recreational and professional events, so that a player can see how a professional acted in a context that is similar to one faced by the recreational player.
- the methods and systems may enable the ability to use metrics and events recorded from a video stream to enable the creation of a recreational fantasy sports game with which users can interact.
- the methods and systems may enable the ability for to recognize specific events or metrics from a recreational game and compare them to similar or parallel events from a professional game to help coach a recreational player or team or for the creation of a highlight reel that features both recreational and professional video cuts.
- the methods and systems disclosed herein may include methods and systems for providing enhanced video content and may include using machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one type for each of the plurality of events; extracting a plurality of video cuts from the video feed and indexing the plurality of video cuts based on at least one type of event determined by the understanding developed by machine learning; and making the indexed and extracted video cuts available to a user.
- the user is enabled to at least one of edit, cut, and mix the video cuts to provide an enhanced video containing at least one of the video cuts.
- the user is enabled to share the enhanced video.
- the methods and systems may further include indexing at least one shared, enhanced video with the semantic understanding of the type of events in that was determined by machine learning.
- the methods and systems may further include using the index information for the shared, enhanced video to determine a similarity between the shared, enhanced video and at least one other video content item.
- the similarity is used to identify additional extracted, indexed video cuts that may be of interest to the user.
- the similarity is used to identify other users who have shared similarly enhanced video.
- the similarity is used to identify other users who are likely to have an interest in the shared, enhanced video.
- the methods and systems may further include recommending at least one of the shared, enhanced video and one of the video cuts based on an understanding of the preferences of the other users.
- the similarity is based at least in part on user profile information for users who have indicated an interest in the video cut and the other video content item.
- the methods and systems disclosed herein may include methods and systems for providing enhanced video content and may include using machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one type for each of the plurality of events; extracting a plurality of video cuts from the video feed and indexing the plurality of video cuts to form an indexed set of extracted video cuts, wherein the indexing is based on at least one type of event determined by the understanding developed by machine learning; determining at least one pattern relating to a plurality of events in the video feed; adding the determined pattern information to the index for the indexed set of video cuts; and making the indexed and extracted video cuts available to a user.
- the user is enabled to at least one of edit, cut, and mix the video cuts to provide an enhanced video containing at least one of the video cuts.
- the user is enabled to share the enhanced video.
- the video cuts are clustered based on the patterns that exist within the video cuts.
- the pattern is determined automatically using machine learning and based on the machine understanding of the events in the video feed.
- the pattern is a highlight event within the video feed.
- the highlight event is presented to the user when the indexed and extracted video cut is made available to the user.
- the user is prompted to watch a longer video feed upon viewing the indexed and extracted video cut.
- a touch screen or other gesture -based interface experience based, at least in part, on extracted semantic events.
- the methods and systems disclosed herein may include methods and systems for machine extracting semantically relevant events from 3D motion/position data captured at a venue, calculating a plurality of metrics relating to the events, and presenting the metrics in a video stream based on the context of the video stream.
- the methods and systems disclosed herein may include methods and systems for producing machine-enhanced video streams and may include taking a video feed from 3D motion and position data from a venue; using machine learning to develop an understanding of at least one first event within the video feed, the understanding including identifying context information relating to the first event; calculating a plurality of metrics relating to the events; and producing an enhanced video stream that presents the metrics in the video stream, wherein the presentation of at least one metric is based on the context information for the event with which the metric is associated in the video stream.
- semantically relevant events determined by machine understanding of 3D motion/position data for an event from a venue can be used to calculate various metrics, which may be displayed in the video stream of the event.
- Context information which may be determined based on the types and sequences of events, can be used to determine what metrics should be displayed at a given position within the video stream. These metrics may also be used to create new options for users to place wagers on or be integrated into a fantasy sports environment.
- the methods and systems disclosed herein may include methods and systems enabling a user to cut or edit video based on machine learned context and share the video clips. These may further include allowing a user to interact with the video data structure to produce an edited video data stream that includes the video data structure. In embodiments, the interaction includes at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the methods and systems may enable the ability for users to interact with video cuts through an interface to enhance the content with graphics or metrics based on a pre-set set of options, and then share a custom cut and enhanced clip.
- the methods and systems may include the ability to automatically find similarity in different video clips based on semantic context contained in the clips, and then cluster clips together or to recommend additional clips for viewing.
- the methods and systems may include the ability to extract contextualized content from a feed of a recreational event to immediately deliver content to players, including comparing a recreational player to a professional player based on machine learned understanding of player types.
- the second screen interface unique to extracted semantic events and user selected augmentations.
- the second screen may display real-time, or near real time, contextualized content.
- the methods and systems disclosed herein may include methods and systems for taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; and producing a video content data structure that includes the associated context information.
- the methods and systems may further include determining a plurality of semantic categories for the context information and filtering a plurality of such video content data structures based on the semantic categories.
- the methods and systems may further include matching the events that occur in one video feed to those that occur in a separate video feed such that the semantic understanding captured in the first video feed can be used to at least one of filter and cut a separate second video feed based on the same events.
- the methods and systems may further include determining a pattern relating to a plurality of the events and providing a content data structure based on the pattern.
- the pattern comprises a plurality of important plays in a sports event that are identified based on comparison to similar plays from previous sports events.
- the pattern comprises a plurality of plays in a sports event that is determined to be unusual based on comparison to video feeds from other sports events.
- the methods and systems may further include extracting semantic events over time to draw a comparison of at least one of a player and a team over time.
- the methods and systems may further include superimposing video of events extracted from video feeds from at least two different time periods to illustrate the comparison.
- the methods and systems may further include allowing a user to interact with the video data structure to produce an edited video data stream that includes the video data structure.
- the interaction includes at least one of editing, mixing, cutting, and sharing a video clip that includes the video data structure.
- the methods and systems may further include enabling users to interact with the video cuts through a user interface to enhance the video content with at least one graphic element selected from a menu of options.
- the methods and systems may further include enabling a user to share the enhanced video content.
- the methods and systems may further include enabling a user to find similar video clips based on the semantic context identified in the clips.
- the methods and systems may further include using the video data structure and the context information to construct modified video content for a second screen that includes the video data structure.
- the content for the second screen correlates to the timing of an event displayed on a first screen.
- the content for the second screen includes a metric determined based on the machine understanding, wherein the metric is selected based on the context information.
- the methods and systems disclosed herein may include methods and systems for displaying contextualized content of a live event on a second screen that correlates to the timing of the live event on the first screen. These may include using the video data structure and the context information to construct modified video content for a second screen that includes the video data structure.
- the content for the second screen correlates to the timing of an event displayed on a first screen.
- the content for the second screen includes a metric determined based on the machine understanding, wherein the metric is selected based on the context information.
- machine extracted metrics and video cuts can be displayed on a second screen, such as a tablet, smart phone, or smart remote control screen, such as showing metrics that are relevant to what is happening, in context, on a main screen.
- a second screen such as a tablet, smart phone, or smart remote control screen, such as showing metrics that are relevant to what is happening, in context, on a main screen.
- the methods and systems disclosed herein may include methods and systems for an ingestion facility adapted or configured to ingest a plurality of video feeds; a machine learning system adapted or configured to apply machine learning on a series of events in a plurality of video feeds in order to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; an extraction facility adapted or configured to automatically, under computer control, extract the content displaying the event and associate the extracted content with the context information; and a video publishing facility for producing a video content data structure that includes the associated context information.
- the methods and systems may further include an analytic facility adapted or configured to determine a plurality of semantic categories for the context information and filter a plurality of such video content data structures based on the semantic categories.
- the methods and systems may further include a matching engine adapted or configured to match the events that occur in one video feed to those that occur in a separate video feed such that the semantic understanding captured in the first video feed can be used to at least one of filter and cut a separate second video feed based on the same events.
- the methods and systems may further include a pattern recognition facility adapted or configured to determine a pattern relating to a plurality of the events and providing a content data structure based on the pattern.
- the methods and systems disclosed herein may include methods and systems for displaying machine extracted, real time, contextualized content based on machine identification of a type of event occurring in a live video stream.
- the methods and systems disclosed herein may include methods and systems for taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; and producing a video content data structure that includes the associated context information.
- the methods and systems disclosed herein may include methods and systems for providing context information in video cuts that are generated based on machine extracted cuts that are filtered by semantic categories.
- the methods and systems disclosed herein may include methods and systems for determining a plurality of semantic categories for the context information and filtering a plurality of the video content data structures based on the semantic categories.
- the methods and systems disclosed herein may include methods and systems for matching the events that occur in one video feed to those that occur in a separate video feed such that the semantic understanding captured in the first video feed can be used to filter and cut a separate second video feed based on these same events.
- the methods and systems disclosed herein may include methods and systems for enabling user interaction with a mobile application that displays extracted content, where the user interaction is modified based on the context of the content (e.g., the menu is determined by context).
- the methods and systems disclosed herein may include methods and systems for enabling an application allowing user interaction with video content and may include an ingestion facility adapted or configured to access at least one video feed, wherein the ingestion facility may be executing on at least one processor; a machine learning facility operating on the at least one video feed to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; an extraction facility adapted or configured to automatically, under computer control, extract the content displaying the event and associate the extracted content with the context information; a video production facility adapted or configured to produce a video content data structure that includes the associated context information; and an application having a user interface by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- the application is a mobile application. In embodiments, the application is at least one of a smart television application, a virtual reality headset application and an augmented reality application.
- the user interface is a touch screen interface. In embodiments, the user interface allows a user to enhance the video feed by selecting a content element to be added to the video feed. In embodiments, the content element is at least one of a metric and a graphic element that is based on the machine understanding. In embodiments, the user interface allows the user to select content for a particular player of a sports event. In embodiments, the user interface allows the user to select content relating to a context involving the matchup of two particular players in a sports event.
- the system takes at least two video feeds from different time periods
- the machine learning facility determines a context the includes a similarity between at least one of a plurality of players and a plurality of plays in the two feeds
- the user interface allows the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the video feed comprises 3D motion camera data captured from a live sports venue.
- the ability of the machine learning facility to develop the understanding is developed by feeding the machine learning facility a plurality of events for which context has already been identified.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interaction with video content and may include taking at least one video feed; applying machine learning on the at least one video feed to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing a mobile application having a user interface by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- the user interface is a touch screen interface.
- the user interface allows a user to enhance the video feed by selecting a content element to be added to the video feed.
- the content element is at least one of a metric and a graphic element that is based on the machine understanding.
- the user interface allows the user to select content for a particular player of a sports event.
- the user interface allows the user to select content relating to a context involving the matchup of two particular players in a sports event.
- the system takes at least two video feeds from different time periods
- the machine learning facility determines a context the includes a similarity between at least one of a plurality of players and a plurality of plays in the two feeds and the user interface allows the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the video feed comprises 3D motion camera data captured from a live sports venue.
- the ability of the machine learning facility to develop the understanding is developed by feeding the machine learning facility a plurality of events for which context has already been identified.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interacting with video content and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing a mobile application by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- machine extracted content may be provided to users via a mobile application, through which the users may display and interact with the content, such as by selecting particular types of content based on a desired semantic category (such as by selecting the category in list, menu, or the like), playing content (including pausing, rewinding, fast forwarding, and the like), and manipulating content (such as positioning content within a display window, zooming, panning, and the like).
- a desired semantic category such as by selecting the category in list, menu, or the like
- playing content including pausing, rewinding, fast forwarding, and the like
- manipulating content such as positioning content within a display window, zooming, panning, and the like.
- the nature of the permitted interaction may be governed by the context information associated with the content, where the context information is based on a machine understanding of the content and its associated context.
- the user may be permitted to select from a set of metrics that are relevant to rebounding, so that the selected metrics from a context-relevant set are displayed on the screen with the content.
- the context is different, such as if the content relates to a series of pick-and-roll plays by a particular player, different metrics may be made available for selection by the user, such as statistics for that player, or metrics appropriate for pick-and-rolls.
- the machine-extracted understanding of an event can be used to customize the content displayed to the user, including to allow the user to select context-relevant information for display.
- the methods and systems disclosed herein may include methods and systems for allowing a user to control a presentation of a broadcast video event, where the options for control are based on a context of the content as determined by machine extraction of semantically relevant events from the content.
- spatiotemporal pattern recognition based, at least in part, on optical XYZ alignment for semantic events.
- verification and refinement of spatiotemporal semantic pattern recognition based, at least in part, on hybrid validation from multiple sources.
- video cut-up based on extracted semantics.
- a video cut-up is a remix made up of small clips of video that are related to each other in some meaningful way.
- the semantic layer enables real-time discovery and delivery of custom cut-ups.
- the semantic layer may be produced in one of two ways: (1) Video combined with data produces semantic layer, or (2) video directly to a semantic layer. Extraction may be through ML or human tagging.
- video cut-up may be based, at least in part, on extracted semantics, controlled by users in a stadium and displayed on a Jumbotron.
- video cut-up may be based, at least in part, on extracted semantics, controlled by users at home and displayed on broadcast TV.
- video cut-up may be based, at least in part, on extracted semantics, controlled by individual users and displayed on the web, tablet, or mobile for that user.
- video cut-up may be based, at least in part, on extracted semantics, created by an individual user, and shared with others. Sharing could be through inter-tablet/inter-device communication, or via mobile sharing sites.
- the methods and systems disclosed herein may include methods and systems for enabling an application allowing user interaction with video content and may include an ingestion facility for taking at least one video feed; a machine learning facility operating on the at least one video feed to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; an extraction facility for automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; a video production facility for producing a video content data structure that includes the associated context information; and an application having a user interface by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- the application is a mobile application. In embodiments, the application is at least one of a smart television application, a virtual reality headset application and an augmented reality application.
- the user interface is a touch screen interface. In embodiments, the user interface allows a user to enhance the video feed by selecting a content element to be added to the video feed. In embodiments, the content element is at least one of a metric and a graphic element that is based on the machine understanding. In embodiments, the user interface allows the user to select content for a particular player of a sports event. In embodiments, the user interface allows the user to select content relating to a context involving the matchup of two particular players in a sports event.
- the system takes at least two video feeds from different time periods
- the machine learning facility determines a context the includes a similarity between at least one of a plurality of players and a plurality of plays in the two feeds and the user interface allows the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the video feed comprises 3D motion camera data captured from a live sports venue.
- the ability of the machine learning facility to develop the understanding is developed by feeding the machine learning facility a plurality of events for which context has already been identified.
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interaction with video content and may include taking at least one video feed; applying machine learning on the at least one video feed to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing a mobile application having a user interface by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- the user interface is a touch screen interface.
- the user interface allows a user to enhance the video feed by selecting a content element to be added to the video feed.
- the content element is at least one of a metric and a graphic element that is based on the machine understanding.
- the user interface allows the user to select content for a particular player of a sports event.
- the user interface allows the user to select content relating to a context involving the matchup of two particular players in a sports event.
- the system takes at least two video feeds from different time periods
- the machine learning facility determines a context the includes a similarity between at least one of a plurality of players and a plurality of plays in the two feeds
- the user interface allows the user to select at least one of the players and the plays to obtain a video feed that illustrates a comparison.
- the user interface includes options for at least one of editing, cutting, and sharing a video clip that includes the video data structure.
- the video feed comprises 3D motion camera data captured from a live sports venue.
- the ability of the machine learning facility to develop the understanding is developed by feeding the machine learning facility a plurality of events for which context has already been identified.
- the methods and systems disclosed herein may include methods and systems for an analytic system and may include a video ingestion facility for ingesting at least one video feed; a machine learning facility that develops an understanding of at least one event within the video feed, wherein the understanding identifies at least a type of the event and a time of the event in an event data structure; a computing architecture enabling a model that takes one or more event data structures as input and applies at least one calculation to transform the one or more event data structures into an output data structure; and a data transport layer of the computing architecture for populating the model with the event data structures as input to the model.
- the output data structure includes at least one prediction.
- the prediction is of an outcome of at least one of a sporting event and at least one second event occurring within a sporting event.
- the video feed is of a live sporting event, wherein the prediction is made during the live sporting event, and wherein the prediction relates to the same sporting event.
- the prediction is based on event data structures from a plurality of video feeds.
- the prediction is used for at least one of placing a wager, setting a line for a wager, interacting with a fantasy program, setting a parameter of a fantasy program, providing insight to a coach and providing information to a fan.
- the model takes inputs from a plurality of data sources in addition to the event data structures obtained from the video feed.
- the methods and systems may further include a pattern analysis facility that takes a plurality of the event data structures and enables analysis of patterns among the event data structures.
- the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the methods and systems may further include a second machine learning facility for refining the model based on outcomes of a plurality of predictions made using the model.
- the methods and systems disclosed herein may include methods and systems for an analytic method and may include ingesting at least one video feed in a computing platform capable of handling video data; developing an understanding of at least one event within the video feed using machine learning, wherein the understanding identifies at least a type of the event and a time of the event in an event data structure; providing a computing architecture that enables a model that takes one or more event data structures as input and applies at least one calculation to transform the one or more event data structures into an output data structure; and populating the model with the event data structures as input to the model.
- the output data structure includes at least one prediction.
- the prediction is of an outcome of at least one of a sporting event and at least one-second event occurring within a sporting event.
- the video feed is of a live sporting event, wherein the prediction is made during the live sporting event, and wherein the prediction relates to the same sporting event.
- the prediction is based on event data structures from a plurality of video feeds.
- the prediction is used for at least one of placing a wager, setting a line for a wager, interacting with a fantasy program, setting a parameter of a fantasy program, providing insight to a coach and providing information to a fan.
- the model takes inputs from a plurality of data sources in addition to the event data structures obtained from the video feed.
- the methods and systems may further include providing a pattern analysis facility that takes a plurality of the event data structures and enables analysis of patterns among the event data structures.
- the pattern analysis facility includes at least one tool selected from the group consisting of a pattern visualization tool, a statistical analysis tool, a machine learning tool, and a simulation tool.
- the methods and systems may further include at least one of providing and using a second machine learning facility to refine the model based on outcomes of a plurality of predictions made using the model.
- the methods and systems disclosed herein may include methods and systems for taking a video feed; using machine learning to develop an understanding of a semantically relevant event within the video feed; indexing video segments of the video feed with information indicating the semantically relevant events identified within the feed by the machine learning; and applying machine learning to a plurality of the semantically relevant events to determine a pattern of events.
- the pattern is within a video feed.
- the pattern is across a plurality of video feeds.
- the pattern corresponds to a narrative structure.
- the narrative structure corresponds to a recurring pattern of events.
- the narrative structure relates to a sporting event and wherein the pattern relates to at least one of a blow-out victory pattern, a comeback win pattern, a near comeback pattern, a back-and-forth game pattern, an individual achievement pattern, an injury pattern, a turning point moment pattern, a close game pattern, and a team achievement pattern.
- the indexed video segments are arranged to support the narrative structure.
- the arranged segments are provided in an interface for developing a story using the segments that follow the narrative structure and wherein a user may at least one of edit and enter additional content for the story.
- summary content for the narrative structure is automatically generated, under computer control, to provide a story that includes the video sequences.
- the methods and systems may further include delivering a plurality of the automatically generated stories at least one of from a defined time period and of a defined type, allowing a user to indicate whether they like or dislike the delivered stories, and using the indications to inform later delivery of at least one additional story.
- the pattern is relevant to a prediction.
- the prediction is related to a wager, and the pattern corresponds to similar patterns that were used to make predictions that resulted in successful wagers in other situations.
- the methods and systems disclosed herein may include methods and systems for machine-extracting semantically relevant events from a video content stream and determining a pattern relating to the events.
- the methods and systems also include providing a content stream based on the pattern.
- the content stream is used to provide coaching information based on the pattern.
- the content stream is used to assist prediction of an outcome in a fantasy sports contest.
- the pattern is used to provide content for a viewer of a sporting event.
- the methods and systems disclosed herein may include methods and systems for machine-extracting semantically relevant events from a video content stream; determining a pattern relating to the events; storing the pattern information with the extracted events; and providing a user with the option to view and interact with the patterns, wherein at least one of the patterns and the interaction options are personalized based on a profile of the user.
- the profile is based on at least one of user indication of a preference, information about actions of the user, and demographic information about the user.
- the pattern comprises at least one of a trend and a statistic that is curated to correspond with the user profile.
- the pattern relates to a comparison of a professional athlete to another athlete.
- the other athlete is the user and the comparison is based on a playing style of the user as determined by at least one of information indicated by the user and a video feed of the user.
- the pattern relates to an occurrence of an injury.
- the pattern information is used to provide coaching to prevent an injury.
- the methods and systems may further include automatically generating, under computer control, an injury prevention regimen based on the pattern and based on information about the user.
- the methods and systems disclosed herein may include methods and systems for machine-extracting semantically relevant events from a video content stream, determining a pattern relating to the events, and providing a content stream based on the pattern.
- the methods and systems may further include determining a pattern relating to a plurality of the events and providing a content data structure based on the pattern.
- machine-extracted information about events and contexts may be used to determine on or more patterns, such as by analyzing time series, correlations, and the like in the machine-extracted events and contexts. For example, tendencies of a team to follow running a certain play with a particular play may be determined by comparing instances of the two plays over time.
- Embodiments may include extracting particularly interesting or potential "game changing" plays by understanding the context of an individual event and comparing it to similar events from previous games.
- Embodiments may include extracting situations or plays that are particularly rare or unique by understanding the context of an individual event and comparing it to similar events from previous games.
- Embodiments may include extracting semantic events over time to draw a comparison of a player's or team's trajectory over time and superimposing video to draw out this comparison.
- the methods and systems disclosed herein may include methods and systems for a model to predict the outcome of a game or events within a game based on a contextualized understanding of a live event for use in betting/fantasy, coaching, augmented fan experiences, or the like.
- the methods and systems disclosed herein may include methods and systems for an analytic system and may include taking a video feed; using machine learning to develop an understanding of at least one first event within the video feed, the understanding including identifying context information relating to the first event; taking a model used to predict the outcome of at least one of a live game and at least one second event within a live game; and populating the model with the machine understanding of the first event and the context information to produce a prediction of an outcome of at least one of the game and the second event.
- the model is used for at least one of placing a wager, setting a line for a wager, interacting with a fantasy program, setting a parameter of a fantasy program, providing insight to a coach and providing information to a fan.
- machine-extracted event and context information can be used to populate one or more predictive models, such as models used for betting, fantasy sports, coaching, and entertainment.
- the machine understanding including various metrics described throughout this disclosure, can provide or augment other factors that are used to predict an outcome. For example, outcomes from particular matchups can be machine extracted and used to predict outcomes from similar matchups in the future. For example, based on the machine understood context of a moment in an individual game, and the machine understanding of similar moments from previous games, a model can be created to predict the outcome of an individual play or a series of plays on which an individual can place a bet or on which a betting line may be set.
- the methods and systems disclosed herein may include methods and systems for suggestions of bets to make based on patterns of previously successful bets. For example, a user may be prompted with an option to place a bet based on previous betting history on similar events or because a particular moment is an opportunistic time to place a bet based on the context of a game and other user generated preferences or risk tolerances.
- the methods and systems disclosed herein may include methods and systems for automated storytelling, such as the ability to use patterns extracted from semantic events, metrics derived from tracking data, and combinations thereof to populate interesting stories about the content.
- the methods and systems disclosed herein may include methods and systems for enabling automated generation of stories and may include taking a video feed; using machine learning to develop an understanding of a semantically relevant event within the video feed, the understanding including identifying context information relating to the event; providing a narrative structure for a story, wherein the narrative structure is arranged based on the presence of semantic types of events and the context of those events; and automatically, under computer control, generating a story following the narrative structure, wherein the story is populated based on a sequence of the machine-understood events and the context information.
- patterns from semantic events may be used to populate stories.
- Various narrative structures can be developed, corresponding to common patterns of events (e.g., stories about blow-out victories, comeback wins, back-and-forth games, games that turned on big moments, or the like).
- Machine extracting of events and contexts can allow identification of patterns in the events and contexts that allow matching to one or more of the narrative structures, as well as population of the story with content for the events, such as video cuts or short written summaries that are determined by the machine extraction (e.g., "in the first quarter, Team A took the lead, scoring five times on the pick-and-roll.”).
- the methods and systems disclosed herein may include methods and systems for enabling a mobile application allowing user interacting with video content and may include taking a video feed; using machine learning to develop an understanding of an event within the video feed, the understanding including identifying context information relating to the event; automatically, under computer control, extracting the content displaying the event and associating the extracted content with the context information; producing a video content data structure that includes the associated context information; and providing a mobile application by which a user can interact with the video content data structure, wherein the options for user interaction are based on the context information.
- machine extracted content may be provided to users via a mobile application, through which the users may display and interact with the content, such as by selecting particular types of content based on a desired semantic category (such as by selecting the category in list, menu, or the like), playing content (including pausing, rewinding, fast forwarding, and the like), and manipulating content (such as positioning content within a display window, zooming, panning, and the like).
- a desired semantic category such as by selecting the category in list, menu, or the like
- playing content including pausing, rewinding, fast forwarding, and the like
- manipulating content such as positioning content within a display window, zooming, panning, and the like.
- the nature of the permitted interaction may be governed by the context information associated with the content, where the context information is based on a machine understanding of the content and its associated context.
- the user may be permitted to select from a set of metrics that are relevant to rebounding, so that the selected metrics from a context-relevant set are displayed on the screen with the content.
- the context is different, such as if the content relates to a series of pick-and-roll plays by a particular player, different metrics may be made available for selection by the user, such as statistics for that player, or metrics appropriate for pick-and-rolls.
- the machine-extracted understanding of an event can be used to customize the content displayed to the user, including to allow the user to select context-relevant information for display.
- the methods and systems disclosed herein may include methods and systems for allowing a user to control the presentation of a broadcast video event, where the options for control are based on a context of the content as determined by machine extraction of semantically relevant events from the content.
- X, Y, and Z data may be collected for purposes of inferring player actions that have a vertical component.
- the methods and systems disclosed herein may employ a variety of computer vision, machine learning, and/or active learning techniques and tools to extract, analyze and process data elements originating from sources, such as, but not limited to, input data sources relating to sporting events and items in them, such as players, venues, items used in sports (such as balls, pucks, and equipment), and the like.
- sources such as, but not limited to, input data sources relating to sporting events and items in them, such as players, venues, items used in sports (such as balls, pucks, and equipment), and the like.
- These data elements may be available as video feeds in an example, such that the video feeds may be captured by image recognition devices, video recognition devices, image and video capture devices, audio recognition devices, and the like, including by use of various devices and components such as a camera (such as a tracking camera or broadcast camera), a microphone, an image sensor, or the like.
- Audio feeds may be captured by microphones and similar devices, such as integrated on or with cameras or associated with independent audio capture systems.
- Input feeds may also include tracking data from chips or sensors (such as wearable tracking devices using accelerometers and other motion sensors), as well as data feeds about an event, such as a play-by-play data feed, a game clock data feed, and the like.
- facial recognition systems may be used to capture facial images of players, such as to assist in recognition of players (such as in cases where player numbers are absent or obscured) and to capture and process expressions of players, such as emotional expressions, micro-expressions, or the like.
- These expressions may be associated with events, such as to assist in machine understanding (e.g., an expression may convey that the event was exciting, meaningful, the like, that it was disappointing to one constituency, that it was not important, or the like).
- Machine understanding may thus be trained to recognize expressions and provide an expression-based understanding of events, such as to augment one or more data structures associated with an event for further use in the various embodiments described herein.
- a video feed may be processed based on a machine understanding of expressions to extract cuts that made players of one team happy.
- a cut showing an emotional reaction (such as by a player, fan, teammate, or coach) to an event may be associated with a cut of the event itself, providing a combined cut that shows the event and the reaction it caused.
- the computer vision, machine learning and/or active learning tools and techniques may receive the data elements from various input feeds and devices as a set of inputs either in real-time (such as in case of a live feed or broadcast) or at a different time (such as in case of a delayed broadcast of the sporting or any other event) without limitations.
- the computer-controlled intelligent systems may process the set of inputs, apply machine learning and natural language processing using artificial intelligence (AI) and natural language processing (NLP) capabilities to produce a set of services and outputs.
- AI artificial intelligence
- NLP natural language processing
- the set of services and outputs may signify spatial-temporal positions of the players and sport accessories/objects such as a bat, ball, football, and the like.
- the set of services and outputs may represent spatial-temporal alignments of the inputs such as the video feeds, etc.
- a broadcast video feed may be aligned in time with another input feed, such as input from one or more motion tracking cameras, inputs from player tracking systems (such as wearable devices), and the like.
- the set of services and outputs may include machine understood contextual outputs involving machine learning or understanding that may be built using various levels of artificial intelligence, algorithmic processes, computer- controlled tasks, custom rules, and the like, such as described throughout this disclosure.
- the machine understanding may include various levels of semantic identification, as well as information of position and speed information for various items or elements, identification of basic events such as various types of shots and screens during a sporting event, and identification of complex events or a sequence of events such as various types of plays, higher level metrics and patterns involving such as game trajectory, style of play, strengths and weaknesses of teams and team members/players from each team, and the like.
- the machine learning tools and input feed alignment may allow automatic generation of content and information such as statistics, predictions, comparisons, and analysis.
- the machine learning tools may further allow to generate outputs based on a user query input such as to determine various predictive analytics for a particular team player in view of historical shots and screens in a particular context, determine possibilities of success and failures in particular zones and game scenarios conditioned to particular user inputs, and the like.
- the machine understanding tools may simulate entire aspects of real-life sporting events on a computer screen utilizing visualization and modeling examples.
- the services and outputs generated by the intelligent computer-controlled systems may be used in a variety of ways such as generation of a live feed or a delayed feed during a sporting event in real time or at a later broadcasting time after the sporting event.
- the services and outputs may allow generating various analyses of statistics, trends, and strategy before events or across multiple events.
- the services and outputs may facilitate an interactive user session to extract contextual details relating to instantaneous sporting sessions of the sporting events in association with user defined queries, constraints, and rules.
- the services and outputs generated by the computer-controlled intelligent systems may enable spatiotemporal analysis of various game attributes and elements for exploring, learning, analyzing such sporting events and utilize analytics results to generate predictive models and predictive analytics for gaming strategy. These services and outputs may provide valuable insights and learnings that are otherwise not visible.
- the methods and systems disclosed herein may employ delay-dependent computer vision and machine learning systems (or the intelligent computer-controlled systems) for providing delay-dependent services and outputs with respect to the occurrence of a sporting event.
- the services and outputs as discussed herein may be employed in different applications with varying time delays relative to the actual occurrence of the sporting event.
- the actual event may occur at a time Tl and the content feeding or broadcasting may occur at a time T2 with a time delay of T2-T1.
- the time delay may be small such as of a few seconds so as the content is useful in a live commentary or augmentation of a live video.
- the machine learning tools may for example utilize real-time services and outputs and benefit from the spatiotemporal features and attributes to generate game patterns and automatic validations during the event itself such as to highlight certain event aspects in the commentary and/or validate momentary sessions when there are confusions during the event for decision making.
- the time delay may be longer in certain situations such as for replays, post-event analysis, predictive modeling, and future strategies, and the like.
- the methods and systems disclosed herein may support provision of the services and outputs at various time delays by determining processing steps and their order of execution according to delay requirements.
- the system may be configured to operate such that the services and outputs may be obtained at arbitrary times with an increasing accuracy or time resolution or such that the system targets specific delay requirements as specified by users or defined in accordance with intended applications. For example, if in an application, computational resources are insufficient to process all frames originating from input devices such as cameras etc. at maximum accuracy at a video frame rate within a desired delay, then instead of processing the input video frames in sequential orders, processing may be ordered in such a way that at any time there is a uniform or approximately uniform distribution of processed frames.
- processing decisions may also be influenced by other computational efficiency considerations for certain tasks that operate on video segments, such as an opportunity to reuse certain computations across successive frames in tracking algorithms.
- processing techniques such as inference and interpolation over processed frames may be used to provide a tracking output whose accuracy and time resolution improves with delay as more frames are processed.
- each component of processing application such as background subtraction, detection of various elements
- the specified time delays may also consider video qualities needed at sending destinations so as to ensure that enough computation resources are allocated for appropriate resolutions and transmission rates at the destinations during broadcasting of the content. In certain cases, a normal resolution may be sufficient while in other cases a higher resolution may be needed.
- the intelligent computer-controlled systems may be capable of defining appropriate resolutions, data transmission rates, and computation resources allocation in view of the delay requirements.
- the methods and systems disclosed herein may facilitate enabling calibration of a moving camera or any other image recognition device via tracking of moving points in a sporting event.
- Existing techniques for finding unknown camera calibration parameters from captured images or videos of sporting events rely on identifying a set of known locations, such as intersections of lines on the court or field.
- calibrating the moving camera as it changes its position or zooms across frames is challenging since there may be only a few of such known locations in the frames.
- the methods and systems disclosed herein may enable finding the calibration parameters of the moving or operator-controlled camera by using positions of moving points located by an associated tracking system.
- these positions may represent locations and spatial coordinates of a player's or a referee's head or hand or legs in the sporting event which may be identified by the tracking system.
- the tracking system may be an optical tracking system or a chip-based tracking system, which may be configured to determine positions of locations tags.
- several other types of camera controlling, calibration, and position determining systems may be employed along with the tracking systems.
- a fixed spotting camera may be used to capture a view and a moving camera contained within the tracking system may be used to capture the positions of the moving points in the frames.
- the moving camera may be configured to perform several functions such as zoom, tilt, pan, and the like.
- the tracking system may be configured to perform calibration and identification of the positions based on a tracking algorithm that may execute pre-defined instructions to compute relevant information necessary to drive the tracking system across the frames.
- the methods and systems disclosed herein may facilitate enabling pre-processing of images from calibrated cameras to improve object detection and recognition.
- the methods and systems disclosed herein may enable providing for accurate detection and recognition of humans, such as players or referees, and objects, such as a ball, a game clock, jersey numbers and the like with better performance and lower complexity.
- the tasks of object detection and recognition may be performed on the basis of knowledge of known calibration parameters of the cameras in the tracking system and known properties of the objects being detected such as their size, orientation, or positions etc. For example, perspectives and distortions introduced by the cameras can be undone by applying a transformation such that the objects being detected may have a consistent scale and orientation in transformed images.
- the transformed images may be used as inputs to detection and recognition algorithms by image processing devices so as to enable faster and more accurate object detection and recognition performance with lower complexity as compared to performing object detection and recognition directly on original images.
- an output generated by the image processing devices may be used as inputs, along with other inputs described herein, to enable or refine the various machine learning and algorithmic capabilities described throughout this disclosure.
- machine learning capabilities may be introduced to build improved processing utilizing machine learning tools as discussed above in the document.
- the methods and systems described herein may be deployed in part or in whole through a machine that executes computer software, program codes, and/or instructions on a processor.
- the processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platforms.
- a processor may be any kind of computational or processing device capable of executing program instructions, codes, binary instructions, and the like.
- the processor may be or include a signal processor, digital processor, embedded processor, microprocessor, or any variant such as a co-processor (math coprocessor, graphic co-processor, communication co-processor and the like) and the like that may directly or indirectly facilitate execution of program code or program instructions stored thereon.
- the processor may enable execution of multiple programs, threads, and codes.
- the threads may be executed simultaneously to enhance the performance of the processor and to facilitate simultaneous operations of the application.
- methods, program codes, program instructions and the like described herein may be implemented in one or more thread.
- the thread may spawn other threads that may have assigned priorities associated with them; the processor may execute these threads based on priority or any other order based on instructions provided in the program code.
- the processor may include memory that stores methods, codes, instructions, and programs as described herein and elsewhere.
- the processor may access a storage medium through an interface that may store methods, codes, and instructions as described herein and elsewhere.
- the storage medium associated with the processor for storing methods, programs, codes, program instructions or other type of instructions capable of being executed by the computing or processing device may include but may not be limited to one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache, and the like.
- a processor may include one or more cores that may enhance speed and performance of a multiprocessor.
- the process may be a dual core processor, quad core processors, other chip-level multiprocessor and the like that combine two or more independent cores (called a die).
- the methods and systems described herein may be deployed in part or in whole through a machine that executes computer software on a server, client, firewall, gateway, hub, router, or other such computer and/or networking hardware.
- the software program may be associated with a server that may include a file server, print server, domain server, Internet server, intranet server and other variants such as secondary server, host server, distributed server, and the like.
- the server may include one or more of memories, processors, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces capable of accessing other servers, clients, machines, and devices through a wired or a wireless medium, and the like.
- the methods, programs or codes as described herein and elsewhere may be executed by the server.
- other devices required for execution of methods as described in this application may be considered as a part of the infrastructure associated with the server.
- the server may provide an interface to other devices including, without limitation, clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers, and the like. Additionally, this coupling and/or connection may facilitate remote execution of program across the network. The networking of some or all of these devices may facilitate parallel processing of a program or method at one or more location without deviating from the scope.
- any of the devices attached to the server through an interface may include at least one storage medium capable of storing methods, programs, code and/or instructions.
- a central repository may provide program instructions to be executed on different devices.
- the remote repository may act as a storage medium for program code, instructions, and programs.
- the software program may be associated with a client that may include a file client, print client, domain client, Internet client, intranet client and other variants such as secondary client, host client, distributed client, and the like.
- the client may include one or more of memories, processors, computer readable media, storage media, ports (physical and virtual), communication devices, and interfaces capable of accessing other clients, servers, machines, and devices through a wired or a wireless medium, and the like.
- the methods, programs or codes as described herein and elsewhere may be executed by the client.
- other devices required for execution of methods as described in this application may be considered as a part of the infrastructure associated with the client.
- the client may provide an interface to other devices including, without limitation, servers, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers, and the like. Additionally, this coupling and/or connection may facilitate remote execution of program across the network. The networking of some or all of these devices may facilitate parallel processing of a program or method at one or more location without deviating from the scope.
- any of the devices attached to the client through an interface may include at least one storage medium capable of storing methods, programs, applications, code and/or instructions.
- a central repository may provide program instructions to be executed on different devices.
- the remote repository may act as a storage medium for program code, instructions, and programs.
- the methods and systems described herein may be deployed in part or in whole through network infrastructures.
- the network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices and other active and passive devices, modules and/or components as known in the art.
- the computing and/or non-computing device(s) associated with the network infrastructure may include, apart from other components, a storage medium such as flash memory, buffer, stack, RAM, ROM, and the like.
- the processes, methods, program codes, instructions described herein and elsewhere may be executed by one or more of the network infrastructural elements.
- the methods, program codes, and instructions described herein and elsewhere may be implemented on a cellular network having multiple cells.
- the cellular network may either be frequency division multiple access (FDMA) network or code division multiple access (CDMA) network.
- FDMA frequency division multiple access
- CDMA code division multiple access
- the cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, and the like.
- the cell network may be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.
- the methods, programs codes, and instructions described herein and elsewhere may be implemented on or through mobile devices.
- the mobile devices may include navigation devices, cell phones, mobile phones, mobile personal digital assistants, laptops, palmtops, netbooks, pagers, electronic books readers, music players and the like. These devices may include, apart from other components, a storage medium such as a flash memory, buffer, RAM, ROM and one or more computing devices.
- the computing devices associated with mobile devices may be enabled to execute program codes, methods, and instructions stored thereon. Alternatively, the mobile devices may be configured to execute instructions in collaboration with other devices.
- the mobile devices may communicate with base stations interfaced with servers and configured to execute program codes.
- the mobile devices may communicate on a peer to peer network, mesh network, or other communications networks.
- the program code may be stored on the storage medium associated with the server and executed by a computing device embedded within the server.
- the base station may include a computing device and a storage medium.
- the storage device may store program codes and instructions executed by the computing devices associated with the base
- the computer software, program codes, and/or instructions may be stored and/or accessed on machine readable media that may include: computer components, devices, and recording media that retain digital data used for computing for some interval of time; semiconductor storage known as random access memory (RAM); mass storage typically for more permanent storage, such as optical discs, forms of magnetic storage like hard disks, tapes, drums, cards and other types; processor registers, cache memory, volatile memory, non- volatile memory; optical storage such as CD, DVD; removable media such as flash memory (e.g., USB sticks or keys), floppy disks, magnetic tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable mass storage, off-line, and the like; other computer memory such as dynamic memory, static memory, read/write storage, mutable storage, read only, random access, sequential access, location addressable, file addressable, content addressable, network attached storage, storage area network, bar codes, magnetic ink, and the like.
- RAM random access memory
- mass storage typically for more permanent storage, such as optical discs, forms of magnetic
- the methods and systems described herein may transform physical and/or intangible items from one state to another.
- the methods and systems described herein may also transform data representing physical and/or intangible items from one state to another.
- machines may include, but may not be limited to, personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication devices, transducers, chips, calculators, satellites, tablet PCs, electronic books, gadgets, electronic devices, devices having artificial intelligence, computing devices, networking equipment, servers, routers, and the like.
- the elements depicted in the flow chart and block diagrams or any other logical component may be implemented on a machine capable of executing program instructions.
- the methods and/or processes described above, and steps thereof, may be realized in hardware, software or any combination of hardware and software suitable for a particular application.
- the hardware may include a general purpose computer and/or dedicated computing device or specific computing device or particular aspect or component of a specific computing device.
- the processes may be realized in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, along with internal and/or external memory.
- the processes may also, or instead, be embodied in an application specific integrated circuit, a programmable gate array, programmable array logic, or any other device or combination of devices that may be configured to process electronic signals. It may further be appreciated that one or more of the processes may be realized as a computer executable code capable of being executed on a machine readable medium.
- the computer executable code may be created using a structured programming language such as C, an object oriented programming language such as C++, or any other high- level or low-level programming language (including assembly languages, hardware description languages, and database programming languages and technologies) that may be stored, compiled or interpreted to run on one of the above devices, as well as heterogeneous combinations of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.
- a structured programming language such as C
- an object oriented programming language such as C++
- any other high- level or low-level programming language including assembly languages, hardware description languages, and database programming languages and technologies
- each method described above and combinations thereof may be embodied in computer executable code that, when executing on one or more computing devices, performs the steps thereof.
- the methods may be embodied in systems that perform the steps thereof, and may be distributed across devices in a number of ways, or all of the functionality may be integrated into a dedicated, standalone device or other hardware.
- the means for performing the steps associated with the processes described above may include any of the hardware and/or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Pure & Applied Mathematics (AREA)
- Computational Mathematics (AREA)
- Mathematical Optimization (AREA)
- Algebra (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Analysis (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
Abstract
La fourniture d'un contenu vidéo amélioré consiste à traiter au moins un flux vidéo par l'intermédiaire d'au moins un algorithme de reconnaissance de motif spatiotemporel qui fait intervenir un apprentissage machine pour développer une compréhension d'une pluralité d'événements et pour déterminer au moins un type d'événement pour chaque événement parmi la pluralité d'événements. Le type d'événement comprend une entrée dans une bibliothèque de relations détaillant une relation entre deux caractéristiques visibles. Une extraction et une indexation d'une pluralité de séquences vidéo du flux vidéo sont effectuées sur la base dudit type d'événement déterminé par la compréhension qui correspond à un événement parmi la pluralité d'événements détectables dans les séquences vidéo. Enfin, une structure de données de contenu vidéo améliorée est générée, automatiquement et sous une commande informatique, à l'aide de la pluralité extraite de séquences vidéo sur la base de l'indexation de la pluralité extraite de séquences vidéo.
Priority Applications (17)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP17851597.9A EP3513566A4 (fr) | 2016-09-16 | 2017-09-15 | Procédés et systèmes de reconnaissance de motif spatiotemporel pour un développement de contenu vidéo |
US16/229,457 US10460177B2 (en) | 2014-02-28 | 2018-12-21 | Methods and systems of spatiotemporal pattern recognition for video content development |
US16/351,213 US10748008B2 (en) | 2014-02-28 | 2019-03-12 | Methods and systems of spatiotemporal pattern recognition for video content development |
US16/525,830 US10832057B2 (en) | 2014-02-28 | 2019-07-30 | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition |
US16/573,599 US10997425B2 (en) | 2014-02-28 | 2019-09-17 | Methods and systems of spatiotemporal pattern recognition for video content development |
US16/675,799 US10713494B2 (en) | 2014-02-28 | 2019-11-06 | Data processing systems and methods for generating and interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US16/795,834 US10769446B2 (en) | 2014-02-28 | 2020-02-20 | Methods and systems of combining video content with one or more augmentations |
US16/925,499 US11380101B2 (en) | 2014-02-28 | 2020-07-10 | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US17/006,962 US11373405B2 (en) | 2014-02-28 | 2020-08-31 | Methods and systems of combining video content with one or more augmentations to produce augmented video |
US17/029,808 US11275949B2 (en) | 2014-02-28 | 2020-09-23 | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition |
US17/117,356 US11120271B2 (en) | 2014-02-28 | 2020-12-10 | Data processing systems and methods for enhanced augmentation of interactive video content |
US17/238,847 US11861905B2 (en) | 2014-02-28 | 2021-04-23 | Methods and systems of spatiotemporal pattern recognition for video content development |
US17/399,570 US11861906B2 (en) | 2014-02-28 | 2021-08-11 | Data processing systems and methods for enhanced augmentation of interactive video content |
US17/848,120 US20220327830A1 (en) | 2014-02-28 | 2022-06-23 | Methods and systems of combining video content with one or more augmentations to produce augmented video |
US17/856,364 US20220335720A1 (en) | 2014-02-28 | 2022-07-01 | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US18/510,439 US20240087316A1 (en) | 2014-02-28 | 2023-11-15 | Methods and systems of spatiotemporal pattern recognition for video content development |
US18/511,906 US20240087317A1 (en) | 2014-02-28 | 2023-11-16 | Data processing systems and methods for enhanced augmentation of interactive video content |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662395886P | 2016-09-16 | 2016-09-16 | |
US62/395,886 | 2016-09-16 | ||
US15/586,379 | 2017-05-04 | ||
US15/586,379 US10521671B2 (en) | 2014-02-28 | 2017-05-04 | Methods and systems of spatiotemporal pattern recognition for video content development |
US201762532744P | 2017-07-14 | 2017-07-14 | |
US62/532,744 | 2017-07-14 |
Related Parent Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/586,379 Continuation US10521671B2 (en) | 2014-02-28 | 2017-05-04 | Methods and systems of spatiotemporal pattern recognition for video content development |
US15/586,379 Continuation-In-Part US10521671B2 (en) | 2014-02-28 | 2017-05-04 | Methods and systems of spatiotemporal pattern recognition for video content development |
US16/229,457 Continuation US10460177B2 (en) | 2014-02-28 | 2018-12-21 | Methods and systems of spatiotemporal pattern recognition for video content development |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/229,457 Continuation-In-Part US10460177B2 (en) | 2014-02-28 | 2018-12-21 | Methods and systems of spatiotemporal pattern recognition for video content development |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018053257A1 true WO2018053257A1 (fr) | 2018-03-22 |
Family
ID=61619742
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2017/051768 WO2018053257A1 (fr) | 2014-02-28 | 2017-09-15 | Procédés et systèmes de reconnaissance de motif spatiotemporel pour un développement de contenu vidéo |
Country Status (2)
Country | Link |
---|---|
EP (1) | EP3513566A4 (fr) |
WO (1) | WO2018053257A1 (fr) |
Cited By (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109600559A (zh) * | 2018-11-29 | 2019-04-09 | 北京字节跳动网络技术有限公司 | 一种视频特效添加方法、装置、终端设备及存储介质 |
US10269140B2 (en) | 2017-05-04 | 2019-04-23 | Second Spectrum, Inc. | Method and apparatus for automatic intrinsic camera calibration using images of a planar calibration pattern |
US10460177B2 (en) | 2014-02-28 | 2019-10-29 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
WO2020154309A1 (fr) * | 2019-01-22 | 2020-07-30 | MGM Resorts International Operations, Inc. | Systèmes et procédés pour diviser un flux vidéo afin de segmenter une activité de joueur en direct |
WO2020176869A1 (fr) | 2019-02-28 | 2020-09-03 | Stats Llc | Système et procédé de ré-identification de joueurs dans une vidéo diffusée |
US10769446B2 (en) | 2014-02-28 | 2020-09-08 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations |
WO2020221022A1 (fr) * | 2019-04-28 | 2020-11-05 | 阿里巴巴集团控股有限公司 | Procédé de recommandation d'objet de service |
US10917702B2 (en) | 2018-12-13 | 2021-02-09 | At&T Intellectual Property I, L.P. | Creating customized short-form content from long-form content |
US20210042960A1 (en) * | 2018-01-18 | 2021-02-11 | Verizon Media Inc. | Computer vision on broadcast video |
CN112399165A (zh) * | 2019-08-14 | 2021-02-23 | 腾讯美国有限责任公司 | 解码方法及装置以及计算机设备和存储介质 |
CN113038272A (zh) * | 2021-04-27 | 2021-06-25 | 武汉星巡智能科技有限公司 | 婴儿视频自动剪辑的方法、装置、设备及存储介质 |
US11113535B2 (en) | 2019-11-08 | 2021-09-07 | Second Spectrum, Inc. | Determining tactical relevance and similarity of video sequences |
US11120271B2 (en) | 2014-02-28 | 2021-09-14 | Second Spectrum, Inc. | Data processing systems and methods for enhanced augmentation of interactive video content |
CN113506195A (zh) * | 2021-06-15 | 2021-10-15 | 深圳十方融海科技有限公司 | 互联网班级管理方法、系统、计算机设备和存储介质 |
US11172248B2 (en) | 2019-01-22 | 2021-11-09 | Tempus Ex Machina, Inc. | Systems and methods for customizing and compositing a video feed at a client device |
US20220179900A1 (en) * | 2018-09-04 | 2022-06-09 | Toyota Connected North America, Inc. | Systems and methods for querying a distributed inventory of visual data |
US11380101B2 (en) | 2014-02-28 | 2022-07-05 | Second Spectrum, Inc. | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
CN115175005A (zh) * | 2022-06-08 | 2022-10-11 | 中央广播电视总台 | 视频处理方法、装置、电子设备及存储介质 |
US20220343253A1 (en) * | 2021-04-27 | 2022-10-27 | Stats Llc | Virtual Coaching System |
CN116383669A (zh) * | 2023-03-18 | 2023-07-04 | 宝钢工程技术集团有限公司 | 一种数据贯通的工厂对象位号标识生成方法及系统 |
US11861906B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Data processing systems and methods for enhanced augmentation of interactive video content |
CN117649608A (zh) * | 2024-01-29 | 2024-03-05 | 阿坝州林业和草原科学技术研究所 | 基于遥感监测的松材线虫病识别系统及方法 |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2021189145A1 (fr) * | 2020-03-27 | 2021-09-30 | Sportlogiq Inc. | Système et procédé de reconnaissance d'activités collectives dans des images et des vidéos à mécanismes d'autoattention |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030093810A1 (en) * | 2001-10-30 | 2003-05-15 | Koji Taniguchi | Video data transmitting/receiving method and video monitor system |
US7085322B2 (en) * | 1998-05-29 | 2006-08-01 | International Business Machines Corporation | Distributed control strategy for dynamically encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel |
US7143083B2 (en) * | 2001-06-12 | 2006-11-28 | Lucent Technologies Inc. | Method and apparatus for retrieving multimedia data through spatio-temporal activity maps |
US20080312010A1 (en) * | 2007-05-24 | 2008-12-18 | Pillar Vision Corporation | Stereoscopic image capture with performance outcome prediction in sporting environments |
US7932923B2 (en) * | 2000-10-24 | 2011-04-26 | Objectvideo, Inc. | Video surveillance system employing video primitives |
US20140058992A1 (en) * | 2012-08-21 | 2014-02-27 | Patrick Lucey | Characterizing motion patterns of one or more agents from spatiotemporal data |
WO2015060691A1 (fr) * | 2013-10-24 | 2015-04-30 | 사회복지법인 삼성생명공익재단 | Système d'assurance qualité pour appareil de radiothérapie et son procédé d'assurance qualité |
US20150248917A1 (en) * | 2014-02-28 | 2015-09-03 | Second Spectrum, Inc. | System and method for performing spatio-temporal analysis of sporting events |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB0105421D0 (en) * | 2001-03-06 | 2001-04-25 | Prozone Holdings Ltd | Sport analysis system and method |
US7657836B2 (en) * | 2002-07-25 | 2010-02-02 | Sharp Laboratories Of America, Inc. | Summarization of soccer video content |
US9186548B2 (en) * | 2009-07-20 | 2015-11-17 | Disney Enterprises, Inc. | Play sequence visualization and analysis |
US9342785B2 (en) * | 2013-11-15 | 2016-05-17 | Disney Enterprises, Inc. | Tracking player role using non-rigid formation priors |
WO2016040833A1 (fr) * | 2014-09-12 | 2016-03-17 | Kiswe Mobile Inc. | Procédés et appareil d'interaction de contenu |
US10536758B2 (en) * | 2014-10-09 | 2020-01-14 | Thuuz, Inc. | Customized generation of highlight show with narrative component |
-
2017
- 2017-09-15 EP EP17851597.9A patent/EP3513566A4/fr not_active Withdrawn
- 2017-09-15 WO PCT/US2017/051768 patent/WO2018053257A1/fr unknown
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7085322B2 (en) * | 1998-05-29 | 2006-08-01 | International Business Machines Corporation | Distributed control strategy for dynamically encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel |
US7932923B2 (en) * | 2000-10-24 | 2011-04-26 | Objectvideo, Inc. | Video surveillance system employing video primitives |
US7143083B2 (en) * | 2001-06-12 | 2006-11-28 | Lucent Technologies Inc. | Method and apparatus for retrieving multimedia data through spatio-temporal activity maps |
US20030093810A1 (en) * | 2001-10-30 | 2003-05-15 | Koji Taniguchi | Video data transmitting/receiving method and video monitor system |
US20080312010A1 (en) * | 2007-05-24 | 2008-12-18 | Pillar Vision Corporation | Stereoscopic image capture with performance outcome prediction in sporting environments |
US20140058992A1 (en) * | 2012-08-21 | 2014-02-27 | Patrick Lucey | Characterizing motion patterns of one or more agents from spatiotemporal data |
WO2015060691A1 (fr) * | 2013-10-24 | 2015-04-30 | 사회복지법인 삼성생명공익재단 | Système d'assurance qualité pour appareil de radiothérapie et son procédé d'assurance qualité |
US20150248917A1 (en) * | 2014-02-28 | 2015-09-03 | Second Spectrum, Inc. | System and method for performing spatio-temporal analysis of sporting events |
Non-Patent Citations (2)
Title |
---|
HYUN-SOOK KIM: "AUTOMATIC CLASSIFICATION OF OFFENSIVE PATTERNS FOR SOCCER GAME HIGHLIGHTS USING NEURAL NETWORKS", MALAYSIAN JOURNAL OF COMPUTER SCIENCE, vol. 15, no. 1, June 2002 (2002-06-01), pages 57 - 67, XP055422797 * |
See also references of EP3513566A4 * |
Cited By (53)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10997425B2 (en) | 2014-02-28 | 2021-05-04 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10460176B2 (en) | 2014-02-28 | 2019-10-29 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11861905B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Methods and systems of spatiotemporal pattern recognition for video content development |
US10460177B2 (en) | 2014-02-28 | 2019-10-29 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10769446B2 (en) | 2014-02-28 | 2020-09-08 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations |
US10521671B2 (en) | 2014-02-28 | 2019-12-31 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11373405B2 (en) | 2014-02-28 | 2022-06-28 | Second Spectrum, Inc. | Methods and systems of combining video content with one or more augmentations to produce augmented video |
US11861906B2 (en) | 2014-02-28 | 2024-01-02 | Genius Sports Ss, Llc | Data processing systems and methods for enhanced augmentation of interactive video content |
US10748008B2 (en) | 2014-02-28 | 2020-08-18 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10755102B2 (en) | 2014-02-28 | 2020-08-25 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10755103B2 (en) | 2014-02-28 | 2020-08-25 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10762351B2 (en) | 2014-02-28 | 2020-09-01 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US11120271B2 (en) | 2014-02-28 | 2021-09-14 | Second Spectrum, Inc. | Data processing systems and methods for enhanced augmentation of interactive video content |
US11380101B2 (en) | 2014-02-28 | 2022-07-05 | Second Spectrum, Inc. | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content |
US11023736B2 (en) | 2014-02-28 | 2021-06-01 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10269140B2 (en) | 2017-05-04 | 2019-04-23 | Second Spectrum, Inc. | Method and apparatus for automatic intrinsic camera calibration using images of a planar calibration pattern |
US10380766B2 (en) | 2017-05-04 | 2019-08-13 | Second Spectrum, Inc. | Method and apparatus for automatic intrinsic camera calibration using images of a planar calibration pattern |
US10706588B2 (en) | 2017-05-04 | 2020-07-07 | Second Spectrum, Inc. | Method and apparatus for automatic intrinsic camera calibration using images of a planar calibration pattern |
US20210042960A1 (en) * | 2018-01-18 | 2021-02-11 | Verizon Media Inc. | Computer vision on broadcast video |
US11694358B2 (en) * | 2018-01-18 | 2023-07-04 | Verizon Patent And Licensing Inc. | Computer vision on broadcast video |
US20220179900A1 (en) * | 2018-09-04 | 2022-06-09 | Toyota Connected North America, Inc. | Systems and methods for querying a distributed inventory of visual data |
US11928149B2 (en) * | 2018-09-04 | 2024-03-12 | Toyota Connected North America, Inc. | Systems and methods for querying a distributed inventory of visual data |
CN109600559A (zh) * | 2018-11-29 | 2019-04-09 | 北京字节跳动网络技术有限公司 | 一种视频特效添加方法、装置、终端设备及存储介质 |
CN109600559B (zh) * | 2018-11-29 | 2021-07-23 | 北京字节跳动网络技术有限公司 | 一种视频特效添加方法、装置、终端设备及存储介质 |
US10917702B2 (en) | 2018-12-13 | 2021-02-09 | At&T Intellectual Property I, L.P. | Creating customized short-form content from long-form content |
US11754662B2 (en) | 2019-01-22 | 2023-09-12 | Tempus Ex Machina, Inc. | Systems and methods for partitioning a video feed to segment live player activity |
WO2020154309A1 (fr) * | 2019-01-22 | 2020-07-30 | MGM Resorts International Operations, Inc. | Systèmes et procédés pour diviser un flux vidéo afin de segmenter une activité de joueur en direct |
US11172248B2 (en) | 2019-01-22 | 2021-11-09 | Tempus Ex Machina, Inc. | Systems and methods for customizing and compositing a video feed at a client device |
AU2020211935B2 (en) * | 2019-01-22 | 2023-06-15 | Infinite Athlete, Inc. | Systems and methods for partitioning a video feed to segment live player activity |
CN113490944A (zh) * | 2019-01-22 | 2021-10-08 | 天时机械公司 | 用于划分视频馈送以分割实况运动员活动的系统和方法 |
US11140328B2 (en) | 2019-01-22 | 2021-10-05 | Tempus Ex Machina, Inc. | Systems and methods for partitioning a video feed to segment live player activity |
EP3912089A4 (fr) * | 2019-02-28 | 2022-12-07 | Stats Llc | Système et procédé de ré-identification de joueurs dans une vidéo diffusée |
WO2020176869A1 (fr) | 2019-02-28 | 2020-09-03 | Stats Llc | Système et procédé de ré-identification de joueurs dans une vidéo diffusée |
US11861850B2 (en) | 2019-02-28 | 2024-01-02 | Stats Llc | System and method for player reidentification in broadcast video |
US11830202B2 (en) | 2019-02-28 | 2023-11-28 | Stats Llc | System and method for generating player tracking data from broadcast video |
US11586840B2 (en) | 2019-02-28 | 2023-02-21 | Stats Llc | System and method for player reidentification in broadcast video |
US11593581B2 (en) | 2019-02-28 | 2023-02-28 | Stats Llc | System and method for calibrating moving camera capturing broadcast video |
US11935247B2 (en) | 2019-02-28 | 2024-03-19 | Stats Llc | System and method for calibrating moving cameras capturing broadcast video |
US11861848B2 (en) | 2019-02-28 | 2024-01-02 | Stats Llc | System and method for generating trackable video frames from broadcast video |
WO2020221022A1 (fr) * | 2019-04-28 | 2020-11-05 | 阿里巴巴集团控股有限公司 | Procédé de recommandation d'objet de service |
CN112399165B (zh) * | 2019-08-14 | 2022-03-25 | 腾讯美国有限责任公司 | 解码方法及装置以及计算机设备和存储介质 |
CN112399165A (zh) * | 2019-08-14 | 2021-02-23 | 腾讯美国有限责任公司 | 解码方法及装置以及计算机设备和存储介质 |
US11113535B2 (en) | 2019-11-08 | 2021-09-07 | Second Spectrum, Inc. | Determining tactical relevance and similarity of video sequences |
US11778244B2 (en) | 2019-11-08 | 2023-10-03 | Genius Sports Ss, Llc | Determining tactical relevance and similarity of video sequences |
CN113038272A (zh) * | 2021-04-27 | 2021-06-25 | 武汉星巡智能科技有限公司 | 婴儿视频自动剪辑的方法、装置、设备及存储介质 |
US20220343253A1 (en) * | 2021-04-27 | 2022-10-27 | Stats Llc | Virtual Coaching System |
CN113506195A (zh) * | 2021-06-15 | 2021-10-15 | 深圳十方融海科技有限公司 | 互联网班级管理方法、系统、计算机设备和存储介质 |
CN113506195B (zh) * | 2021-06-15 | 2024-03-08 | 深圳十方融海科技有限公司 | 互联网班级管理方法、系统、计算机设备和存储介质 |
CN115175005A (zh) * | 2022-06-08 | 2022-10-11 | 中央广播电视总台 | 视频处理方法、装置、电子设备及存储介质 |
CN116383669A (zh) * | 2023-03-18 | 2023-07-04 | 宝钢工程技术集团有限公司 | 一种数据贯通的工厂对象位号标识生成方法及系统 |
CN116383669B (zh) * | 2023-03-18 | 2024-04-16 | 宝钢工程技术集团有限公司 | 一种数据贯通的工厂对象位号标识生成方法及系统 |
CN117649608A (zh) * | 2024-01-29 | 2024-03-05 | 阿坝州林业和草原科学技术研究所 | 基于遥感监测的松材线虫病识别系统及方法 |
CN117649608B (zh) * | 2024-01-29 | 2024-03-29 | 阿坝州林业和草原科学技术研究所 | 基于遥感监测的松材线虫病识别系统及方法 |
Also Published As
Publication number | Publication date |
---|---|
EP3513566A4 (fr) | 2019-09-11 |
EP3513566A1 (fr) | 2019-07-24 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11861905B2 (en) | Methods and systems of spatiotemporal pattern recognition for video content development | |
US11778244B2 (en) | Determining tactical relevance and similarity of video sequences | |
US10832057B2 (en) | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition | |
US11373405B2 (en) | Methods and systems of combining video content with one or more augmentations to produce augmented video | |
US11380101B2 (en) | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content | |
US11861906B2 (en) | Data processing systems and methods for enhanced augmentation of interactive video content | |
US11275949B2 (en) | Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition | |
US11120271B2 (en) | Data processing systems and methods for enhanced augmentation of interactive video content | |
EP3513566A1 (fr) | Procédés et systèmes de reconnaissance de motif spatiotemporel pour un développement de contenu vidéo | |
WO2019183235A1 (fr) | Procédés et systèmes de reconnaissance de motif spatiotemporel pour développement de contenu vidéo | |
US20220335720A1 (en) | Data processing systems and methods for generating interactive user interfaces and interactive game systems based on spatiotemporal analysis of video content | |
US20220327830A1 (en) | Methods and systems of combining video content with one or more augmentations to produce augmented video | |
US20240031619A1 (en) | Determining tactical relevance and similarity of video sequences |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17851597 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2017851597 Country of ref document: EP Effective date: 20190416 |