WO2017201608A1  System and method for evaluating team game activities  Google Patents
System and method for evaluating team game activities Download PDFInfo
 Publication number
 WO2017201608A1 WO2017201608A1 PCT/CA2017/050253 CA2017050253W WO2017201608A1 WO 2017201608 A1 WO2017201608 A1 WO 2017201608A1 CA 2017050253 W CA2017050253 W CA 2017050253W WO 2017201608 A1 WO2017201608 A1 WO 2017201608A1
 Authority
 WO
 WIPO (PCT)
 Prior art keywords
 game
 player
 team
 method
 players
 Prior art date
Links
Classifications

 A—HUMAN NECESSITIES
 A63—SPORTS; GAMES; AMUSEMENTS
 A63B—APPARATUS FOR PHYSICAL TRAINING, GYMNASTICS, SWIMMING, CLIMBING, OR FENCING; BALL GAMES; TRAINING EQUIPMENT
 A63B71/00—Games or sports accessories not covered in groups A63B1/00  A63B69/00
 A63B71/06—Indicating or scoring devices for games or players, or for other sports activities

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06Q—DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
 G06Q10/00—Administration; Management
 G06Q10/10—Office automation, e.g. computer aided management of electronic mail or groupware; Time management, e.g. calendars, reminders, meetings or time accounting

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
 H04L12/00—Data switching networks

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
 H04L12/00—Data switching networks
 H04L12/28—Data switching networks characterised by path configuration, e.g. local area networks [LAN], wide area networks [WAN]
 H04L12/46—Interconnection of networks
 H04L12/4604—LAN interconnection over a backbone network, e.g. Internet, Frame Relay
 H04L12/462—LAN interconnection over a bridge based backbone
 H04L12/4625—Single bridge functionality, e.g. connection of two networks over a single bridge
Abstract
Description
SYSTEM AND METHOD FOR EVALUATING TEAM GAME ACTIVITIES
CROSSREFERENCE TO RELATED APPLICATION(S)
[0001] This application claims priority to U.S. Provisional Patent Application No.
62/341 ,358 filed on May 25, 2016, the contents of which are incorporated herein by reference.
TECHNICAL FIELD
[0002] The following relates to systems and methods for evaluating players and team game activities.
DESCRIPTION OF THE RELATED ART
[0003] An important objective in the field of sports statistics is to understand which actions contribute to winning in what situation. Since sports statistics have begun entering the field of "big data", there are increasing opportunities for largescale machine learning to model complex sports dynamics. Existing techniques that rely on Markov Game models typically aim to compute optimal strategies or policies [14] (e.g., minimax or equilibrium strategies).
[0004] One of the main tasks for sports statistics is evaluating the performance of teams and players [20]. A common approach is to assign action values, and sum the
corresponding values each time a player takes the respective action. A simple and widely used example in ice hockey is the "+/ score": for each goal scored by (against) a player's team when he/she is on the ice, add +1 (subtract 1) point. Researchers have developed several extensions of "+/ score" for hockey [2, 22, 16]. The National Hockey League has also begun publishing advanced player statistics such as the Corsi (shot attempts) and Fenwick (unblocked shot attempts) ratings.
[0005] There exist several methods that aim to improve the basic "+/ score" with statistical techniques [2, 7, 22]. A common approach is to use regression techniques where an indicator variable for each player is used as a regressor for a goalrelated quantity (e.g., logodds of a goal for the player's team vs. the opposing team). The regression weight measures the extent to which the presence of a player contributes to goals for his/her team or prevents goals for the other team. However, these approaches are found to consider only goals, without considering other actions.
[0006] An alternative game model, the Total Hockey Rating (THoR) [16], assigns a value to all actions, not only goals. Actions are evaluated based on whether or not a goal occurred in the following 20 seconds after an action. For penalties, the duration of the penalty was used as the look ahead window. The THoR assumes a fixed value for every action and does not account for the context in which an action takes place. Furthermore, the 20 second window restricts the look ahead value of each action.
[0007] Markov Decision Process (MDP)type models have been applied in a number of sports settings, such as baseball, soccer and football [3]. For example, in reference [3] spatialtemporal tracking data for basketball are used to build the "Pointwise" model for valuing player decisions and player actions.
[0008] In general, it has been found that MDPtype models have been limited to the use of temporal order of the game states in a discrete space. That is, the MDP is typically used in its conventional way, i.e., identifying an optimal policy for a critical situation in a sport or game. In addition, those models are found to be limited to a small number of states and actions (a low dimensional space) [21].
[0009] Moreover, player comparison and ranking is considered a very difficult task that requires deep domain knowledge. The difficulty is not only in defining appropriate key metrics for players, but also in finding a group of players who have similar playing styles. From the scouting perspective, a scout would need to watch multiple games of a player to come up with a conclusion about the skills and style of a young talent. However, it is not possible for the scouts to watch all the games from all the leagues around the world, making it difficult to rely on scouting reports to obtain the knowledge required to properly compare and rank players.
SUMMARY
[0010] The following relates to a system and method for creating and evaluating computational models for games (e.g., team or individual sports games, etc.), team performance, and individual player performance evaluation in such games.
[0011] In one aspect, there is provided a method for evaluating a team game, the method comprising obtaining data associated with the team game, the information comprising at least one individual player activity, at least one team activity, at least one game event, and a location in space and time for each of the events and activities;
generating quantitative values for the data associated with the team game; and evaluating either or both an individual player and a team using the quantitative values.
[0012] In another aspect, there is provided a method for grouping players in a game based on a respective playing style, the method comprising: obtaining data associated with the game, the information comprising at least one individual player or team activity, and at least one game event; and automatically generating a plurality of clusters of one or more players based on a presence and/or activities associated with each player.
[0013] The clusters can be generated by: obtaining information about a player's presence; generating a presence map for the players; grouping presence maps based on similarity in spatial and/or temporal characteristics; and forming the clusters.
[0014] In other aspects, there are provided computer readable media and electronic devices and/or systems for performing the above methods.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Embodiments will now be described by way of example only with reference to the appended drawings wherein:
[0016] FIG. 1 A is a schematic block diagram of a system for obtaining data and building and evaluating a game model;
[0017] FIG. 1 B is a schematic block diagram of a system for evaluating a game using information about the game situation and by building and using a game model;
[0018] FIG. 2 is a schematic diagram illustrating a highvalue trajectory for a successful play in a sports game;
[0019] FIG. 3 is a graph depicting a correlation between a goal ratio and team impact.
[0020] FIG. 4 is a schematic view of an ice rink divided into twelve regions for generating player heat maps;
[0021] FIG. 5A is an example activity heat map for Player A;
[0022] FIG. 5B is an example activity heat map for Player A's cluster;
[0023] FIG. 6A is an example activity heat map for Player B;
[0024] FIG. 6B is an example activity heat map for Player B's cluster;
[0025] FIG. 7 is a chart illustrating learned clusters matched to player categories;
[0026] FIG. 8A is an overlay of an ice rink showing learned regions for "block" events;
[0027] FIG. 8B is an overlay of an ice rink showing learned regions for "reception" events;
[0028] FIG. 9 is an example of a state action trajectory according to a Markov game model;
[0029] FIG. 10A is an overlay of an ice rink with a drilldown analysis for Player A ; and [0030] FIG. 10B is an overlay of an ice rink with a drilldown analysis for Player B. DETAILED DESCRIPTION
[0031] The following provides a method that uses a contextaware approach to value player actions, locations, and individual or team performance in games, particularly for sports games and matches. The developed models can use machine learning and artificial intelligence (Al) techniques to incorporate context and lookahead metrics. Team
performance can be assessed as the aggregate value of actions by the team's players. Model validation shows that the total team action value provides a strong indicator or predictor of team success. Although the following examples are directed to icehockey, the principles described herein can also be used for any other games in which there exists a set of meaningful events and actions in a defined space, resulting in a measurable outcome such as a goal or score.
[0032] The following applies such Al techniques to model the dynamics of team sports, for example using the Markov Game Model formalism described in reference [14], and related computational techniques such as a dynamic programming value iteration algorithm. In this way, the system described herein develops and relies on models that learn how a game is actually played, and optimal strategies can be considered as implicit natural outcomes of the learnt models.
[0033] It has been found that the methods that are only context regressorbased only take into account which players are on the ice when a goal is scored. As shown below, such regressionbased methods can also be combined with the Markov Game Model as herein described, to capture how team impact scores depend on the presence or absence of individual players.
[0034] In contrast to previous techniques, the presently described Markov Gamebased learning method is not restricted to any particular time window for a look ahead.
[0035] Also, the system described herein in part proposes a contextaware model. As an example, a goal (an action) is typically considered to be more valuable in a tiedgame situation than when the scorer's team is already four goals ahead [18]. Richer state spaces therefore capture more of the context of an action. In addition, previously reported methods compute action scores based on immediate positive consequences of an action (e.g. goals following a shot). However, an action may have mediumterm and/or ripple effects rather than immediate consequences in terms of visible rewards like goals. Therefore evaluating the impact of an action requires look ahead. Longterm look ahead can be particularly important in ice hockey because evident rewardslike goals occur infrequently [15]. For example, if a player receives a penalty, this leads to a manpower disadvantage for his/her team, known as a powerplay for the other team. It may be considered easier to score a goal during a powerplay, but this does not mean that a goal will be scored immediately after the penalty is assessed (or at all). The dynamic programming value iteration algorithm of Markov Decision Processes provides a computationally efficient way to perform unbounded look ahead.
[0036] The systems and methods described herein therefore use game context and locations associated with game events in order to learn the value of each action and event executed by players of the game. The learned game model provides information about the likely consequences of actions. The basic model and algorithms can also be adapted to study different outcomes of interest, such as goals and penalties. In addition, the model can be used for team performance evaluation, team performance prediction, game outcome prediction, and grouping players based on a playing style. In addition, players can be ranked within groups of similar players based the values of their activities throughout a game.
[0037] It has also been recognized that player comparison and ranking is a very difficult task that requires deep domain knowledge. The difficulty is not only in defining appropriate key metrics for players, but also in finding a group of players who have similar playing styles. Following forming the groups of similar players, game models and performance metrics can be applied to assess players' skills.
[0038] The systems and methods described herein also use the location pattern of the players, i.e., where they tend to be present or play (or where they take their actions) in a game. Clusters can be formed using machine learning techniques which take into account the location information, however the prior information about players' roles and styles can be added to the models to generate more refined clusters.
[0039] An exemplary embodiment focuses on measuring how much a player's actions contribute to the outcome of the game at a given location and time. This can be performed by assigning a value to each action depending on where and when the action takes place using a Markov decision process model. Once the values for the actions and game events are assigned, players can be ranked according to the aggregate value of their actions, and compared to others in their cluster. In this study, the value of a player's action is measured as its impact on his team's chance of "scoring the next goal"; the resulting player metric is called his or her Scoring Impact (SI). The "scoring the next goal" metric can be chosen as the end goal of a sequence of game events because it can be clearly defined as a measurable objective for a team. However, the developed model is not necessarily dependent on this outcome and any other event can be used as the end state of the Markov process, for example shots (from a certain location) in a hockey game, passes, or any possible player activity in a game.
Learned Game Model and System Configuration
[0040] Turning now to the figures, FIG. 1 A illustrates an example of a system 8 that can gather or receive, and utilize media data such as video, images, audio clips, sensory data about player locations, body pose, actions, etc. (e.g., Mi , M_{2}, etc. shown in this example) from any number of gameplay locations (e.g., rinks Ri , R_{2}, etc. shown in this example) to enable a computing system 4 to model game play and game events using a game model 10, in order to generate quantitative data outputs and/or to display such data outputs.
[0041] FIG. 1 B is a schematic block diagram of the system 8 that builds and uses a game model 10 for team sports, and which illustrates the general flow of information through the process described herein. As illustrated above, the system 8 can include any combination of software, hardware, firmware, etc.; and programs the model 10 into software program. The system 8 receives information 14 about the current game situation, individual and/or group activities and events along with location and their temporal order and generates an output 20 that provides the quantitative values enabling one to determine what has been learned by the model. Game states and actions are evaluated using the quantitative values 12, 16, and 18 as will be explained in greater detail below.
[0042] FIG. 2 demonstrates a highvalue trajectory that represents one generic successful play by a home team. In this example, the play takes place in the top half of an ice rink during an ice hockey game, and location coordinates are rescaled to bring nodes closer together. The nodes are labelled with the conditional values of a state for the home team, e.g., the probability that the home team scores the next goal, as described below. Edges are labelled with actions, and with the impact of the action, which is the difference in conditional probabilities. FIG. 3, also described in greater detail below, is a graph showing the correlation between a "Goal Ratio" (averaged over all matches in this example) and "Team Impact" (averaged over the first k matches in this example).
[0043] The following relates to methods and systems for developing a game model that considers both space and time components of player locations and their actions and game events. The exemplary implementation described herein develops game models for ice hockey and certain aspects are directed to a specific sport. This exemplary embodiment uses Markov Models to formalize the ice hockey game and compute the values of states and actions during a game. The following also provides examples of how individual player actions can be evaluated and how performance of teams can be measured.
[0044] A Markov Game [14], sometimes referred to as a stochastic game, is defined by a set of states, S , and a collection of action sets, A _ _{0}ne for each agent in the
environment. State transitions are controlled by the current state and a list of actions, one action from each agent. For each agent, there is an associated reward function mapping a state transition to a reward. An overview of how the presently described Markov Game model fills in this schema is as follows.
[0045] There are two teams, which are referred to herein as the Home Team, H and the Away Team, A . _{n} each state, only one team performs an action, although not in a turn based sequence. This reflects the way the data record actions. Thus at each state of the Markov Game, exactly one player from one team can perform the action and the other players from the other team chooses Nooperation, which implies that either they do nothing or their activity is not in the action space, A . The following generic notation for all states is provided, based in part on references [23, 14].
[0046] Occ(s, a) is the number of times that action a occurs in state s as observed in the playbyplay data.
[0047] Occ(s,a, s') is the number of occurrences of action a in state s being immediately followed by state s' as observed in the playbyplay data. (s, s') forms an edge with label ^{a} in the transition graph of the Markov Game model.
[0048] The transition probability function TP \s a mapping of S x A x S— » (0,l] .
Occis ci s'
[0049] It is estimated using the observed transition frequency
Occ(s, a)
[0050] One can begin by defining the state space, S and then the action space, A _
State space, S jn a sport game is normally defined in terms of handselected features that are intuitively relevant for the game dynamics, such as the goal differential and penalties in ice hockey, which are referred to as context features. An exemplary state space for ice hockey may contain:
[0051] Goal Differential (GD), which is calculated as Number of Home Goals  Number of Away Goals; and [0052] Manpower Differential ( D) , which specifies whether the teams are at even strength (EV) , the acting team is shorthanded (SH) or in a powerplay (PP), etc.
The action space, A , can be a multidimensional space coding the type of players' actions, information about their team, location of that action in space, and the time that an action happens. For instance, block(home,region3) denotes the action that the home team blocks the puck in the blockregion 3 (see FIG . 8A). FIG . 9 shows a possible stateaction trajectory, which Table 1 below describes in playbyplay format.
Table 1: StateAction Trajectory in PlaybyPlay Format
[0053] Previous research on Markov process models related to ice hockey has used only context features. A Markov process model can answer questions such as how goal scoring or penalty rates depend on the game context [24] . However, the presently described system also focuses on the impact of a player's actions on a game. One can next introduce the action space for the Markov model.
[0054] A strength of Markov Game modelling is value iteration, which can be applied to many reward functions depending on what results are of interest. This exemplary embodiment focuses on goal scoring , specifically scoring the next goal. This choice produces an interpretable value function: the value of a state for a team is the chance that the team scores the next goal.
[0055] The next goal objective can be represented in the Markov Game model as follows.
[0056] For any state s e S where a Home team scores a goal, one can set:
R_{H} (s, ct) := 1 . Similarly, when a goal is scored for the Away team : R_{A} (s, ct) := \ . For other states the reward is equal to 0. The reward differential is defined by:
[0057] i?(s, a) := ¾ (s, a)  i?_{A} (s, a) .
[0058] After a goal is scored, the next state is an absorbing state (no transitions from this state). [0059] With these definitions, the expected reward represents the probability that if play starts in a state S , a random walk through the state space of unbounded length ends with a goal for either the Home or the Away team. The reward differential is the difference in these probabilities. It is consistent with the practice in zerosum game theory of assigning a single value to a game outcome: +1 for the maximizer, 1 for the minimizer, ^{0} for a tie [23].
[0060] In order to encode the location information in a computationally efficient way, the continuous location space can be discretized into a discrete set. The disadvantage of this approach is that it can lose some information about the exact location of an action event. The computational advantage is that algorithms for discrete Markov models can be employed. The statistical advantage is that discretization requires neither parametric assumptions (e.g. Gaussian or Poisson distribution), nor stationarity assumptions that treat different locations as the same. To minimize the loss of information, the discrete location bins can be learned by clustering occurrences of each action type. There is therefore a separate clustering for each action type. Learning the clusters has the advantage over a fixed discretization that clusters are allocated to areas of high density; some action types occur very rarely in some parts of the ice hockey rink. However, the system described herein is not limited to the use of a specific discretization technique and continuous locations can be used to build the game model.
[0061] Recall that the value of a state, V(s) , is the expected reward when the game starts in state ^{s} . If each state appeared at most once between goals, one could compute the goal value function for, say the home team, simply as follows: count the number of state occurrences that were followed by a home goal, and divide by the total number of all state occurrences.
[0062] However, a state may appear any number of times between goals. That is, the transition graph may contain loops. In that case the value function can be computed by an iterative procedure known as value iteration. It may be observed that any value function satisfies the Bellman equation:
[0063] V(s) =∑P(s, a, s') yy(s')
[0064] Value iteration starts from an initial value assignment— typically 0 for each state "then applies the Bellman equation as an update equation. One can use an undiscounted value function for the value iteration. In the hockey domain, one can run value iterations for each of the three reward functions R_{H} , R_{A} , R and stop the iteration when the value difference stops changing (i.e. when the value function for R converges). In the experimental evaluation a relative convergence of 1% was used as the convergence criterion. With this convergence criterion, value iteration converges within 14 steps. This means that looking ahead more than 14 steps changes the estimate of the goal scoring differential between
Home and Away by less than l^{0//}° .
[0065] Given the learned models of the game, the following provides experimental examples of how the model evaluates sport player actions in context. This illustrates some results for the hockey domain.
[0066] As an exemplary application, the model can be used for a planning task, to find successor states that are most likely to lead to a goal. FIG. 2 shows a highvalue trajectory where a state is connected to its highest value successor (for the Home team). The play takes place in the top half of the rink. Location coordinates are rescaled to bring nodes closer. Nodes are labelled with the conditional values of a state for the home team, i.e., probability that the home team scores the next goal. Edges are labelled with actions, and with the impact of the action, which is the difference in conditional probabilities.
[0067] One can evaluate an action in a contextaware way by considering its expected reward after executing it in a given state. This is known in reinforcement learning as the Q value [19]:
[0069] The following also considers a conditional Qvalue. The conditional Qvalue for the home team is the Qvalue for the home team performing an action, divided by the sum of the Qvalue for the home team and the Qvalue for the away team (and similarly for the away team). The conditional Qvalue corresponds to conditioning on the event that at least one of the team scores a goal within the next 14 time steps (a look ahead horizon). It measures the expected reward of a team relative to its opponents, in other words, the advantage that a team enjoys over its opponent in a given match state. For example, with reward = scoring the next goal, a conditional Qvalue of p for the home team means that, given that one of the teams will score the next goal within the next 14 time steps, the chance is p that the home team manages the next goal.
[0070] The model can be used to evaluate team performance. The general idea is that team strength can be estimated as an aggregate of individual action values, and then correlated with ground truth metrics such as numbers of goals scored. For example, an Expected Goal metric scores each shot according to its chance of leading to a goal
(immediately). The sum of goal probabilities for each shot is the team's Expected Goal metric. The Expected Goal metric is a special case of the value function, where look ahead is equal to one and the only action type taken into consideration is a shot. Consequently, the goal impact metric and more generally, the impact of an action can be defined as:
[0071] impact s, a) = Q s,a) V s)
[0072] When the reward function is defined by scoring the next goal, the goal impact measures how much a player's action changes the chance that his/her team scores the next goal, given the current state of the match. The team goal impact in a game is the sum of goal impacts over all actions in the match by the players of a team. One can examine correlations between the Average Goal Ratio, Average Team Impact, Average Team Impact with Lookahead = 1 , Average Team Impact without location, and Average Team Value Difference with Lookahead = 1 which are defined as follows:
[0073] Average Goal Ratio is defined as the number of the goals by the team divided by the total number of goals by either side. For each team, we compute the average goal ratio over all games.
[0074] Average Team Impact is the average goal impact for a team, over all the games.
[0075] Average Team Impact with Lookahead = 1 is the average team impact using a value function with lookahead = 1 step (rather than 14).
[0076] Average Team Impact without Location is the average team impact where all locations are treated the same.
[0077] Average Team Value Difference with Lookahead = 1 is the sum of action values,
Q(s,a) , for each action taken by the team in a game, minus the sum of action values taken by their opponent in the same game. With lookahead = 1 , the action value sums are dominated by probability of a shot being successful, which is the basis of the Expected Goals Ratio metric.
[0078] Table 2 below shows the correlation coefficients P between Goal Ratio and the Impact metrics. The full team impact metric shows an impressive correlation of 0.7.
Reducing the look ahead to only 1 step can significantly reduce the information that the impact metric provides about a team's result, down to a mere 0.09 correlation. Without the location information, the impact metric still gives meaningful results, but is much less informative with a correlation of 0.21. The value difference is in fact more informative with singlestep lookahead, at a correlation of 0.34. Overall, it can be seen that the full team impact metric manages to extract by far the most information relevant to predicting a team's performance.
 Correlations Between Team Impact in the Full Model vs. Restricted Models. The range of the correlation coefficient is [1 +1]
[0079] The correlation between goal ratio and the Team Impact in a single game was also computed and the correlation was found to be p = 0.45 . This correlation coefficient indicates that the impact values carry substantial information about the outcome of a game.
[0080] Correlations computed not from the entire dataset, but from initial observations were also considered. FIG. 3 illustrates the correlation coefficients between the final Goal Ratio, averaged over all matches, and a moving average of the Team Impact Total, averaged over the first k matches. The correlation is close to 0.5 after 10 matches, which is less than half the number of total matches for all teams in the dataset that was used. This means that after a relatively small number of observed matches, the Average Team Impact carries substantive information about the final Goal Ratio performance of a team.
LocationBased Player Clustering and Ranking
[0081] In the following, location information is used to first generate clusters of players who are similar and then compute a Scoring Impact (SI) metric based on the players' actions at different locations. The rationale behind clustering players before ranking them is intuitive, for example, since typically a defenseman is not compared directly to a forward. In the following examples, a forward is compared to a forward while a defenseman should be compared to a defenseman. Although comparing players in the same position may be considered trivial for anyone who knows the game, building a purely datadriven approach to generate clusters of players without using any prior information has been found to pose challenges. To build the player clusters, the following uses the location pattern of the players, i.e., where they tend to play. This generates clusters in an unsupervised fashion, which results in groups of players with similar styles and roles.
[0082] Once the clusters are formed, any metric can be developed to rank the players and evaluate their impact on the game outcome. The following focuses on measuring how much a player's actions contribute to the outcome of the game at a given location and time. This is performed by assigning a value to each action depending on where and when the action takes place using a Markov decision process model. For example, the value of a pass depends on where it is taken and it has to be rewarded if it ends up in maintaining the puck possession. Once the values for the actions and game events are assigned, players can be ranked according to the aggregate value of their actions, and compared to others in their cluster. In a study that has been conducted, the value of a player's action is measured as its impact on his team's chance of scoring the next goal; the resulting player metric is called his Scoring Impact (SI) as noted above.
[0083] The experimental results indicate that the SI correlates with plausible alternative metrics such as a player's total points and salary. However, SI suggests some
improvements over those metrics as it captures more information about the game events. The results can be illustrated by identifying players that highly impact their team scoring chances, yet draw a low salary compared to others in their cluster.
[0084] For the system 8, player clustering can be performed by using the affinity propagation algorithm [25]. This algorithm groups players by clustering heat maps that represent their location patterns. To compute the probability that a team scores the next goal given the current state of the game, a Markov Decision Process is developed to model hockey games. As opposed to approaches for player performance assessment that are based on using aggregate action counts as features, the presently described modelbased method has several advantages, including:
[0085] Capturing the game context: the states in the model capture the context of actions in a game. For example, a goal is more valuable in a tiedgame situation than when the scorer's team is already four goals ahead.
[0086] Lookahead and mediumterm values: modeling game trajectories provides a lookahead to the mediumterm consequences of an action. Looking ahead to the medium term consequences allows one to assign a value to every action. This is particularly important in continuousflow games like hockey because evident rewards like goals occur infrequently. For example, if a player receives a penalty, the likelihood increases that the opposing team will score a goal during the power play at some point, but this does not mean that they will score immediately.
[0087] Player and team impact: The aggregate impact of a player can be broken down into his average impact at specific game states. Since game states include a highlevel of context detail, the model can be used to find the game situations in which a player's decisions carry especially high or low values, compared to other players in his cluster. This kind of drilldown analysis explains, and goes beyond, a player's overall ranking. The following provides what has been discovered to be the first examples of drilldown analysis for two players (Players A and B, based on reallife players).
[0088] It can be appreciated that while the present examples focus on players, the same approach can be used to cluster and analyze the performance of lines and teams.
Hockey Dataset
[0089] The data used by the system 8 can be generated from videos using computer vision techniques including player tracking and activity recognition. While the principles herein are applicable to any game, the following describes a particular data set used to model a hockey game and hockey players. Table 3 shows the dataset statistics for the 20152016 NHL season. The dataset includes playbyplay information of game events and player actions for the entire season. Every event is marked with a continuous time stamp, the xy location, and the player that carries out the action of the event. The playbyplay event data records 13 general action types. Table 4 shows an example playbyplay dataset. The table utilizes adjusted spatial coordinates where negative numbers refer to the defensive zone of the acting player, positive numbers to his offensive zone. To illustrate, FIG. 4 shows a schematic layout of an ice hockey rink. Adjusted Xcoordinates run from  100 to +100, and Ycoordinates from 42.5 at the bottom to 42.5 at the top, and the origin is at the ice center.
Table 3: Dataset statistics for 20152016 Season
Table 4: Sample PlaybyPlay Data LocationBased Player Clustering
[0090] Hockey is considered to be a fastpaced game where players of all roles act in all parts of the ice hockey rink. The presently described player clustering method is based on each player's distribution of action locations across the rink. To represent the action location pattern of a player, one can divide the rink into a fixed number of regions, as shown in FIG. 4.
[0091] This division uses four horizontal and three vertical regions, corresponding to the traditional center, left and right wings. For each player, the region frequency is the total number of actions he or she has performed in a region, divided by the total number of his or her actions. Converting counts to frequencies avoids conflating the level of a player's activity with the location of his actions. One can apply the affinity propagation algorithm [2] to the player frequency vectors to obtain a player clustering.. Affinity propagation has been found to produce nine player clusters: four clusters of forwards, four clusters of defensemen, and one cluster of goalies. It is interesting to note that the clustering is an unsupervised process.
[0092] FIG. 5A shows the 12region activity heat map for Player A, and FIG. 5B represents the heat map for the cluster that player belongs to. Similarly, FIG. 6A shows the heat map for a different player, namely Player B, and FIG. 6B depicts the average heat map for that player's cluster. The average heat map represents the average frequency of the game events which are happening in that region, over all players in the cluster. The heat maps show that Player B and other players in his cluster tend to play a defensive role on the left wing, whereas Player A and other players in his cluster play a more offensive role, mostly on the right wing.
[0093] It is important to compare the learned clusters with the known player types. FIG. 7 shows that the clusters match the basic grouping into defensive players and forwards. The algorithm discovers this grouping only from the game event location information without being given any prior or explicit information about the player's official position. Forwards are commonly divided into centers, left wing players and right wing players. The learned forward clusters match this division to some extent. For instance, clusters 5 and 7 contain mainly but not only centers, cluster 6 contains mainly but not only leftwingers, and cluster 8 contains mainly but not only rightwingers. This indicates not only that the clusters match the conventional player positions, but also that they provide information beyond those predefined positions. [0094] As such, player clustering can be performed using the affinity propagation algorithm. To simplify the computations for clustering, one can discretize the continuous time and location spaces, such as dividing the rink into an arbitrary chosen number of regions, and count the number of specific actions taken place from each region or the amount of time a payer is present in a given location and normalize those counted values.
[0095] Applying a clustering algorithm to the location information of where players are taking their actions results in clusters of players who are have tendencies in playing in similar locations and/or taking similar actions during the games. In this exemplary embodiment, applying the affinity propagation algorithm resulted in nine player clusters which are consistent with the conventional player roles in the icehockey as illustrated in FIG. 7.
[0096] Once the clusters are formed, a highresolution largescale Markov game model quantifies the impact of all events on scoring the next goal. The aggregate impact of an action provides a principled effective way to assess player performance. Breaking down the aggregate impact allows the analyst to pinpoint the exact situations in which a player's decisions tends to deviatepositively or negativelyfrom comparable players. Statistical modelling could further enhance drilldown analysis by identifying which features of the game context and of a player's actions predict a high addedimpact.
[0097] The impact of an action is defined as the extent to which the action changes the conditional value of the acting player's team at a state. The scoring impact metric for a player is defined as their total impact over all player's actions.
[0098] The players' scoring impact metric shows a strong correlation with other important metrics, such as points, time on ice, and salary. This correlation increases by computing the metric for comparable players rather than all players.
[0099] The following Table 5 shows the correlation between SI and time on ice (per game). For example, the correlation between SI and time on ice is 0.83 overall, and increases to 0.89 and 0.92 for the clusters shown in the table. The SI is also temporally consistent, i.e., a player's SI metric in the first half of the season correlates strongly with his SI metric in the second half (p = 0.77).
Table 5: Correlation Between Scoring Impact and Time on Ice [00100] The player's ranking can be explained by how he performs in specific game context. This breakdown makes the ranking interpretable because it explains the specific observations that led to the rating and pinpoints where a player's effectiveness deviates from comparable players. The basis of the approach is to find the game contexts in which a player's expected impact differs the most from a random player in his cluster. This metric is referred to herein as the player's added impact. This analysis looks for game contexts where the player shows an unusually high or low added impact.
Markov Game Model
[00101] As indicated above, a Markov model is a dynamic model that represents how a hockey game moves from one game state to the next. A sequence of state transitions constitutes a trajectory. The parameters of a (homogeneous) Markov chain are transition probabilities P(s's) where s is the current state and s' the next state. Previous Markov chain models for ice hockey have included goal differential and/or manpower differential in the state space. Then the transition probabilities represent how goal scoring and penalty drawing rates depend on the current goal and manpower differentials. This approach can measure the impact of those actions that directly affect the state variables, i.e., goals and penalties. Markov decision processes and Markov game models include both states and actions, which allows us to measure the impact of all actions. The parameters of the presently described Markov game model are stateaction transition probabilities of the form P(s', a' s, a) where a is the current action and a' is the next action. The model therefore describes stateaction trajectories as illustrated in FIG 9.
[00102] The Markov model represents the probability that a given action occurs at a given location on the rink. To model the action occurrence probability, one can discretize the rink space into a discrete set of regions. One option for generating discrete regions is to use a fixed grid, such as the one shown in FIG. 4. However, the problem with a fixed grid is that different types of actions tend to be distributed in different locations. For example, shots are found to rarely occur in the defensive zone, whereas blocked shots often do. Therefore, using the same grid for shots and blocks is neither statistically nor computationally efficient. Instead, the system 8 has been used to learn from the data a separate discretization tailored to each action, by applying affinity propagation to cluster the locations of occurrences of a given action type. FIG. 8A shows the resulting regions for blocks, and FIG. 8B for receptions. In each figure, the cluster mean is shown with an occurrence label indicating how many actions are happening in each region. The figures also show the impact of the actions on scoring the next goal for each region, averaged over the game contexts. Those numbers are derived from the developed Markov game model
[00103] Important quantities in the model specify the joint stateaction distribution P(s', a'\s, a) that an action a e A occurs at the game state s, and is followed by game state s e S and action a. Because the distribution of the next action and its location depends on the most recent action and its location, the action distribution represents spatial and temporal dynamics. For example, the transition probability of 21 % in the second row of Table 1 includes the probability that play moves from carryregion 4 to passregion 2, given the current game context. One can refer to a stateaction pair as a game context, so P(s',a'ls,a) models a context transition probability. Decomposing this probability as
P{s', a'\s, a) = P(s' \a', s, a) x P(a'\s, a), it can be seen that it combines two quantities of interest:
[00104] (1) the state transition probabilities P(s'\a', s, a) that describe how game states evolve given players' actions.
[00105] (2) The action distribution P(a'\s, a) that describes how a random player acts in a given game context.
[00106] One can estimate the actionstate distribution using the observed occurrence counts n(s', a', s, a)„ which record how often action a' and state s' follows state s and action oin the dataset. For simplicity, one use n also for marginal occurrence counts, for example
a' , s, a). The maximum likelihood estimates can be computed as follows:n(s', a', a, s)
P(s', a' \s, a) = ' ' ' ^{J}
n(s, a)
Action Values and Scoring Impact
[00107] In the presently described model 10, the agents are a generic home team and a generic away team, not individual players, similar to previous game models for hockey. This is appropriate for the goal of assigning generic values to all action events. Herein, the Markov model has been used to quantify how a player's action, given a game context, affects the probability that his or her team scores the next goal. A similar approach can be followed to quantify the impact of actions on other outcomes of interest, such as winning the game and penalties. A key feature of a Markov model is that it quantifies not only the immediate but also the mediumterm impact of an action.
[00108] For 7 = home or away, let P(T scores next\s, d) denote the probability derived from the model, that after action a, the team 7 scores the next goal, before the opposing team T . For a point in a game, it is possible that a play sequence ends with neither team scoring. Therefore another quantity of interest is the conditional probability that a team scores given that one of the two teams scores next. This can be referred to as the conditional value of a game context for team T:
P(T scores next \s, a)
CV_{r}(s, a) =
P(T scores next \s, a) + P(T scores next \s, a)
[00109] The conditional value is an appropriate quantity for evaluating actions since the goal of an action is to improve a team's position relative to their opponent. The impact of an action is defined as the extent to which the action changes the conditional value of the acting player's team at a state.
[00110] FIG. 8A shows the impact of a "Block" by region, averaged over game contexts, and in FIG. 8B, "Receptions". The scoring impact metric for a player is defined as their total impact over all their actions and formulated as follows:
Impacted'; s, a) = ^ CV_{r}(s', a') X P(s' \a' , s, a)— CV_{T}{s, a)
s'
S/j = ^ iij(a', s, a) X Impact(a'; s, a) = ^ iij(s, ) ^{x} ^ Impacted' ; s, a) x Pj(a' s, a) a',s,a s,a a'
[00111] where Pj(a' s, a) = "^{ί(}° is the action distribution for player /. The occurrence counts ηι(α' , s' , S, CL) record how many times the game reaches the state s' and player / takes action a' after state s and a player (not necessarily ;) took action a. The second expression for the scoring impact shows that the SI metric can be interpreted as the expected impact of a player given a game context (s,a), weighted by how often the player reaches the context.
[00112] The SI metric shows a strong correlation with other important metrics, such as points, time on ice, and salary. This correlation increases by computing the metric for comparable players rather than all players. Table 4 (discussed above) shows the correlation between SI and time on ice (per game). For example, the correlation between SI and time on ice is 0.83 overall, and increases to 0.89 and 0.92 for the clusters shown in the table. The SI is also temporally consistent [18], i.e., a player's SI metric in the first half of the season correlates strongly with his SI metric in the second half (p = 0.77).
[00113] Given the strong correlation of the scoring impact (SI) metric to the game outcome, one can combine the scoring impacts (Sis) of all the players on one team and compare to another team to determine the expected outcome of the game, and predict the winning team. It can be appreciated that different methods of aggregation can be applied to either the SI or action values to predict the outcome of a game.
[00114] For the example clusters used herein, one can also discuss the topranked player and some undervalued players.
DrillDown Analysis
[00115] The following breakdown makes the ranking interpretable because it explains the specific observations that led to the rating and pinpoints where a player's effectiveness deviates from comparable players. The basic approach is to find the game contexts in which a player's expected impact differs the most from a random player in his cluster. Referring to this metric as the player's added impact, Aj(s, a), computed as follows:
E, (s, a) = lmpact{a' ; s, a) P: {a' \s, a)
∑ rij (s, a)
— : E, (s, )
jec n_{c} s, a)
[00116] where C is the cluster of player / and n_{c} (s, a) =∑_{;eC} ^ (s, ).. Drilldown analysis looks for game contexts where the player shows an unusually high or low added impact. For Player A, his highest added impact is in the first period, with even score and manpower, after his team has managed a reception in region 1. Among action types, the highest added impact stems from Block. FIG. 10A compares Player A's region distribution for Blocks with those of a random player from his cluster. In the specified game context, a Block has the most scoring impact in the rightwing offensive region 3. For this game context, 50% of Player A's Blocks occur in this highimpact region, compared to only 19.6% of Blocks for a random player from his cluster.
[00117] For Player B, the highest added impact is in the third period, with even score and manpower, after his team has managed a pass in region 4. FIG. 10B compares Player B's region distribution for Receptions with those of a random player from his cluster. In the specified game context, a Reception has the most scoring impact in the leftwing offensive region 1. For this game context, 37.5% of Player B's Receptions occur in this highimpact region, compared to only 14.6% of Receptions for a random player from his cluster.
Summary
[00118] The systems and methods described herein propose a pure datadriven approach based on clustering and Markov decision process to support the way that scouts and managers evaluate players. Location information of the game events and player actions can be used to identify players with similar styles and roles and rank them based on their impact on scoring the next goal. This work supports "applestoapples" comparisons of similar players. Once the clusters are formed, a highresolution largescale Markov game model quantifies the impact of all events on scoring the next goal. The aggregate impact of an action provides a principled effective way to assess player performance. Breaking down the aggregate impact allows the analyst to pinpoint the exact situations in which a player's decisions tends to deviatepositively or negativelyfrom comparable players. Statistical modelling could further enhance drilldown analysis by identifying which features of the game context and of a player's actions predict a high addedimpact.
[00119] Numerous specific details are set forth in order to provide a thorough understanding of the examples described herein. However, it will be understood by those of ordinary skill in the art that the examples described herein may be practiced without these specific details. In other instances, wellknown methods, procedures and components have not been described in detail so as not to obscure the examples described herein. Also, the description is not to be considered as limiting the scope of the examples described herein.
[00120] It will be appreciated that the examples and corresponding diagrams used herein are for illustrative purposes only. Different configurations and terminology can be used without departing from the principles expressed herein. For instance, components and modules can be added, deleted, modified, or arranged with differing connections without departing from these principles. The steps or operations in the flow charts and diagrams described herein are just for example. There may be many variations to these steps or operations without departing from the principles discussed above. For instance, the steps may be performed in a differing order, or steps may be added, deleted, or modified.
[00121] Although the above principles have been described with reference to certain specific examples, various modifications thereof will be apparent to those skilled in the art as outlined in the appended claims.
[00122] It will also be appreciated that any module or component exemplified herein that executes instructions may include or otherwise have access to computer readable media such as storage media, computer storage media, or data storage devices (removable and/or nonremovable) such as, for example, magnetic disks, optical disks, or tape. Computer storage media may include volatile and nonvolatile, removable and nonremovable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of computer storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by an application, module, or both. Any such computer storage media may be part of the system 8, any component of or related to the system 8, etc., or accessible or connectable thereto. Any application or module herein described may be implemented using computer readable/executable instructions that may be stored or otherwise held by such computer readable media.
References
[00123] [2] Brian Macdonald: An Improved Adjusted PlusMinus Statistic for NHL Players. 201 1 . URL http://www.sloansportsconference.com/wpcontent/uploads/201 1/08/An lmprovedAdjustedPlusMinusStatisticforNHLPlayers.pdf.
[00124] [3] Dan Cervone, Alexander DAmour, Luke Bornn, Kirk Goldsberry:
"POINTWISE: Predicting points and valuing decisions in real time with NBA optical tracking data", 8th Annual MIT Sloan Sports Analytics Conference, February, 2014.
[00125] [7] R.B. Gramacy, ST. Jensen, M. Taddy: "Estimating player contribution in hockey with regularzed logistic regression.", Journal of Quantitative Analysis in Sports, pp. 971 1 1 , 2013.
[00126] [14] Michael L Littman: "Markov games as a framework for multiagent reinforcement learning", Proceedings of the eleventh international conference on machine learning, pp. 157—163, 1994.
[00127] [15] D. Lock, M. Schuckers: Beyond +/: A rating system to compare NHL players.. 2009. Presentation at joint statistical meetings.
[00128] [16] Michael Schuckers, James Curro: "Total Hockey Rating (THoR): A comprehensive statistical rating of National Hockey League forwards and defensemen based upon all onice events", 7th Annual MIT Sloan Sports Analytics Conference, 2013.
[00129] [18] Stephen Pettigrew: "Assessing the offensive productivity of NHL players using ingame win probabilities", 9th Annual MIT Sloan Sports Analytics Conference, 2015.
[00130] [19] Richard S. Sutton, Andrew G. Barto: Reinforcement learning : an introduction . MIT Press, 1998. [00131] [20] Robert P. Schumaker, Osama K. Solieman, Hsinchun Chen: Research in Sports Statistics in Sports Data Mining. Springer US, 2010. URL
http://dx.doi.Org/10.1007/978 1 441967305 4.
[00132] [21] Routley, K. & Schulte, O. (2015), A Markov Game Model for Valuing Player Actions in Ice Hockey, in Uncertainty in Artificial Intelligence (UAI), pp. 782791.
[00133] [22] Nathan Spagnola: The Complete PlusMinus: A Case Study of The Columbus Blue Jackets. 2013.
[00134] [23] Stuart Russell, Peter Norvig: Artificial Intelligence: A Modern Approach. Prentice Hall, 2010.
[00135] [24] A.C. Thomas, S.L. Ventura, S. Jensen, S. Ma: "Competing Process Hazard Function Models for Player Ratings in Ice Hockey", The Annals of Applied Statistics, pp. 14971524, 2013.
[00136] [25] Frey, B. J. & Dueck, D. (2007), 'Clustering by passing messages' Frey, B. J. & Dueck, D. (2007), Science 315(5814), 972976.
Claims
Priority Applications (2)
Application Number  Priority Date  Filing Date  Title 

US201662341358P true  20160525  20160525  
US62/341,358  20160525 
Applications Claiming Priority (3)
Application Number  Priority Date  Filing Date  Title 

EP17801858.6A EP3465498A1 (en)  20160525  20170227  System and method for evaluating team game activities 
CA3025382A CA3025382A1 (en)  20160525  20170227  System and method for evaluating team game activities 
US16/198,953 US20190091541A1 (en)  20160525  20181123  System and Method for Evaluating Team Game Activities 
Related Child Applications (1)
Application Number  Title  Priority Date  Filing Date 

US16/198,953 Continuation US20190091541A1 (en)  20160525  20181123  System and Method for Evaluating Team Game Activities 
Publications (1)
Publication Number  Publication Date 

WO2017201608A1 true WO2017201608A1 (en)  20171130 
Family
ID=60411126
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

PCT/CA2017/050253 WO2017201608A1 (en)  20160525  20170227  System and method for evaluating team game activities 
Country Status (4)
Country  Link 

US (1)  US20190091541A1 (en) 
EP (1)  EP3465498A1 (en) 
CA (1)  CA3025382A1 (en) 
WO (1)  WO2017201608A1 (en) 
Citations (4)
Publication number  Priority date  Publication date  Assignee  Title 

US20090325709A1 (en) *  20080626  20091231  Microsoft Corporation  Game Clan Matchmaking 
US20120142429A1 (en) *  20101203  20120607  Muller Marcus S  Collaborative electronic game play employing player classification and aggregation 
US20120283007A1 (en) *  20110503  20121108  Sony Computer Entertainment America Llc  Special regional online video game forum based on location 
US20140330734A1 (en) *  20130501  20141106  Mercer (US) Inc.  Systems and methods for identifying hidden talent 

2017
 20170227 EP EP17801858.6A patent/EP3465498A1/en active Pending
 20170227 CA CA3025382A patent/CA3025382A1/en active Pending
 20170227 WO PCT/CA2017/050253 patent/WO2017201608A1/en active Application Filing

2018
 20181123 US US16/198,953 patent/US20190091541A1/en active Pending
Patent Citations (4)
Publication number  Priority date  Publication date  Assignee  Title 

US20090325709A1 (en) *  20080626  20091231  Microsoft Corporation  Game Clan Matchmaking 
US20120142429A1 (en) *  20101203  20120607  Muller Marcus S  Collaborative electronic game play employing player classification and aggregation 
US20120283007A1 (en) *  20110503  20121108  Sony Computer Entertainment America Llc  Special regional online video game forum based on location 
US20140330734A1 (en) *  20130501  20141106  Mercer (US) Inc.  Systems and methods for identifying hidden talent 
NonPatent Citations (2)
Title 

KURT R. ET AL.: "A Markov Game Model for Valuing Player Actions in Ice Hockey", 31ST CONFERENCE ON UNCERTAINTY IN ARTIFICIAL INTELLIGENCE, 7 December 2015 (20151207), XP055441142, Retrieved from the Internet <URL:http://auai.org/uai2015/proceedings/papers/130.pdf> [retrieved on 20170606] * 
SKINNER, B. ET AL.: "A Method for Using Player Tracking Data in Basketball to Learn Player Skills and Predict Team Performance", PLOS ONE, 9 September 2015 (20150909), pages 1  15, XP055441146, Retrieved from the Internet <URL:http://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0136393&type=pri ntable> [retrieved on 20170606] * 
Also Published As
Publication number  Publication date 

US20190091541A1 (en)  20190328 
EP3465498A1 (en)  20190410 
CA3025382A1 (en)  20171130 
Similar Documents
Publication  Publication Date  Title 

De Vany  Hollywood economics: How extreme uncertainty shapes the film industry  
Needham et al.  Tracking multiple sports players through occlusion, congestion and scale  
Weber et al.  A data mining approach to strategy prediction  
Olson  Comparison of three multicriteria methods to predict known outcomes  
Dai et al.  POMDPbased control of workflows for crowdsourcing  
Perše et al.  A trajectorybased analysis of coordinated team activity in a basketball game  
US20180137364A1 (en)  Methods, systems and software programs for enhanced sports analytics and applications  
Song et al.  Competitive forces and strategic choice decisions: An experimental investigation in the United States and Japan  
Bartlett et al.  Analysing team coordination patterns from player movement trajectories in soccer: methodological considerations  
Dereszynski et al.  Learning probabilistic behavior models in realtime strategy games  
Zhu et al.  Error detection and impactsensitive instance ranking in noisy datasets  
US20120016633A1 (en)  System and method for automatic detection of anomalous recurrent behavior  
Memmert et al.  Current approaches to tactical performance analyses in soccer using position data  
Synnaeve et al.  A Bayesian model for opening prediction in RTS games with application to StarCraft  
Chan et al.  A model for predicting customer value from perspectives of product attractiveness and marketing strategy  
Synnaeve et al.  A Bayesian model for plan recognition in RTS games applied to StarCraft  
Bakkes et al.  Player behavioural modelling for video games  
Min et al.  A compound framework for sports results prediction: A football case study  
Lasek et al.  The predictive power of ranking systems in association football  
Baumer et al.  The sabermetric revolution: Assessing the growth of analytics in baseball  
EP3111659A1 (en)  System and method for performing spatiotemporal analysis of sporting events  
Lucey et al.  quality vs quantity: Improved shot prediction in soccer using strategic features from spatiotemporal data  
Gerla  The psychology of predatory pricing: Why predatory pricing pays  
Lipovetzky et al.  Classical planning with simulators: Results on the atari video games  
Lucey et al.  How to get an open shot: Analyzing team movement in basketball using tracking data 
Legal Events
Date  Code  Title  Description 

ENP  Entry into the national phase in: 
Ref document number: 3025382 Country of ref document: CA 

NENP  Nonentry into the national phase in: 
Ref country code: DE 

121  Ep: the epo has been informed by wipo that ep was designated in this application 
Ref document number: 17801858 Country of ref document: EP Kind code of ref document: A1 

ENP  Entry into the national phase in: 
Ref document number: 2017801858 Country of ref document: EP Effective date: 20190102 