JP7296546B2 - Play analysis device and play analysis method - Google Patents

Description

The present disclosure relates to a ball game play analysis device and a play analysis method.

"Data Volley" is known as volleyball analysis software. Data Volley is used by analysts familiar with the software, who enter data into Data Volley that describes the condition of the team's players based on subjective judgment.

Patent Literature 1 discloses a scouting system with improved operability for analysts. Specifically, in Patent Document 1, the position information of each player is acquired using image analysis, and the input player situation, each position information of each player, and a still image of a desired scene are stored in association with each other. It discloses building a database in the device to display each player's position, a still image of the desired scene, and the player's situation. This invention solves the problems of Data Volley's software that the training of scorers (analysts) is required and the data accuracy is low because each player's position information is input by visual and subjective judgment.

In addition, conventional volleyball analysis software sequentially captures images of receiving scenes for serves, inputs skill evaluations of each player, and aggregates and displays the number of times each player has received, success rate of receiving, and the like. A user (such as a team manager) analyzes the team's strengths and weaknesses by looking at the aggregated results.

JP 2004-351097 A

Xina CHENG, Norikazu IKOMA, Masaaki HONDA and Takeshi IKENAGA "Multi-view 3D Ball Tracking with Abrupt Motion Adaptive System Model, Anti-occlusion Observation and Spatial Density based Recovery in Sports Analysis", IEICE TRANS. FUNDAMENTALS, VOL.E94-A, NO.1 JANUARY 2011

A user analyzing a ball game such as volleyball may wish to analyze in detail a particular action of a particular player (eg, a setter's toss in volleyball). However, with conventional techniques, such detailed analysis is difficult.

An object of the present disclosure is to provide a play analysis apparatus and method capable of analyzing in detail specific actions of specific players in ball games.

A play analysis device according to one aspect of the present disclosure is a play analysis device that analyzes a play of a ball game, and calculates a trajectory of a moving object in the ball game using a plurality of play videos captured by a plurality of cameras. , detecting an action related to the ball game, the player who performed the action, and the position at which the player performed the action, based on changes in the trajectory of the moving body, and correlating the action, the player, and the position. a ball game video analysis unit that generates play information, and a plurality of pieces of play information of a target player who has performed a specific action; and a play analysis unit that generates a play analysis image aggregated and shown on the image, wherein the play analysis unit adds the specific action to each position where the specific action is performed in the play analysis image. and a plurality of attribute images are arranged as the state of the attribute images, the attribute images are the type related to the specific action, the speed of the ball related to the specific action, the height of the ball related to the specific action, and the specific action represents at least two attribute information of the distances associated with .

A play analysis method according to an aspect of the present disclosure is a play analysis method for analyzing a play of a ball game, in which the trajectory of a moving object in the ball game is calculated using a plurality of play videos captured by a plurality of cameras. , detecting an action related to the ball game, the player who performed the action, and the position at which the player performed the action, based on changes in the trajectory of the moving body, and correlating the action, the player, and the position. A plurality of pieces of play information of a target player who has performed a specific action are acquired, and from the plurality of pieces of play information, execution positions and states of the specific actions are aggregated on a zone image of the court. a plurality of attribute images as the state of the specific action at each position where the specific action is performed in the play analysis image, and It represents at least two pieces of attribute information of a type related to a specific action, a ball speed related to the specific action, a height of the ball related to the specific action, and a distance associated with the specific action.

In addition, these general or specific aspects may be realized by devices, methods, integrated circuits, computer programs, or recording media, and any of the systems, devices, methods, integrated circuits, computer programs, and recording media may be implemented. may be implemented in any combination.

According to the present disclosure, specific actions of specific players in ball games can be analyzed in detail.

1 is a diagram showing an overview of a ball game video analysis system according to Embodiment 1; FIG. 1 is a diagram showing an example of the configuration of a ball game video analysis device according to Embodiment 1; FIG. 4 is a diagram showing an example of play information according to Embodiment 1; FIG. 4 is a diagram showing an example of a detection region and an actor region according to Embodiment 1; FIG. FIG. 4 is a diagram showing another example of detection regions and actor regions according to Embodiment 1; FIG. 10 is a diagram for a case where the same player is determined in the same player determination process according to Embodiment 1; FIG. 10 is a diagram showing a case where different players are determined in the identical player determination process according to Embodiment 1; 4 is a flowchart showing action determination processing according to Embodiment 1; 4 is a flowchart showing actor recognition processing according to Embodiment 1; FIG. 10 is a diagram showing a configuration example of a play analysis system according to Embodiment 2; FIG. FIG. 11 is a diagram showing a configuration example of a play analysis device according to Embodiment 2; FIG. 10 is a diagram showing an example of a UI (User Interface) generated by the play analysis device according to Embodiment 2; FIG. 10 is a diagram showing an example of a toss play analysis image according to the second embodiment; FIG. 10 is a diagram showing an example of a play information list according to Embodiment 2; FIG. FIG. 10 is a diagram showing an example of a score information list according to Embodiment 2; FIG. FIG. 11 is a first diagram for explaining a procedure for correcting play information according to Embodiment 2; FIG. 10 is a second diagram for explaining the play information correction procedure according to the second embodiment; FIG. 13 is a third diagram for explaining the play information correction procedure according to the second embodiment; It is a figure showing an example of hardware constitutions concerning an embodiment of this indication.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art.

It should be noted that the accompanying drawings and the following description are provided for a thorough understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter.

(Embodiment 1)
<Ball game video analysis system>
First, an overview of a ball game video analysis system according to one embodiment will be described with reference to FIG.

The ball game video analysis system 1 is a system that analyzes video of a ball game and recognizes a player who has taken an action on a moving object used in the ball game. A moving body used in ball games is typically a ball, but may be a shuttle or the like used in badminton. In this embodiment, volleyball, which is one of the ball games, will be described as an example. However, the ball game video analysis system 1 can be applied to various ball games such as soccer, baseball, table tennis, basketball, tennis, rugby, American football, lacrosse, and badminton. Also, the ball game video analysis system 1 can be applied to a moving object having a shape that does not correspond to the concept of a "ball", such as an ice hockey puck. In other words, the ball game video analysis system 1 can be applied to any game as long as the score or victory or defeat is determined by a team composed of a plurality of players performing actions on a moving object.

A ball game video analysis system 1 includes a plurality of cameras 3 ( 3 A, 3 B, 3 C, 3 D), a display device 4 , an input device 5 and a ball game video analysis device 100 .

A plurality of cameras 3 are installed at different positions. For example, as shown in FIG. 1, each camera 3 is installed at a position where the court 10 can be photographed from a different viewpoint (angle of view) from a high place. Although the number of cameras 3 is four in FIG. 1, the present embodiment is not limited to this, and the number of cameras 3 may be any number as long as it is two or more. By using two or more cameras 3, the three-dimensional position of the ball can be calculated.

Each camera 3 is communicably connected to the ball game video analysis device 100 via a cable or wirelessly. Each camera 3 captures the situation during the ball game and generates a captured image. Then, each camera 3 generates a moving image frame 201 (see FIG. 2) from a plurality of captured images. Then, each camera 3 transmits the video frame 201 to the ball game video analysis device 100 . The moving image frame 201 is composed of a plurality of photographed images, for example, MP4, H.264, etc. 264, H. H.265, or compressed based on a standard such as Motion JPEG.

The display device 4 is communicably connected to the ball game video analysis device 100 via a cable or wirelessly, and displays an image output from the ball game video analysis device 100 . The display device 4 is, for example, a liquid crystal display or an organic EL display.

The input device 5 is communicably connected to the ball game video analysis device 100 via a cable or wirelessly, receives an operation from the user, and transmits the operation information to the ball game video analysis device 100 . The input device 5 is, for example, a keyboard, mouse, microphone and/or touch panel. Note that the input device 5 and the display device 4 may be an integrated device.

The ball game video analysis device 100 uses the video frames captured by each camera 3 to identify the player who performed the action on the ball. In volleyball, actions performed on the ball include serves, receptions, digs, tosses, attacks, and blocks. Hereinafter, a player who performs an action on the ball may be referred to as an "actor".

In the present embodiment, as shown in FIG. 1, the center point of the surface of the court 10 is the origin O, the axis parallel to the surface of the court 10 and parallel to the net 11 is the X axis, and the surface of the court 10 is parallel. The axis perpendicular to the net 11 is the Y-axis, and the axis perpendicular to the surface of the court 10 is the Z-axis. Regarding the X-axis, the direction away from the referee 12 is the positive direction, and the direction closer to the referee 12 is the negative direction. As for the Y-axis, the left direction as viewed from the referee 12 is the positive direction, and the right direction is the negative direction. The positive direction of the Z-axis is the direction away from the surface of the court 10 . That is, the Z-axis coordinate z corresponds to the height from the surface of the court 10 .

<Ball game video analysis device>
Next, the ball game video analysis device 100 will be described with reference to FIG.

The ball game video analysis device 100 includes a video reception unit 101, a trajectory calculation unit 102, an action determination unit 103, an action frame selection unit 104, a detection area setting unit 105, a player detection unit 106, and a number recognition unit 107. , a result output unit 108 and a storage unit 109 .

The video receiving unit 101 receives moving image frames 201 transmitted from each of the cameras 3A to 3D and stores them in the storage unit 109. FIG.

The trajectory calculation unit 102 applies the method disclosed in Non-Patent Document 1, for example, to a plurality of moving image frames 201 stored in the storage unit 109, and calculates the time when the moving image frame 201 was shot (hereinafter referred to as “frame time ”), the three-dimensional position (x, y, z) and velocity of the ball are calculated. Then, the trajectory calculation unit 102 generates ball trajectory information 202 by associating the frame time with the three-dimensional position and speed of the ball, and stores the generated ball trajectory information 202 in the storage unit 109 .

The action determination unit 103 determines from the ball trajectory information 202 whether or not an action has occurred with respect to the ball. When the action determination unit 103 determines that an action has occurred, the action determination unit 103 generates action information 203 by associating the frame time when the action occurred (hereinafter referred to as "action frame time") with the type of the action that occurred. Stored in the storage unit 109 . Details of the action determination unit 103 will be described later.

The action frame selection unit 104 selects, from among the plurality of video frames 201 stored in the storage unit 109, the video frame corresponding to the action frame time included in the action information 203 and the frame time near the action frame time. Select the corresponding video frame as the action frame. Details of the action frame selection unit 104 will be described later.

The detection area setting unit 105 sets a detection area, which is an area where player detection is performed, for the action frame selected by the action frame selection unit 104 . Details of the detection area setting unit 105 will be described later.

The player detection unit 106 detects an area in which the player is captured (hereinafter referred to as “player area”) from the image within the detection area set by the detection area setting unit 105 . When the player detection unit 106 detects a plurality of player areas, the player detection unit 106 selects the player area in which the player (actor) who performed the action is captured as the actor area. Details of the player detection unit 106 will be described later.

The number recognition unit 107 recognizes the actor's uniform number from the image within the actor area detected by the player detection unit 106 . The number recognition unit 107 then associates the action frame time with the uniform number of the recognized actor to generate actor information 204 and stores it in the storage unit 109 . Details of the number recognition unit 107 will be described later.

The result output unit 108 associates the ball trajectory information 202 , the action information 203 , and the actor information 204 to generate play information 205 and stores it in the storage unit 109 . For example, in the action information 203, the action frame time T and the action type "attack" are associated, and in the actor information 204, the action frame time T and the actor's uniform number "14" are associated. The output unit 108 generates play information 205 as shown in FIG. That is, as shown in FIG. 3, the result output unit 108 generates play information 205 in which the action type "attack" and the uniform number "14" of the actor are associated with the frame time T of the ball trajectory information 202. Generate. As a result, the user or other device uses the play information 205 so that the player (actor) with the uniform number “14” at the frame time T will hit the ball at the three-dimensional position (x _T , y _T , z _T ). On the other hand, it can be known that the "attack" of the speed " _ST (km/h)" has been performed.

<Details of the action determination part>
Next, the details of the action determination unit 103 will be described.

The action determination unit 103 calculates the trajectory of the ball from the three-dimensional position and velocity of the ball for each frame time included in the ball trajectory information 202 . Then, when the calculated change in the trajectory of the ball satisfies a predetermined condition, the action determination unit 103 determines that an action has occurred with respect to the ball. When the action determination unit 103 determines that an action has occurred, the action determination unit 103 sets the frame time that matches the condition as the action frame time.

Note that the action determination unit 103 may determine the type of action based on changes in the trajectory of the ball, the three-dimensional position and speed of the ball, ball game rules, and the like. Action types in volleyball include serve, reception (receiving on serve), digging (receiving on attack), toss, attack, and block. For example, the action determination unit 103 determines that the trajectory of the ball detected for the first time from the start of the analysis has a movement component in the Y-axis direction (long side direction of the court shown in FIG. 1), and the velocity component of the ball in the Y-axis direction is If it is within the predetermined range, the action type is determined as "serve". As another example, the action determination unit 103 determines that the trajectory of the ball straddles the coordinates of the net 11 on the Y-axis after the "serve" and the change in the three-dimensional position of the ball changes from falling to rising (that is, , when the change in the coordinate in the Z-axis direction turns positive), the action type is determined to be "reception". According to the rules of volleyball, the action of receiving a "serve" is a "reception", so making decisions based on such rules makes it possible to distinguish between a "reception" and a "dig".

Similarly, the action determination unit 103 determines another action type based on the determination criteria according to the rules of the game when the change in the trajectory of the ball or the like satisfies a predetermined condition.

In the above description, three-dimensional information is used to determine whether an action has occurred, but two-dimensional or one-dimensional information may be used to determine whether an action has occurred. For example, when judging the occurrence of a "serve" under the above conditions, it is possible to make the judgment only from the analysis start time and the information in the Y-axis direction without using the information in the X-axis direction and the Z-axis direction.

Further, the action determination unit 103 may generate the action information 203 by associating the action frame time and the action type with the three-dimensional position of the ball corresponding to the action frame time. This is because the information on the three-dimensional position of the ball is used in the action frame selection unit 104 and the like, which will be described later.

<Details of the action frame selection part>
Next, the details of the action frame selection unit 104 will be described.

The action frame selection unit 104 first selects at least two cameras 3 closer to the three-dimensional position of the ball corresponding to the action frame time. For example, the action frame selection unit 104 selects the cameras 3A and 3B when the y-coordinate of the ball is 0 or more (when the ball is on the left side of the court 10 in FIG. 1), and selects the cameras 3A and 3B when the y-coordinate of the ball is less than 0. If the ball is on the right side of court 10 in FIG. 1, select cameras 3C and 3D. As a result, the camera 3 that is highly likely to capture the uniform number of the player's uniform is selected.

Further, the action frame selection unit 104 selects the video frame 201 corresponding to the action frame time T in the action information 203 from the video frames 201 of each camera 3 selected above, and the frame time preceding the action frame time T (for example, A video frame 201 corresponding to frame time T-3) and a video frame 201 corresponding to a frame time later than action frame time T (for example, frame time T+3) are selected as action frames.

By selecting a plurality of action frames in this way, even if an actor's uniform number cannot be recognized in one action frame, the actor's uniform number may be recognized in other action frames. That is, the possibility of recognizing the actor's uniform number increases.

In this way, by selecting the moving image frame 201 that is separated by two frames or more as an action frame, even if the actor's uniform number cannot be recognized in one action frame, the posture of the actor can be recognized in the other action frames. has changed and may recognize the actor's uniform number. That is, the possibility of recognizing the actor's uniform number increases.

It should be noted that separating the moving image frames selected as action frames by two frames or more is merely an example. If it is possible to extract a plurality of frames in which the posture of the actor has changed and the actor is highly likely to be filmed, the video frame 201 that is three or more frames away or the video frame that is only one frame away 201 may be extracted as an action frame. However, if the action frame time T is too far away, the actor who performed the action is more likely to be out of the video frame 201 . Therefore, for example, in the case of the moving image frame 201 corresponding to the moving image of 60 fps, it is preferable that the upper limit of the frame interval is about 10 frames.

Also, the action frame selection unit 104 determines how many frames of the moving image frame 201 separated from the action frame time T are to be extracted as the action frame according to the frame rate of the moving image that is the source of the moving image frame. good too. For example, two frames at 60 fps correspond to one frame at 30 fps, so the action frame selection unit 104 selects the video frame 201 that is two frames away for 60 fps, and the video frame 201 that is one frame away for 30 fps. . As a result, moving image frames 201 corresponding to approximately the same real time can be extracted as action frames even if the fps are different.

Also, the action frame selection unit 104 may switch whether or not to select the moving image frame 201 separated by how many frames or more as the action frame according to the type of action determined by the action determination unit 103 . Since various actions are performed in ball games, the ease with which an actor's posture changes depends on the action. Therefore, if the action frames selected for an action that greatly changes the posture are too far apart from each other, the moving image frame 201 in which the actor is not captured may be selected as the action frame. On the other hand, if the action frames selected for an action whose posture is difficult to change are too close to each other, the moving image frame 201 in which the actor in almost the same posture is shot will be selected as the action frame, and all the action frames will have uniform numbers. It may not be recognized. Therefore, the action frame selection unit 104 selects a video frame 201 closer to the action frame time T as an action frame for an action with a larger change in the actor's posture, and selects a video frame 201 closer to the action frame time T for an action with a smaller change in the actor's posture. A video frame 201 away from is selected as an action frame. As a result, the above problems can be alleviated.

Also, the ball game video analysis apparatus 100 may manually accept a setting from the user as to whether the moving image frame 201 separated by more than one frame should be selected as an action frame. Even for the same action, it is difficult to automatically and completely predict the degree of change in the actor's posture. Therefore, it is useful to be able to change the action frame selection criteria according to the user's judgment.

Note that even if the action frame selection unit 104 does not select the moving image frame 201 corresponding to the frame time before or after the action frame time T as the action frame, the present embodiment can be realized. It is possible. Also, the action frame selection unit 104 may select more video frames 201 as action frames, or may select fewer video frames 201 as action frames. In addition, the action frame selection unit 104 does not evenly select moving image frames from the frame times before and after the action frame time T, but selects action frames corresponding to only one frame time before or after the action frame time T. may

<Details of the detection area setting part>
Next, details of the detection area setting unit 105 will be described.

Detection region setting unit 105 sets a detection region according to the three-dimensional position of the ball at action frame time T for each action frame selected by action frame selection unit 104 . Next, the setting of detection areas in volleyball will be described with reference to FIGS. 4 and 5. FIG.

When the coordinate z (height) of the ball at the action frame time T is equal to or greater than a predetermined threshold value (for example, 2 m), the detection region setting unit 105 sets the ball image 301A to the upper side of the detection region 300A as shown in FIG. (Upper end) A detection area 300A is set for the action frame so as to be positioned in the center. This is because when the coordinate z (height) of the ball is equal to or greater than a predetermined threshold, there is a high possibility that an actor exists under the ball. Note that the detection area 300A may be set at a position where the ball image 301A is shifted from the exact center of the upper side by one to several pixels. The "center of the upper side" in this specification includes such a position.

On the other hand, when the coordinate z (height) of the ball at the action frame time T is less than a predetermined threshold value (for example, 2 m), the detection area setting unit 105 changes the ball image 301B to the detection area 300B as shown in FIG. A detection area 300B is set with respect to the action frame so as to be positioned at the center of . This is because when the coordinate z (height) of the ball is less than a predetermined threshold, there is a high possibility that the actor exists in the vicinity of the ball. Note that the detection area 300B may be set at a position shifted from the exact center of the ball image 301B by one to several pixels. The "center" in this specification shall also include such a position.

In this way, by setting the detection area 300 in the detection area setting unit 105 and detecting the actor area 400 from within the detection area 300 in the player detection unit 106, the processing load can be reduced rather than detecting the actor area from the entire action frame. is reduced, which in turn shortens the processing time. As a result, the ball game video analysis apparatus 100 can recognize the actor's uniform number almost in real time.

It should be noted that the size of the detection area 300 is preferably slightly larger than the size occupied by the player in the video frame when the player is closest to the camera 3 during the competition. This is because athletes during the competition take various postures, such as opening their legs wide and sliding, and the movement distance between frames is large.

<Details of player detection unit>
Next, details of the player detection unit 106 will be described.

First, the player detection unit 106 detects a region in which a player (person) is photographed (hereinafter referred to as “player region”) from each detection region 300 . For example, the player detection unit 106 detects a player region from within the detection region 300 using a player detection model (neural network) obtained by learning a plurality of player images during a game by deep learning.

Next, the player detection unit 106 selects a player area in which the actor is photographed from the detected player areas as an actor area. For example, if one player area is detected, the player detection unit 106 selects the player area as the actor area 400, and if there are multiple detected player areas, , select one player area as the actor area 400 .

(A1) When a plurality of player regions are detected from the action frame at action frame time T, the player detection unit 106 selects the player region closest to the three-dimensional position of the ball at action frame time T in the three-dimensional space as Actor region 400 is selected.

(A2) When a plurality of player areas are detected from an action frame before (eg, frame time T−3) or after (eg, frame time T+3) action frame time T, in the three-dimensional space, the above (A1) Select the player area closest to the selected actor area 400 as the actor area 400

Note that the player detection unit 106 estimates the posture of the actor in the actor region 400, and leaves the actor region 400 when it is estimated that the posture is facing the camera 3. Region 400 may be discarded. As a result, the actor regions 400 in which the uniform number 500 is photographed remain, and the actor regions 400 in which the uniform number 500 is not photographed are discarded.

<<Same Player Judgment>>
Note that the player detection unit 106 may determine whether the actor regions 400 detected from each action frame as described above belong to the same player. Hereinafter, this determination process will be referred to as "same player determination process".

Next, an example of same-player determination processing will be described with reference to FIGS. 6A and 6B. FIG. 6A is a diagram showing a case where the players are determined to be the same player, and FIG. 6B is a diagram showing a case where the players are determined to be different players.

First, as shown in FIG. 6A, the player detection unit 106 detects a vector line 402A extending from the position of the camera 3A to the position of the upper side center 401A of the actor region 400C detected from the action frame captured by the camera 3A at frame time t. Calculate Similarly, as shown in FIG. 6A, the player detection unit 106 detects a vector line extending from the position of the camera 3B to the position of the upper side center 401B of the actor region 400D detected from the action frame captured by the camera 3B at frame time t. 402B is calculated.

Next, when the minimum distance 403A between the vector line 402A and the vector line 402B is less than a predetermined threshold, the player detection unit 106 determines that the actor region 400C and the actor region 400D belong to the same player, When the minimum distance 403A is equal to or greater than a predetermined threshold, it is determined that the actor areas 400C and 400D belong to different players. For example, in the case of FIG. 6A, since the minimum distance 403A is less than the predetermined threshold, the player detection unit 106 determines that the actor areas 400C and 400D belong to the same player. On the other hand, in the case of FIG. 6B, since the minimum distance 403B is equal to or greater than the predetermined threshold, the player detection unit 106 determines that the actor areas 400D and 400E belong to different players.

This determination will be described in more detail below.

The same part of the same player exists on the same three-dimensional coordinates. Therefore, as shown in FIG. 6A, even if the three-dimensional coordinates of the cameras 3A and 3B are different, the vector lines 402A and 402B ideally intersect at one point (same part of the same player). Although the vector lines may deviate slightly depending on the accuracy of extraction of the actor regions 400C and 400D, the possibility of the vector lines 402A and 402B being extremely separated is low as long as the same person is detected. Therefore, when the shortest distance between vector lines 402A and 402B is less than a threshold, player detection unit 106 determines that actor region 400C and actor region 400D belong to the same player.

On the other hand, as shown in FIG. 6B, when different players are detected in actor regions 400D and 400E, parts (401B and 401C) that appear to be the same exist at clearly different coordinates in the three-dimensional space. . As a result, vector lines 402B and 402C pass through a position that is skewed to each other, and the minimum distance 403B between vector lines is clearly greater than if the same player were being detected. Therefore, when the shortest distance between the vector lines 402B and 402C is equal to or greater than a threshold, the player detection unit 106 determines that the actor regions 400D and 400E belong to different players. For example, in FIG. 6B, a player with uniform number "13" is detected in actor area 400D, and a taller player with uniform number "33" is detected in actor area 400E. In this case, the vector line 402C passes through a position clearly higher in the Z-axis direction than the vector line 402B. Therefore, the shortest distance between the vector line 402B and the vector line 402C increases, indicating that different players are detected in the actor areas 400D and 400E.

Note that the player detection unit 106 may leave the actor regions 400 determined to belong to the same player and discard the actor regions 400 determined to belong to a different player. This reduces the possibility that a different player is erroneously detected as the actor area 400, so that the number recognition unit 107 reduces the possibility that different uniform numbers 500 are recognized. That is, the recognition accuracy of the uniform number 500 of the actor is improved.

<Details of the number recognition part>
Next, the details of the number recognition unit 107 will be described.

The number recognition unit 107 recognizes the actor's uniform number 500 from each actor area 400 . For example, the number recognition unit 107 recognizes the actor's uniform number 500 from within the actor area 400 using a uniform number detection model (neural network) that has been trained by deep learning on the number images of the uniforms of a plurality of players. do.

If the uniform numbers 500 recognized from each actor area 400 do not match, the number recognition unit 107 may determine the actor's uniform number 500 by majority vote. For example, if four of the uniform numbers 500 recognized from the six actor areas 400 are "13" and two are "10", the number recognition unit 107 determines that the number of matches is four. The uniform number "13" with the largest number is determined as the uniform number 500 of the actor.

Further, the number recognition unit 107 may perform processing in two steps, first detecting the image area of the number from the actor area 400 and then recognizing the number itself from the detected image area.

In addition, the number recognition unit 107 calculates the likelihood (for example, a value of 0 or more to 1 or less) indicating the likelihood of the recognized uniform number candidate for each actor region 400, and the result of adding the likelihood is the highest. A large candidate may be determined to be the actor's uniform number. In the majority case described above, the likelihood of the number recognized does not affect the decision, so the uniform number "16" recognized from one actor region 400 is 90% correct, and from two actor regions 400 Even if the recognized uniform number "10" is 30% correct, the correct uniform number is determined to be "10". On the other hand, by making a decision taking likelihood into consideration in this way, the likelihood of the recognized number can be reflected in the decision of the number.

<Processing flow>
Next, a processing flow in the ball game video analysis apparatus 100 will be described. The processing flow is roughly divided into action determination processing and actor recognition processing.

<<Action Judgment Processing>>
Next, action determination processing will be described with reference to the flowchart shown in FIG.

The video receiving unit 101 receives moving image frames 201 transmitted from the cameras 3A to 3D and stores them in the storage unit 109 (S101). The trajectory calculation unit 102 calculates the three-dimensional position and velocity of the ball at each frame time from the plurality of moving image frames 201 stored in the storage unit 109, and generates ball trajectory information 202 (S102).

The action determination unit 103 determines whether or not an action has occurred with respect to the ball from the ball trajectory information 202 (S103). If it is determined that no action has occurred (S103: NO), the flow returns to S101. When it is determined that an action has occurred (S103: YES), the action determination unit 103 generates action information 203 (S104) and generates a thread for actor recognition processing (S105). Then the flow returns to S101.

<<Actor recognition processing>>
Next, the actor recognition processing will be described with reference to the flowchart shown in FIG. The actor recognition process corresponds to the thread process generated in S105 of FIG.

The action frame selection unit 104 selects at least two cameras 3 based on the three-dimensional position of the ball and the positions of the cameras 3 included in the action information 203 (S201).

Next, the action frame selection unit 104 selects an action frame corresponding to the action frame time and action frames corresponding to frame times before and after the action frame time from the video frames 201 of each camera 3 selected in S201. (S202).

Next, the detection area setting unit 105 sets the detection area 300 for each action frame selected in S202 (S203).

Next, the player detection unit 106 detects a player area from each detection area 300 set in S203 (S204). Then, the player detection unit 106 selects the actor area 400 from among the player areas detected in S204 (S205). Then, the player detection unit 106 uses the actor area 400 of each camera 3 selected in S201 to perform same player determination processing (S206). Note that the same player determination process may be omitted.

Next, the number recognition unit 107 recognizes the uniform number 500 from each actor area 400 and determines the uniform number 500 of the actor (S207). When the recognized uniform numbers 500 do not match, the number recognition unit 107 determines the uniform number 500 having the largest number as the uniform number 500 of the actor, as described above.

Next, the number recognition unit 107 generates actor information 204 that associates the action frame time T with the uniform number 500 of the actor detected in S207, and stores it in the storage unit 109 (S208). Then, the flow ends (END).

In this way, the ball game video analysis apparatus 100 repeatedly executes action determination processing, and when it is determined that an action has occurred, generates a thread for actor recognition processing and recognizes the uniform number 500 of the actor. As a result, the actor recognition process is executed only when an action has occurred with respect to the ball, so the processing load on the ball game video analysis apparatus 100 can be reduced.

<Summary of Embodiment 1>
In Embodiment 1, ball game video analysis apparatus 100 receives moving image frames 201 from cameras 3A to 3D, calculates the trajectory of the ball from the plurality of received moving image frames 201, and calculates changes in the calculated trajectory of the ball. Based on this, it is determined whether or not an action has occurred by the player with respect to the ball, and if an action has occurred, the video frame at the timing when the action has occurred is selected from among the plurality of video frames 201 as an action frame. , the player (actor) who performed the action from the action frame is recognized by uniform number 500, for example.

This makes it possible to identify the player (actor) who performed the action on the ball based on the moving image frame of the ball game.

Further, in Embodiment 1, a detection area 300 corresponding to the three-dimensional position of the ball at the timing when the action occurs is set in the action frame, and an actor area 400 in which the actor is photographed from the detection area 300 is set. may be detected and actors may be recognized from the actor domain 400 .

As a result, the actor detection process is performed only within the detection area 300 , and the uniform number 500 recognition process is performed only within the actor area 400 . Therefore, the processing load can be reduced compared to the case where the actor detection process and the uniform number recognition process are performed on the entire moving image frame. Therefore, the ball game video analysis apparatus 100 can recognize the actor almost in real time.

(Embodiment 2)
Embodiment 2 describes a play analysis system that analyzes a player's play using the play information 205 generated by the ball game video analysis apparatus 100 described in Embodiment 1. FIG. Constituent elements common to those in the first embodiment are denoted by the same reference numerals, and descriptions thereof are omitted.

FIG. 9 shows a configuration example of the play analysis system 2. As shown in FIG. The play analysis system 2 includes a plurality of cameras 3A, 3B, 3C, 3D, a ball game video analysis device 100, a play analysis device 1000, a play information DB 1100, a play information management device 1200, a display device 1300, and an input device. 1400. The cameras 3A to 3D and the ball game video analysis device 100 are as described in the first embodiment.

The ball game video analysis device 100, the play analysis device 1000, and the play information management device 1200 are connected via a wired or wireless network. Also, the play information management device 1200 and the play information DB 1100 are connected by a predetermined cable. Note that the play information DB 1100 may be built in a storage device (not shown) within the play information management device 1200 . Also, the play analysis device 1000, the display device 1300 and the input device 1400 are connected by a predetermined cable or wireless I/F.

The play information DB 1100 manages the play information 205 generated by the ball game video analysis device 100 as a database. The play information management device 1200 registers the play information 205 generated by the ball game video analysis device 100 in the play information DB 1100 . Also, the play information management device 1200 acquires the play information 205 from the play information DB 1100 or corrects the play information 205 in the play information DB 1100 based on instructions from the play analysis device 1000 .

The play analysis device 1000 generates a UI 2000 (see FIG. 11) relating to ball game play analysis based on the play information 205 in the play information DB 1100, and displays it on the display device 1300 (for example, a liquid crystal display). Also, the play analysis device 1000 executes processing related to the play information 205 based on an operation performed on the UI 2000 through the input device 1400 (for example, a mouse or the like). Note that the play analysis device 1000, the display device 1300, and the input device 1400 may be an integrated device. In this case, the display device 1300 and the input device 1400 may be devices that serve both as input and display, such as touch panel displays.

A case where the ball game is volleyball will be described below as an example. However, Embodiment 2 is not limited to volleyball, and can be applied to various ball games such as table tennis or badminton.

FIG. 10 shows a configuration example of a play analysis device 1000. As shown in FIG. FIG. 11 shows an example of UI 2000 generated by play analysis device 1000 .

The play analysis device 1000 has a UI control section 1001 , a video control section 1002 , a play information control section 1003 , a score information control section 1004 and a play analysis section 1005 .

A UI control unit 1001 generates a UI 2000 and displays it on the display device 1300 . Also, the UI control unit 1001 receives an operation for the UI 2000 through the input device 1400 .

The image control unit 1002 displays a ball game play image 2100 on the UI 2000 . The play image 2100 may be a real-time image being captured by the cameras 3A to 3D, or may be a recorded image of past play. Note that when receiving a camera switching instruction from the user, the video control unit 1002 displays the play video 2100 captured by the switching destination camera.

The play information control unit 1003 arranges the contents of the play information 205 generated during one rally in chronological order and displays them as a play information list 2200 on the UI 2000 . Details of the play information control unit 1003 and the play information list 2200 will be described later.

The score information control unit 1004 arranges the contents of the score information in chronological order and displays them on the UI 2000 as a score information list 2300 . One piece of score information includes the score of one's own team and the score of the opponent's team after one rally. One piece of score information is associated with one or more pieces of play information 205 generated during the rally. Details of the score information control unit 1004 and the score information list 2300 will be described later.

The play analysis unit 1005 generates a play analysis image 2400 (see FIG. 12), which is an image showing the analysis result of the play of the player selected for analysis, and displays it on the UI 2000. FIG. Details of the play analysis unit 1005 will be described later.

<Play analysis image>
Next, a toss play analysis image 2400 will be described with reference to FIG. The toss play analysis image 2400 is an image for analyzing the toss play of the analysis target player (typically the setter).

The toss play analysis image 2400 is based on the front zone of the target player's side of the court. Note that the user can operate the cursor button 2410 to move the area of the play analysis image 2400 for the court.

The play analysis unit 1005 acquires from the play information DB 1100 the play information 205 in which the actor number is the number of the player to be analyzed and the action type is "toss". Then, the play analysis unit 1005 identifies the position where the analysis target player performed the toss from the ball coordinates of the acquired play information 205 .

The play analysis unit 1005 arranges the attribute image 2401 at the position where the target player performed the toss in the play analysis image 2400 of the toss. The attribute image 2401 has, for example, a shape including a circle and a bar, and visually expresses a plurality of different attribute information regarding the toss by the color of the circle and the length of the bar. The toss-related attribute information includes, for example, the type of toss, the speed of the ball during the toss, the height of the ball during the toss, and the movement distance of the ball during the toss.

- The color of the inner circle 2402 indicates the type of toss. For example, light blue indicates a toss to the right, red indicates a toss to the back, yellow indicates a toss to the left, pink indicates a quick, and blue indicates a two-attack. In addition, in FIG. 12, the difference in color is represented by the difference in filling pattern.
- The color of the outer circle 2403 excluding the area of the inner circle 2402 indicates the speed of the ball during the toss. For example, green indicates 20 km/h or less, yellow indicates 20 to 25 km/h, and red indicates 25 km/h or more. In addition, in FIG. 12, the difference in color is represented by the difference in filling pattern.
- The size (diameter) of the outer circle 2403 indicates the height of the ball when tossed. For example, a small outer circle is less than 2 m, a medium outer circle is less than 2-3 m, and a large outer circle is 3 m or more.
- The length of the bar 2404 indicates the distance traveled by the ball during the toss. For example, a long bar is 4 m or more, a medium bar is 2 to less than 4 m, and a short bar is less than 2 m. Note that the length of the bar 2404 is not limited to the distance the ball moves during the toss, but may be the distance acquired along with the toss, for example, the distance from the net of the attack position, which is an action linked to the toss. .

This allows the user to visually grasp and analyze the toss play of the target player from the toss play analysis image 2400 . Note that the above description of the play analysis image 2400 for toss can also be applied to actions other than toss. For example, the color of the inner circle 2402 indicates the type of attack, the color of the outer circle 2403 indicates the speed of the ball during the attack, the size of the outer circle 2403 indicates the height of the ball during the attack, and the length of the bar 2404. may indicate the distance traveled by the ball when attacking.

Note that the toss play analysis image 2400 may be displayed when the user performs the following operation.
(Step S11) When the user presses a toss analysis button (not shown) on the UI 2000, the UI control unit 1001 displays a player list.
(Step S12) The user selects a toss analysis target player from the displayed list of players.
(Step S<b>13 ) The play analysis unit 1005 acquires a plurality of pieces of play information 205 regarding the selected target player from the play information DB 1100 . Then, the play analysis unit 1005 identifies the toss attribute information from the acquired play information 205 and generates a toss play analysis image 2400 .
(Step S<b>14 ) The play analysis unit 1005 displays the generated play analysis image 2400 of the toss on the UI 2000 .

<Play information list>
Next, the play information list 2200 will be described with reference to FIG.

As described above, the play information list 2200 displays the contents of the play information 205 generated during one rally arranged in chronological order. The play information list 2200 has a play button 2201 , a team 2202 , a player number 2203 and an action 2204 for each play information 205 .

The play button 2201 is a button for playing back the play video 2100 associated with the play information 205 associated with the play button 2201 . When the user presses the play button 2201 , the video control unit 1002 plays the play video 2100 near the frame time of the play information 205 corresponding to the play button 2201 . Thereby, the user can easily check the play of the player corresponding to the play information 205 .

Team 2202 indicates whether the player who performed action 2204 is his team or the opponent's team. For example, "a" represents your team and "e" represents the opponent's team. Also, the team 2202 is a button, and the play information control unit 1003 displays buttons "a" and "e" for correction when the team 2202 is pressed.

The player number 2203 indicates the number (uniform number) of the player who performed the action 2204 . Also, the player number 2203 is a button, and when the player number 2203 is pressed, the play information control unit 1003 displays a button for correcting the player number. The button for correcting the player number may be a button for inputting numbers. Alternatively, the player number correction button may be a list of player numbers registered in the team.

Action 2204 indicates the type of action. For example, "S" for serve, "R" for reception, "E" for toss, "A" for attack, "D" for dig, and "B" for block. Also, the action 2204 is a button, and when the action 2204 is pressed, the play information control unit 1003 selects "S", "R", "E", "A", "D" for correction. and "B" buttons are displayed (see FIGS. 16 and 17).

For example, the first line of the play information list 2200 shown in FIG. 13 indicates that the player with uniform number "10" of his team ("a") held a reception "R".

<Score information list>
Next, the score information list 2300 will be described with reference to FIG.

As described above, the score information list 2300 displays the contents of the score information corresponding to each rally in one set arranged in chronological order. The score information list 2300 has a play button 2301, a score 2302, and a check box 2303 for each piece of score information.

The play button 2301 is a button for playing back the play video 2100 related to the score information associated with the play button 2301 . When the user presses the play button 2301 , the video control unit 1002 plays the rally play video 2100 corresponding to the play button 2301 .

A score 2302 indicates the scores of the own team and the opponent's team after the rally. In addition, the score 2302 can be selected (clicked), and when the score 2302 is selected, the score information control unit 1004 displays the play information list 2200 associated with the rally (score information) of the score 2302. display.

Check boxes 2303 are for the user to turn on and off checks for rallies by score. For example, in a real-time situation such as during a game, even if an error is found in the contents of the play information list 2200 (team 2202, player number 2203, action 2204), the next rally will start immediately. The user can identify that the contents of the play information list 2200 are in a provisional state by turning on the check box 2303 (input of identification information) corresponding to the rally in preparation for re-verification (correction) later. make it This allows the user to later collectively re-verify the rallies for which the check boxes 2303 have been turned on. For example, the score information control unit 1004 restricts and displays only the rallies for which the check box 2303 is turned on according to an instruction from the user, so that the user can perform correction work efficiently. Also, the number of check boxes 2303 is not limited to one. For example, as shown in FIG. 14, two or more check boxes may be provided for one piece of score information for each application.

<Correction of play information>
Next, a procedure for correcting the play information 205 will be described with reference to FIGS. 15, 16 and 17. FIG. It should be noted that the correction procedure is only an example, and correction by other procedures is also possible.

(Step S21) While viewing the play information list 2200 that is updated in real time during the rally, the user performs the following operation when determining that the play information 205 related to the rally needs to be corrected. That is, the user turns on the check box 2303 corresponding to the score 2302 of the rally in the score information list 2300 .

(Step S22) As shown in FIG. 15, the user selects the score 2302 with the check box 2303 turned on after one set or match is finished.

(Step S23) As shown in FIG. 15, the score information control section 1004 displays the play information list 2200 associated with the score 2302 (score information) selected in step S22.

(Step S24) As shown in FIG. 16, the user presses an action 2204 (for example, "A") to be corrected from the play information list 2200. FIG.

(Step S25) As shown in FIG. 16, the play information control unit 1003 detects pressing of the action 2204, and displays a button 2210 for correcting the action.

(Step S26) As shown in FIG. 17, the user selects a button (for example, "E") corresponding to the action after correction from among the buttons 2210 for correction.

(Step S27) The play information control unit 1003 corrects the play information 205 with an action corresponding to the correction button (for example, "E") selected in step S26. For example, as shown in FIG. 17, an action 2204 "A" before correction in the play information 205 is corrected to an action 2204 "E" after correction.

According to the above procedure, errors in the play information 205 can be easily corrected. Note that the correction target is not limited to the action 2204, and the team 2202 and player number 2203 can also be corrected in the same way. In addition, regardless of whether or not the check box 2303 is corrected, when the play information 205 whose check box 2303 is in the on state is confirmed, the state of the check box 2303 may be turned off in conjunction with this.

<Summary of Embodiment 2>
In Embodiment 2, the play analysis device 1000 includes a play analysis unit 1005 that uses a plurality of pieces of play information 205 to generate a play analysis image 2400 showing each position where a specific player has performed a specific action. Note that the play information 205 may be generated by the ball game video analysis device 100 . For example, the ball game video analysis device 100 calculates the trajectory of a moving body (ball) in a ball game using a plurality of play videos captured by a plurality of cameras 3A to 3D, and based on changes in the trajectory of the moving body , an action related to the ball game, the player who performed the action, and the position where the player performed the action are detected, and play information 205 is generated in which the action, the player, and the position are associated with each other.

As a result, since the play analysis image 2400 displays each position where the specific player performed the specific action, the user can analyze the specific action of the specific player in detail from the play analysis image 2400. can be done.

Further, in Embodiment 2, the play analysis device 1000 includes a play information control unit 1003 that arranges and displays information on players and actions included in a plurality of pieces of play information 205 in chronological order. The play information 205 may be generated by the ball game video analysis device 100 . For example, the ball game video analysis device 100 calculates the trajectory of the moving body of the ball game using a plurality of play videos captured by the plurality of cameras 3A to 3D, and based on changes in the trajectory of the moving body, An action and the player who performed the action are detected, and play information 205 in which the action and the player are associated is generated.

As a result, the play analysis device 1000 can display the player and action information (play information list 2200 ) included in the pieces of play information 205 automatically generated by the ball game video analysis device 100 . Therefore, the user can check the play performed by each player during the ball game without performing complicated operations.

In addition, the play information control unit 1003 displays the information of the players and actions included in at least one piece of the play information 205 in association with the score in real time, arranging them in chronological order, and using the check boxes 2303, displays the information in score units. It will be possible to later identify that the player and action information that has been played is in a provisional state.

As a result, the play analysis device 1000 can collectively perform the verification later even when there are multiple verifications of the player and action information (play information list 2200) included in the multiple pieces of automatically generated play information 205. I can confirm. Therefore, when the contents of the automatically generated play information 205 are incorrect, the user can easily and efficiently confirm and correct the error.

The embodiments according to the present disclosure have been described above in detail with reference to the drawings, and the functions of the devices 100, 1000, and 1200 described above can be realized by computer programs.

FIG. 18 is a diagram showing the hardware configuration of a computer that implements the functions of each device by a program. This computer 3100 includes an input device 3101 such as a keyboard, mouse, or touch pad, an output device 3102 such as a display or speaker, a CPU (Central Processing Unit) 3103, a GPU (Graphics Processing Unit) 3104, a ROM (Read Only Memory) 3105, Read information from storage media such as RAM (Random Access Memory) 3106, hard disk drive or SSD (Solid State Drive) 3107, DVD-ROM (Digital Versatile Disk Read Only Memory) or USB (Universal Serial Bus) memory A reading device 3108 and a transmitting/receiving device 3109 for communicating via a network are provided, and each unit is connected by a bus 3110 .

Then, the reading device 3108 reads the program from the recording medium recording the program for realizing the functions of the respective devices, and stores the program in the storage device 3107 . Alternatively, the transmitting/receiving device 3109 communicates with a server device connected to the network, and causes the storage device 3107 to store the program for realizing the functions of the above devices downloaded from the server device.

Then, the CPU 3103 copies the program stored in the storage device 3107 to the RAM 3106, sequentially reads out instructions included in the program from the RAM 3106, and executes them, thereby realizing the functions of the respective devices.

Each functional block used in the description of the above embodiments is typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them. Although LSI is used here, it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Also, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure connections and settings of circuit cells inside the LSI may be used.

Furthermore, if an integration technology that replaces the LSI appears due to advances in semiconductor technology or another derived technology, the technology may naturally be used to integrate the functional blocks. Application of biotechnology, etc. is possible.

One aspect of the present disclosure is useful in systems for analyzing ball games.

1 ball game video analysis system 2 play analysis system 3A, 3B, 3C, 3D cameras 4, 1300 display device 5, 1400 input device 100 ball game video analysis device 101 video reception unit 102 trajectory calculation unit 103 action determination unit 104 action frame selection unit 105 Detection area setting unit 106 player detection unit 107 number recognition unit 108 result output unit 109 storage unit 201 video frame 202 ball trajectory information 203 action information 204 actor information 205 play information 1000 play analysis device 1001 UI control unit 1002 video control unit 1003 play information Control unit 1004 Score information control unit 1005 Play analysis unit 1100 Play information DB
1200 play information management device 2000 UI
2100 play video 2200 play information list 2300 score information list 2400 play analysis image

Claims (4)

A play analysis device for analyzing a play of a ball game,
A trajectory of the moving body in the ball game is calculated using a plurality of play videos captured by a plurality of cameras, and based on changes in the trajectory of the moving body, an action related to the ball game, the player who performed the action, and a ball game video analysis unit that detects the position where the player performs the action and generates play information that associates the action, the player, and the position;
A plurality of pieces of play information of a target player who has performed a specific action is acquired, and a play analysis image is generated in which the execution position and state of the specific action are aggregated and shown on a zone image of the court from the plurality of pieces of play information. a play analysis unit that generates ;
The play analysis unit arranges a plurality of attribute images as the state of the specific action at each position where the specific action is performed in the play analysis image,
The attribute image includes at least two pieces of attribute information of a type related to the specific action, a speed of the ball related to the specific action, a height of the ball related to the specific action, and a distance associated with the specific action. show,
Play analysis equipment. The attribute image has the shape of a circle and a bar, and represents the plurality of attribute information in the form of the circle and the bar.
The play analysis device according to claim 1 . the ball game is volleyball, the specific action is toss,
The color of the circle represents the type of the toss,
the color of the circumference of the circle represents the velocity of the ball with respect to the toss;
the width of the perimeter indicates the height of the ball with respect to the toss;
the length of the bar represents the distance traveled by the ball with respect to the toss;
The play analysis device according to claim 2 . A play analysis method for analyzing the play of a ball game, comprising:
calculating the trajectory of the moving object of the ball game using a plurality of play videos taken by a plurality of cameras;
Detecting an action related to the ball game, the player who performed the action, and the position where the player performed the action, based on changes in the trajectory of the moving object;
Generate play information that associates the action, the player, and the position,
A plurality of pieces of play information of a target player who has performed a specific action is acquired, and a play analysis image is generated in which the execution position and state of the specific action are aggregated and shown on a zone image of the court from the plurality of pieces of play information. generate and
arranging a plurality of attribute images as the state of the specific action at each position where the specific action is performed in the play analysis image;
The attribute image includes at least two pieces of attribute information of a type related to the specific action, a speed of the ball related to the specific action, a height of the ball related to the specific action, and a distance associated with the specific action. show,
Play analysis method.

Images