JP7429887B2 - Ball game video analysis device, ball game video analysis method, and computer program - Google Patents

Description

The present disclosure relates to a ball game video analysis device, a ball game video analysis method, and a computer program.

BACKGROUND ART Conventionally, "Data Volley" has been commercially available as volleyball analysis software, and a technique is known that converts the status of players on a team into data based on the subjective judgment of an analyst familiar with this software.

In recent years, cameras have been installed in each of the four corners overlooking the court, and these four cameras are synchronized to capture footage of the play, track the movement of the ball, and detect the ball, without relying on analysts' operations. There is a known technology that obtains three-dimensional positional information of a player and simultaneously outputs analysis data regarding the player's jersey number and the type of action included in the play (serve, reception, dig, toss, attack, block) (Patent Document 1).

International Publication No. 2019/225415

According to Patent Document 1, high-performance dedicated cameras with the same specs are installed at each of the four corners overlooking the court, and the video captured by each dedicated camera is configured to be captured in synchronization.

By the way, a simple ball game video analysis device can be configured using a camera that is nearby (for example, a camera installed in a mobile terminal such as a smartphone or tablet) instead of a dedicated camera, and can be used to practice a ball game (for example, practicing a volleyball serve). ), there is a desire to use it for However, if the camera is not a dedicated camera, there is a problem in that the movement trajectory of a moving object (for example, a ball) related to a ball game cannot be identified due to differences in performance between cameras used for ball game video analysis.

An object of the present disclosure is to provide a technology that can identify the movement trajectory of a moving object related to a ball game using a nearby camera without using a dedicated camera specialized for ball game video analysis.

A ball game video analysis device according to the present disclosure includes a receiving unit that receives a first video and a second video that are taken from different positions of a person's action toward a moving object used in a ball game; A reference frame image, which is a frame image in which a specific sub-action constituting the action was shot, is identified from each of the second videos, and a first reference frame image, which is the reference frame image identified from the first video, is identified. a reference frame specifying unit that generates synchronization difference information that is information indicating a time difference between a reference frame image and a second reference frame image that is the reference frame image identified from the second moving image; an analysis unit that corrects a synchronization difference between frame images between the first video and the second video based on the synchronization information and identifies a movement trajectory of the moving body, and the reference frame identification unit generates and outputs a reference frame confirmation UI that presents the first reference frame image and the second reference frame image to the user, and the analysis unit extracts the reference frame image from the reference frame confirmation UI. If a user's instruction not to change is received, the synchronization difference information is confirmed, and if a user's instruction to change the reference frame image is received from the reference frame confirmation UI, the first reference frame is fixed. The image and candidate images of the second reference frame image are extracted, and a user operation for selecting the reference frame image from the candidate images is accepted.

In the ball game video analysis method according to the present disclosure, a device receives a first video and a second video that are taken from different positions of a person's action toward a moving object used in a ball game, and A reference frame image, which is a frame image in which a specific sub-action constituting the action was shot, is identified from each of the second videos, and a first reference frame image, which is the reference frame image identified from the first video, is identified. Generate synchronization difference information that is information indicating a time difference between a reference frame image and a second reference frame image that is the reference frame image identified from the second video, and When correcting the synchronization deviation of the frame images with the second moving image based on the synchronization deviation information, specifying the movement trajectory of the moving body, and specifying the reference frame image, the first reference frame image and If a reference frame confirmation UI that presents the second reference frame image to the user is generated and output, and a user's instruction not to change the reference frame image is received from the reference frame confirmation UI, the synchronization error occurs. When the information is determined and a user instruction to change the reference frame image is received from the reference frame confirmation UI, candidate images of the first reference frame image and the second reference frame image are extracted. , accepts a user's operation to select the reference frame image from the candidate images.

A computer program according to the present disclosure receives a first video and a second video taken from different positions of a person's action on a moving object used in a ball game, and receives a first video and a second video taken from different positions, and compares the first video and the second video. From each, specify a reference frame image that is a frame image in which a specific sub-action constituting the action was photographed, and identify a first reference frame image that is the reference frame image identified from the first video and the Synchronization difference information, which is information indicating a time difference between the second reference frame image, which is the reference frame image specified from the second video, is generated, and synchronization difference information between the first video and the second video is generated. When correcting the synchronization difference between the frame images based on the synchronization information, analyzing the movement trajectory of the moving object, and identifying the reference frame image, the first reference frame image and the second reference frame image are corrected based on the synchronization difference information. generating and outputting a reference frame confirmation UI that presents a frame image to a user, and when receiving an instruction from the user not to change the reference frame image from the reference frame confirmation UI, determining the synchronization shift information; When a user's instruction to change the reference frame image is received from the reference frame confirmation UI, candidate images of the first reference frame image and the second reference frame image are extracted, and candidate images are extracted from the candidate images. A user operation for selecting the reference frame image is accepted.

Note that these comprehensive or specific aspects may be realized by a system, a device, a method, an integrated circuit, a computer program, or a recording medium, and any of the systems, devices, methods, integrated circuits, computer programs, and recording media may be implemented. It may be realized by any combination.

According to the present disclosure, by using a reference frame confirmation UI that confirms and changes automatically identified reference frame images, a nearby camera can be used without using a dedicated camera specialized for ball game video analysis. It is possible to identify the movement trajectory of a moving object related to a ball game.

A diagram showing a configuration example of a ball game video analysis system according to the present embodiment Diagram to explain how to locate the ball Block diagram showing a configuration example of a ball game video analysis device Diagram for explaining how to identify a reference frame image Flowchart showing an example of preliminary work Sequence chart showing an example of processing related to preliminary work Sequence chart showing an example of processing related to preliminary work when an operation to change the reference frame image is performed in FIG. 6 Diagram showing a detailed example of the reference frame confirmation UI (User Interface) Diagram showing a detailed example of the reference frame change UI Diagram showing an example of displaying analysis results A diagram showing the hardware configuration of a computer that implements the functions of the ball game video analysis device according to the present disclosure through a program. A diagram showing the hardware configuration of a computer that implements the functions of a first form terminal, a second form terminal, and a display terminal according to the present disclosure by a program.

Embodiments of the present disclosure will be described in detail below with appropriate reference to the drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters and redundant explanations of substantially the same configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

(This embodiment)
<Configuration of ball game video analysis system>
FIG. 1 is a diagram showing an example of the configuration of a ball game video analysis system according to the present embodiment.

The ball game video analysis system 10 is a system that analyzes a video shot of a ball game and analyzes the trajectory of a moving object used in the ball game. In this embodiment, volleyball, which is one of the ball games, will be explained as an example. However, the ball game video analysis system 10 is applicable not only to volleyball but also to various ball games such as soccer, baseball, table tennis, basketball, tennis, rugby, American football, lacrosse, badminton, or ice hockey. Further, the moving object targeted by the ball game video analysis system 10 is typically a ball. However, the moving object targeted by the ball game video analysis system 10 is not limited to a ball, and may be of a shape that does not fit the concept of a "ball", such as a badminton shuttle or an ice hockey puck, for example.

The ball game video analysis system 10 includes a first mobile terminal 20A, a second mobile terminal 20B, a display terminal 30, and a ball game video analysis device 100. The first mobile terminal 20A, the second mobile terminal 20B, the display terminal 30, and the ball game video analysis device 100 can send and receive information to and from each other through the communication network N. The communication network N is configured by, for example, a wireless communication network (wireless LAN (Local Area Network), 4G, 5G, etc.), an Internet network, or a combination thereof.

The first mobile terminal 20A and the second mobile terminal 20B are wireless communication terminals equipped with a camera. Examples of the first mobile terminal 20A and the second mobile terminal 20B are a mobile phone, a smartphone, a tablet, a mobile PC, or the like.

The first mobile terminal 20A and the second mobile terminal 20B photograph the actions of a person (hereinafter referred to as a "player") playing a ball game. In this embodiment, a volleyball serve, which is one of the player's actions, will be explained as an example. However, the player's action is not limited to a volleyball serve, but may also be a dig, a reception, a toss, an attack, a block, etc. Alternatively, the action of a person playing a ball game may be a specific action in the above sports other than volleyball.

The first mobile terminal 20A and the second mobile terminal 20B, for example, take videos of a player's serve practice from different directions, and generate a first video 201A and a second video 201B, respectively. For example, as shown in FIG. 1, the first mobile terminal 20A photographs the player from the rear right, and the second mobile terminal 20B photographs the player from the rear left.

The first mobile terminal 20A and the second mobile terminal 20B transmit the generated first video 201A and second video 201B to the ball game video analysis device 100 via the communication network N, respectively.

The display terminal 30 displays the analysis results by the ball game video analysis device 100. Note that a display example of the analysis results will be described later (see FIG. 10). Examples of the display terminal 30 are a mobile phone, a smartphone, a tablet, or a mobile PC. The display terminal 30 may be read as a third mobile terminal. Although a configuration in which the display terminal 30 for displaying the analysis results is separately provided has been illustrated, either the first mobile terminal 20A or the second mobile terminal 20B may be used as a terminal for displaying the analysis results.

The ball game video analysis device 100 analyzes the movement trajectory of the ball in three-dimensional space based on the first video 201A and second video 201B received from the first mobile terminal 20A and the second mobile terminal 20B, respectively. . The movement trajectory of the ball in the three-dimensional space can be generated by arranging the three-dimensional coordinates of the ball in time series.

As shown in FIG. 2, the three-dimensional coordinates of the ball at time t can be calculated from the frame image F1 at time t in the first video 201A and the frame image F2 at time t in the second video 201B. . That is, as shown in FIG. 2, a vector V1 extending from the position U1 of (the camera of) the first mobile terminal 20A to the position B1 of the ball in the frame image F1 and the position of (the camera of the second mobile terminal 20B) The intersection with the vector V2 extending from U2 to the ball position B2 in the frame image F2 becomes the three-dimensional coordinate Pa of the ball.

Here, the time code (time information) that the first mobile terminal 20A gives to each frame image of the first video 201A, and the time code that the second mobile terminal 20B gives to each frame image of the second video 201B. If the time code is completely accurate, the three-dimensional coordinates of the ball can be accurately specified using the above-described method using the frame images F1 and F2 to which the time code of time t has been added.

However, when using a general-purpose camera such as a camera mounted on a mobile terminal rather than a dedicated camera specialized for ball game video analysis as shown in Patent Document 1, the first mobile terminal 20A and the second mobile The terminal 20B is not necessarily able to provide an accurate time code. The reason for this will be explained below.

Since the ball in sports related to ball games moves at high speed, in order to identify the three-dimensional position of the ball, it is necessary to shoot at a high frame rate of 30 fps or higher, for example, and combine the frame image shot by the first mobile terminal 20A with the second frame image. Synchronization precision in milliseconds is required between the frame image captured by the mobile terminal 20B.

Here, it is possible to synchronize the internal clocks of the first mobile terminal 20A and the second mobile terminal 20B using NTP (Network Time Protocol). However, since the synchronization accuracy of NTP is on the order of seconds, there is a possibility that a synchronization difference on the order of milliseconds will occur between the internal clock of the first mobile terminal 20A and the internal clock of the second mobile terminal 20B. Therefore, with this method, the three-dimensional coordinates of a ball moving at high speed may not be specified with sufficient accuracy.

Alternatively, it is possible to synchronize the internal clocks of the first mobile terminal 20A and the second mobile terminal 20B using a satellite positioning system such as GPS (Global Positioning System). However, with this method, the camera of a mobile terminal that does not have a GPS function cannot be used. Furthermore, if a GPS signal cannot be received in an indoor facility or the like, there is a possibility that a synchronization difference will occur between the internal clock of the first mobile terminal 20A and the internal clock of the second mobile terminal 20B. Therefore, with this method, the three-dimensional coordinates of a ball moving at high speed may not be specified with sufficient accuracy.

Alternatively, when the first mobile terminal 20A and/or the second mobile terminal 20B assigns a time code, a synchronization error may occur. For example, if the processing performance of the second mobile terminal 20B is lower than that of the first mobile terminal 20A, the processing load of time code assignment on the mobile terminal 20B is large, and overall it is a little slower than the first mobile terminal 20A. There is a possibility that a time code will be added. In this case, even if the time codes of frame image F1 and frame image F2 are the same, frame image F2 may actually be photographed a little earlier than frame image F1. In the frame image F2 taken a little earlier, the ball is positioned at B3, and as shown in FIG. 2, the three-dimensional coordinates of the ball also shift to Pb. Therefore, the calculated movement trajectory of the ball also deviates.

Therefore, the ball game video analysis device 100 according to the present embodiment has a function of identifying the synchronization shift of frame images between the first mobile terminal 20A and the second mobile terminal 20B. Then, the ball game video analysis device 100 calculates the movement trajectory of the ball in consideration of the identified synchronization deviation. As a result, the movement trajectory of the ball can be calculated using a general-purpose camera such as a camera mounted on a mobile terminal, rather than a dedicated camera specialized for ball game video analysis as shown in Patent Document 1. This will be explained in detail below.

<Configuration of ball game video analysis device>
FIG. 3 is a block diagram showing a configuration example of ball game video analysis device 100 according to the present embodiment.

The ball game video analysis device 100 includes a moving image receiving section 101 , an image converting section 102 , a reference frame specifying section 103 , a reference frame changing section 104 , an analyzing section 105 , and a result display section 106 .

The video receiving unit 101 receives the first video 201A from the first mobile terminal 20A, and receives the second video 201B from the second mobile terminal 20B. Examples of video formats are MPEG4, TS file, H. 264, H. It is 265 mag. Each frame image constituting the first moving image 201A is given a time code by the first mobile terminal 20A. A time code is given to each frame image constituting the second moving image 201B by the second mobile terminal 20B. As described with reference to FIG. 1, the first video 201A and the second video 201B include, for example, a scene in which a player hits a serve.

The image conversion unit 102 converts each frame image of the first moving image 201A into a still image, and generates a first frame image group 202A. The image conversion unit 102 converts each frame image of the second moving image 201B into a still image, and generates a second frame image group 202B. Examples of still image formats include JPEG, GIF, and TIFF.

The reference frame identifying unit 103 identifies frame images (hereinafter referred to as "reference frame images") that serve as a reference for frame synchronization from the first frame image group 202A and the second frame image group 202B, respectively. . Hereinafter, the reference frame image specified from the first frame image group 202A will be referred to as a first reference frame image, and the reference frame image specified from the second frame image group 202B will be referred to as a second reference frame. It's called an image.

The reference frame identifying unit 103 identifies a frame image in which a specific sub-action constituting an action is photographed as a reference frame image. For example, the reference frame specifying unit 103 selects a frame image in which a moment when the hand that hits the serve (sub-action) hits the ball among a group of frame images in which a series of movements of a serve (action) is captured. Specify as the reference frame image. Next, an example of a method for specifying a reference frame image will be described with reference to FIG. 4.

As shown in FIG. 4, the reference frame specifying unit 103 sets a ball detection area 211A for the first frame image group 202A. The reference frame specifying unit 103 sets a ball detection area 211B for the second frame image group 202B. This reduces the ball detection area, so the time required for ball detection processing can be shortened. Note that the detection area for the ball does not need to be set.

The reference frame specifying unit 103 detects a ball from each frame image of the first frame image group 202A using a known method. For example, the reference frame identifying unit 103 detects a ball from a frame image by pattern matching. Alternatively, the reference frame identifying unit 103 detects a ball from the frame image using deep learning learned in advance for ball detection. The reference frame identifying unit 103 similarly detects a ball from each frame image of the second frame image group 202B.

The reference frame specifying unit 103 calculates a two-dimensional movement trajectory of the ball based on the position of the ball detected from each frame image of the first frame image group 202A. Then, the reference frame identifying unit 103 identifies the first reference frame image based on the change in the two-dimensional movement trajectory of the ball. For example, the reference frame identifying unit 103 identifies the first reference frame image through the following processes S11 to S13.

(S11) The reference frame identification unit 103 selects a first frame image in which the amount of movement of the ball in the X direction is greater than or equal to a predetermined threshold (that is, the ball has moved significantly in the horizontal direction) from the first frame image group 202A. Extract. As a result, it is possible to extract the first frame image in which the ball is flying toward the net immediately after the hand that hits the serve hits the ball.

(S12) The reference frame specifying unit 103 determines that among a plurality of first frame images included in a certain period before the first frame image extracted in S11, the amount of movement of the ball in the Y-axis direction is greater than or equal to a predetermined threshold. The first frame image that has changed and is closest to the first frame image extracted in S11 is identified. This makes it possible to identify the first frame image at the moment when the hand that served the ball hit the ball, among the multiple first frame images in which the ball is rising or falling after being thrown up for a serve. .

(S13) The reference frame identifying unit 103 sets the first frame image identified in S12 as the first reference frame image.

The reference frame identifying unit 103 identifies the second reference frame image by the same process as described above. Then, the reference frame specifying unit 103 extracts a time code corresponding to the first reference frame image (hereinafter referred to as "first reference time code") and a time code corresponding to the second reference frame image (hereinafter referred to as "second reference time code"). "Reference time code") is written in the synchronization difference information 203.

Note that the reference frame identifying unit 103 may identify the reference frame image using an action and sub-action different from serve. For example, the reference frame specifying unit 103 may use a frame image captured at the moment when the player's hand attacking (or digging, receiving, tossing, blocking, etc.) hits the ball as the reference frame image.

Further, the reference frame specifying unit 103 may write a value obtained by subtracting the second reference time code from the first reference time code (hereinafter referred to as “synchronization time difference”) to the synchronization shift information 203. The synchronization time difference written in this synchronization shift information 203 corresponds to the magnitude of the synchronization shift of frame images between the first mobile terminal 20A and the second mobile terminal 20B. For example, when the synchronization time difference is approximately 0, this indicates that the time code provided by the first mobile terminal 20A and the time code provided by the second mobile terminal 20B tend to be almost the same. If the synchronization time difference is a positive value, this indicates that the time code provided by the second mobile terminal 20B tends to be later than the time code provided by the first mobile terminal 20A. If the synchronization time difference is a negative value, this indicates that the time code provided by the first mobile terminal 20A tends to be later than the time code provided by the second mobile terminal 20B.

Returning to the explanation of FIG. 3.

The reference frame image automatically identified by the reference frame specifying unit 103 using the method described above is not necessarily the correct reference frame image. Therefore, the reference frame specifying unit 103 provides a UI (User Interface) 310 (hereinafter referred to as "reference frame confirmation UI") for the user to confirm the reference frame image. Note that details of the reference frame confirmation UI 310 will be described later (see FIG. 8).

The reference frame changing unit 104 provides a UI (hereinafter referred to as "reference frame changing UI") 320 for the user to change the reference frame image. Note that details of the reference frame change UI 320 will be described later (see FIG. 9). When the reference frame image is changed through the reference frame change UI 320, the reference frame changing unit 104 changes the synchronization shift information 203 based on the changed reference frame image.

The analysis unit 105 uses the synchronization difference information 203 from a first video 201A and a second video 201B that capture a serve action by a player, for example, to identify a movement trajectory of a ball in a three-dimensional space related to the action, Analysis result information 204 including information indicating the movement trajectory of the ball is generated. Note that the first video 201A and second video 201B that are analyzed by the analysis unit 105 are different from the first video 201A and second video 201B that were shot to generate the synchronization difference information 203 described above. It can be anything.

For example, the analysis unit 105 calculates each camera parameter of the first mobile terminal 20A and the second mobile terminal 20B (based on a perspective projection transformation matrix for converting world coordinates into a camera image) obtained in advance through calibration processing. , the position of the ball in the first frame image, and the position of the ball in the second frame image that is synchronized with the first frame image, which is corrected by the synchronization difference information 203. The three-dimensional coordinates of the ball at the timing when the image and the second frame image were photographed are identified. The analysis unit 105 specifies the movement trajectory of the ball in the three-dimensional space by arranging the three-dimensional coordinates of the ball identified in this way in chronological order. The analysis unit 105 generates analysis result information 204 including information indicating the movement trajectory of the identified ball. As a result, the movement trajectory of a ball in three-dimensional space can be analyzed with sufficient accuracy even when a camera mounted on a mobile terminal is used instead of a dedicated camera specialized for ball game video analysis.

The result display unit 106 displays the analysis result information 204 on, for example, the display terminal 30. Note that the result display unit 106 may display the analysis result information 204 on the first mobile terminal 20A and/or the second mobile terminal 20B.

<Preliminary work>
FIG. 5 is a flowchart showing an example of preliminary work. The preliminary work is work performed before the analysis unit 105 analyzes the actual action, and is, for example, work performed to generate the synchronization difference information 203.

The user installs the first mobile terminal 20A and the second mobile terminal 20B at positions where they can photograph the player and the ball from different directions (S101).

The user pairs the first mobile terminal 20A and the second mobile terminal 20B (S102). Pairing information may be transmitted to the ball game video analysis device 100.

The user instructs the first mobile terminal 20A and the second mobile terminal 20B to start photographing related to the preliminary work (S103). Thereby, the first mobile terminal 20A and the second mobile terminal 20B start photographing.

The player hits a serve (S104). Thereby, the first mobile terminal 20A and the second mobile terminal 20B can respectively photograph a first video 201A and a second video 201B in which the player hits a serve and the ball flies. Note that the user and the player may be the same person.

The user instructs the first mobile terminal 20A and the second mobile terminal 20B to stop photographing related to the preliminary work (S105). The first mobile terminal 20A and the second mobile terminal 20B each receive the instruction to stop shooting, and transmit the first video 201A and the second video 201B to the ball game video analysis device 100. The ball game video analysis device 100 specifies the first reference frame image from the first video 201A and the second reference frame image from the second video 201B, and displays the reference frame confirmation UI 310 on the display terminal 30.

The user checks the first reference frame image and the second reference frame image from the reference frame confirmation UI 310 displayed on the display terminal 30 (S106). For example, when changing the reference frame image, the user instructs the ball game video analysis device 100 to change the reference frame image through the reference frame confirmation UI 310. In this case, the ball game video analysis device 100 displays the reference frame change UI 320 on the display terminal 30.

Note that instructions to start and stop photographing in S103 and S105 above may be given from either one of the mobile terminals. For example, when an instruction to start and stop shooting is given from the first mobile terminal 20A, the instruction to start and stop shooting is automatically transmitted to the second mobile terminal 20B via the ball game video analysis device 100. It's fine.

The user operates the reference frame change UI 320 displayed on the display terminal 30 to change the reference frame image (S107). Note that details of S106 and S107 will be described later (see FIGS. 7 and 8).

Through the above processing, the ball game video analysis device 100 can generate the synchronization shift information 203 based on the first reference frame image and the second reference frame image that have been confirmed and corrected by the user.

<Processing related to preliminary work>
FIG. 6 is a sequence chart showing an example of processing related to preliminary work. FIG. 6 corresponds to detailed explanation of S106 to S107 in FIG. 5.

The video receiving unit 101 receives the first video 201A from the first mobile terminal 20A (S201), and receives the second video 201B from the second mobile terminal 20B (S202).

The image conversion unit 102 generates a first frame image group 202A from the first moving image 201A, and generates a second frame image group 202B from the second moving image 201B (S203).

The reference frame identifying unit 103 searches the first frame image group 202A for a first reference frame image in which a specific sub-action constituting an action has been photographed (S204). For example, as described above, the reference frame specifying unit 103 photographs the moment when the player's serve hand hits the ball based on changes in the two-dimensional (X direction, Y direction) movement trajectory of the ball. A first reference frame image is searched for.

Similarly to the above, the reference frame identifying unit 103 searches for the second reference frame image from the second frame image group 202B (S205). At this time, the reference frame specifying unit 103 may set the second frame image group 202B after a time code that is a predetermined time back from the time code of the first reference frame image as the search range for the second reference frame image. Thereby, the time required for the search can be shortened compared to the case where the entire second frame image group 202B is set as the search range.

The reference frame specifying unit 103 uses the time code of the first reference frame image (first reference time code) and the time code of the second reference frame image (second reference time code) as synchronization shift information 203. (S206).

The reference frame changing unit 104 transmits the first reference frame image, the second reference frame image, and information for displaying the reference frame confirmation UI 310 to the display terminal 30 (S207).

The display terminal 30 displays the first reference frame image, the second reference frame image, and the reference frame confirmation UI 310 (S208). Note that details of the reference frame confirmation UI 310 will be described later (see FIG. 8).

When the user performs an operation to the effect that the reference frame image is not changed on the reference frame confirmation UI 310 displayed in S208, the display terminal 30 sends instruction information indicating no change to the ball game video analysis device 100. Send (S209).

When the reference frame changing unit 104 receives instruction information indicating no change, it determines the contents of the synchronization shift information 203 written in S206 (S210). Then, the ball game video analysis device 100 ends this process.

FIG. 7 is a sequence chart illustrating an example of processing related to preliminary work when an operation to change the reference frame image in FIG. 6 is performed.

When the user performs an operation to change the reference frame image on the reference frame confirmation UI 310 displayed in S208, the display terminal 30 sends instruction information indicating that the reference frame image has been changed to the ball game video analysis device 100. Send (S221).

When the reference frame changing unit 104 receives instruction information indicating that there is a change, the reference frame changing unit 104 performs the following process. That is, the reference frame changing unit 104 extracts a plurality of first frame images taken before and after the first reference frame image searched in S204 from the first frame image group 202A, and The first frame image is set as a candidate for the first reference frame image.

Similarly, the reference frame changing unit 104 extracts a plurality of second frame images taken before and after the second reference frame image searched in S205 from the second frame image group 202B, and The second frame image is set as a candidate for the second reference frame image. Then, the reference frame change unit 104 displays the plurality of first candidate frame candidates and their time codes, the plurality of second reference frame image candidates and their time codes, and the reference frame change UI 320. information is transmitted to the display terminal 30 (S222).

The display terminal 30 displays the reference frame change UI 320 (S223). Note that details of the reference frame change UI 320 will be described later (see FIG. 9).

The user operates the reference frame change UI 320 displayed in S223 to select the changed first reference frame image from among the plurality of first reference frame image candidates, and selects the changed first reference frame image from among the plurality of first reference frame image candidates. The changed second reference frame image is selected from among the reference frame image candidates (S224).

The display terminal 30 displays the time code of the first reference frame image after the change selected in S224 (hereinafter referred to as the "first reference time code after the change") and/or the time code of the second reference frame image after the change. The time code of the reference frame image (hereinafter referred to as the "changed second reference time code") is transmitted to the ball game video analysis device 100 (S225).

The reference frame changing unit 104 overwrites the synchronization difference information 203 with the changed first reference time code and/or the changed second reference time code received from the display terminal 30, and removes the synchronization difference. The contents of the information 203 are determined (S226). As a result, the contents of the synchronization difference information 203 automatically generated by the reference frame specifying unit 103 are overwritten with the time code of the reference frame image changed by the user through the reference frame changing unit 104. Then, the ball game video analysis device 100 ends this process.

<Details of UI for checking reference frame>
FIG. 8 is a diagram showing a detailed example of the reference frame confirmation UI.

As shown in FIG. 8, the reference frame confirmation UI 310 includes a first image display area 210A, a second image display area 210B, an OK button 312, a change button 313, and a redo button 314. Note that the reference frame confirmation UI 310 does not need to include either the change button 313 or the redo button 314.

The display terminal 30 displays the first reference frame image transmitted from the ball game video analysis device 100 in S207 of FIG. 6 in the first image display area 210A. The display terminal 30 displays the second reference frame image transmitted from the ball game video analysis device 100 in S207 of FIG. 6 in the second image display area 210B. As shown in FIG. 8, the first image display area 210A and the second image display area 210B may be arranged side by side. However, the manner in which the first image display area 210A and the second image display area 210B are arranged is not limited to being arranged side by side. For example, the first image display area 210A and the second image display area 210B may be displayed as separate windows.

When the OK button 312 is pressed, the display terminal 30 transmits instruction information indicating no change in S209 of FIG. 6 to the ball game video analysis device 100. The user can confirm that both the first reference frame image displayed in the first image display area 210A and the second reference frame image displayed in the second image display area 210B are captured by the subaction correctly. If the image is an image that has been edited, the OK button 312 may be pressed. Thereby, as shown in S210 of FIG. 6, the contents of the synchronization difference information 203 are determined.

When the change button 313 is pressed, the display terminal 30 transmits instruction information indicating that there is a change in S221 of FIG. 7 to the ball game video analysis device 100. The user can correctly determine whether at least one of the first reference frame image displayed in the first image display area 210A and the second reference frame image displayed in the second image display area 210B is a subaction. If the image is not a photographed image, the change button 313 may be pressed. Thereby, as shown in S223 in FIG. 7, the reference frame change UI 320 is displayed on the display terminal 30.

When the redo button 314 is pressed, the display terminal 30 transmits instruction information indicating redo of the preliminary work to the ball game video analysis device 100. When receiving this instruction, the ball game video analysis device 100, the first mobile terminal 20A, and the second mobile terminal 20B redo the preliminary work from S103 in FIG. 5. For example, if the serve in S104 fails, the first reference frame image and/or the second reference frame image clearly shows a ball other than the ball served by the player (for example, a ball lying on the floor or another player). If the user recognizes a ball being practiced (such as a ball being practiced), the user may press the redo button 314.

Note that the first reference frame images include three first frame images immediately before the first reference frame image searched in S204, and three first frame images immediately after the first reference frame. The frame image may be included. In this case, there are a total of seven first reference frame images, including the first reference frame image searched in S204. Similarly, the second reference frame image includes three second frame images immediately before the second reference frame image searched in S205 and three second frame images immediately after the second reference frame image. A second frame image may be included. In this case, there are a total of seven second reference frame images, including the second reference frame image searched in S205.

In this case, the display terminal 30 displays seven first reference frame images and seven second reference frame images in the first image display area 210A and the second image display area 210B, respectively. It may be synchronized and displayed continuously.

Note that the above seven images are just an example, and the number of the plurality of first and second reference frame images may be any number. Further, the plurality of first and second reference frame images are not limited to being consecutive one frame at a time, and may be skipped at predetermined frame intervals.

This allows the user to not only check the synchronization between one first reference frame image and the second reference frame image, but also to check the synchronization between the first reference frame image and the second reference frame image. You can easily check the synchronization between

Although an example has been shown in which the reference frame confirmation UI 310 shown in FIG. 8 is displayed on the display terminal 30, this reference frame confirmation UI 310 may be displayed on either the first mobile terminal 20A or the second mobile terminal 20B. may be displayed. Alternatively, the reference frame confirmation UI 310 for the first reference frame image may be displayed on the first mobile terminal 20A, and the reference frame confirmation UI 310 for the second reference frame image may be displayed on the second mobile terminal 20B. .

<Details of UI for changing reference frame>
FIG. 9 is a diagram showing a detailed example of the reference frame change UI.

As shown in FIG. 9, the reference frame change UI 320 includes a first image display area 210A, a second image display area 210B, an OK button 322, and a return button 323. Furthermore, the reference frame change UI 320 includes a 1 frame return button 324A, an N frame return button 325A, a 1 frame advance button 326A, and an N frame advance button 327A, which correspond to the first image display area 210A. Here, N is a predetermined integer of 2 or more. Further, the reference frame change UI 320 includes a 1 frame return button 324B, an N frame return button 325B, a 1 frame advance button 326B, and an N frame advance button 327B, which correspond to the second image display area 210B.

When the return button 323 is pressed, the display terminal 30 returns to displaying the reference frame confirmation UI 310 shown in FIG. 8 .

When the one-frame return button 324A is pressed, the display terminal 30 displays a first reference frame image candidate that is one frame earlier than the first reference frame image candidate currently displayed in the first image display area 210A. , is displayed in the first image display area 210A. Note that the same applies to the one-frame return button 324B corresponding to the second image display area 210B.

When the N frame return button 325A is pressed, the display terminal 30 displays a first reference frame image candidate that is N frames earlier than the first reference frame image candidate currently displayed in the first image display area 210A. , is displayed in the first image display area 210A. Note that the same applies to the N frame return button 325B corresponding to the second image display area 210B.

When the advance one frame button 326A is pressed, the display terminal 30 selects a first reference frame image candidate one frame later than the first reference frame image candidate currently displayed in the first image display area 210A. , is displayed in the first image display area 210A. Note that the same applies to the advance one frame button 326B corresponding to the second image display area 210B.

When the N frame advance button 327A is pressed, the display terminal 30 selects a first reference frame image candidate N frames later than the first reference frame image candidate currently displayed in the first image display area 210A. , is displayed in the first image display area 210A. Note that the same applies to the advance N frame button 327B corresponding to the second image display area 210B.

The user presses each of the above-mentioned buttons 324A to 327A and 324B to 327B so that the image in which the subaction has been correctly captured is displayed in both the first image display area 210A and the second image display area 210B. You can operate it.

When the OK button 322 is pressed, the display terminal 30 displays the time code of the candidate for the first reference frame image being displayed in the first image display area 210A (the first reference time code after the change), and The time code of the candidate for the second reference frame image being displayed in the second image display area 210B (the changed second reference time code) is transmitted to the ball game video analysis device 100. This process corresponds to S225 in FIG. The user may press the OK button 322 after performing an operation such that an image in which the sub-action has been correctly captured is displayed in both the first image display area 210A and the second image display area 210B.

<Analysis result screen>
FIG. 10 is a diagram showing a display example of the analysis results.

The first mobile terminal 20A and the second mobile terminal 20B photograph the player's serve practice, and generate and transmit a first video 201A and a second video 201B, respectively. Note that the first video 201A and the second video 201B may be different from the first video 201A and the second video 201B shot for preliminary work.

The analysis unit 105 corrects the synchronization difference of the frame images between the first video 201A and the second video 201B in which serve practice is shot based on the synchronization difference information 203, and then calculates the movement trajectory of the ball. The analysis result information 204 is generated. For example, in the synchronization difference information 203, if the second reference time code is later than the first reference time code, the analysis unit 105 analyzes the difference between the second reference time code and the first reference time code. After making corrections to delay each frame image of the first video 201A by the difference time, each frame image of the first video 201 and each frame image of the second video 201B are synchronized, and the ball is Analyze the movement trajectory. As a result, the delay in assigning the time code by the second mobile terminal 20B is corrected, and the movement trajectory of the ball can be calculated with sufficient accuracy.

The result display unit 106 causes the display terminal 30 to display an analysis result screen 330 based on the analysis result information 204. For example, as shown in FIG. 10, on the analysis result screen 330, the movement trajectory 331 of the ball indicated by the analysis result information 204 may be displayed superimposed on the playback of the first video 201A or the second video 201B. . In addition, the analysis result screen 330 may display the height and speed of the ball hit by the hand at the moment it leaves the hand. In addition, the analysis result screen 330 may display the height and speed of the ball at the moment the ball passes over the net.

This allows the player to check in detail the movement of the ball during the serve as well as the movement trajectory of the ball due to the serve from the analysis result screen 330.

In addition, in the above, as a preliminary work, a video for synchronization adjustment is taken in advance, synchronization adjustment and confirmation is performed, and then a video is taken for analysis processing, and the video with the trajectory of the ball etc. superimposed is used as the analysis result. An example of displaying this is explained. However, in the present embodiment, the synchronization adjustment and analysis processing are performed continuously on the first video shot, for example, without distinguishing between the video for synchronization adjustment and the video for analysis processing. Good too. Furthermore, detection of synchronization deviation may be automated and synchronization adjustment may be performed at any timing while performing analysis processing on sequentially shot videos.

<Modified example>
In the above description, the first time code added to the first frame image by the first mobile terminal 20A and the second time code added to the second frame image by the second mobile terminal 20B are used as information indicating the frame synchronization deviation. An example using the time code No. 2 has been explained. However, information indicating a shift in frame synchronization is not limited to time codes.

For example, as information indicating a frame synchronization shift, the frame number of the first frame image (hereinafter referred to as "first frame number") in the first video 201A generated by the first mobile terminal 20A, and the frame number of the first frame image in the first video 201A generated by the first mobile terminal 20A, A frame image (hereinafter referred to as "second frame number") in the second moving image 201B generated by the mobile terminal 20B may be used. In this case, the reference frame identifying unit 103 identifies the first frame number (hereinafter referred to as "first reference frame number") of the first reference frame image and the second frame number (hereinafter referred to as "first reference frame number") of the second reference frame image. (referred to as “second reference frame number”) may be written in the synchronization difference information 203. Then, the reference frame changing unit 104 selects the frame number of the changed first reference frame image (hereinafter referred to as "the changed first reference frame number") selected through the reference frame changing UI 320, and/or , the frame number of the changed second reference frame image (hereinafter referred to as “changed second reference frame number”) may be overwritten on the synchronization shift information 203.

<Hardware configuration>
The embodiments according to the present disclosure have been described above in detail with reference to the drawings. Note that the functions of the first mobile terminal 20A, the second mobile terminal 20B, the display terminal 30, and the ball game video analysis device 100 described above can be realized by a computer program.

FIG. 11 is a diagram showing the hardware configuration of a computer that implements the functions of the ball game video analysis device 100 by a program.

This computer 1000A includes an input device 1001 such as a keyboard, mouse, or touch pad, an output device 1002 such as a display or speakers, a CPU (Central Processing Unit) 1003, a GPU (Graphics Processing Unit) 1004, a ROM (Read Only Memory) 1005, RAM (Random Access Memory) 1006, storage device 1007 such as HDD (Hard Disk Drive), SSD (Solid State Drive) or flash memory, DVD-ROM (Digital Versatile Disk Read Only Memory) or USB (Universal Serial Bus) memory, etc. A reading device 1008 that reads information from a recording medium, and a transmitting/receiving device 1009 that communicates by wire or wirelessly via a communication network N are provided, and each part is connected by a bus 1010.

Then, the reading device 1008 reads the program from the recording medium recording the program for realizing the functions of each of the above devices, and stores the program in the storage device 1007. Alternatively, the transmitting/receiving device 1009 communicates with a server device connected to the communication network N, and causes the storage device 1007 to store programs downloaded from the server device to implement the functions of each of the devices described above.

Then, the CPU 1003 copies the program stored in the storage device 1007 to the RAM 1006, and sequentially reads out and executes instructions included in the program from the RAM 1006, thereby realizing the functions of each of the devices described above.

FIG. 12 is a diagram showing the hardware configuration of a computer that implements the functions of the first mobile terminal 20A, the second mobile terminal 20B, and the display terminal 30 using programs. Note that the components already explained in FIG. 11 may be given the same reference numerals and the explanation will be omitted.

This computer 1000B includes an input device 1001, an output device 1002, a CPU 1003, a GPU 1004, a ROM 1005, a RAM 1006, a storage device 1007, a reading device 1008, a transmitting/receiving device 1009, and a camera device 1011, and each part is connected by a bus 1010.

The camera device 1011 includes an image sensor and generates captured images (moving images and still images). The generated captured image is stored in the storage device 1007. The captured image stored in the storage device 1007 may be transmitted to an external server or the like via the transmitting/receiving device 1009 and the communication network N.

Each functional block used in the description of the above embodiment is typically realized as an LSI, which is an integrated circuit. These may be integrated into one chip individually, or may be integrated into one chip including some or all of them. Although it is referred to as an LSI here, it may also be called an IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Furthermore, if an integrated circuit technology that replaces LSI emerges due to advancements in semiconductor technology or other derived technology, then of course the functional blocks may be integrated using that technology.

(Summary of this disclosure)
A ball game video analysis device (100) according to the present disclosure receives a first video (201A) and a second video (201B) that are taken from different positions of a person's action on a moving object used in a ball game. (101), from each of the first moving image and the second moving image, specifying a reference frame image that is a frame image in which a specific sub-action constituting the action is photographed; synchronization shift information (203) that is information indicating a time difference between a first reference frame image that is a reference frame image identified from the second moving image and a second reference frame image that is a reference frame image identified from the second video; a reference frame identification unit (103) that generates a frame image, and an analysis unit that corrects synchronization deviation of frame images between the first moving image and the second moving image based on the synchronization deviation information and identifies the movement trajectory of the moving object. (105).

According to this configuration, synchronization shift information indicating the time shift between the first reference frame image and the second reference frame image in which a specific sub-action is photographed is generated. Therefore, even if the camera that shot the first video and the second video is not a dedicated camera specialized for ball game video analysis, the difference between the first video and the second video can be determined based on the synchronization information. Since the synchronization deviation between frame images can be corrected, the trajectory of the moving body can be specified with sufficient accuracy.

Further, the reference frame identification unit (103) may generate and output a reference frame confirmation UI (310) that presents the first reference frame image and the second reference frame image to the user.

According to this configuration, since the user can check the first reference frame image and the second reference frame image through the reference frame confirmation UI, it is possible to prevent out-of-synchronization information from being generated due to an incorrect reference frame image. .

Further, when an instruction to change the reference frame is received from the user through the reference frame confirmation UI (310), the reference frame change instruction for the user to change the first reference frame image and/or the second reference frame image is provided. It may further include a reference frame changing unit (104) that generates and outputs the UI (320).

According to this configuration, the user can change the first reference frame image and/or the second reference frame image through the reference frame change UI (320), so that the synchronization information can be generated using the correct reference frame image.

In addition, the reference frame identification unit (103) generates and outputs a reference frame confirmation UI (310) that receives an instruction as to whether to reshoot the first video (201A) and the second video (201B). You may do so.

According to this configuration, the user can reshoot the first video and second video for identifying the reference frame image through the reference frame confirmation UI, so that synchronization deviation information is not generated due to an incorrect reference frame image. This can be prevented.

Further, the first video and the second video may be taken by a first mobile terminal (20A) and a second mobile terminal (20B) each equipped with a camera and a wireless communication device. .

According to this configuration, the movement trajectory of the moving body can be analyzed with sufficient accuracy using the first mobile terminal and the second mobile terminal, rather than a dedicated camera specialized for ball game video analysis.

The ball game video analysis device (100) also sends the first reference frame image, the second reference frame image, and information indicating the results of the analysis by the analysis unit (105) to the third mobile terminal (30). It may further include a result display section (106) for output.

According to this configuration, the information indicating the analysis result is displayed on the third mobile terminal (30), so the user can move the first mobile terminal (20A) and the second mobile terminal (20B) to information indicating the analysis results can be confirmed from the third mobile terminal (30) without touching the first mobile terminal and the second mobile terminal.

Further, the ball game may be volleyball, the moving object may be the volleyball ball, the action may be a volleyball serve, and the subaction may be the moment when the hand of the person who served hits the ball.

According to this configuration, a frame image captured at the moment when the hand of the person who served the ball hits the ball can be specified as the reference frame image. Therefore, even if the camera that captured the first video and the second video is not a dedicated camera specialized for ball game video analysis, the movement trajectory of the volleyball ball can be specified with sufficient accuracy.

Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can come up with various changes, modifications, substitutions, additions, deletions, and equivalent examples within the scope of the claims, and It is understood that it falls within the technical scope of the present disclosure. Further, each component in the embodiments described above may be arbitrarily combined without departing from the spirit of the invention.

The technology of the present disclosure can be used to analyze sports using moving objects.

10 Ball game video analysis system 20A First mobile terminal 20B Second mobile terminal 30 Display terminal 100 Ball game video analysis device 101 Video reception section 102 Image conversion section 103 Reference frame identification section 104 Reference frame change section 105 Analysis section 106 Result display section 201A First video 201B Second video 202A First frame image group 202B Second frame image group 203 Synchronization shift information 204 Analysis result information 210A First image display area 210B Second image display area 211A, 211B Ball Detection area 310 Reference frame confirmation UI
312 OK button 313 Change button 314 Redo button 320 UI for changing reference frame
322 OK button 323 Back button 324A, 324B 1 frame return button 325A, 325B N frame return button 326A, 326B 1 frame advance button 327A, 327B N frame advance button 330 Analysis result screen 331 Ball movement trajectory 1000A, 1000B Computer 1001 Input device 1002 Output device 1003 CPU
1004 GPU
1005 ROM
1006 RAM
1007 Storage device 1008 Reading device 1009 Transmitting/receiving device 1010 Bus 1011 Camera device

Claims (7)

a receiving unit that receives a first video and a second video captured from different positions of a person's action on a moving object used in a ball game;
From each of the first moving image and the second moving image, a reference frame image that is a frame image in which a specific sub-action constituting the action is photographed is specified, and the reference frame image specified from the first moving image is Generate synchronization shift information that is information indicating a time difference between a first reference frame image that is a frame image and a second reference frame image that is the reference frame image specified from the second moving image. a reference frame identification section;
an analysis unit that corrects a synchronization difference between frame images between a first video and a second video based on the synchronization information and identifies a movement trajectory of the moving body ;
The reference frame specifying unit includes:
generating and outputting a reference frame confirmation UI that presents the first reference frame image and the second reference frame image to the user;
The analysis section includes:
When receiving a user instruction not to change the reference frame image from the reference frame confirmation UI, confirming the synchronization deviation information;
When a user's instruction to change the reference frame image is received from the reference frame confirmation UI, candidate images of the first reference frame image and the second reference frame image are extracted, and candidate images are extracted from the candidate images. accepting a user operation to select the reference frame image;
Ball game video analysis device. The reference frame identification unit generates and outputs the reference frame confirmation UI that receives an instruction as to whether to reshoot the first video and the second video.
The ball game video analysis device according to claim 1. The first moving image and the second moving image are respectively shot by a first mobile terminal and a second mobile terminal equipped with a camera and a wireless communication device.
The ball game video analysis device according to claim 1 or 2 . Further comprising a result display unit that outputs the first reference frame image, the second reference frame image, and information indicating the results of analysis by the analysis unit to a third mobile terminal.
The ball game video analysis device according to claim 3 . the ball game is volleyball;
The moving object is the volleyball ball,
The action is the serve of the volleyball,
The sub-action is the moment when the hand of the person who served the serve hits the ball;
The ball game video analysis device according to any one of claims 1 to 4 . The device is
Receive a first video and a second video that are taken from different positions of a person's action on a moving object used in a ball game,
From each of the first moving image and the second moving image, a reference frame image that is a frame image in which a specific sub-action constituting the action is photographed is specified, and the reference frame image specified from the first moving image is Generate synchronization shift information that is information indicating a time difference between a first reference frame image that is a frame image and a second reference frame image that is the reference frame image specified from the second moving image. ,
Correcting a synchronization difference between frame images between a first video and a second video based on the synchronization information, and identifying a movement trajectory of the moving object;
When identifying the reference frame image,
generating and outputting a reference frame confirmation UI that presents the first reference frame image and the second reference frame image to the user;
When receiving a user instruction not to change the reference frame image from the reference frame confirmation UI, confirming the synchronization deviation information;
When a user's instruction to change the reference frame image is received from the reference frame confirmation UI, candidate images of the first reference frame image and the second reference frame image are extracted, and candidate images are extracted from the candidate images. accepting a user operation to select the reference frame image;
Ball game video analysis method. Receive a first video and a second video that are taken from different positions of a person's action on a moving object used in a ball game,
From each of the first moving image and the second moving image, a reference frame image that is a frame image in which a specific sub-action constituting the action is photographed is specified, and the reference frame image specified from the first moving image is Generate synchronization shift information that is information indicating a time difference between a first reference frame image that is a frame image and a second reference frame image that is the reference frame image specified from the second moving image. ,
Correcting the synchronization deviation of frame images between the first video and the second video based on the synchronization information, and analyzing the movement trajectory of the moving object;
When identifying the reference frame image,
generating and outputting a reference frame confirmation UI that presents the first reference frame image and the second reference frame image to the user;
When receiving a user instruction not to change the reference frame image from the reference frame confirmation UI, confirming the synchronization deviation information;
When a user's instruction to change the reference frame image is received from the reference frame confirmation UI, candidate images of the first reference frame image and the second reference frame image are extracted, and candidate images are extracted from the candidate images. accepting a user operation to select the reference frame image;
A computer program that causes a computer to perform a process.

Images