WO2024135232A1

WO2024135232A1 - Impact detection device, impact detection method, and program

Info

Publication number: WO2024135232A1
Application number: PCT/JP2023/042297
Authority: WO
Inventors: 良平田嶋; 拓也中島; 和彦山本; 栄美伊藤
Original assignee: ヤマハ株式会社
Priority date: 2022-12-22
Filing date: 2023-11-27
Publication date: 2024-06-27
Also published as: JP2024089828A

Abstract

The present invention comprises: a first acquisition unit 101 which acquires moving image data that includes sound recorded at the time of a swing of a user and is obtained by capturing the swing at a plurality of frames; a second acquisition unit 102 which acquires reference motion data that represents a time transition of the swing and has an identifier attached thereto, the identifier indicating the time of impact in the swing; an identification unit 104 which identifies, from the moving image data, a section including a frame corresponding to the timing of attaching the identifier in the reference motion data; and a determination unit 105 which determines whether a frame including a hitting ball sound is present in the section and, when it is determined that the frame including the hitting ball sound is present in the section, determines that the swing with an impact is included in the moving image data.

Description

Impact detection device, impact detection method and program

This disclosure relates to, for example, an impact detection device, an impact detection method, and a program.

　Traditionally, in sports and the like, a swing with a certain pattern is filmed and the filmed video data is analyzed. This makes it possible to evaluate and diagnose the swing. An example of such technology is a technology that reduces the processing cost required for image recognition of video data related to a swing, and improves the accuracy of analysis of movements using the results of image recognition (see, for example, Patent Document 1).

JP 2021-125075 A

However, the technology described in Patent Document 1 has the problem that it is difficult to distinguish between a swing that does not involve impact (hitting), specifically a simple practice swing or a missed swing, and a swing that actually involves impact.
In consideration of such circumstances, one aspect of the present disclosure has an object to provide a technique that makes it easy to distinguish between a practice swing or a whiff swing and a swing that results in impact.

An impact detection device according to one aspect of the present disclosure includes a first acquisition unit that acquires video data including sounds recorded when a user swings, the video data being a plurality of frames of the swing; a second acquisition unit that acquires reference motion data showing the time progression of a swing by a model, the reference motion data being given an identifier indicating the timing of impact in the swing; an identification unit that identifies a section of the video data that includes a frame corresponding to the timing of the identifier in the reference motion data; and a determination unit that determines whether or not a frame including a hitting sound is present in the section, and determines that the video data includes a swing with impact if it is determined that a frame including a hitting sound is present in the section.

1 is a diagram illustrating an example of a usage state of an impact detection device according to an embodiment. FIG. 2 is a diagram illustrating a hardware configuration of the impact detection device. FIG. 2 is a functional block diagram of the impact detection device. FIG. 2 is a diagram illustrating an example of video data. FIG. 11 is a diagram illustrating an example of reference motion data. 4 is a flowchart showing an operation of the impact detection device. FIG. 13 is a diagram illustrating impact detection in the relationship between video data and audio data. FIG. 13 is a diagram showing an example of video data showing a series of swings. FIG. 13 is a diagram showing the extraction of video data. FIG. 13 is a diagram showing an example of playback of video data.

Hereinafter, an impact detection device according to an embodiment of the present disclosure will be described with reference to the drawings.
In each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, since the embodiments described below are preferred specific examples, various technically preferable limitations are added, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

FIG. 1 is a diagram showing an example of a usage state of the impact detection device 1 according to the embodiment. The impact detection device 1 is, for example, an information processing device with a shooting function, specifically, a smartphone. The impact detection device 1 shoots a swing accompanied by an impact by a user U, for example, a golf swing, and analyzes the shot video data. Note that the impact detection device 1 may be a device other than a smartphone, such as a portable terminal device or a personal computer, that has a function of analyzing video data.

FIG. 2 is a diagram showing the hardware configuration of the impact detection device 1. Impact detection by the impact detection device 1 is achieved through the cooperation of hardware and software. The impact detection device 1 includes a processing device 10, a photographing device 11, an operation input device 12, a display device 13, a storage device 14, and a communication device 15.

The processing device 10 is composed of one or more arithmetic processing circuits, such as a CPU (Central Processing Unit), and controls all elements of the impact detection device 1, which is an information processing device. The processing device 10 may also be composed of circuits such as a DSP (Digital Signal Processor) and an ASIC (Application Specific Integrated Circuit) in addition to a CPU.

The photographing device 11 photographs the swing of the user U and outputs video data including the photographed audio and video.
The operation input device 12 accepts operations by a user. The operation input device 12 is, for example, a plurality of operators pressed by the user or a touch panel that detects contact by the user, and is overlaid on the image display screen of the display device 13. Note that the user who operates the operation input device 12 may be the user who made the swing, or may be another user.

The display device 13 is composed of, for example, a liquid crystal panel or an organic EL panel, and displays various images under the control of the processing device 10. For example, the photographed swing and an editing screen for that swing are displayed on the display device 13.

The storage device 14 is a single or multiple memories configured with known recording media such as magnetic recording media or semiconductor recording media, and stores the programs executed by the processing device 10, various data used by the processing device 10, and video data of the swing. The storage device 14 may be configured with a combination of multiple types of recording media. Also, a portable recording medium that can be attached to and detached from the impact detection device 1, or an external recording medium (e.g., online storage) with which the impact detection device 1 can communicate via a network may be used as part of the storage device 14.

The communication device 15 communicates with a server etc. via a network under the control of the processing device 10.

3 is a block diagram illustrating a functional configuration of the processing device 10. The processing device 10 executes a program stored in the storage device 14 to function as a plurality of elements (a control unit 100, a first acquisition unit 101, a second acquisition unit 102, an analysis unit 103, an identification unit 104, and a determination unit 105) for detecting an impact frame in video data where an impact occurs. After detecting an impact frame in the video data, the processing device 10 also functions as an editing unit 106 for editing the video data and a playback unit 107 for playing the video data.
It should be noted that some of the functions of the processing device 10 may be implemented by another device, for example, a server connected via a network.

Fig. 4 is a diagram showing an example of video data. As shown in the figure, the video data is an object of impact detection, and is a golf swing by a user U, captured by, for example, the impact detection device 1. The video data includes a video portion made up of a plurality of frames Fr, and an audio portion synchronized with the video portion, although this is omitted in Fig. 4. In the video portion, a swing image is recorded for each frame Fr, and in the audio portion, sounds picked up during the swing are recorded.
Each frame Fr of the video data is associated with a frame number as a serial number from the start frame.

In the figure, the address, top, impact and finish of typical swing movements are shown as (A), (C), (E) and (G), respectively.
Furthermore, the moving image data is, for example, in the MPEG format, but any format is acceptable as long as it includes a video portion consisting of a series of multiple frames and an audio portion synchronized with the video portion.

The impact detection device 1 refers to the reference motion data to detect impact frames in the video data. Here, we will explain the reference motion data.

FIG. 5 is a diagram for explaining the reference action data.
The reference motion data is data showing a model/example swing. In detail, the reference motion data is data showing a time transition of three-dimensional coordinates of each joint J of skeletal information when the model M swings. The reference of the three-dimensional coordinates is, for example, the center of gravity coordinates of both feet of the model M, and the three-dimensional coordinates of each joint J are defined for each frame Fr by relative coordinates from the center of gravity coordinates.
Each frame Fr of the reference action data is associated with a frame number as a serial number from the start frame, similar to the video data.

Examples of the joint J include a left knee, a right knee, a left hip joint, a right hip joint, a left shoulder joint, a right shoulder joint, a left elbow, and a right elbow.
In the figure, the joints J of a right-handed model M when swinging are shown as black circles when viewed from the front (east, if the ball is hit in the direction of north). Representative parts of the swing, including the address, take-back, top, downswing, impact, follow-through, and finish, are illustrated, in that order, as (1) to (7).
Among the frames of the swing indicated by the reference motion data, the frame indicating the impact is tagged in advance with an identifier to distinguish it from the other frames.

Since the reference motion data indicates the three-dimensional coordinates of each joint J, it is possible to view the swing of the model M from any point by specifying a viewpoint.
Generally speaking, swings differ depending on the attributes of the user U, such as the gender, dominant hand, generation (age), and golf skill (professional, semi-professional, amateur). For this reason, multiple types of reference motion data are prepared according to these attributes and organized into a database. The database of reference motion data may be configured to be stored in the storage device 14 or may be configured to be stored in a server on the cloud.
In order to improve the ease of searching the database of reference motion data, a tree diagram showing the similarity of swings (time transition of joint coordinates) indicated by the reference motion data may be created in advance.

Next, we will explain the operation of the impact detection device 1 to detect impact frames from video data.

FIG. 6 is a flowchart showing the operation of the impact detection device 1. This operation is executed when, for example, a user U operates an icon corresponding to an application program that detects impact frames on the impact detection device 1.

When the application program is executed, the control unit 100 instructs the first acquisition unit 101 to acquire video data for which impact frames are to be detected (step Sa11).

The instructed first acquisition unit 101 displays, for example, a list of one or more video data on the display device 13, and prompts the user U to select a video file that is to be detected for an impact frame. When the user U selects a video file, the first acquisition unit 101 acquires the selected video file as a video file that is to be detected for an impact frame.
The video file may be acquired from the storage device 14 or an external device via a network. The video files stored in the storage device 14 include the most recent video file captured by the imaging device 11 and video files captured in the past.

The control unit 100 transfers the video data acquired by the first acquisition unit 101 to the analysis unit 103. The analysis unit 103 analyzes the time transition of the joint coordinates of the user swinging in the video data by using a known method.
The analysis unit 103 transfers information indicating the analyzed changes in the joint coordinates over time to the second acquisition unit 102 .

The second acquisition unit 102 searches a database to identify and acquire the reference motion data that is most similar to the time progression of the joint coordinates obtained by analyzing the video data (step Sa12). Note that the reference motion data that is most similar to the time progression of the joint coordinates obtained by analysis may be identified using, for example, Subsequence Dynamic Time Warping.

The control unit 100 transfers the video data acquired by the first acquisition unit 101 and the reference action data acquired by the second acquisition unit 102 to the identification unit 104. The identification unit 104 determines the correlation between each frame of the video data and each frame of the reference action data, specifically, which frame of the reference action data each frame of the video data corresponds to.

The upper column of Figure 7 is a diagram showing the correlation between each frame of the video data and each frame of the reference action data; specifically, black dots indicate which frame of the reference action data each frame of the video data corresponds to.
For example, frame (A) in the video data corresponds to the frame of address (1) in the reference motion data. Similarly, frame (E) in the video data corresponds to the frame of impact (5) in the reference motion data. Also, frame (G) in the video data corresponds to the frame of finish (7) in the reference motion data.

After determining the correlation, the identification unit 104 identifies a section S from, for example, 0.3 seconds earlier in time to, for example, 0.3 seconds later in time, based on frame (E) of the video data corresponding to impact (5) in the reference action data, as a section in the video data that is likely to include an impact frame (step Sa13).
The identification unit 104 transfers information indicating the identified section S to the determination unit 105 .

Next, the determination unit 105 determines whether or not there is one impact sound in the audio portion of the section S identified by the identification unit 104 in the video data (step Sa14). An impact sound is a sound whose volume exceeds a threshold within an extremely short period of time (e.g., 0.01 seconds). The lower section of Figure 7 shows an example waveform showing the audio portion of the section S identified by the identification unit 104, which is an example of a case in which there are two impact sounds in the section S.

If the determination unit 105 determines that only one impact sound is present in the audio portion of section S (if the determination result in step Sa14 is "Yes"), it detects the frame in the video portion of section S in which the impact sound reaches its peak value as an impact frame (step Sa15).

If it is determined that there is more than one impact sound in the audio portion of section S (if the determination result in step Sa14 is "No"), the determination unit 105 then determines whether there are two or more impact sounds in the audio portion of section S (step Sa16).

If it is determined that there are two or more impact sounds present in the audio portion of the section S (if the determination result of step Sa16 is "Yes"), the determination unit 105 determines that, among the two or more impact sounds present in the section S, the impact sound having the timing closest to frame (E) of the video data is the impact that occurred during the swing by the user U. Then, the determination unit 105 identifies the frame in which the impact sound reaches its peak value as the impact frame (step Sa17).

At a golf driving range, many users practice golf simultaneously. For this reason, many impact sounds can occur in a short period of time. The lower section of Figure 7 shows an example of a case where impact sounds occur at timings a and b in the audio portion of section S. In this case, if the volume of the impact sound generated at timing b is greater than the volume of the impact sound generated at timing a, there is a possibility that the impact in user U's swing will be erroneously detected as occurring at timing b.

In contrast, in this embodiment, it is not the volume of the impact sound that is used to determine that the impact sound occurred in the swing by user U, but rather the timing a closest to frame (E). Therefore, even when multiple users are practicing at the same time, it is possible to accurately identify the impact frame in user U's swing.

Note that if there is not one impact sound in the audio portion of section S (the determination result in step Sa14 is "No") but two or more impact sounds (the determination result in step Sa16 is "No"), this means that no impact sound is present in the audio portion of section S. Therefore, if the determination result in step Sa16 is "No", the determination unit 105 notifies the control unit 100 that there is no impact frame in the video data, and the control unit 100 executes error processing (step Sa18).
In this embodiment, when the swing by the user U is merely a practice swing, no impact sound is generated in the section S, and therefore error processing is executed in step Sa18. For this reason, in a swing that does not include an impact sound, frame (E) is not identified as an impact frame.
Examples of error processing include displaying a message on the display device 13 indicating that there is no impact frame in the video data, or that the swing in the video data is a practice swing, etc.

This embodiment not only makes it possible to distinguish between practice swings and misses and swings that result in actual impact, but also makes it possible to accurately detect impact frames captured from users' swings, even when multiple people are swinging simultaneously.

In the embodiment, the explanation is based on the assumption that the user U takes one swing, but in reality, the swing is repeated multiple times from the start of filming to the end of filming. If the video data includes multiple swings, the impact frame can be identified for each swing.

8 is a diagram showing the correlation between each frame of the video data and each frame of the reference motion data when the video data includes multiple swings. The correlation shown in the figure is an example in which one video data includes three swings.
In this example, the audio portion is not shown, but if an impact sound occurs near frame (E) of each swing, the frame in which the impact sound occurs is detected as the impact frame.

In the case where the video data includes a plurality of swings, after the impact frame is identified for each swing, the editing unit 106 may edit the video data to extract each swing.
Specifically, when frame (E) is identified as an impact frame among the three swings in the video data, the editing unit 106 extracts the period from 2.0 seconds before frame (E) to 2.0 seconds after frame (E) as one swing, as shown in FIG. 9 .

In the illustrated example, three pieces of video data showing the swing are extracted. The extracted video data are associated with information such as the shooting date and time.
In addition, 2.0 seconds before frame (E) is an example of the first time, and 2.0 seconds after frame (E) is an example of the second time. The first time and the second time may be set to any time by the user U.

When video data is extracted for each swing, the playback unit 107 may be configured to play two videos side by side for comparison, as shown in FIG. 10, for example. In a configuration in which two videos are played side by side, if the frames (E) identified as the impact frame are aligned as shown in the figure, the user U can easily grasp the difference in the swing before and after impact in the two videos.

The number of pieces of video data to be played back is not limited to two, but may be three or more. Furthermore, the video data to be played back is not limited to that stored in the storage device 14, but may be video data stored in a server on the cloud.
For example, by comparing one's swing with the swing of a more advanced player, it becomes easier to grasp the inferior points of one's swing. Also, by comparing a past swing with a current swing, it is possible to grasp the degree of improvement. Note that the playback unit 107 is not limited to playback at a constant speed, and may play back in slow motion, fast forward, or frame by frame.

In the embodiment, the application program for identifying the impact frame is executed in response to an operation by the user U, but the application program may be executed in response to, for example, completion of filming of the swing. In this configuration, the first acquisition unit 101 acquires video data of the filmed swing as a detection target for impact.
In addition, in the embodiment, the analysis of the video data is configured to be performed by the analysis unit 103 of the impact detection device 1, but the analysis may also be performed by a device other than the impact detection device 1, for example an external device, and the second acquisition unit 102 may acquire information indicating the time progression of the joint coordinates, which is the analysis result.

In the embodiment, a golf swing is given as an example of a swing, but the invention can also be applied to swings that involve impact sounds, such as tennis, baseball, and table tennis.

From the above description, preferred aspects of the present disclosure can be understood as follows, for example:

An impact detection device according to one aspect (aspect 1) of the present disclosure includes a first acquisition unit that acquires video data including sounds recorded when a user swings, the video data being a plurality of frames of the swing; a second acquisition unit that acquires reference motion data showing the time progression of a swing by a model, the reference motion data being given an identifier indicating the timing of impact in the swing; an identification unit that identifies a section of the video data that includes a frame corresponding to the timing of the identifier in the reference motion data; and a determination unit that determines whether or not a frame including a hitting sound is present in the section, and determines that the video data includes a swing with impact if it is determined that a frame including a hitting sound is present in the section.

According to aspect 1, first, a section is identified in the video data that includes a frame that corresponds to a timing in the reference motion data that is assigned an identifier indicating an impact, and second, if a frame including a hitting sound is present in that section, it is determined that the video data includes a swing with impact. In other words, if a frame including a hitting sound is not present in that section, it is determined that the video data does not include a swing with impact. Therefore, according to aspect 1, it is possible to distinguish between a swing that does not include impact, such as a practice swing or a whiff, and a swing that includes impact.

In the impact detection device according to a specific aspect 2 of aspect 1, when there are two or more frames including a hitting sound in the section, the determination unit determines the frame closest to the timing at which the identifier is assigned to be the impact frame.
According to the second aspect, when there are two or more frames including a hitting sound in a section, the frame closest to the timing when the identifier was added is determined to be the impact frame. Therefore, the frame including the hitting sound by the target user can be identified with high accuracy without being affected by the hitting sounds by other users.

In the impact detection device relating to a specific aspect 3 of aspect 1, a plurality of pieces of reference motion data are prepared for each model, and the second acquisition unit acquires one piece of reference motion data from the plurality of pieces of reference motion data that is similar to the swing shown in the video data.
According to the third aspect, the section is identified by using, from among the plurality of reference motion data, reference motion data similar to the swing shown in the video data.

In the impact detection device according to specific aspect 4 of aspect 3, the reference motion data is data indicating the time progression of the joint coordinates in the model, and an analysis unit is provided that analyzes the time progression of the joint coordinates of the user in the video data, and the second acquisition unit acquires, from the plurality of reference motion data, the reference motion data having the time progression that is most similar to the time progression of the joint coordinates of the user in the video data.

An impact detection device relating to a specific aspect 5 of aspect 2 has an editing unit that edits the video data, and the editing unit cuts out frames from the video data that are a first time ahead to a second time behind the impact frame identified by the identification unit.
According to the fifth aspect, the editing unit cuts out from the video data a frame that is a first time before the impact frame and a frame that is a second time after the impact frame.

In the impact detection device according to a specific aspect 6 of aspect 5, when the video data includes two or more swings, the editing unit cuts out the video data for each of the swings. According to aspect 6, a series of video data including multiple swings is cut out for each swing.

The impact detection device according to aspect 7, which is a specific example of aspect 6, includes a playback unit that aligns and plays two or more pieces of video data extracted by the editing unit. According to aspect 7, swings can be compared by playing two or more videos.

The impact detection method according to the eighth preferred aspect of the present disclosure causes a computer to execute the steps of: acquiring video data including sounds recorded when a user swings, the video data being a multiple-frame shot of the swing; acquiring reference motion data showing the time progression of a swing by a model, the reference motion data being given an identifier indicating the timing of impact in the swing; identifying a section of the video data that includes a frame corresponding to the timing of the identifier in the reference motion data; and determining whether or not a frame including a hitting sound is present in the section, and determining that the video data includes a swing with impact if it is determined that a frame including a hitting sound is present in the section.

The program according to a preferred aspect 9 of the present disclosure causes a computer to function as a first acquisition unit that acquires video data including sounds recorded when a user swings, the video data being a plurality of frames of the swing; a second acquisition unit that acquires reference motion data showing the time progression of a swing by a model, the reference motion data being given an identifier indicating the timing of impact in the swing; an identification unit that identifies a section of the video data that includes a frame corresponding to the timing of impact in the reference motion data that is given the identifier; and a determination unit that determines whether or not a frame including a hitting sound is present in the section, and determines that the video data includes a swing with impact if it is determined that a frame including a hitting sound is present in the section.

1...impact detection device, 10...processing device, 11...imaging device, 12...operation input device, 13...display device, 14...storage device, 15...communication device, 100...control unit, 101...first acquisition unit, 102...second acquisition unit, 103...analysis unit, 104...identification unit, 105...determination unit, 106...editing unit, 107...playback unit.

Claims

A first acquisition unit that acquires video data including a sound recorded when a user swings, the video data being a plurality of frames of the swing;
A second acquisition unit that acquires reference motion data indicating a time transition of a swing by a model, the reference motion data being assigned an identifier indicating a timing of an impact in the swing;
an identification unit that identifies a section including a frame corresponding to a timing at which the identifier is assigned in the reference action data, from the video data;
determining whether or not a frame including a hitting sound is present in the section;
a determination unit that, when it is determined that a frame including a hitting sound exists in the section, determines that the video data includes a swing including an impact;
An impact detection device having an impact detection means.
When there are two or more frames including a hitting sound in the section,
The determination unit is
The impact detection device according to claim 1 , wherein a frame closest to the timing at which the identifier is attached is determined to be an impact frame.
A plurality of sets of the reference motion data are prepared for each model,
The second acquisition unit is
The impact detection device according to claim 1 , further comprising: acquiring one piece of reference motion data similar to the swing shown in the video data from among the plurality of pieces of reference motion data.
the reference motion data is data indicating a time transition of joint coordinates in the model,
an analysis unit that analyzes a time transition of a joint coordinate of a user in the video data;
The second acquisition unit is
Among the plurality of reference motion data,
The impact detection device according to claim 3 , further comprising: acquiring reference motion data having a time transition that is most similar to a time transition of the joint coordinates of the user in the video data.
An editing unit that edits the video data,
The editorial department:
The impact detection device according to claim 2 , wherein frames from a first time ahead to a second time behind the impact frame specified by the specification unit are extracted from the video data.
If the video data includes two or more swings,
The editorial department:
The impact detection device according to claim 5 , wherein the video data is extracted for each swing.
The impact detection device according to claim 6 , further comprising a playback unit that arranges and plays back two or more pieces of video data cut out by the editing unit.
On the computer,
acquiring video data including sounds recorded during a user's swing, the video data being a plurality of frames of the swing;
A process unit for acquiring reference motion data showing a time progression of a swing by a model, the reference motion data being assigned an identifier showing a timing of an impact in the swing;
identifying a section of the video data that includes a frame corresponding to a timing at which the identifier is assigned in the reference action data;
determining whether or not a frame including a hitting sound is present in the section;
determining that the video data includes a swing involving an impact when it is determined that a frame including a ball-hitting sound exists in the section;
An impact detection method that causes the
Computer,
A first acquisition unit that acquires video data including a sound recorded when a user swings, the video data being a plurality of frames of the swing;
a second acquisition unit that acquires reference motion data indicating a time progression of a swing by a model, the reference motion data being assigned an identifier indicating a timing of impact in the swing;
an identification unit that identifies a section including a frame corresponding to a timing at which the identifier is assigned in the reference action data, from the video data; and
a determination unit that determines whether or not a frame including a hitting sound is present in the section, and when it is determined that a frame including a hitting sound is present in the section, determines that the video data includes a swing involving an impact;
A program that functions as a