JP6039500B2 - Video playback device and video playback program - Google Patents

Description

  The present invention relates to a video playback apparatus and a video playback program that play back two systems of video in synchronization.

  It is extremely important to check your own movements taken during exercises in sports. It is also anticipated that the learning effect will be enhanced by comparing the behavior that serves as a model and one's own motion, or by comparing the current motion and the motion that is being performed well (see, for example, Non-Patent Document 1).

  Note that checking the motion of a moving object from video images is referred to as video feedback. Also, a model operation or a favorable operation is called a reference operation, and a video of the reference operation is called a reference operation video. In addition, an action of a human or the like that is a comparison target with the reference action video is called a current action, and a picture of the current action is called a current action picture.

Hirofumi Terai, Masanobu Tate, Influence of practice using video feedback on batting technology, Sports Performance Research, 3, 138-152, 2011

  As described above, it is possible to check the action by shooting the action with video, but generally, it takes time until the visual confirmation of the video after the action, so the memory of the action decreases during that time. End up. As a result, there is a problem that the effect of video feedback is reduced. In addition, it is possible to use a mirror, but there are cases where the operation itself is changed by looking at the mirror.

  In addition, video feedback by comparing the reference operation with the current operation has a problem that it is difficult to prepare a state in which the reference operation and the current operation can be easily compared. In particular, since the comparison process becomes complicated due to a change in the background, a change in the lighting environment, and a change in clothes, there is also a problem that it is difficult to compare high-speed operations.

  Further, when two images are compared, since it is necessary to finish the synchronization process within a short delay, a high-speed calculation process must be performed. Further, when the speed of the reference operation and the current operation are different, there is a problem that it is difficult to compare even if the reproduction is simply synchronized.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a video playback apparatus and a video playback program that can easily compare a reference operation with a current operation.

  The present invention provides a reference motion video reading means for reading a reference motion video, a synchronization designation information reading means for reading synchronization designation information indicating a timing to be synchronized in the reference motion video, and the reference specified by the synchronization designation information. Reference motion video feature acquisition means for acquiring image features in a motion video, current motion video feature acquisition means for acquiring features of an image in a current motion video to be compared with the reference motion video, and the reference motion According to the similarity obtained as a result of comparing the feature of the image in the reference motion video acquired by the video feature acquisition unit and the feature of the image in the current motion video acquired by the current motion video feature acquisition unit, Synchronization timing specifying means for specifying a synchronization timing in the current operation video; and a delay width acquisition for acquiring a delay width to be delayed of the current operation video. It means, wherein the current operation image which is delayed in accordance with the delay spread, and the reference operation image, characterized in that it comprises a synchronization reproducing means for reproducing in synchronization at the synchronization timing.

  According to the present invention, the synchronization timing specifying unit thins out the processing for obtaining the similarity by comparing the feature of the image in the reference motion video with the feature of the image in the current motion video according to the similarity. It is characterized by.

  According to the present invention, the synchronization timing specifying unit compares the feature of the image in the reference motion video with the feature of the image in the current motion video according to a detection result of a state occurring prior to the synchronization timing. Thus, the process for obtaining the similarity is thinned out.

  In the present invention, the synchronization designation information reading means reads synchronization designation information indicating two different timings to be synchronized in the reference motion video, and the synchronization playback means is configured to read the reference motion video and the reference motion video at the two timings. The reproduction time is expanded and contracted so that the current operation video is synchronized.

  The present invention is characterized in that the synchronization timing specifying means uses an image feature based on Motion History Image.

  The present invention is a video playback program for causing a computer to function as the video playback device.

  According to the present invention, it is possible to easily compare the reference operation with the current operation, so that the effect of maximizing the effect of video feedback can be obtained.

It is a block diagram which shows the structure of 1st Embodiment of this invention. It is explanatory drawing which shows operation | movement of the video delay part 3 shown in FIG. It is a block diagram which shows the structure of 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the synchronizer 9 shown in FIG. It is a flowchart which shows the processing operation of the synchronizer 9 in 4th Embodiment of this invention. FIG. 6 is a diagram illustrating a relationship between similarity and similarity calculation timing when processing is thinned by the processing operation illustrated in FIG. 5. It is explanatory drawing which shows the processing operation of the video reproduction apparatus by 5th Embodiment. It is a flowchart which shows operation | movement of the synchronizer 9 in the video reproduction apparatus by 5th Embodiment. It is explanatory drawing which shows the operation | movement in 2nd Embodiment. It is explanatory drawing which shows the operation | movement in 6th Embodiment.

<First Embodiment>
A video playback apparatus according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the first embodiment. The video playback apparatus according to the first embodiment plays back with a delay of a predetermined time. In FIG. 1, reference numeral 1 denotes a video acquisition unit that inputs a video to be reproduced with a delay. Reference numeral 2 denotes a delay width acquisition unit that inputs information on a delay width to be delayed. Reference numeral 3 denotes a video delay unit that delays the video input in the video acquisition unit 1 by the delay width input in the delay width acquisition unit 2. Reference numeral 4 denotes a video display unit that displays the video delayed by the video delay unit 3 and is configured by a display device.

  Next, the operation of the video playback apparatus shown in FIG. 1 will be described with reference to FIG. FIG. 2 is an explanatory diagram showing the operation of the video delay unit 3 shown in FIG. First, the video acquisition unit 1 receives a video from a camera or the like for each frame and sends it to the video delay unit 3. On the other hand, the delay width acquisition unit 2 receives delay width information d specified by the user. The delay width information d is information indicating how many seconds the video is delayed.

  Next, the video delay unit 3 includes a memory array B (B [0] to B [NB-1] serving as a buffer therein and performs the following processing. Here, the NB delays in the video delay unit 3. The maximum delay width (corresponding to the number of arrays in the array B) The video delay unit 3 converts the frame images stored in the arrays B [0] to B [NB-2] into the array B, respectively. [1] to B [NB-1] are sequentially copied, and the video delay unit 3 stores the image newly acquired from the video acquisition unit 1 in the array B [0]. By outputting a frame image of width information d × frame rate of video] to the video display unit 4, a video delayed by d seconds is displayed on the video display unit 4. In the example illustrated in FIG. -1] (maximum delay width) to output to the video display unit 4. By performing the processes, the image display unit, so that the image of d seconds delay is displayed.

  As described above, the frame image acquired from the video acquisition unit 1 is temporarily stored in the buffer in the video delay unit 3, and the frame image delayed for a predetermined time is output to the video display unit 4, thereby arbitrarily delayed video. Can be played.

Second Embodiment
Next, a video playback apparatus according to the second embodiment of the present invention will be described. FIG. 3 is a block diagram showing the configuration of the second embodiment. The video reproduction apparatus according to the second embodiment uses the operation information at the synchronization timing designated during the reference operation and searches the video of the current operation to synchronize the reference operation and the current operation temporally. In FIG. 3, the same parts as those of the video reproduction apparatus shown in FIG. Reference numeral 5 denotes a reference operation reading unit that reads the manufacture of the reference operation and holds it inside.

  Reference numeral 6 denotes a synchronization timing information acquisition unit that receives information designating a timing at which the video of the current operation is to be synchronized with the video of the reference operation read by the reference operation reading unit 5. The synchronization timing information acquisition unit 6 may read information specifying the timing from a file or may input it from a keyboard.

  Reference numeral 7 denotes a reference motion video feature acquisition unit that acquires an image feature in an operation in a reference motion video at a specified synchronization timing. The image characteristics of the operation are not particularly limited because various methods such as an optical flow, an optical flow histogram, and a motion history image can be applied.

Reference numeral 8 denotes a current motion video feature acquisition unit that acquires a motion image feature using the same technique as the reference motion video feature acquisition unit 7. Reference numeral 9 denotes a comparison between the image feature of the motion in the current motion video acquired by the current motion video feature acquisition unit 8 and the motion image feature of the reference motion video acquired in the reference motion video feature acquisition unit 7. Thus, the synchronization unit extracts the timing when the same operation is performed in the current operation and the reference operation. This is represented as a synchronization timing, and is represented by a set of (t _c , _tr ).

  Next, the operation of the video playback apparatus shown in FIG. 3 will be described. First, the reference operation reading unit 5 reads a video of the reference operation from the outside and holds it inside. Next, the synchronization timing information acquisition unit 6 receives information that designates the timing at which the video of the current operation is to be synchronized with the video of the reference operation read by the reference operation reading unit 5 from the outside. In response to this, the reference motion video feature acquisition unit 7 acquires the image feature of the frame image in the reference motion video specified by the designated synchronization timing and outputs it to the synchronization unit 9.

  On the other hand, the video acquisition unit 1 acquires a video of the current operation from the outside for each frame, and outputs the video to the current motion video feature acquisition unit 8 and the video delay unit 3. The currently operating video feature acquisition unit 8 acquires an image feature from the frame output from the video acquisition unit 1 and outputs it to the synchronization unit 9.

  Next, the synchronization unit 9 compares the image feature output from the reference motion video feature acquisition unit 7 with the image feature output from the current motion video feature acquisition unit 8, and performs the same operation for the reference motion and the current motion. Extract the timing that was done. Detailed processing operations of the synchronization unit 9 will be described later.

  Next, the video display unit 4 reads the current operation video delayed by the video delay unit 3 and the reference operation reading unit 5 so that the synchronization timing designated by the synchronization unit 9 is displayed at the same timing. Display the video of the reference operation in synchronization.

Next, the detailed operation of the synchronization unit 9 shown in FIG. 3 will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the synchronization unit 9 shown in FIG. Here, the current time t, also the reference operation synchronization timing and t _r. First, the synchronization unit 9 compares the image feature of the reference operation image and the image feature of the current operation image, it obtains the similarity s _t (Step S1). Then, the synchronization unit 9, the similarity _{s t} is equal to or threshold _{th detec} greater than (Step S2). If the similarity s _t is greater than the threshold th _{detec as} a result of the determination, the synchronization unit 9 determines whether or not the similarity s _{t-1 at} the previous time (t−1) is greater than the current similarity s _t . Is determined (step S3). The result of this determination, a similarity _{s t-1} is greater than the similarity _{s t,} is set as the synchronization timing _{(t-1, t r)} ( step S4). By these two determination processes, the peak of the similarity of the current operation can be found. As a result of the determination in steps S2 and S3, if the condition is not satisfied, nothing is done and the process is terminated. By the processing operation shown in FIG. 4, the reference operation synchronization timing can find the most similar currently operating in t _r. At the synchronization timing set here, the reference operation and the current operation are synchronized and displayed on the video display unit 4.

  In this way, the video of the reference operation serving as a model is read, and the synchronization designation information during the reference operation is read, and the video of the current motion during the delayed playback and the video of the reference motion are synchronized at the designated synchronization point. It becomes possible to reproduce.

<Third Embodiment>
Next, a video playback apparatus according to a third embodiment of the present invention will be described. The video playback apparatus according to the third embodiment uses MHI (Motion History Image) as an image feature in the reference motion video feature acquisition unit 7 and the current motion video feature acquisition unit 8 shown in FIG. MHI represents the direction and history of motion, and the document “A. Bobick and J. Davis, The Representation and Recognition of Action Using Temporal Templates, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 3 , 2001, pp.257-267. Etc., and detailed description thereof is omitted here.

  MHI is obtained by the following process. First, a difference binarized image D (t) between the image I (t) at the current time t and the image I (t−1) one hour before is obtained. If the difference is greater than or equal to the threshold value, it takes a binary value of 255, otherwise it takes a binary value of 0. The pixel value is subtracted by a from each pixel of MHI M (t−1) one time before to obtain M ′ (t−1). Then, MHI of the current time is obtained by M (t) ← M ′ (t−1) + D (t).

<Fourth embodiment>
Next, a video playback apparatus according to a fourth embodiment of the present invention will be described. The video playback apparatus according to the fourth embodiment speeds up by reducing the calculation load by not performing the process of obtaining the similarity for a certain frame when the similarity is equal to or less than a predetermined threshold. It is.

In the video reproduction apparatus of the present embodiment, flag is introduced as a variable for adjusting the thinning width. The initial value of flag is 0. Then, in addition to the threshold value th _detec for image detection having a high degree of similarity, a threshold value th _skip for determining whether or not to thin out the processing is newly introduced. As a result, when the degree of similarity is smaller than th _skip, it is assumed that an image to be detected having a high degree of similarity will not appear for a while, and processing is skipped for a certain period of time. If the similarity is greater than the threshold th _skip, there is a high possibility that there will be an image to be detected in the near future, so the similarity is calculated in the next frame without skipping.

  FIG. 5 is a flowchart showing the processing operation of the synchronization unit 9 shown in FIG. First, the synchronization unit 9 determines whether or not the flag is 0 (step S11). If the flag is 0, the process proceeds to the next. Otherwise, the flag is decreased by 1 (step S18), and the synchronization process is terminated.

Then, the synchronization unit 9 compares the current operation image feature and the reference operation image feature of the current time, determine the similarity s _t (step S12). Then, the synchronization unit 9 determines whether the similarity s _t is smaller than a predetermined threshold th _skip (step S13). The result of this determination, the similarity _{s t} is when the threshold _{th skip} smaller, set N to flag, and ends the synchronization process (step S17). N is a value determined in advance by experiments or the like. If N is increased, the processing interval is widened, and if N is decreased, the processing interval is narrowed.

On the other hand, when the similarity s _t is not smaller than the threshold th _skip , the synchronization unit 9 determines whether or not the similarity s _t is larger than the threshold th _detec (step S14). If the similarity s _t is greater than the threshold th _{detec as} a result of the determination, the synchronization unit 9 determines whether or not the similarity s _{t-1 at} the previous time (t−1) is greater than the current similarity s _t . Is determined (step S15). The result of this determination, a similarity _{s t-1} is greater than the similarity _{s t,} updates the synchronization timing of the _{(t-1, t r)} ( step S16). As a result of the determination in steps S14 and S15, if the condition is not satisfied, nothing is done and the process is terminated.

  As described above, when detecting images having similar operations for synchronization, the processing can be speeded up by thinning out processing with an appropriate time width.

<Fifth Embodiment>
Next, a video playback apparatus according to a fifth embodiment of the present invention will be described. The video playback apparatus according to the fifth embodiment uses an event (characteristic state) that precedes the synchronization timing and improves the thinning efficiency of the processing, thereby increasing the speed. FIG. 6 is a diagram showing the relationship between the similarity and the similarity calculation timing when processing is thinned out by the processing operation shown in FIG. In FIG. 6, the horizontal axis represents time and the vertical axis represents similarity. The vertical line on the horizontal axis represents the time when the similarity calculation is performed. As shown in FIG. 6, when the similarity is equal to or less than th _skip , the processing is thinned out. When the similarity is equal to or greater than th _skip , the similarity is calculated for each frame.

  FIG. 7 is an explanatory view showing the processing operation of the video reproduction apparatus according to the fifth embodiment. First, the state (preceding state) that occurs prior to the synchronization timing is searched. Then, when the similarity of the preceding state search becomes a certain level or more, the similarity of the preceding state is examined at the next time (the portion surrounded by a circle of reference A shown in FIG. 7). The processing is skipped based on the change in the preceding state at two consecutive times. In addition, it is designated in advance how many hours ahead of the current change time at two consecutive times in the preceding state will appear.

FIG. 8 is a flowchart showing the operation of the synchronization unit 9 in the video playback apparatus according to the fifth embodiment. First, the synchronization unit 9 determines the value of FlagB (step S21). As a result of the determination, if the value is 0, the synchronization unit 9 compares the image feature of the reference operation image and the image feature of the current operation image, it obtains the similarity S _t (step S22).

Then, the synchronization unit 9, the similarity _{s t} is equal to or threshold _{th detec} greater than (step S23). Result of the judgment, if not greater than the threshold _{th detec} similarity _{s t,} is set to 0 FlagA, the process ends by setting the -1 FLAGB (step S24). On the other hand, if the similarity s _t is greater than the threshold th _detec , the synchronization unit 9 determines whether the similarity s _{t-1 at} the previous time (t-1) is greater than the current similarity s _t. (Step S25). Result of the judgment, if the similarity _{s t-1} is not greater than the similarity _{s t,} is set to 0 FlagA, the process ends by setting the -1 FLAGB (step S27). On the other hand, if the similarity s _t−1 is greater than the similarity s _t , (t−1, _tr ) is updated as the synchronization timing (step S26). Then, 0 is set in Flag A and Flag B, and the process is terminated (step S28).

Next, in Step S21, when the value of FlagB is smaller than 0, the synchronization unit 9 determines whether FlagA is 0 (Step S29). If Flag A is not 0 as a result of this determination, 1 is subtracted from Flag A and the process is terminated (Step S30). On the other hand, if FlagA is 0, the synchronization unit 9 compares the image feature of the image feature and the preceding operation the image currently operating the image, obtains the similarity p _t (step S31). Then, the synchronization unit 9 determines whether or not the similarity p _t−1 is smaller than the threshold th _skip (step S32). As a result of this determination, if the degree of similarity p _t−1 is not smaller than the threshold th _skip , N is set in FlagA and the process is terminated (step S33). This N is a predetermined value determined by experiments or the like. On the other hand, if the similarity _{p t-1} is smaller than the threshold _{th skip,} the synchronization unit _{9, p t-1} -p _t is equal to or larger than the threshold value th (step S34). As a result of the _{determination,} if _{p t-1} -p _t is not greater than the threshold th, the process ends by setting the _{N 1} to FLAGB (step S35).

_Then, the larger _{p t-1} -p _t is the threshold th, the synchronization unit _{9, p t-1} -p _t is determined whether the threshold -th smaller (step S36). The result of this _{determination, p t-1} -p _t is not less than the threshold -th, the process ends by setting the _{N 2} to FLAGB (step S37). On the other _{hand, p t-1} -p _t is less than the threshold value -th, the process ends by setting the _{N 3} to FLAGB (step S38). N ₁ , N ₂ , and N ₃ are all values determined in advance by experiments or the like.

  Next, in Step S21, when the value of FlagB is larger than 0, the synchronization unit 9 subtracts 1 from FlagB and ends the process (Step S39).

  In FIG. 8, Flag A is a flag for controlling the thinning width defined by the preceding state. As shown in FIG. 7, in the fifth embodiment, the thinning width (arrows indicated by symbols E, F, and G) is changed due to the change in the similarity of the detection of the preceding event (arrows indicated by symbols B, C, and D). Change. To achieve this, the flag is FlagA. Flag B is a flag for thinning out the detection of the preceding state, and plays the same role as Flag in the second embodiment.

  As described above, when detecting an operation for synchronization, it is possible to speed up the processing by detecting a state that occurs prior to the synchronization time.

<Sixth Embodiment>
Next, a video reproducing apparatus according to the sixth embodiment of the present invention will be described. If the speed of movement is different, it is difficult to compare the reference movement with the current movement. In order to solve this problem, the video playback apparatus according to the sixth embodiment designates two synchronization timings and expands / contracts between the synchronization timings so that even operations with different speeds can be easily compared.

The synchronization timing information acquisition unit 6 shown in FIG. 3 acquires synchronization timings (t _rs , t _re ) at two times, and the reference motion video feature acquisition unit 7 acquires the image features of the reference motion video for each. The synchronization unit 9 synchronizes the image features of each reference motion video. This can be realized by performing the processing of the synchronization unit 9 of the second to fourth embodiments for each synchronization timing. As a result, the synchronization unit 9 of the sixth embodiment acquires two sets of synchronization timings {(t ₁ , t _rs ), (t ₂ , t _re )}. The video display unit 4 displays the reference operation with time expansion and contraction so that the same video is reproduced at the synchronization timing.

  This operation will be described with reference to FIGS. FIG. 9 is an explanatory diagram showing an operation in the second embodiment. FIG. 10 is an explanatory diagram showing the operation in the sixth embodiment. In the second embodiment, since a single synchronization timing is used, as shown in FIG. 9, even if the current operation is a slower movement than the reference operation, synchronization is performed only at the synchronization timing. Therefore, the reference operation is started after the start of the current operation, and it is difficult to confirm the operation.

  On the other hand, according to the sixth embodiment, as shown in FIG. 10, it is possible to synchronize at two points of the operation start time and the operation end time, so that the operation can be performed even when the operation speed is different. It becomes possible to confirm in synchronization.

  In this way, the video of the reference operation as a model is read, the synchronization designation information at two times is read from the reference operation, and the current operation during delayed playback and the model operation are synchronized with time expansion and contraction. It will be possible to play.

  As described above, since it is possible to easily compare the reference operation with the current operation, the effect of video feedback can be maximized.

  You may make it implement | achieve the video reproduction apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  It can be applied to applications where it is essential to easily compare the reference operation with the current operation when two systems of images are reproduced in synchronization.

  DESCRIPTION OF SYMBOLS 1 ... Video acquisition part, 2 ... Delay width acquisition part, 3 ... Video delay part, 4 ... Video display part, 5 ... Reference operation reading part, 6 ... Synchronization timing information acquisition , 7... Reference motion video feature acquisition unit, 8... Current motion video feature acquisition unit, 9.

Claims (6)

A reference action video reading means for reading the reference action video;
Synchronization designation information reading means for reading synchronization designation information indicating timing to be synchronized in the reference operation video;
Reference motion video feature acquisition means for acquiring a feature of an image in the reference motion video specified by the synchronization designation information;
A current motion video feature acquisition means for acquiring a feature of an image in a current motion video that is to be compared with the reference motion video;
The similarity obtained as a result of comparing the feature of the image in the reference motion video acquired by the reference motion video feature acquisition means with the feature of the image in the current motion video acquired by the current motion video feature acquisition means A synchronization timing specifying means for finding a peak of similarity that is most similar based on and specifying a synchronization timing in the current operation video;
A delay width acquisition means for acquiring a delay width to be delayed of the current operation video;
Each image for each frame of the current motion video stored in each of the memories having the number of arrays corresponding to the maximum delay width is sequentially stored in the memory in the order of the array, and the newly acquired image in the first memory Synchronous reproduction means for reproducing the current operation video delayed according to the delay width by storing a frame image of the current operation video and the reference operation video in synchronization with the synchronization timing. A video playback device characterized by the above. The synchronization timing specifying means includes
2. The video according to claim 1, wherein, according to the similarity, a process for obtaining a similarity by comparing a feature of an image in the reference motion video with a feature of an image in the current motion video is thinned out. Playback device. The synchronization timing specifying means includes
In accordance with a detection result of a state that occurs prior to the synchronization timing, a process for obtaining a similarity by thinning out a feature of the image in the reference motion video and a feature of the image in the current motion video is thinned out The video reproduction apparatus according to claim 1. The synchronization designation information reading means reads synchronization designation information indicating two different timings to be synchronized in the reference operation video,
The synchronous reproduction means includes
4. The video according to claim 1, wherein at the two timings, playback is performed with a playback time expanded and contracted so that the reference motion video and the current motion video are synchronized. 5. Playback device. The synchronization timing specifying means includes
5. The video reproduction apparatus according to claim 1, wherein an image feature based on Motion History Image is used. 6.   A video playback program for causing a computer to function as the video playback device according to any one of claims 1 to 5.

Images