JP2008047998A - Moving video reproducer, and moving video reproducing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform feature parameter extraction for sound data recorded together with a moving video, and to perform synchronization of two moving videos from the similarity of feature parameters. <P>SOLUTION: The moving video reproducer comprises a means (2) for inputting a moving video file with sound; a means (3) for extracting the feature parameters of sound data in the moving video file with sound; a means (3) for generating a sound label file corresponding to the moving video file with sound consisting of the information about a label by labeling the feature parameters extracted by the extraction means; means (5, 6) for storing two moving video files with sound inputted by the input means, and storing two sound label files corresponding, respectively, to the two moving video files with sound generated by the generation means; and means (8, 9) for comparing/contrasting the information of the label included in two sound label files, and reproducing the frame of two moving video files with sound while synchronizing. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a moving image reproducing apparatus and a moving image reproducing method, for example, to a moving image reproducing device and a moving image reproducing method capable of simultaneously reproducing and comparing two moving images obtained by shooting golf swings and the like.

  As a conventional moving image reproducing apparatus of this type, for example, “an image recording / reproducing apparatus and an image recording / reproducing method thereof” described in Patent Document 1 below are known. Hereinafter, this technology is referred to as prior art 1. In this prior art 1, the playback start points of each of the two moving images are artificially specified, and the playback start points of the respective images are aligned and played back simultaneously. Yes. Therefore, if the two images are, for example, an instructor's video as a model and a video of the subject of a comparatively symmetric lesson, the difference in swing between the two can be visually distinguished, and effective lessons are performed. be able to.

  The disadvantage of the above-mentioned prior art 1 is that it takes time and effort to deny the trouble because it designates “artificial” playback start points of two moving images. Therefore, in the “moving image playback method and moving image playback system” described in Patent Document 2 below, the playback start point is automatically specified as follows. Hereinafter, this technique is referred to as Conventional Technique 2.

  That is, in the related art 2, the similarity of each frame of the moving image B with respect to the reference frame of the moving image A is calculated, the referenced frame of the moving image B is determined based on the similarity, and the reference frame and the referenced frame are determined. And the moving image A and the moving image B are simultaneously reproduced from the reproduction start points.

  Here, the “similarity” in the conventional technique 2 is calculated by a similarity calculation unit (see the similarity calculation unit 21 in the literature). In the similarity calculation unit, the similarity S is calculated from the color feature of each frame of the two moving images. There is also a statement that the similarity S may be obtained from the absolute difference value for the intensity data of the sound of one frame.

Japanese Patent Laid-Open No. 10-145724 JP-A-8-106543

  However, although the above-described conventional technique 2 is excellent in that the playback start point can be automatically specified and the operation can be simplified and the operation can be simplified, the following points are provided. There is a problem to be solved.

  For example, in a golf swing, it is necessary to follow several processes such as address, back swing, down swing, impact, and follow swing, and each process must synchronize two images, but the color characteristics are Since these processes do not change so much, the scene for each process cannot be specified. Therefore, for example, it is impossible to take out the moment of impact of two moving images and synchronize them.

  In the prior art 2, there is a description that “similarity S is obtained from an absolute difference value for sound intensity data”, and from this description, frames in which loud sounds are recorded can be synchronized. Although it is understood that “loud sound” includes not only impact sound but also other sounds (noises such as cheers and applause), the “sound strength data” alone would not necessarily expect the moment of impact, etc. The scene synchronization effect cannot be obtained.

  Therefore, the present invention provides a moving image playback apparatus and a moving image playback method that extract feature parameters of acoustic data recorded together with a moving image and perform synchronous playback of the two moving images based on the similarity of the feature parameters. It is to provide.

The invention described in claim 1 is an input means for inputting a moving image file with sound, an extracting means for extracting feature parameters of acoustic data of the moving image file with sound, and labeling the feature parameter extracted by the extracting means. Generating means for generating an audio label file corresponding to the moving image file with sound, comprising the information of the label, and storing at least two moving image files with sound input by the input means, and generating by the generating means Storage means for storing two sound label files corresponding to each of the two sound-added moving image files, and the two sound-added moving image files by comparing and contrasting the label information contained in the two sound label files. And a synchronous playback means for playing back the frames in synchronization with each other. It is a reproduction apparatus.
The invention described in claim 2 is an input step of inputting a moving image file with sound, an extracting step of extracting feature parameters of acoustic data of the moving image file with sound, and labeling the feature parameter extracted by the extracting step. A generation step of generating an acoustic label file corresponding to the moving image file with sound, comprising the label information, and storing at least two moving image files with sound input by the input step, and generating by the generation step Storing the two sound label files corresponding to each of the two sound-added moving image files, and comparing and contrasting the information of the labels included in the two sound label files, the two sound-added movie files. Including a synchronized playback step of synchronizing and playing back frames of It is a raw way.

  In the present invention, the two videos are synchronously reproduced based on the similarity of the acoustic data recorded together with the video. For example, when applied to video reproduction such as a golf lesson video, the instructor's video For both the lesson and the subject video, it is possible to synchronize and play these two videos at the same time using the impact sound and wind noise of the swing as a clue so that effective lessons can be performed. Become.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted that the specific details or examples in the following description and the illustrations of numerical values, character strings, and other symbols are only for reference in order to clarify the idea of the present invention, and the present invention may be used in whole or in part. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

  FIG. 1 is a configuration diagram of a moving image playback apparatus in the embodiment. In this figure, the moving image playback apparatus 1 includes a moving image input unit 2 with sound, an acoustic label creation unit 3, a distribution unit 4, a sample data storage unit 5, a comparison data storage unit 6, a data reading unit 7, a frame synchronization unit 8, A synthetic moving image reproduction unit 9, a display unit 10, and an audio output unit 11 are provided.

  The details of each part will be described. First, the moving image input unit 2 with sound is a part for taking in a moving image file with sound, such as a golf swing, and specifically, a video camera, It is a part having a corresponding function. Alternatively, it is a part that takes in the moving image file with sound through storage means such as a hard disk for recording a moving image file with sound taken separately by a video camera or the like, or communication means such as a network.

  The “moving image file with sound” input by the moving image input section 2 with sound is at least the following two moving images. That is, one moving image is a model moving image with sound serving as a model, and the second moving image is a comparative moving image with sound compared with the sample moving image with sound. Here, taking a golf swing as an example, the above one video (example video with audio) is a video recording the swing of an instructor etc. with audio, and the second video (comparison video with audio) is a lesson. This is a video recording the subject's swing with sound.

  The moving image file with sound (the sample moving image file with sound and the comparative moving image file with sound) input by the moving image input unit 2 with sound is supplied to the acoustic label creation unit 3 and the sorting unit 4.

The acoustic label creating unit 3 has an acoustic sample to which an acoustic label is attached in advance.
FIG. 2 is a conceptual diagram of a moving image file and an acoustic label. In this figure, each frame image of a moving image file arranged in time order from left to right is drawn on the upper stage. Here, a movie file of a golf swing is taken as an example. In this case, each frame image can be divided into processes such as an address, a back swing, a down swing, an impact, and a follow through.

  In the figure, the waveform of the acoustic data recorded with the moving image is schematically shown in the middle stage. This waveform is composed of a silent part of almost only background noise and a sounded part consisting of, for example, a wind noise of a golf club or a hitting sound at the moment of impact.

  For example, in the illustrated example, the silent part from the address to the completion of the backswing is labeled as “label 1”. Further, the wind noise of the subsequent downswing is labeled as “Label 2”. Further, the impact hitting sound is labeled as “label 3”. In addition, the follow-through wind noise immediately after the impact is labeled as “Label 4 and Label 5”. The silent part of the last follow-through is labeled as “Label 6”.

  These label names need only be unique names for one acoustic data, and may be sequential numbers as shown in the figure (“label 1” to “label 6”). It may be an artificially input explicit name or a character string corresponding to the meaning. For example, in the illustrated example, “silA” is the explicit name of label 1, “swing” is the explicit name of label 2, “impact” is the explicit name of label 3, and “clubA” is the explicit name of label 4. , “ClubB” is added as the explicit name of the label 5, and “silB” is added as the explicit name of the label 6. The meanings of these explicit names are that “silA” and “silB” indicate silence, “swing” is the wind noise of the downswing, “impact” is the impact sound, “clubA” and “clubB” are immediately after the impact, respectively. Represents wind noise.

  The voice label creation unit 3 performs voice analysis on a number of such labeled acoustic samples to extract feature parameters corresponding to each label, and inputs these features for unlabeled acoustic data. It has a function of searching for a location corresponding to a parameter and labeling the corresponding portion.

Here, as a speech analysis method, that is, a feature parameter extraction method for acoustic data, for example, filter bank analysis or linear predictive coding (linear
Various methods such as predictive cording) are known. The method applied to the acoustic label creating unit 3 is not particularly limited to this. For example, when using filter bank analysis, this analysis method is based on a spectrum by FFT (Fast Fourier Transform). , The output of the band filter bank arranged at equal intervals on the mel scale is extracted, the output is logarithmically transformed, and the inverse Fourier transform is performed, so that the sound called MFCC (Mel Frequency Cepstrum Coefficient) is obtained. Extract feature parameters for data.

  As described above, the acoustic label creating unit 3 extracts the characteristic parameters of the acoustic data of the supplied moving image and generates a corresponding acoustic label file. One acoustic label file is generated for each of the supplied moving images (example moving image file with sound 5a and comparative moving image file with sound 6a). That is, an acoustic label file (hereinafter referred to as “example acoustic label file 5b”) of the sample movie file with sound 5a and an acoustic label file (hereinafter referred to as “comparison acoustic label file 6b”) of the comparison movie file with sound 6a are generated.

  The distribution unit 4 is generated by the moving image file with sound (the sample moving image file with sound 5a / compared moving image file with sound 6a) input by the moving image input unit with sound 2 and the acoustic label creating unit 3 described above. The sound label file (example sound label file 5b / comparison sound label file 6b) is a part that is distributed and supplied to the example data storage unit 5 and the comparison data storage unit 6 based on user designation.

  That is, when the user designation is “example movie”, the movie file with audio (in this case, the sample movie file with audio 5a) input by the above-described movie-with-audio input unit 2 and the above-described acoustic label. When the acoustic label file generated by the creation unit 3 (in this case, the model acoustic label file 5b) is supplied to the model data storage unit 5, while the user designation is “comparison video”, the above-mentioned A moving image file with sound (in this case, a comparison moving image file with sound 6a) input by the moving image input section 2 with sound, and an acoustic label file (in this case, a comparative sound label file 6b) generated by the sound label creating section 3 ) Is supplied to the comparison data storage unit 6.

  Each of the model data storage unit 5 and the comparison data storage unit 6 is a large-capacity storage device composed of a hard disk, a nonvolatile semiconductor storage device, a magnetic disk, or the like. In the figure, the model data storage unit 5 and the comparison data storage unit 6 are drawn as separate bodies. However, this requires that the sample data storage space and the comparison data storage space are independent of each other. Is conceptually shown and need not be physically separated.

  The data reading unit 7 is a part that reads sample data and comparison data from the sample data storage unit 5 and the comparison data storage unit 6 in response to a reproduction instruction by the user, and compares the sample data with the read sample data. Data is supplied to the frame synchronization unit 8.

The frame synchronization unit 8 generates two moving images (an example movie file with sound 5a / a comparison movie file with audio) based on the example data and comparison data read from the example data storage unit 5 and the comparison data storage unit 6. 6a) is a part that takes frame synchronization, and this frame synchronization is performed by comparing and collating the characteristic parameters of the acoustic label files 5b and 6b corresponding to the respective moving image files 5a and 6a.

  FIG. 3 is a conceptual diagram of an acoustic label file, where (a) shows a model acoustic label file 5b, and (b) shows a comparative acoustic label file 6b. In the figure, although not particularly limited, the model sound label file 5b and the comparative sound label file 6b are text files composed of six lines in the order of the labels 1 to 6, respectively. The format of each line is “Fs Fe Lname”, where Fs and Fe are frame numbers of the moving image file corresponding to the acoustic label, and Lname is a label name (for example, the above-described explicit name). Here, Fs is the start frame number of the sound indicated by the Lname, and Fe is the end frame number of the sound indicated by the Lname.

  For example, the first line of the model acoustic label file 5b is “0 38 silA”, and this means that the frame corresponding to the label name “silA” in the model video file 5a with audio is “frame 0”. This indicates that the frame is up to “frame 38”. Similarly, the second line of the model acoustic label file 5b is “38 52 swing”, and this means that the frame corresponding to the label name “swing” in the sample video file with sound 5a is “frame 38”. To “frame 52”.

  This is the same for the comparative sound label file 6b. For example, the first line of the comparative sound label file 6b is “0 52 silA”, which means that the label name in the comparison video file with sound 6a is the same. The frames corresponding to “silA” are “frame 0” to “frame 52”. Similarly, the second line of the comparative audio label file 6b is “52 64 swing”, which means that the frames corresponding to the label name “swing” in the audio comparison video file 6a are “frame 52” to “ This indicates that the frame is up to 64 ″.

  Comparing the two illustrated acoustic label files (example acoustic label file 5b and comparative acoustic label file 6b), the same label names are aligned, and the start frame number (Fs) and the end frame number for each label name. Both (Fe) or any one of them is different. The frame synchronization unit 8 collates the label names of the two acoustic label files, and uses the same frame name line for the start frame number (Fs) and end frame number (Fe). By synchronizing the frames of each video file so that they match in the file 5a / comparative video file with sound 6a), it becomes possible to simultaneously play the frames of interest (for example, the moment of impact) of the two video files. .

  FIG. 4 is a conceptual diagram of frame synchronization. In this figure, the vertical axis represents the frame number i of the reference side moving image, and the horizontal axis represents the frame number j of the reproduction speed adjustment side moving image. For example, if the reference-side moving image is a sample moving image file 5a with sound and the playback speed adjustment-side moving image is a comparative moving image file 6a with sound, in this case, the frame of interest of the sounded moving image file 5a (for example, the moment of impact) And the reproduction speed of the comparative moving image file with sound 6a are adjusted (frame thinning or frame interpolation) so that the frame of interest in the comparative moving image file with sound 6a matches.

  In the figure, the solid line in the figure shows the case where the playback speed is not adjusted, and in this case, the playback frame number of the reference side moving image and the playback frame number of the playback speed adjustment side moving image have a one-to-one correspondence. On the other hand, the alternate long and short dash line in the figure is the case where the playback speed is adjusted. In this case, for example, while the frame 4 of the reference side moving image is being played, the frame 5 of the playback speed adjusting side movie is being played. In addition, frame 5 of the reference side moving image → frame 6 of the reproducing speed adjusting side moving image, frame 6 of the reference side moving image → frame 7 of the reproducing speed adjusting side moving image, frame 7 of the reference side moving image → frame 8 of the moving speed adjusting side moving image. As described above, the frame number of the moving image on the playback speed adjustment side is shifted by one.

  In this way, the frame synchronization unit 8 specifies that the frame number j of the playback speed adjustment side video is synchronized with the frame number i of the reference side video, and these i and j are It is determined exclusively based on the characteristic parameters (see the above-mentioned labels 1 to 6) for the acoustic data of the model video file with audio 5a and the comparative video file with audio 6a. That is, simply speaking, i and j are determined so that the frame of interest (for example, the moment of impact) of the sample video file 5a with audio matches the frame of interest of the comparative video file 6a with audio. is there.

  Based on i and j determined by the frame synchronization unit 8, the synthesized moving image reproduction unit 9 synthesizes and reproduces each frame of the sample movie file with sound 5a and the comparison movie file with sound 6a on one screen. The display unit 10 displays the synthesized image, and the audio output unit 11 outputs the acoustic data of the reference side moving image (the sample moving image file 5a with sound or the comparison moving image file 6a with sound).

  FIG. 5 is a diagram showing a display example of a composite image, in which a model image is displayed on the left side and a comparison image is displayed on the right side simultaneously. As shown in this display example, in the present embodiment, the feature parameter extraction of the acoustic data recorded together with the moving image is performed, and the two moving images are synchronized from the similarity of the feature parameter. The moment of remarkable impact can be displayed on one screen at the same time, the instructor's swing and the lesson's swing can be compared, and more effective lessons can be performed. .

  In addition, although the example applied to a golf lesson was shown in the above description, it is not limited to this use. In short, any video can be applied as long as it is a comparison of various moving images that emit characteristic sounds at the necessary points on the time axis.

  In addition, in the above description, a plurality of comparative video files with audio (in the case of a golf lesson, for example, a plurality of comparative video files with audio for each instructor) are collected and analyzed comprehensively to make a single comparison. Although the acoustic label file 6b is generated, the present invention is not limited to this. The analysis may be performed outside in advance and only the resulting characteristic parameters may be included. In this case, you may make it perform said analysis process in the arbitrary external apparatuses (for example, personal computer etc.) which have the function corresponded to the analysis of the acoustic label preparation part 3 of FIG.

It is a block diagram of the moving image reproducing device in the embodiment. It is a conceptual diagram of a moving image file and an acoustic label. It is a conceptual diagram of an acoustic label file. It is a conceptual diagram of frame synchronization. It is a figure which shows the example of a display of a synthesized image.

Explanation of symbols

1 video playback device 2 video input unit with audio (input means)
3 Acoustic label creation unit (extraction means, generation means)
5 Model data storage (storage means)
5a Model video file with audio (video file with audio)
5b Model acoustic label file (acoustic label file)
6 Comparison data storage unit (storage means)
6a Comparison video file with audio (video file with audio)
6b Comparative acoustic label file (acoustic label file)
8 Frame synchronization unit (synchronous playback means)
9 Composite image playback unit (synchronous playback means)

Claims (2)

An input means for inputting a video file with sound;
Extracting means for extracting feature parameters of acoustic data of the moving image file with sound;
Generating means for labeling the feature parameter extracted by the extracting means and generating an acoustic label file corresponding to the moving image file with sound, comprising the information of the label;
Storage means for storing at least two moving image files with sound input by the input means, and storing two acoustic label files corresponding to each of the two moving image files with sound generated by the generating means;
A video playback apparatus comprising: synchronous playback means for comparing and contrasting the label information contained in the two audio label files and playing back the frames of the two video files with audio in synchronization. Input process to input video file with audio,
An extraction step of extracting feature parameters of the acoustic data of the video file with audio;
Generating the acoustic label file corresponding to the moving image file with sound, which is labeled with the feature parameter extracted by the extraction step and includes the information of the label;
Storing at least two moving image files with sound input by the input step, and storing two acoustic label files corresponding to each of the two moving image files with sound generated by the generating step;
And a synchronized playback step of synchronizing and playing back the frames of the two video files with audio by comparing and contrasting the label information contained in the two acoustic label files.

Images