JPH08106543A - Method and system for reproducing moving image - Google Patents

Abstract

PURPOSE: To provide the method and system for moving image reproducing which easily search another moving image which should be reproduced simultaneously with one moving image and should be compared with it. CONSTITUTION: Display parts A and B of a moving image reproducing part 30 load frames 40 of a moving image A and frames 41 of a moving image B to a memory 8 to display them on a display device. A user designates one of frames of the moving image A as a reference frame. A similarity degree calculation part 21 of an automatic search part 20 calculates the degree of similarity between the reference frame 40 and each frame 41 of the moving image B and reports it to a part 22 which determines the frame to be referred to. This part 22 determines the frame to be referred to based on degrees of similarity of respective frames of the moving image B and reports it to a moving image reproducing part 30. An operation control part 31 of the moving image reproducing part 30 takes the reference frame 40 and the frame 41 to be referred to of the moving image B as the reproducing start positions and reports them to display parts A32 and B33.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image reproducing method and a moving image reproducing system, and more particularly, it is possible to reproduce a certain moving image at the same time and easily find another moving image to be compared. The present invention relates to a moving image reproducing method and a moving image reproducing system.

[0002]

2. Description of the Related Art In the study of movement (analysis, learning, practice),
It is effective to compare the basic movements with the movements under study. For example, in learning a golf swing, cue a video tape recording a golf swing that serves as a model and pause it, then cue a video tape recording a golf swing of your own and pause it. Then both are started playing at the same time and both golf swings are compared.

[0003]

As described above, conventionally, after temporarily stopping the image of the basic motion, the operation of reproducing, stopping, and reversely reproducing the moving image in which the motion of the research object is recorded is repeated. Then, the user searches for an image corresponding to the temporarily stopped image and starts searching. However, this cueing work is
There is a problem that is a heavy burden on the user. Therefore, an object of the present invention is to provide a moving image reproducing method and a moving image reproducing system that can easily reproduce a certain moving image at the same time and locate another moving image to be compared.

[0004]

According to a first aspect of the present invention, the present invention calculates the degree of similarity of each frame in a moving image B to a reference frame in a moving image A, and the moving image B is calculated based on the similarity. Of the moving image A and the moving image B are simultaneously reproduced from the reproduction start positions of the reference frame and the referenced frame. Provide a playback method.

According to a second aspect of the present invention, in the moving image reproducing method having the above structure, the reference frame in the moving image A is defined as the similarity of each frame in the moving image B to the reference frame in the moving image A. There is provided a method of reproducing a moving image, which is characterized by calculating the similarity of the image data or the acoustic data of each frame in the moving image B with respect to the image data or the acoustic data.

According to a third aspect of the present invention, in the moving image reproducing method having the above-mentioned structure, the feature amount of the image data or acoustic data of the reference frame in the moving image A and the image data of each frame in the moving image B are used. Alternatively, a method for reproducing a moving image is provided, which is characterized in that the degree of similarity is calculated from the difference between the characteristic amount of the acoustic data.

According to a fourth aspect of the present invention, in the method of reproducing a moving image having the above structure, a graph of the calculated similarity is displayed on the screen, and the extreme value of the graph closest to the position pointed by the user on the screen is displayed. A method of reproducing a moving image, characterized in that a frame corresponding to a point is determined as a referenced frame.

According to a fifth aspect of the present invention, in the moving image reproducing method having the above structure, a plurality of frames having a high degree of similarity are respectively determined as the referenced frames of the moving image B,
The reference frame and the referenced frame are respectively set as reproduction start positions, and simultaneously reproducing the moving image A and the moving image B from the reproduction start positions is repeated in order from the referenced frame having a high degree of similarity. A method of reproducing a moving image is provided.

According to a sixth aspect, the present invention relates to a similarity calculating means for calculating the similarity of each frame in the moving image B to the reference frame in the moving image A, and the moving image B of the moving image B based on the similarity. The apparatus further comprises: a referenced frame determining means for determining a referenced frame; and a simultaneous playback means for simultaneously playing back the moving image A and the moving image B with the reference frame and the referenced frame as playback start positions. Provided is a moving image playback system.

According to a seventh aspect of the present invention, in the moving image reproducing system having the above-mentioned configuration, the similarity calculating means includes each of the moving image B corresponding to the image data or the acoustic data of the reference frame in the moving image A. Provided is a moving image reproduction system characterized by calculating a similarity between image data of a frame or sound data.

According to an eighth aspect of the present invention, in the moving image reproducing system, the similarity calculating means includes the feature amount of the image data or acoustic data of the reference frame in the moving image A and the moving image B in the moving image B. There is provided a moving image reproduction system characterized by calculating a degree of similarity from a difference between a feature amount of image data or acoustic data of each frame.

According to a ninth aspect of the present invention, in the moving image reproducing system having the above-mentioned configuration, the referenced frame determining means displays a graph of similarity on the screen and displays it on the screen at the position pointed by the user. There is provided a moving image reproduction system characterized in that a frame corresponding to an extreme point of the nearest graph is determined as a referenced frame.

According to a tenth aspect of the present invention, in the moving image reproducing system having the above structure, the referenced frame determining means determines each of a plurality of frames having a high degree of similarity as a referenced frame of the moving image B. The simultaneous reproduction means sets the reference frame and the referenced frame as reproduction start positions, and reproduces the moving image A and the moving image B from the reproduction start positions at the same time from the referenced frame having a high degree of similarity. Provided is a moving image reproduction system characterized by repeating in order.

[0014]

In the moving image reproducing method according to the first aspect,
The similarity of each frame in the moving image B to the reference frame in the moving image A is calculated, the referenced frame of the moving image B is determined based on the similarity, and the reference frame and the referenced frame are respectively determined. The moving image A and the moving image B are simultaneously reproduced from the reproduction start positions. This eliminates the need for the user to search for the frame (referenced frame) in the moving image B corresponding to the reference frame in the moving image A, and the burden on the user can be reduced. If the moving image A is a part of a certain moving image and the moving image B is another part of the same moving image, the present invention can be applied to one moving image. Further, if the moving image B is a plurality of different moving images, the present invention can be applied to three or more moving images.

In the moving image reproducing method according to the second aspect, the degree of similarity is calculated based on the image data or the acoustic data in the moving image. As a result, it becomes possible to cue a frame of an image similar to the image of the reference frame. Further, it becomes possible to find a frame of sound similar to the sound of the reference frame.

In the moving image reproducing method according to the third aspect, the degree of similarity is calculated from the difference between the characteristic amounts of the image data or the acoustic data. Since the difference between the feature amounts of the image data or the sound data can be calculated by a simple calculation, the similarity can be calculated at high speed even with a small computer such as a personal computer.

In the moving image reproducing method according to the fourth aspect, the graph of the calculated similarity is displayed on the screen, the user is made to point on the screen, and the extreme point of the graph closest to the position pointed by the user ( The frame corresponding to the maximum point when the similarity becomes larger and the similarity value becomes smaller, and the minimum point when the similarity becomes smaller and similar value is determined as the referenced frame. This makes it possible to determine the correct referenced frame even if there are a plurality of referenced frame candidates. In addition, the user is
There is no need to perform complicated operations such as reproduction, stop, and reverse reproduction of the moving image B, and only the approximate position of the graph is pointed, so there is almost no burden.

In the moving picture reproducing method according to the fifth aspect, a plurality of frames having a high degree of similarity are respectively determined as the referenced frames of the moving picture B, and the referenced frames having a high degree of similarity are sequentially referred to as the reference frames. Repeating the simultaneous reproduction of the moving image A and the moving image B from the reproduction start positions of the referenced frame and the reproduction start positions thereof, respectively. Thus, even if there are a plurality of referenced frame candidates, the moving image A and the moving image B can be simultaneously reproduced in order.

In the moving image reproducing system according to the sixth to tenth aspects, the moving image reproducing method according to the first to fifth aspects can be suitably implemented.

[0020]

An embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to this.

First Embodiment FIG. 1 is a schematic block diagram showing the configuration of the first embodiment of the moving image reproducing system of the present invention. In the moving image reproducing system 100, a display device 1 such as a CRT is used.
Displays the output screen of the computer 2, and the speaker 1-1
Outputs audio. Commands for the computer 2 can be input using the pointing device 3 and the keyboard 4. A video reproducing device 5 such as an optical disk or a video deck is controlled by the computer 2 via a control line 10. The video signal of the moving image A output from the video reproduction device 5 is sequentially converted into digital image data by the video input device 6 and input to the computer 2. The digital image data of the moving image A input to the computer 2 is stored in the external information storage device 11 as one file. Further, the video signal of the moving image B output from the video reproduction device 5 is sequentially converted into digital image data by the video input device 6 and input to the computer 2. Digital image data of the moving image B input to the computer 2 is stored in the external information storage device 11 as one file. At the time of reproducing the moving image, the frames of the moving image A and the moving image B are loaded from the external information storage device 11 into the memory 8.

FIG. 2 is an explanatory diagram of a moving image reproduction process executed by the computer 2. The moving image reproducing section 30 is C
Configured as a function of the program executed by PU9,
It is composed of an operation control unit 31, a display unit A32, and a display unit B33. The display unit A and the display unit B load the frame 40 of the moving image A and the frame 41 of the moving image B into the memory 8 and display them on the display device 1. The operation control unit 31 will be described later. The user designates one of the frames of the moving image A as a reference frame. Automatic cueing unit 20
Is configured as a function of a program executed by the CPU 9, and includes a similarity calculation unit 21 and a referenced frame determination unit 22. The similarity calculation unit 21 determines the reference frame (40) of the moving image A and each frame 4 of the moving image B designated by the user.
The similarity of 1 is calculated, and the referenced frame determination unit 22 is notified. The referenced frame determination unit 22 determines the referenced frame based on the similarity of each frame of the moving image B, and notifies the moving image reproduction unit 30 of the determined referenced frame.

The operation control section 31 of the moving picture reproducing section 30 sets the reference frame (40) of the moving picture A and the referenced frame (41) of the moving picture B as reproduction start positions, and displays them on the display section A32 and the display section B33. Notice. The display unit A32 sequentially loads the frames 40 of the moving image A into the memory 8 from the notified reproduction start position (reference frame) and displays them on the display device 1. The display unit B33 displays the frame 41 of the moving image B from the notified reproduction start position (referenced frame).
Are sequentially loaded into the memory 8 and displayed on the display device 1 in synchronization with the moving image A.

FIG. 3 is a view showing an example of a display screen in a moving image reproducing process. The screen of the display device 1 has 2
One window A32-1, and one window B32-2 are displayed side by side. In the window A 32-1, the display unit A
The frame 40 of the moving image A loaded from the external storage device 11 into the memory 8 is displayed by 32. Window B3
On 2-2, the frame 41 of the moving image B loaded from the external storage device 11 to the memory 8 is displayed by the display unit B33.

Selection A in the operation panel window 31-1
When the button 31-1c is turned on, the operations of the file selection button 31-h, the scroll bar 31-1a, and the scan button group 31-1b are valid only for the window A32-1. When the selection B button 31-1d is turned on, the file selection button 31-h and the scroll bar 3
The operations of 1-1a and the scan button group 31-1b are valid only for the window B32-2. Choice AB
When button 31-1e is turned on, scroll bar 31
-1a and the operation of the scan button group 31-1b are valid for both the window A32-1 and the window B32-2. These buttons 31-1c, 31-1
d, 31-1e, when any button is turned on,
The others are off.

The file selection button 31-1h is used to activate the window A32-1 or the window B3.
This is for designating the file name of the moving image displayed in 2-2. After turning on the file selection button 31-1h, a file name is input from the keyboard 4.
The scroll bar 31-1a is for moving the frame of the moving image displayed in the valid window A32-1 or / and window B32-2. For example, if the knob 31-1i is dragged to the left end, the first frame is displayed, and if it is dragged to the right end, the last frame is displayed. Scan button group 31-1b
Is a button group having the same meaning as the operation button group of a normal VCR. From the left button in order, "call the first frame and stop", "call the last frame and stop", "playback start", "pause", "reverse playback start", "frame reverse feed", "frame advance" It corresponds to the operation of "Stop".

The rectangular area button 31-1g is for setting the rectangular area 32-1b in the window A 32-1. After turning on the rectangular area button 31-1g, the cursor 34 is dragged diagonally to the rectangular area 32.
Set -1b. Similarity calculation command button 31-1f
Is for the user to instruct to calculate the similarity of each frame of the moving image B with respect to the image in the rectangular area 32-1b (the image of the reference frame of the moving image A).

The operation procedure of the user is shown below. (1) After turning on the selection A button 31-1c and then turning on the file selection button 31-1h, the keyboard 4
Input the file name of the moving image A from. (2) Since the image of the moving image A is displayed in the window A 32-1, the reference frame is determined by operating the scroll bar 31-1a and the scan button group 31-1b.
Next, after turning on the rectangular area button 31-1g, the cursor 34 is dragged diagonally to set the rectangular area 32-1b. (3) After turning on the selection B button 31-1d and then turning on the file selection button 31-1h, the keyboard 4
Input the file name of the moving image B from. (4) Since the image of the moving image B is displayed in the window B 32-2, confirm it. Next, the similarity calculation command button 31-1f is turned on. (5) As described later, the similarity graph window (see FIG.
22-2a) and the similarity graph (22-2 in FIG. 9).
b) is displayed. Then, the neighborhood of the maximum point in the similarity graph is pointed. (6) The selection AB button 31-1e is turned on, and then the "playback start" button of the scan button group 31-1b is turned on.

FIG. 4 is an explanatory diagram of a file storage format of a moving image. This file 42 has one header 42h
And a plurality of frame data sections 42f, 42f, ... FIG. 5 is an explanatory diagram of the image data structure of one frame. The frame 43 is composed of pixel data of image width × image height, and each pixel data is composed of a red component pixel value, a green component pixel value, and a blue component pixel value. The moving image file may be held in a data compression format.

FIG. 6 is a detailed block diagram of the similarity calculator 21. The similarity calculation unit 21 includes a difference absolute value calculation unit 21-1, a threshold value storage unit 21-4, a threshold value determination unit 21-2, and an area update unit 21-3. The difference absolute value calculation unit 21-1 performs the calculation of the following equation between the reference frame (40) of the moving image A and each frame of the moving image B, focusing on the color feature. Here, the red component pixel value of the pixel (i, j) of the reference frame is Rij, the green component pixel value is Gij,
Let the blue component pixel value be Bij. Further, the red component pixel value of the pixel (i, j) of the processing target frame of the moving image B is rij, the green component pixel value is gij, and the blue component pixel value is bij. The evaluation value is Eij. Further, (i, j) is a rectangular area 32-1.
The range is b. Eij = | Rij-rij | + | Gij-gij | + | Bij-bij
| The threshold value storage unit 21-4 transfers the threshold value T to the threshold value determination unit 21-2. The threshold value determination unit 21-
2 performs the calculation of the following equation between the evaluation value Eij and the threshold value T. Aij = 1 ... Eij <T, = 0 ... Eij ≧ T The area updating unit 21-3 obtains the similarity S by the following equation. S = ΣAij This similarity S means the area of a region similar to the reference frame in the frame of the moving image B. Therefore, the more similar, the larger the similarity S is.

Similar to the above, the degree of similarity may be obtained from the absolute difference value for the sound intensity data of one frame. The degree of similarity in this case means the time of a sound similar to the reference frame in the frame of the moving image B.

FIG. 7 is a detailed block diagram of the referenced frame determining unit 22. This referenced frame determination unit 22
Similarity graph creation unit 22-1 and frame number determination unit 2
2-2. The similarity graph creation unit 22-1
Associates the frame number with the similarity S to create the similarity table 50 as shown in FIG. 8 and passes it to the frame number determination unit 22-2. The frame number determination unit 22-2
Is the similarity graph window 22-2 as shown in FIG.
a is displayed on the display device 1, and the similarity graph 22-2b is displayed in the similarity graph window 22-2a based on the similarity table 50. Then, when the user points a position in the vicinity of the similarity graph 22-2b, the similarity graph 2 closest to the position is displayed.
The frame number corresponding to the maximum point of 2-2b is obtained, and the frame number is output to the moving image reproducing unit 30.

According to the moving image reproducing system 100 described above, it is not necessary to repeat the reproduction, stop, and reverse reproduction of the moving image B, and the moving image B desired to be reproduced simultaneously with the moving image A is compared.
You can easily find the cue. Further, it is possible to perform the cueing with higher accuracy than in the case where the user repeatedly performs the reproduction, stop, reverse reproduction, and the like of the moving image B to perform the cueing.

Further, the moving image reproducing system 100 described above.
According to this, since a rectangular area (processing area) can be set in the reference frame, it is possible to narrow down to a portion to be particularly compared and start the search. In addition, since the similarity is calculated from the absolute difference value, a simple calculation is required, the similarity can be calculated quickly even when the number of frames of the moving image B is large, and it can be realized by a small computer such as a personal computer, which is low cost. Can be promoted. Further, since the user points in the vicinity of the maximum point of the similarity graph 22-2b, the similarity graph 22-2
Even if there is a possibility that there are a plurality of maximum points of the same magnitude in -2b, it is possible to accurately find the maximum point.

-Second Embodiment-Similarity table 5 in the referenced frame determination unit 22.
It is also possible to select the highest frame number from 0 to the similarity S and notify the moving image reproducing unit 30 of it. In this case, display of the similarity graph and pointing by the user are unnecessary.

-Third Embodiment-In the referenced frame determination unit 22, the similarity table 5
It is assumed that a plurality of frame numbers having a high degree of similarity S are selected from 0, and the frame numbers are notified to the moving image reproducing unit 30 in descending order of the degree of similarity S. In addition, the operation control unit 31 of the moving image reproduction unit 30.
In the above, the display unit B is notified of the frame numbers as referenced frames in the notified order, and the moving image A and the moving image B are simultaneously reproduced from each referenced frame. This is useful when the same motion is repeatedly included in the moving image B.

[0037]

According to the moving picture reproducing method and the moving picture reproducing system of the present invention, the moving picture B to be compared can be easily found at the same time as the moving picture A is reproduced.
Further, it is possible to perform the cueing with higher accuracy than in the case where the user repeatedly performs the reproduction, stop, reverse reproduction, and the like of the moving image B to perform the cueing. Moreover, if the similarity is calculated from the image data and the acoustic data, it is possible to perform the cue based on the similarity of the images and the similarity of the sounds. Further, if the similarity is calculated from the difference between the feature amounts of the image data or the acoustic data, a simple calculation is sufficient, so that the similarity can be calculated quickly even if the number of frames of the moving image B is large, and a personal computer is used. It can be realized with a small computer such as. Further, by displaying a graph of the degree of similarity and pointing to the user, even if there are a plurality of similar frames, it is possible to accurately find the beginning.

[Brief description of drawings]

FIG. 1 is a configuration block diagram of a moving image reproduction system according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of a moving image reproduction process of the present invention.

FIG. 3 is an exemplary diagram of a display screen in a moving image reproduction process.

FIG. 4 is an explanatory diagram of a file storage format of a moving image.

FIG. 5 is an explanatory diagram of a 1-frame image data structure.

FIG. 6 is a detailed block diagram of a similarity calculation unit.

FIG. 7 is a detailed block diagram of a referenced frame determination unit.

FIG. 8 is a view showing an example of a similarity table.

FIG. 9 is a view showing an example of a display screen of a similarity graph.

[Explanation of symbols]

 1 Display Device 2 Computer 3 Pointing Device 4 Keyboard 5 Video Playback Device 6 Video Input Device 8 Memory 9 CPU 11 External Information Storage Device 20 Automatic Cueing Unit 21 Similarity Calculation Unit 22 Referenced Frame Determining Unit 30 Moving Image Playback Unit 50 Similar Degree table

Front page continuation (72) Inventor Hiroi Ueda 1-280 Higashi Koigokubo, Kokubunji, Tokyo Inside the Central Research Laboratory, Hitachi, Ltd.

Claims (10)

[Claims] 1. A similarity of each frame in a moving image B to a reference frame in a moving image A is calculated, a referenced frame of the moving image B is determined based on the similarity, and the reference frame and the referenced frame are determined. A method of reproducing a moving image, characterized in that the reference frame is set as a reproduction start position, and the moving image A and the moving image B are simultaneously reproduced from the reproduction start positions. 2. The moving image reproducing method according to claim 1, wherein the similarity of each frame in moving image B to the reference frame in moving image A is image data or sound of the reference frame in moving image A. A method of reproducing a moving image, which comprises calculating a degree of similarity of image data or acoustic data of each frame in the moving image B with respect to the data. 3. The moving image reproducing method according to claim 2, wherein the feature amount of the image data or acoustic data of the reference frame in the moving image A and the feature amount of the image data or acoustic data of each frame in the moving image B. A method of reproducing a moving image, characterized in that the degree of similarity is calculated from the difference from the amount. 4. The moving image reproducing method according to claim 1, wherein a graph of the calculated similarity is displayed on the screen, and the graph closest to the position pointed by the user on the screen is displayed. A method of reproducing a moving image, characterized in that a frame corresponding to an extreme point is determined as a referenced frame. 5. The moving image reproducing method according to claim 1, wherein a plurality of frames having a high degree of similarity are each determined as a referenced frame of the moving image B, and the reference frame and the reference frame are combined. The referenced frame is set as the reproduction start position, and the moving image A is generated from those reproduction start positions.
And a moving image B are simultaneously reproduced, in order from a referenced frame having a high degree of similarity, in order to reproduce the moving image. 6. A similarity calculation means for calculating the similarity of each frame in the moving image B with respect to the reference frame in the moving image A, and a referenced frame for determining a referenced frame of the moving image B based on the similarity. A moving image reproducing system comprising: a frame determining unit; and a simultaneous reproducing unit that simultaneously reproduces the moving image A and the moving image B by using the reference frame and the referenced frame as reproduction start positions. 7. The moving image reproducing system according to claim 6, wherein the similarity calculation means is a moving image B for image data or acoustic data of a reference frame in the moving image A.
A moving image reproduction system characterized by calculating a similarity between image data or audio data of each frame in the frame. 8. The moving image reproducing system according to claim 7, wherein the similarity calculating means calculates the feature amount of the image data or acoustic data of the reference frame in the moving image A and each frame of the moving image B. A moving image reproducing system characterized by calculating a degree of similarity from a difference between a feature amount of image data or sound data. 9. The moving image reproducing system according to claim 6, wherein the referenced frame determining unit displays a graph of similarity on a screen and a user points on the screen. A moving image reproducing system characterized in that a frame corresponding to an extreme point of a graph closest to a position is determined as a referenced frame. 10. The moving image reproducing system according to claim 6, wherein the referenced frame determining means sets a plurality of frames having a high degree of similarity as referenced frames of the moving image B. The simultaneous reproduction means determines that the reference frame and the referenced frame are respectively reproduction start positions and simultaneously reproduces the moving image A and the moving image B from the reproduction start positions. A moving image playback system characterized by repeating in order from a frame.