JP2010268325A - Video comparison device, video comparison method, video comparison program and recording medium of video comparison program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To simply perform alignment of scenes which are desired to be compared of a standard video and a reference video to facilitate comparison of a plurality of videos. <P>SOLUTION: A spatiotemporal image generation part 12 generates spatiotemporal images of the standard video and the reference video for comparison and playback, and a video presentation part 15 displays them. When a standard point to be noticed in the spatiotemporal image of the standard video and a reference point corresponding to the standard point in the spatiotemporal image of the reference video are instructed, display positions are adjusted so that the positions of the standard point and the reference point match to each other to present the spatiotemporal images. Otherwise, the standard point is synchronized with the reference point to play back and output the standard video and the reference video. Otherwise, a reference point corresponding to the standard point is automatically detected. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an apparatus and a method for performing comparative reproduction of videos so that the contents can be viewed while comparing the contents of a plurality of videos.

  In the conventional technology that performs comparative playback of multiple videos, in the interface that presents multiple video playback players, after the playback of the reference video is stopped, the reference video to be compared is played back or frame-by-frame playback is performed. Reproduction control such as rewinding is performed, and synchronized reproduction is performed after adjusting to the time of the scene representing the same content as the reproduction time of the reference video (Non-patent Document 1).

Satoshi Ichimura, Ryota Nakamura, Masahito Ito, Takaya Uda, Kazuya Tahu, Atsushi Matsushita, “Multi-view Video Synchronized Playback System for Distance Training,” IPSJ Transactions Vol. 45, No. 10, pp. 2335-2343, 2004.

  The method using the interface of the video playback player of the prior art has a problem that it takes time and effort to operate the reference video several times before moving the playback time of the reference video to the same scene as the standard video. In particular, in the case where the same video is viewed several times from various viewpoints for the purpose of analysis or investigation, the inefficiency in accessing an arbitrary scene according to the prior art becomes a big problem.

  An object of the present invention is to solve the above-mentioned problems, and to provide a means for facilitating the comparison of a plurality of images so that a scene to be compared can be easily aligned with a reference image and a reference image. And

  In the present invention, in order to solve the above-described problems, a spatio-temporal image that allows an overview of video content is generated, and a video scene to be compared on the spatio-temporal image can be easily determined. In other words, the present invention displays two spatio-temporal images generated from the video to be compared on the screen, accepts designation of the corresponding location from the user, and uses the received location information, for example, the designated location is moved up and down. The display position of the two spatio-temporal images is adjusted according to a predetermined rule such as shifting the display position of the spatio-temporal image in the time axis direction so that it is the same position in the It is characterized in that the comparison by can be easily performed.

  Specifically, the present invention relates to a spatio-temporal image generation that generates a spatio-temporal image in which temporal changes and spatial changes are expressed as still images for each of the first video and the second video to be compared and reproduced. Means for displaying the spatio-temporal image on a display device, and means for receiving from the input device an input for instructing a point of interest as a reference point in the spatio-temporal image of the first video displayed on the display device; , Means for receiving or automatically detecting an input indicating a reference point corresponding to the indicated reference point from the input device in the spatiotemporal image of the second video, the reference point and the reference point The display position of the spatiotemporal image of the first video and the second video is adjusted so as to be in a predetermined positional relationship and presented to the display device, or the reference point and the reference point are Synchronize before The first image and the second image, characterized in that it comprises a means for reproducing output to the display device.

  As described above, according to the present invention, when pointing a reference point of interest to a spatio-temporal image of a reference image, a frame image corresponding to the video content of the reference point is presented in the reference image. Can be compared and played. In addition, synchronized playback of the reference video and the reference video can be performed only by pointing corresponding positions in the spatiotemporal images of the reference video and the reference video.

It is a figure which shows the structure of the image | video comparison apparatus which concerns on one Example of this invention. It is a figure which shows the example of a production | generation of a spatiotemporal image. It is a figure which shows the example of a production | generation of a spatiotemporal image. It is a process flowchart (the 1) at the time of an image | video comparison. It is a figure which shows the example of the comparison presentation of an image | video. It is a figure which shows the example which indicated the location corresponding to a spatiotemporal image with a line. It is a figure which shows the example of the comparison presentation of an image | video. It is a process flowchart (the 2) at the time of a video comparison.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a configuration of a video comparison apparatus according to an embodiment of the present invention. A video comparison apparatus 10 shown in FIG. 1 includes an access request receiving unit 11, a spatiotemporal image generation unit 12, a video management unit 13, an image information storage unit 14, and a video presentation unit 15. These are realized by a computer including a CPU, a storage device, and the like, a software program, and the like. Further, an input device 20 such as a pointing device and a display device 30 for presenting a video are connected to the video comparison device 10.

  The access request receiving unit 11 receives various requests from the user. When a request for selecting a video to be presented is received, information for identifying the video (such as a file name) is output to the spatiotemporal image generation unit 12 and a signal for requesting the presentation of the spatiotemporal image is output to the video presentation unit 15. Output. When a request for pointing to a spatiotemporal image is received, pointing information is output to the video presentation unit 15.

  When receiving the file name specifying the video from the access request receiving unit 11, the spatiotemporal image generation unit 12 issues a request to the video management unit 13 to acquire the video. A spatiotemporal image is generated from the acquired video, and the generated spatiotemporal image is output to the image information storage unit 14. A spatio-temporal image is a representation of spatial information and temporal information of video content as a single still image.

  As an example of spatio-temporal image generation, for example, in the case of a video shot while moving a camera to track a moving object such as a person or a vehicle, the fluctuation between the two frames before and after It is separated into fluctuations and fluctuations due to object movement, and the movement of the camera between two frame images is detected from the fluctuation of the background, and the area excluding the moving object area from the subsequent frame image is the opposite direction to the detected camera movement The panorama background image is generated by performing processing for all the frame images of the video so that the background matches the previous frame image, and the frame image at an appropriate time interval is generated. It is effective to superimpose the moving object region at.

  FIG. 2 shows an example in which a spatio-temporal image is generated from a video shot while moving the camera so that a person is positioned at the center of the screen. The first video to be compared is called a reference video, and the second video is called a reference video.

  FIG. 2A shows main frame images of a reference video or a reference video to be compared. A panorama image created using an image obtained by removing the moving object region of the person who is ski jumping from the series of frame images is the panorama background image shown in FIG. An image obtained by superimposing an image of a person who is ski jumping at an appropriate time interval on the panoramic background image is a spatio-temporal image shown in FIG.

  In addition, in the case of an image obtained by shooting an object moving with a fixed camera, an average image of all frame images or an image after median filtering for selecting the median value of each pixel is generated as a background image. It is effective to extract a moving object region from the difference between the image and each frame image and superimpose the object region on the background image. FIG. 3 shows an example in which a spatio-temporal image is generated from a video obtained by photographing a person walking.

  FIG. 3A shows the main frame images of the comparison target video when the camera is fixed. In this video, a walking person is moving from the left side of the screen to the right side. An image obtained by removing the moving object area of the walking person from the series of frame images is the background image shown in FIG. An image in which an image of a walking person is superimposed on the background image at an appropriate time is a spatiotemporal image shown in FIG.

  When generating a spatio-temporal image from a specified video, for example, segmentation of the video is performed according to a scene change or a change in the camera movement direction, and a plurality of spatio-temporal images are generated for each video segment. The implementation as presented is also suitable.

  The video management unit 13 stores and manages the video in advance, and receives the information specifying the video from the spatio-temporal image generation unit 12 or the video presentation unit 15 and outputs the corresponding video respectively.

  The image information storage unit 14 stores and manages the spatiotemporal image received from the spatiotemporal image generation unit 12, and when the output request is received from the video presentation unit 15, outputs the corresponding spatiotemporal image to the video presentation unit 15. To do.

  When the video presentation unit 15 receives a signal requesting presentation of a spatiotemporal image from the access request receiving unit 11, the video presentation unit 15 issues a request to the image information storage unit 14 to acquire the spatiotemporal image, and outputs it to the display device 30 for use. Present to the person. When a request for pointing to a spatiotemporal image is received from the access request receiving unit 11, necessary video and image information are acquired from the video management unit 13 and the image information storage unit 14 and presented to the user. To do.

  Next, a description will be given of what kind of request is accepted by the access request accepting unit 11 and what information is presented to the video presenting unit 15 when the currently presented video is compared with another video. To do. The reference video currently presented is video Va, the spatiotemporal image Ka of the video Va is Ka, the reference video for comparison is video Vb, and the spatiotemporal image Vb is Kb.

  FIG. 4 shows a processing flow of the first embodiment when comparative reproduction of the video Va currently presented and another video Vb is performed. It is assumed that the spatiotemporal image Ka has already been generated.

  First, the access request receiving unit 11 receives the reference video Vb to be presented (step S101). Next, the spatiotemporal image generation unit 12 generates a spatiotemporal image Kb of the video Vb received by the access request receiving unit 11 (step S102). The video presentation unit 15 presents the spatiotemporal image Ka and the spatiotemporal image Kb on the screen (step S103). Subsequently, the access request accepting unit 11 accepts the pointing of the reference point of interest in the spatiotemporal image Ka input from the input device 20 (step S104). When the access request receiving unit 11 receives the reference point pointing, the video presenting unit 15 presents a frame image of the video Va corresponding to the reference point pointed to by the spatiotemporal image Ka (step S105).

  An example of presentation by the video presentation unit 15 is shown in FIG. This figure shows the spatio-temporal image Ka and the spatio-temporal image Kb arranged vertically in step S103, and then in step S104, the pointing of the reference point of interest in the spatio-temporal image Ka is accepted. The frame image of the corresponding video Va is shown on the left side.

  The access request reception unit 11 receives a reference point corresponding to the reference point of interest in the reference video as pointing to the spatiotemporal image Kb (step S106). For example, when the input device 20 is a mouse, this pointing can be easily performed by dragging the mouse shown below. In step S104, when the reference point is pointed to the spatio-temporal image Ka, the mouse button is moved as it is and dragged to the corresponding position (reference point) of the spatio-temporal image Kb. You can release them. Note that the clicked position may be determined as a reference point by clicking the mouse on the spatio-temporal image Kb after the mouse is clicked to specify the reference point on the spatio-temporal image Ka instead of the mouse drag operation. Good.

  FIG. 6 shows a result of associating a reference point of interest in the reference video Va with a reference point corresponding to the reference video Vb on the spatiotemporal image. In the figure, the locations associated with the two spatiotemporal images are visualized at both ends of the line.

  The video presentation unit 15 changes the section of the video Vb presented in the spatiotemporal image Kb so that the position pointed to the spatiotemporal image Ka and the spatiotemporal image Kb has a predetermined positional relationship (step S107). An example of the change of the video section is shown in FIG. FIG. 7 shows an example in which the video section presented in the spatiotemporal image Kb is changed so that the position pointed to by the spatiotemporal image Ka and the spatiotemporal image Kb is in the vertical direction.

  The video presentation unit 15 presents a frame image of the video Vb corresponding to the reference point pointed to the spatiotemporal image Kb (step S108). In FIG. 7, the frame image of the video Vb is shown in the lower left part.

  As described above, the playback times of the two videos can be adjusted by a simple operation such as drawing a line with the points of interest in the spatio-temporal image of each video as both ends. Thereafter, these two videos can be reproduced in synchronization.

  Next, a description will be given of a second embodiment in which comparative reproduction of the video Va currently presented and another video Vb is performed. FIG. 8 shows a processing flow of the second embodiment. In the processing flow shown in FIG. 8, steps S201 to S205 and S207 to S208 are the same as steps S101 to S105 and S107 to S108 in the processing flow of the first embodiment shown in FIG. This is different from the embodiment. Therefore, the process of step S206 will be described below.

  In step S206 in the second embodiment, instead of accepting pointing by the access request accepting unit 11 as in step S106 in FIG. 4, the position in the reference video of the scene of interest is automatically processed by image processing on the spatiotemporal image. To detect.

  The automatic detection method is as follows. A point pointed as a reference point of interest in the spatiotemporal image Ka received in step S204 in FIG. In the spatio-temporal image Ka, an image feature amount representing a color or density gradient distribution for a pixel in a predetermined range centered on the reference point P is extracted as a feature amount of the reference point P. Similarly, in the spatio-temporal image Kb, when an image feature amount for a predetermined range is obtained around a certain point, a reference point Q that is most similar to the feature amount of the reference point P is detected. In the first embodiment, the line having both the reference point P and the reference point Q as both ends is regarded as the line designated by the user through the mouse operation, and the subsequent processing is performed.

  With the above processing, it is possible to perform comparative reproduction of the scene of interest simply by pointing the reference point P corresponding to the scene of interest with respect to the spatiotemporal image over which the entire reference video can be viewed.

  The video comparison apparatus 10 can be provided with the processing functions of both the first embodiment and the second embodiment. For example, the position of the reference point Q automatically detected in the second embodiment is determined by the user. It is also possible to change arbitrarily by a drag operation or the like.

  The above-described processing for video comparison can be realized by a computer and a software program, and the program can be recorded on a computer-readable recording medium or provided through a network.

DESCRIPTION OF SYMBOLS 10 Image | video comparison apparatus 11 Access request reception part 12 Space-time image generation part 13 Image | video management part 14 Image information memory | storage part 15 Image | video presentation part 20 Input device 30 Display apparatus

Claims (4)

A spatio-temporal image generation means for generating a spatio-temporal image representing a temporal change and a spatial change as a still image for each of the first video and the second video to be compared and reproduced;
Means for displaying the spatiotemporal image on a display device;
Means for receiving from the input device an input for instructing a point of interest as a reference point in the spatiotemporal image of the first video displayed on the display device;
Means for accepting, from the input device, or automatically detecting an input indicating a reference point corresponding to the indicated reference point in the spatiotemporal image of the second video;
The display position of the spatiotemporal image of the first video and the second video is adjusted so that the position of the reference point and the reference point is in a predetermined positional relationship, and presented to the display device, or the reference A video comparison device comprising: means for reproducing and outputting the first video and the second video to the display device by synchronizing a point with the reference point. A video comparison method executed by a video comparison device having a display device and an input device,
A spatio-temporal image generation process for generating a spatio-temporal image representing a temporal change and a spatial change as a still image for each of the first video and the second video to be compared and reproduced;
Displaying the spatiotemporal image on the display device;
Receiving from the input device an input indicating a point of interest as a reference point in the spatiotemporal image of the first video displayed on the display device;
In the spatiotemporal image of the second video, a process of receiving or automatically detecting an input indicating a reference point corresponding to the indicated reference point from the input device;
The display position of the spatiotemporal image of the first video and the second video is adjusted so that the position of the reference point and the reference point is in a predetermined positional relationship, and presented to the display device, or the reference A method of reproducing and outputting the first video and the second video to the display device by synchronizing a point and the reference point. Computer
A spatio-temporal image generation means for generating a spatio-temporal image representing a temporal change and a spatial change as a still image for each of the first video and the second video to be compared and reproduced;
Means for displaying the spatiotemporal image on a display device;
Means for receiving, from the input device, an input for instructing a point of interest as a reference point in the spatiotemporal image of the first video displayed on the display device;
Means for accepting, from the input device, or automatically detecting an input indicating a reference point corresponding to the indicated reference point in the spatiotemporal image of the second video;
The display position of the spatiotemporal image of the first video and the second video is adjusted so that the position of the reference point and the reference point is in a predetermined positional relationship, and presented to the display device, or the reference As means for reproducing and outputting the first video and the second video to the display device by synchronizing a point and the reference point,
Video comparison program to make it function. Computer
A spatio-temporal image generation means for generating a spatio-temporal image representing a temporal change and a spatial change as a still image for each of the first video and the second video to be compared and reproduced;
Means for displaying the spatiotemporal image on a display device;
Means for receiving from the input device an input for instructing a point of interest as a reference point in the spatiotemporal image of the first video displayed on the display device;
Means for accepting, from the input device, or automatically detecting an input indicating a reference point corresponding to the indicated reference point in the spatiotemporal image of the second video;
The display position of the spatiotemporal image of the first video and the second video is adjusted so that the position of the reference point and the reference point is in a predetermined positional relationship, and presented to the display device, or the reference As means for reproducing and outputting the first video and the second video to the display device by synchronizing a point and the reference point,
A computer-readable recording medium on which a video comparison program for functioning is recorded.

Images