JP2005084809A - System and server for summary picture distribution summary picture reproduction client, picture summary method, and program and recording medium thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily generate a summary picture that can be understood extremely easily, based on a user's request, and to speedily distribute the summary picture. <P>SOLUTION: In the summary picture distribution system, having a server 1 for generating the summary picture of a picture requested from a client 2, the server 1 comprises a picture inputting section 11 for inputting picture information to be distributed for storing at a picture database 16; an operation registration section 12 for registering a target operation picture by receiving operation information to be summarized sent from the client 2; a distribution dealing section 13 for receiving distribution request information sent from the client 2 for an analysis; a picture summary 15 for generating the summary picture to be distributed, based on the analysis result; a picture distribution section 14 for distributing summary picture information to the client 2, based on the summary picture generation result; and the picture database 16 for retaining information regarding the picture stored by the picture inputting section 11. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a summary video bulletin system for distributing summary video over a network and a method for generating summary video used herein.

  Terrestrial digital broadcasting is scheduled to start in 2003, but such digital broadcasts can be viewed in a different way from current analog broadcasting. In conventional analog broadcasting, the user mainly watches live video in real time, and at most, it is recorded on a VCR and viewed. On the other hand, in digital broadcasting, for example, when a sports video is viewed from the middle, video expressions such as summarizing and presenting the progress so far and catching up with a live video are considered. Recently, it has become possible to record long-term video on a low-priced, large-capacity hard disk drive, but the total viewing time of the user does not change significantly at any time, so the video can be summarized efficiently. It is even more important to present them. Here, summarization is to generate a video that is compressed on the time axis from the target video, and deletes sequences that are not of interest to the user from the long-time target video, leaving only the sequences of interest It is.

  As a conventional video summarization technique, there is a storyboard in which the first images of scene changes in which the pixel values of consecutive frames are significantly different within the frame are arranged. For example, the technique described in Non-Patent Document 1 obtains the similarity of storyboard images and generates a scene transition graph in which a series of scenes are aggregated in a relationship graph.

  However, the storyboard itself does not express the contents of the scene, but expresses information related to the structure of the video, and is insufficient for generating a summary video.

  On the other hand, techniques for summarizing video using scene segmentation and camera motion have also been developed. For example, the technique described in Non-Patent Document 2 is a video skimming method using scene segmentation. This method is characterized in that it uses not only image information but also audio and closed captions.

However, the portion selected as the summary video based on the scene segmentation and camera motion is not necessarily what the user wants. In the above-mentioned digital broadcasting environment, the video is adapted to personal preference. There is a need for summarization techniques.
M.M. Yeung etc. , "Extracting story units video browsing and navigation." IEEE ICMCS, pp. 296-305 (1996) M.M. Smith and T.W. Kanade, “Video skimming and characterization through the combination of image and language undertechniques.” IEEE CVPR, pp. 775-781 (1997)

  SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and an object of the present invention is to provide a summary video that is easy to see and understand in accordance with a user's request in a summary video distribution system realized on a network. It is to be able to be generated. Another object is to enable the summary video generated in this way to be distributed on the network at high speed.

  Therefore, according to the summary video distribution system, summary video distribution server, summary video playback client, and video summarization method of the present invention, the viewer registers the video as a summary video key at the center at the start and end points by viewing the video. When the viewer requests a summary video, the summary video is extracted by extracting only a portion similar to the key video.

  That is, the video summarization method of the present invention is a method of summarizing the video in a server that distributes the video, the step of obtaining the motion video to be summarized, and the number of frames constituting the motion video to be summarized. A step of obtaining a video requested to be distributed; a step of obtaining a target video for each number of frames from the search result; and A step of calculating a difference between the pixel values, a step of assigning a frame number to the pixel value when the difference of the pixel values exceeds a threshold value, and a minimum frame number between all the frames as the pixel value. Generating an image to be performed, calculating a correlation value between the generated images, and determining whether to distribute the target video based on the correlation value A method, based on the determination result, and a step of generating a video summary from the target image is determined to be delivered.

The summary video distribution system of the present invention is a summary video distribution system including a server that generates a summary video of a video requested by a client,
The server inputs video information to be distributed, stores the video information in a video database, and operation registration for registering the target motion video by receiving summary target motion information sent from the client A distribution response unit that receives and analyzes distribution request information sent from the client, a video summarization unit that generates a summary video to be distributed based on the analysis result, and a result based on the summary video generation result A video distribution unit that distributes the summary video information to the client, and a video database that holds information about the video stored by the video input unit.

  As the video summarizing unit, a step of acquiring the selected summary target motion video from the motion registration unit based on the analysis result in the distribution response unit, and a step of acquiring the number of frames constituting the summary target motion video Calculating a difference between pixel values between all frames, assigning a frame number to the pixel value when the pixel value difference exceeds a threshold value, Generating a reference image having a frame number as a pixel value; searching a video requested for distribution from the video database based on an analysis result of the distribution response unit; Obtaining a target image of the image, calculating a difference between the pixel values between all frames, and if the difference between the pixel values exceeds a threshold value, Assigning a frame number to the image, generating a target image having the smallest frame number as a pixel value across all frames, and a correlation value between these images based on an inquiry between the target image and the reference image A step of determining whether to distribute the target video constituting the target image based on the correlation value, and a summary video from the target video determined to be distributed based on the determination result And generating a step.

  As said client, the distribution request information including the information about the summarization target operation information including the information about the start and end of the video to be summarized or the information about the summarization target operation to be distributed with respect to the operation instruction that accepted the input from the user And a display requesting unit for receiving summary video information distributed from the server and displaying the information.

The summary video distribution server of the present invention is a summary video distribution server that contacts a client via a network and distributes a summary video of a video requested by the client,
It has a video database that holds information about the video to be distributed,
Store video information to be distributed in the video database, receive summary target operation information sent from the client, register the target operation video, and receive distribution request information sent from the client The summary video to be distributed is generated based on the analysis result, and the summary video information is distributed to the client based on the generation result.

Further, the summary video playback client of the present invention is a summary video playback client operated by a user,
Input / output means operated by the user;
A video database that holds information related to the video to be distributed is stored, the video information to be distributed is stored in this video database, and the operation video to be processed by receiving the summary operation information sent from the client A summary video distribution server that receives and analyzes the distribution request information sent from the client and generates a summary video to be distributed based on the analysis result,
With respect to the operation instruction that has received an input from the user, generation of summarization target operation information including information on the start and end of the video to be summarized or distribution request information including information on the summarization target operation to be distributed is generated. The summary video information is transmitted to the summary video distribution server, and then the summary video information transmitted from the server is received and stored, and this information is displayed.

  The summary video distribution system, summary video distribution server, summary video playback client, and video summarization method of the present invention can be realized by a computer and a program, and the program is recorded on a computer-readable recording medium and provided through a network. Is also possible. Examples of the recording medium include a flexible disk, MO, ROM, memory card, CD, DVD, and removable disk.

  Therefore, according to the present invention, in a summary video distribution system realized on a network, it is possible to easily generate a summary video that is very easy to see and understand in accordance with a user's request. The generated summary video can be distributed on the network at high speed.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a schematic configuration diagram of a summary video distribution system according to an embodiment of the present invention.

  The summary video distribution system of this embodiment includes a summary video distribution server 1 and a summary video playback client 2. These are physically connected via a network. In the following description, it is assumed that the network connecting the summary video distribution server 1 and the summary video playback client 2 is the Internet.

  The summary video distribution server 1 includes a video input unit 11, an operation registration unit 12, a distribution response unit 13, a video distribution unit 14, a video summary unit 15, and a video database 16.

  The video input unit 11 implements a function of inputting video information to be distributed and storing it in the video database 16.

  The video database 16 stores video information to be distributed input by the video input unit 11.

  The action registration unit 12 realizes a function of receiving the summary target action information sent from the summary video reproduction client 2 and registering the target action video. The motion video is generated by acquiring the video from the start to the end of the motion from the video information stored in the video database 16, and is registered together with the identifier.

  The distribution response unit 13 realizes a function of receiving and analyzing distribution request information sent from the summary video reproduction client 2.

  The video summarization unit 15 realizes a function of generating a summary video to be distributed based on the analysis result of the distribution response unit 13.

  The video distribution unit 14 realizes a function of distributing summary video information to the summary video reproduction client 2 based on the generation result of the video summarization unit 15.

  The summary video distribution server 2 may be an information processing device such as a personal computer or a workstation. Video information to be distributed, summary video information, operation video, summary target operation information, and distribution request information are stored in a storage device such as a magnetic disk device or an optical disk device included in these information processing apparatuses. At this time, each of the processing units, that is, the video input unit 11, the operation registration unit 12, the distribution response unit 13, the video distribution unit 14, and the video summary unit 15 is actually in the form of a program in a memory included in the information processing apparatus. Stored and executed.

  The summary video playback client 2 includes summary target operation information including information about the start and end of a video to be summarized with respect to an operation instruction that has received an input from the user 20 or information about a summary target operation to be distributed. A data request unit 21 that generates a distribution request information and transfers it to the summary video distribution server 1; a display control unit 22 that receives the summary video information distributed from the summary video distribution server 1 and performs a display process of the information; Is provided.

  The summary video reproduction client 2 includes means for inputting and outputting as an interface with the user 20 in the summary video distribution system, and accepts a user operation instruction. The illustrated summary video reproduction client 2 includes a data request unit 21, a display control unit 22, and an input / output device 23.

  The data request unit 21 realizes a function of transferring an operation instruction that has received an input to the summary video distribution server 1. This is, for example, summary request operation information including information on the start and end of a video to be summarized or distribution request information including information for identifying a motion video to be distributed, receiving an operation instruction, Appropriate information is generated and transferred.

  The display control unit 22 implements a function of receiving the summary video information distributed from the summary video distribution server 1 and performing a display process of this information.

  The summary video playback client 2 is realized by an input device such as a keyboard, a mouse, and a joystick that serves as an interface with the user, and an output device such as a CRT display, and may be realized by an information processing device such as a personal computer. In this case, each processing unit, that is, the data request unit 21 and the display control unit 22 described above can be realized by a program stored and executed in the memory of such an information processing apparatus.

  Next, a process for distributing the summary video will be described. Registration of the operation video to be summarized is executed according to the procedure of the block chart shown in FIG.

  The summary video distribution server 1 inputs video information to be distributed in the video input unit 11 and stores it in the video database 16. Subsequently, the summary video reproduction client 2 receives an operation instruction from the user through the input / output device 23 in the data request unit 21. Then, summary operation information including information on the start and end of the video to be summarized is generated and transferred to the summary video distribution server 1. The summary video distribution server 1 receives the summary target motion information sent from the summary video playback client 2 at the motion registration unit 12 and stores the video from the start to the end of the motion in the video database 16. It is obtained from and created with the identifier. By repeating this, a plurality of motion images are registered.

  Subsequently, the reproduction of the summary video is executed according to the procedure of the block chart shown in FIG.

  The summary video distribution server 1 transfers the registered motion video to the summary video playback client 2 in the motion registration unit 12. The summary video reproduction client 2 receives the operation video distributed from the summary video distribution server 1 in the display control unit 22 and performs display processing. On the other hand, the operation instruction from the user by the input / output device 23 is received. Subsequently, the data request unit 21 generates distribution request information including information on the summary target operation to be distributed, and transfers the generated information to the summary video distribution server 1.

  The summary video distribution server 1 receives and analyzes the distribution request information sent from the summary video reproduction client 2 at the distribution response unit 13. The video summarizing unit 15 generates a summary video to be distributed based on the analysis result.

  The video distribution unit 14 distributes the summary video information to the summary video playback client 2 based on the generation result. The summary video reproduction client 2 receives the summary video information distributed from the summary video distribution server 1 in the display control unit 22 and performs display processing.

  Next, processing when generating a summary video, which is one function of the present embodiment, will be described. FIG. 4 is a flowchart for explaining the procedure for generating the summary video executed by the video summary unit 15 of the summary video distribution server 1.

  In step 1, an operation video to be summarized (hereinafter, a summary target motion video) is acquired.

  In step 2, the number of frames constituting the summary target motion video is acquired.

  In step 3, the video requested for distribution is searched.

  In step 4, the video for each number of frames in step 2 is acquired from the search result and set as a target video.

  In step 5, a difference image from the previous frame is created for each of the summary target motion video and the target video.

  In step 6, when the difference in pixel value exceeds a threshold value for each pixel of each difference image, a frame number is assigned to the pixel value. On the other hand, 0 is assigned when the threshold is not exceeded.

  In step 7, for each pixel of each total difference image, an image having the smallest frame number as both prime values is generated.

  In step 8, a correlation value between the generated images is calculated.

  In step 9, it is determined whether or not to distribute the target video based on the correlation value.

  In step 10, a summary video is generated from the target video determined to be distributed based on the determination result.

  The threshold value to be used may be determined in advance, or may be dynamically calculated by using a statistically calculated value such as an average value, a frequency, a deviation value, and the like. Since the method is not particularly different from the conventional method, detailed description thereof is omitted here. Here, the correlation value may be calculated using a normalized cross-correlation value (for example, a correlation value represented by the following equation) used in normal image processing. If the reference image is T (x, y) and the input image is I (x, y), the correlation value R (x, y) can be expressed by the following equation.

  Next, a specific example of summary video reproduction in the summary video distribution system described so far will be described. Here, a case where a video of a baseball game is distributed is taken as an example. Here, a case where a summary video composed only of scenes pitched by a pitcher called Pitch by Pitch will be described.

  The video of a baseball game that is broadcast in general is a scene that is not interesting, such as a video that is not related to the game, because the pitcher throws, the batter hits, and the offense and defense are repeated for several hours. Can be viewed very comfortably.

  The process until registering the operation video to be summarized is as follows.

  The summary video distribution server 1 inputs the video of the baseball game to be distributed as video information in the video input unit 11 and stores it in the video database 16.

  On the other hand, on the screen of the summary video playback client 2 as shown in FIG. 5, the user selects “video that has started pitching” to be summarized. The data request unit 21 in the summary video playback client 2 generates summary target operation information related to this, and transfers it to the summary video distribution server 1.

  The summary video distribution server 1 receives the summary target motion information sent from the summary video playback client 2 and registers the target motion video in the motion registration unit 12. For example, when registering a plurality of actions, such as a scene where a home run is hit, a similar operation is repeated to register a plurality of action videos.

  Subsequently, the processing until the summary video is reproduced is as follows. On the screen of the summary video playback client 2 as shown in FIG. 6, the user 20 selects a video that is performing a summary target operation to be distributed. The data request unit 21 of the summary video reproduction client 2 generates distribution request information related to this and transfers it to the summary video distribution server 1. The summary video distribution server 1 receives and analyzes the distribution request information at the distribution response unit 13. Next, the video summary unit 15 generates a summary video to be distributed based on the analysis result.

  Processing for generating a summary video in the video summary unit 15 will be described. A summary target motion video relating to a pitching start motion to be summarized is acquired and the number of frames is acquired. Next, a reference image is generated from the summary target motion video. This procedure will be described by taking the simple waving operation shown in FIG. 7 as an example. The difference between the pixel values between the frames is calculated, and pixels whose difference exceeds the threshold value are detected. Here, when this pixel is expressed by binary values expressed in white, for example, since the hand has not moved in the state of frame number 1 801, it is true black as indicated by 802. Since the hand moves in the frame number 803 and the frame number 20 805, white pixels as indicated by reference numerals 804 and 806 are detected. A frame number is assigned to the pixels detected in this way, and a reference image having a minimum frame number as a pixel value across all frames is generated. As shown in FIG. 8, reference images corresponding to 901, 903, and 905 are 902, 904, and 906, respectively. Here, the pixel value to which the frame number is assigned is shown in correspondence with the gray scale. As indicated by 906, the movement is expressed by gradation from white to black. In the case of a pitching start operation as shown in FIG. 5, a reference image as shown in FIG. 9 is obtained.

  On the other hand, the video of the baseball game requested to be distributed is searched, and the target video for each number of frames is acquired. A target image is generated in the same procedure. A correlation value between the reference image and the target image is calculated. Here, an example of the transition of the correlation value is shown together with the time axis. Plotting with the vertical axis representing the correlation value and the horizontal axis representing time, the characteristic diagram shown in FIG. 10 is obtained. A portion corresponding to this peak is a scene in which a pitching operation is performed, and it can be confirmed that pitching is repeated at a certain interval. Here, since the pitch by pitch video is generated, the video corresponding to the peak is determined as the video to be distributed. Then, by connecting these, a summary video is generated. The summary video information generated in this way is delivered from the video delivery unit 14 to the summary video playback client 2. The summary video playback client 2 receives this summary video information at the display control unit 22 and displays it on the screen of the summary video playback client 2.

  Although the summary image distribution system of the present invention has been specifically described based on the above embodiment, the present invention is not limited to such an embodiment, and it can be changed without departing from the gist thereof. Not too long.

  In this embodiment, an example has been described in which the same video information is processed from the input of video information to be distributed to the playback of the summary video. However, the video information or the session information of the summary video playback client is stored in a general database. It goes without saying that the same implementation is possible for a plurality of video information and a plurality of summary video playback clients by monitoring using the functions of.

  Furthermore, it is needless to say that the video summarization method described in the above embodiment can configure the processing steps shown in FIGS. 1 to 4 by a computer program and cause the computer to execute the program. Further, a recording medium such as a flexible disk, an MO, a ROM, a memory card, a CD, a DVD, and a removable medium that can be read by the computer to realize the function of the computer or to cause the computer to execute the processing process. It can be recorded on a disk and stored or distributed. Further, the program can be provided via a network such as the Internet or electronic mail.

  Then, the present invention can be implemented by installing the program from these recording media into a computer, or by downloading the program from a network and installing the program into the computer. However, installation on a computer is a computer unit, and when there are multiple computers to be installed due to multiple devices and systems, it is natural that the program is installed for each necessary processing part. is there. In this case, the program may be recorded on a recording medium corresponding to a computer, or downloaded via a network.

1 is a schematic configuration diagram of a summary video distribution system according to an embodiment of the present invention. The block chart explaining registration of the operation | movement image | video used as the summary object. Block chart explaining the playback of the summary video. The flowchart explaining the procedure which produces | generates a summary image | video. The figure which showed the Example of this invention. The figure which showed the Example of this invention. The figure which showed the Example of this invention. The figure which showed the Example of this invention. The figure which showed the Example of this invention. The characteristic view which showed the time-dependent transition of the correlation value of a reference screen and an object screen.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Summary video delivery server 10 ... Video content, 11 ... Video input part, 12 ... Operation registration part, 13 ... Delivery response part, 14 ... Video delivery part, 15 ... Video summary part 2 ... Summary video reproduction client 20 ... User , 21 ... Data request unit, 22 ... Display control unit, 23 ... I / O device

Claims (8)

A method for summarizing the video in a server that distributes the video,
Obtaining a motion video to be summarized;
Obtaining the number of frames constituting the operation video to be summarized;
Searching for the video requested to be distributed;
Obtaining a target video for each number of frames from the search results;
Calculating a difference between each pixel value between all frames for the summary target motion video and the target video;
Assigning a frame number to the pixel value when the difference between the pixel values exceeds a threshold;
Generating an image having a minimum frame number as a pixel value across all frames;
Calculating a correlation value between the generated images;
Determining whether to distribute the target video based on the correlation value; and
And a step of generating a summary video from a target video determined to be distributed based on the determination result. A summary video distribution system including a server that generates a summary video of a video requested by a client,
The server inputs video information to be distributed, stores the video information in a video database, and operation registration for registering the target motion video by receiving summary target motion information sent from the client A distribution response unit that receives and analyzes distribution request information sent from the client, a video summarization unit that generates a summary video to be distributed based on the analysis result, and a result based on the summary video generation result A summary video distribution system comprising: a video distribution unit that distributes summary video information to the client; and a video database that stores information about the video stored by the video input unit. The video summarization unit acquires the selected summary target motion video from the motion registration unit based on the analysis result in the distribution response unit, and acquires the number of frames constituting the summary target motion video. Calculating a difference between pixel values between all frames, assigning a frame number to the pixel value when the pixel value difference exceeds a threshold value, and a minimum frame between all frames. Generating a reference image having a pixel value as a number; searching a video requested for distribution from the video database based on an analysis result of the distribution response unit; The step of acquiring the target video, the step of calculating the difference between the pixel values between all the frames, and if the difference between the pixel values exceeds a threshold, Assigning a frame number, generating a target image having a minimum frame number as a pixel value across all frames, and a correlation value between these images based on an inquiry between the target image and the reference image A step of determining whether to distribute the target video constituting the target image based on the correlation value, and a summary video from the target video determined to be distributed based on the determination result The summary video distribution system according to claim 2, further comprising: The client receives summary request operation information including information related to a start and end of a video to be summarized with respect to an operation instruction that has received an input from a user, or distribution request information including information related to a summary target operation to be distributed. The data request part which produces | generates and transfers to the said server, The display control part which receives the summary video information delivered from the said server, and performs the display process of this information is provided, The Claim 2 or 3 characterized by the above-mentioned. The summary video distribution system described. A summary video distribution server that communicates with a client via a network and distributes a summary video of a video requested by the client,
It has a video database that holds information about the video to be distributed,
Store video information to be distributed in the video database, receive summary target operation information sent from the client, register the target operation video, and receive distribution request information sent from the client And a summary video distribution server that generates summary video to be distributed based on the analysis result and distributes summary video information to the client based on the generation result. A summary video playback client operated by a user,
Input / output means operated by the user;
A video database that holds information related to the video to be distributed is stored, the video information to be distributed is stored in this video database, and the operation video to be processed by receiving the summary target operation information sent from the client A summary video distribution server that receives and analyzes the distribution request information sent from the client and generates a summary video to be distributed based on the analysis result,
With respect to the operation instruction that has received an input from the user, generation of summarization target operation information including information on the start and end of the video to be summarized or distribution request information including information on the summarization target operation to be distributed is generated. A summary video reproduction client, wherein the summary video information is transferred to the summary video distribution server, and then summary video information transmitted from the server is received and stored, and display processing of the information is performed. A program for executing, by a computer, a method of summarizing the video in a server for distributing video,
Obtaining a motion video to be summarized;
Obtaining the number of frames constituting the operation video to be summarized;
Searching for the video requested to be distributed;
Obtaining a target video for each number of frames from the search results;
Calculating a difference between each pixel value between all frames for the summary target motion video and the target video;
Assigning a frame number to the pixel value when the difference between the pixel values exceeds a threshold;
Generating an image having a minimum frame number as a pixel value across all frames;
Calculating a correlation value between the generated images;
Determining whether to distribute the target video based on the correlation value; and
A program for executing a video summarization method by a computer, comprising: generating a summary video from a target video determined to be distributed based on the determination result. A computer-readable recording medium having recorded thereon a program for executing a method for summarizing the video in a server for distributing the video,
Obtaining a motion video to be summarized;
Obtaining the number of frames constituting the operation video to be summarized;
Searching for the video requested to be distributed;
Obtaining a target video for each number of frames from the search results;
Calculating a difference between the pixel values between all frames for the summary target motion video and the target video;
Assigning a frame number to the pixel value when the difference between the pixel values exceeds a threshold;
Generating an image having the smallest frame number as a pixel value across all frames;
Calculating a correlation value between the generated images;
Determining whether to distribute the target video based on the correlation value;
And a step of generating a summary video from a target video determined to be distributed based on the determination result. A recording medium storing a program for executing a video summarization method by a computer.