JP2011066487A - Apparatus, method and program for preparing moving picture thumbnail - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract a plurality of moving picture thumbnails having meanings from a moving picture. <P>SOLUTION: A calculation means 21 extracts still images from a moving picture VD<SB>i</SB>at a fixed interval. The number of still images to be extracted is previously set and the time length of the moving picture is equally divided by the number of still images to extract the still images. A selection means 22 receives feature values from the calculation means 21 and selects a still image to be used as a thumbnail and still images to be re-extracted. As to a still image selected by the selection means 22 as a still image not to be used as a thumbnail (still image to be re-extracted), a re-extraction means 23 re-extracts still images around the still image concerned and calculates the high frequency component amount of the re-extracted still images. When a still image satisfying a selection condition exists in the re-extracted still images, the still image is used as a thumbnail instead of the still images to be re-extracted. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an apparatus, a method, and a program for creating a moving image thumbnail.

  In recent years, it has become possible to easily prepare a video processing environment at a low price due to the high performance of video equipment, the spread and high performance of personal computers (hereinafter referred to as PCs), and the low price of storage devices such as hard disks. In addition, due to broadband, the threshold for disclosure and transmission on the Web via the network is lowered, and the number of accesses and contents of video distribution sites such as You Tube and Nico Nico Douga (registered trademark) are increasing.

  Video thumbnails are used on such video distribution sites. The moving image thumbnail is a still image representing the content of the moving image extracted from the moving image, and is generally displayed together with the title of the moving image when a list of a plurality of moving images is displayed.

  A moving image is an aggregate of a plurality of still images, and various means can be considered from which part to extract. If it is simply extracted, the top image may be used. However, in many cases, the top of the content is a copyright or a single color image, and in that case, the content content cannot be inferred from the thumbnail, and the meaningless thumbnail is obtained.

  Therefore, measures are taken such as determining the extraction position after a certain time from the beginning, or specifying the location by the user who uploaded the content by the image analysis (see Patent Documents 1 and 2).

  However, in the case of a video distribution server that uploads a lot of videos taken at home, etc., normal videos start video recording before shooting the target object. The first part of the video is a so-called play part, and the main part of the video follows, so simply extracting the top part or extracting the still image X seconds from the top is The original purpose of the thumbnail to represent is not achieved. However, if the positioning was done manually, if the number of videos increases, the effort will not be negligible, so the upload itself can be deceived, so there is a way to automatically extract meaningful thumbnails. Needed.

  The present invention has been made to solve such a problem, and an object thereof is to automatically extract a plurality of meaningful moving image thumbnails from moving images.

The moving image thumbnail creating apparatus of the present invention includes an image extracting unit that extracts a plurality of still images from a moving image, a feature amount calculating unit that calculates feature amounts of the extracted plurality of still images, and the calculated feature amount. A selection means for selecting a still image to be used as a thumbnail and a still image to be re-extracted, a re-extraction means for re-extracting a still image from the periphery of the still image to be re-extracted, and a still image to be used as the thumbnail And a thumbnail generation unit that generates a thumbnail using the image and the re-extracted still image.
The moving image thumbnail creating method of the present invention includes an image extracting step of extracting a plurality of still images from a moving image, a feature amount calculating step of calculating feature amounts of the extracted plurality of still images, and the calculated feature amount. A selection process for selecting a still image to be used as a thumbnail and a still image to be re-extracted, a re-extraction step for re-extracting a still image from the periphery of the still image to be re-extracted, and a still image to be used as the thumbnail A thumbnail generation step of generating a thumbnail using the image and the re-extracted still image.
The movie thumbnail creation program of the present invention is a program for causing a computer to function as each means of the movie thumbnail creation device of the present invention.

  According to the present invention, a still image to be used as a thumbnail and a still image to be re-extracted are selected on the basis of the feature amount of the still image extracted from the moving image, and the still image is reproduced from the periphery of the still image to be re-extracted. By extracting, a plurality of meaningful video thumbnails can be automatically extracted from the video.

It is a figure which shows the moving image delivery system to which this invention is applied. It is a hardware block diagram of the moving image delivery server in FIG. It is a functional block diagram of the moving image thumbnail creation apparatus of the 1st Embodiment of this invention. It is a flowchart which shows the process of each means in FIG. It is a functional block diagram of the moving image thumbnail creation apparatus of the 2nd Embodiment of this invention. It is a flowchart which shows the process of each means in FIG. It is a functional block diagram of the moving image thumbnail production apparatus of the 3rd Embodiment of this invention. It is a flowchart which shows the process of each means in FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[Video distribution system]
As a preferred application target of the present invention, a moving image distribution system that actually extracts and displays a plurality of thumbnails from moving images will be described as an example. As shown in FIG. 1, the present system is provided as a system including a moving image distribution server 1 and a plurality of (three illustrated) client PCs 2-1 to 2-3. Each client PC 2-1 to 2-3 can upload a moving image to the moving image distribution server 1 via the network 3 such as the Internet, and the uploaded moving image can be viewed from the client PCs 2-1 to 2-3. The video distribution server 1 extracts a plurality of thumbnails from the video when the video is uploaded, responds to a search request from the client PCs 2-1 to 2-3, and displays this video list as a search result. Use extracted thumbnails.

[Video distribution server]
As shown in FIG. 2, the video distribution server 1 includes a bus 10 as a hardware, a CPU (Central Processing Unit) 11 connected to the bus 10, a memory 12, an external storage device 13, an input device 14, and a display. It comprises a device 15 and an I / F (interface) 16.

  The CPU 11 controls the operation of the moving image distribution server 1. The memory 12 includes a ROM (Read Only Memory) for storing a program executed by the CPU 11 at startup, necessary data, and the like, a RAM (Random Access Memory) for configuring a work area of the CPU 11, and the like. The external storage device 13 includes an HDD (Hard Disk Drive), and stores various application programs including a moving image thumbnail creation program, work data, file data, and the like.

  The input device 14 includes a keyboard, a mouse, and the like, and is used to perform various key operations and instructions for the moving image distribution server 1. The display device 15 is composed of a liquid crystal display or the like, and displays a screen for operating the moving image distribution server 1 and an operation state.

  The I / F 16 is a network I / F for connecting the moving image distribution server 1 to a network (not shown), a DVI (digital video interface) for connecting a video camera, or the like.

  The moving image thumbnail creating apparatus according to the embodiment of the present invention is realized by the CPU 11 executing a moving image thumbnail creating program installed in the moving image distribution server 1. Hereinafter, the first to fourth embodiments of the moving image thumbnail creation apparatus will be described in order.

[First Embodiment]
As shown in FIG. 3, the moving image thumbnail creating apparatus according to the present embodiment includes a calculation unit 21, a selection unit 22, and a re-extraction unit 23. These means are functional blocks realized by the CPU 11 in FIG. 2 executing the moving image thumbnail creation program.

The calculating unit 21 extracts a plurality of still images from the input moving image VD _{i at a} predetermined interval, and calculates a feature amount of the extracted still images. The selection unit 22 selects a still image to be used as a thumbnail and a still image to be re-extracted based on the feature amount calculated by the calculation unit 21. The re-extraction means 23 extracts still images around the still image to be re-extracted based on a predetermined reference. A thumbnail VD _o is generated from the still image used as the thumbnail selected by the selection unit 22 and the still image extracted by the re-extraction unit 23.

FIG. 4 shows a flowchart of processing of each means in FIG.
As shown in the figure, first, the calculation means 21 extracts one or more still images at regular intervals from the moving image VD _i (step S1). At this time, the number of images to be extracted is set in advance, and the time length of the moving image is equally divided by the number of images to be extracted. At this time, it is possible to give a certain width to the top location where extraction starts. For example, assuming that one equally divided time length is X, 0 ≦ first position of extraction start <X. This place may be set in advance or may be random. A set number of still images are extracted at equal intervals from that position. By giving X a width, it is possible to deal with a moving image having a play at the head portion taken at home by setting X to be larger.

  Next, the calculation means 21 calculates the feature amount of each extracted still image (step S2). Any feature may be used, but here, the image is frequency-converted by two-dimensional Fourier transform and its high-frequency component is obtained. In general, an image with a complex pattern contains a lot of high-frequency components, and a checkered pattern is one of many examples. On the other hand, a flat image, that is, a cut of only black at the beginning of a movie or the like contains almost no high-frequency component, but only a direct current component. Using this, it is possible to specify a meaningless image by calculating the amount of high frequency components and comparing them.

  Discrete Fourier Transform (DFT) is often used for image compression processing. In DFT, high-speed calculation is possible when the width and height of an image are both powers of two. JPEG (Joint Photographic Experts Group), which is a form of image file, uses DCT (Discrete Cosine Transform). This is a DFT transform based on a cosine function sequence, and DFT is a complex sequence. Then, there are merits of becoming a real number sequence and concentrating signals on DC components and low frequency components. When using DFT, complex sequences need to be powered, but with DCT, it is only necessary to take a simple sum of high-frequency components, and MPEG (Moving Picture Experts Group), a form of video file, Since DCT is used for signal processing, it is convenient for extraction of the amount of high frequency components.

  The selection unit 22 receives the feature amount from the calculation unit 21, and selects a still image to be used as a thumbnail and a still image to be re-extracted (step S3). When the high-frequency component amount is used as a selection criterion, if it is extremely low, there is a high possibility that the image is meaningless. Accordingly, when the threshold value of the high frequency component is set as a fixed value, depending on the input moving image, there is a risk that all the images having extremely low high frequency components are judged as meaningless images. Therefore, in this embodiment, the average of all the high-frequency components from the calculation means 21 is taken, and if it is equal to or higher than the average, it is used as a thumbnail. Although the screen is fairly coarse, it is possible to select a still image to be used as a thumbnail and a still image to be re-extracted for general purpose and at high speed.

  For a still image (still image to be re-extracted) that is determined not to be used as a thumbnail by the selection unit 22, the re-extraction unit 23 extracts (re-extracts) the surrounding still image and calculates the amount of high-frequency components. (Step S4).

  Then, it is determined whether or not there is a still image that satisfies the selection condition of the selection means 22, that is, a still image having a high-frequency component amount equal to or higher than the average value (step S5), and if there is (S5: Yes). This is used as a thumbnail instead of the still image to be re-extracted (step S6).

  If there is no still image that satisfies the selection condition in the vicinity (S5: No), a still image having a higher amount of high-frequency components is used among the surrounding still images (step S7). If there is no still image having a higher frequency component amount than the still image to be re-extracted in the re-extracted still image, the original still image (still image to be re-extracted) is used. Alternatively, the time difference from the re-extracted still image may be weighted to emphasize the extraction at equal intervals.

  By repeating steps S4 to S7 until there are no more still images to be re-extracted (step S8: No), this process is performed for all the still images not selected by the selection means 22, and finally A fixed number of thumbnails are always output.

  According to the moving image thumbnail creating apparatus of the present embodiment, thumbnails are extracted at equal intervals, so that still images can be uniformly extracted from the entire moving image, and a thumbnail group that can clearly understand the contents of the moving image can be created. Further, the feature amount of the extracted thumbnail is calculated, and when there is a meaningless image such as a black image, only the effective thumbnail can be extracted at high speed by re-extraction.

[Second Embodiment]
In the first embodiment, the calculation means 21 performs still image extraction at equal intervals. Movies such as movies have logical breaks in content, and a DVD (Digital Versatile Disk) is a collection of a plurality of chapters into a single movie, and this chapter is a logical break. Or if it is further simplified and the camera is switched to another scene, it may be a break, that is, a cut point, and the cut point is mechanically detected by calculating the feature amount of each frame. There is also a method. In addition, in the case of a video shot at home or the like, shooting start / end is often performed several times in one video, and this can be easily detected by taking a difference in feature amount. The section delimited by the cut points is a scene, and although it can not be generally stated, it is almost the same video in the scene, and if one thumbnail is extracted from each scene, the whole video will be The flow can be almost grasped or guessed.

  Therefore, in the present embodiment, cut points are detected or cut point information is received from the outside, and thumbnails are extracted at intervals of the cut points. In most cases this is not evenly spaced and the number of extracted thumbnails is equal to the number of scenes. The extraction point is a certain distance from each cut point, that is, if the shortest scene length is Y, the thumbnail of each scene is extracted from 0 ≦ extraction position <Y from the cut point. Alternatively, it may be extracted as described above from the end of each scene. Alternatively, in consideration of the scene length, for example, extraction may be performed from 30% of the position of each scene length, which are characteristics of the system.

  As shown in FIG. 5, the moving image thumbnail creating apparatus according to the present embodiment includes a calculation unit 31, a selection unit 32, and a re-extraction unit 33.

The calculation unit 31 extracts a plurality of still images for each scene from the input moving image VD _i and calculates the feature amount of the extracted still image. The selection unit 32 selects a still image to be used as a thumbnail and a still image to be re-extracted based on the feature amount calculated by the calculation unit 31. The re-extraction means 33 extracts a still image from the same scene as the still image to be re-extracted based on a predetermined reference. A thumbnail VD _o is generated from the still image used as the thumbnail selected by the selection unit 32 and the still image extracted by the re-extraction unit 33.

FIG. 6 shows a flowchart of processing of each means in FIG.
As shown in the figure, first, the calculation means 31 extracts still images from the moving image VD _{i by} the number of scenes based on the cut point information (step S11). Next, the calculation means 31 calculates the feature amount of each extracted still image, similarly to step S2 of the first embodiment (FIG. 4) (step S12).

  The selection unit 32 receives the feature amount from the calculation unit 31, and selects a still image to be used as a thumbnail and a still image to be re-extracted (step S13), as in step S3 of the first embodiment (FIG. 4). ).

  For a still image (still image to be re-extracted) that is determined not to be used as a thumbnail by the selection unit 32, the re-extraction unit 33 extracts (re-extracts) a still image of the same scene and calculates the amount of high-frequency components. (Step S14).

  Then, it is determined whether or not there is a still image that satisfies the selection condition of the selection means 32, that is, a still image having a high-frequency component amount equal to or higher than the average value (step S15), and if there is (S15: Yes). This is used as a thumbnail instead of the still image to be re-extracted (step S16).

  If there is no still image satisfying the selection condition in the same scene (S15: No), a still image having a higher amount of high frequency components is used among the still images of the same scene (step S17). If there is no still image having a higher frequency component amount than the still image to be re-extracted in the re-extracted still image, the original still image (still image to be re-extracted) is used.

  By repeating steps S14 to S17 until there are no more still images to be re-extracted (step S18: No), this processing is performed on all the still images not selected by the selection means 32, and finally A fixed number of thumbnails are always output.

  According to the present embodiment, thumbnails can be created from meaningful still images of each scene by using cut points instead of equal intervals.

[Third Embodiment]
In the first embodiment, a high frequency component amount is used as a feature amount, and selection is performed by comparison with an average value of the amount. In the present embodiment, by using the similarity between still images instead of the feature amount, an object is to extract a wide variety of thumbnails excluding similar images. For example, in the case of a news program, generally, after a newscaster reads out the content of the report, the video is switched to the video on the spot. At this time, in the first embodiment, there is a possibility that only the still images of the newscaster may be picked up. In this case, it is a group of thumbnails that certainly represent the contents of the program, but the contents are not interesting as the contents in almost the same still images. It will be. Therefore, the similarity is used to deal with such a video.

  As shown in FIG. 7, the moving image thumbnail creating apparatus according to the present embodiment includes a calculation unit 41, a selection unit 42, and a re-extraction unit 43.

Similar to the calculation means 21 of the first embodiment (FIG. 3), the calculation means 41 extracts a plurality of still images from the input moving image VD _{i at a} predetermined interval, and calculates the similarity between the extracted still images. calculate. The selection unit 42 selects a still image to be used as a thumbnail and a still image to be re-extracted based on the similarity calculated by the calculation unit 41. The re-extraction means 43 extracts a still image from the periphery of the still image to be re-extracted based on a predetermined reference. A thumbnail VD _o is generated from the still image used as the thumbnail selected by the selection unit 43 and the still image extracted by the re-extraction unit 43.

FIG. 8 shows a flowchart of processing of each means in FIG.
As shown in the figure, first, the calculation means 41 extracts one or more still images at regular intervals from the input image VD _i as in step S1 of the first embodiment (FIG. 4) (step S21). Next, the calculation means 41 calculates the similarity between the extracted still images (step S22).

  Various studies have been conducted on the similarity of images, but any method may be used. As a simple example, there is a method of taking a difference for each pixel and using a mean square error or an SN ratio. In order to improve the processing speed, there are also methods such as comparison after reducing the image, or dividing the image into several regions and calculating the feature amount for each region, and comparing by that. It is suitable to use properly according to the situation.

  As the procedure for calculating the similarity between still images, the procedure for calculating the similarity with the first extracted still image is the simplest. As another procedure, a process of calculating a similarity with another still image on the basis of one of the extracted still images and further adding the similarities to obtain a sum of the similarities is performed. It is possible to change the still image to be executed and adopt a combination that minimizes the sum of the similarities.

  The selection unit 32 receives the similarity from the calculation unit 41, and selects a still image to be used as a thumbnail and a still image to be re-extracted (step S23).

  For the still image (still image to be re-extracted) that is determined not to be used as a thumbnail by the selection means 42, the re-extraction means 43 extracts (re-extracts) the surrounding still image and calculates the similarity ( Step S24).

  Then, it is determined whether or not there are still images satisfying the selection condition of the selection means 42, for example, still images having a similarity equal to or less than the average value (step S25). Are used as thumbnails instead of still images to be re-extracted (step S26).

  If there is no object that satisfies the selection condition in the vicinity (S25: No), a still image with a lower similarity is used among the surrounding still images (step S27). If there is no still image having a lower similarity than the still image to be re-extracted in the re-extracted still image, the original still image (still image to be re-extracted) is used.

  By repeating steps S24 to S27 until there are no more still images to be re-extracted (step S28: No), this processing is performed for all the still images not selected by the selection means 42, and finally A fixed number of thumbnails are always output.

  According to the present embodiment, a thumbnail group with a wide variety can be obtained by creating thumbnails from still images having as low a similarity as possible.

[Fourth Embodiment]
In the first to third embodiments, the re-extraction means always performs re-extraction, and when a better still image cannot be re-extracted, the original still image is used, and as a result, the total number of extracted images does not change. There is an advantage that the same number of thumbnails can be output. However, this has the disadvantage that it may contain thumbnails that are not necessary to understand the content.

  Therefore, in this embodiment, if a better still image is not found as a result of re-extraction, the re-extraction means does not output a still image. As a result, the total number of thumbnails is reduced, but at least the high-frequency component amount in the first and second embodiments is greater than the average value, that is, one or more thumbnails are similar in the third embodiment. However, one or more thumbnails closer to the beginning are extracted, and in any case, at least one thumbnail output is guaranteed. Accordingly, although the total number of thumbnails varies, it can be said that even the number of thumbnails can be evaluated as a feature of the video content.

  In the first to third embodiments, the present embodiment can be realized by omitting step S7 in FIG. 4, S17 in FIG. 6, and S27 in FIG.

  According to the present embodiment, when a still image satisfying the selection criterion is not found when re-extraction is performed, the still image closest to the selection criterion is not used, but it is not used as it is not found. Thus, although the number of thumbnails is reduced, the quality of the entire thumbnail is improved.

  21, 31, 41 ... calculating means, 22, 32, 42 ... selecting means, 23, 33, 43 ... re-extracting means.

JP 2004-147204 A JP 2004-274171 A

Claims (9)

Image extraction means for extracting a plurality of still images from a video;
First feature value calculating means for calculating the feature values of the extracted still images;
Selection means for selecting a still image to be used as a thumbnail and a still image to be re-extracted based on the calculated feature amount;
Re-extraction means for re-extracting a still image from the periphery of the still image to be re-extracted;
Thumbnail generation means for generating a thumbnail using the still image used as the thumbnail and the re-extracted still image;
An apparatus for creating a moving image thumbnail, comprising: In the video thumbnail creation device according to claim 1,
The moving image thumbnail creating apparatus, wherein the image extracting means extracts at a predetermined interval. In the video thumbnail creation device according to claim 1,
The moving image thumbnail creating apparatus, wherein the image extracting means extracts based on cut point information of a moving image. In the moving image thumbnail creation device according to any one of claims 1 to 3,
A second feature amount calculating unit configured to calculate a feature amount of the re-extracted still image, wherein the thumbnail generation unit uses a still image whose feature amount satisfies a predetermined criterion among the re-extracted still images; An apparatus for creating a moving image thumbnail. In the video thumbnail creation device according to claim 4,
The thumbnail generating means uses a still image whose feature amount is closest to the predetermined reference when there is no still image whose feature amount satisfies the predetermined reference in the re-extracted still image. Creation device. In the video thumbnail creation device according to claim 4,
The thumbnail creation unit uses only a still image to be used as the thumbnail when there is no still image whose feature amount satisfies a predetermined standard among the re-extracted still images. In the moving image thumbnail creation device according to any one of claims 1 to 6,
A moving image thumbnail creating apparatus using a similarity between extracted still images and a similarity between re-extracted still images instead of the feature amount. An image extraction process for extracting a plurality of still images from a video;
A feature amount calculating step of calculating feature amounts of the extracted still images;
A selection process for selecting a still image to be used as a thumbnail and a still image to be re-extracted based on the calculated feature amount;
A re-extraction step of re-extracting the still image from the periphery of the still image to be re-extracted;
A thumbnail generation step of generating a thumbnail using the still image used as the thumbnail and the re-extracted still image;
A method for creating a moving image thumbnail, comprising:   The program for functioning a computer as each means of the moving image thumbnail production apparatus described in any one of Claims 1-7.

Images