JP2004192527A - Animation summarizing device, animation summarizing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an animation summarizing device, an animation summarizing method, and a program for summarizing in correspondence with human subjectivity (sensibility) when summarizing animation. <P>SOLUTION: The device comprises a still image creating means for obtaining a still image from targeted animation, based on a prescribed rule; a noticed region calculating means for calculating a pattern in a noticed region of the obtained still image; and a summarized still image selecting means for selecting the still image to configure the summary, according to the prescribed rule, based on the pattern in the noticed region of the obtained still image. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a moving picture summarizing apparatus, a moving picture summarizing method, and a program for creating a moving picture summary, and more particularly to a moving picture summarizing method capable of forming a summarization in accordance with human subjectivity (sensitivity). The present invention relates to an apparatus, a method of creating a summary of a moving image, and a program.
[0002]
[Prior art]
2. Description of the Related Art Recently, in various fields such as the Internet and DVD video, the opportunity to handle moving image information (hereinafter, also simply referred to as “moving image”) on a computer has been increasing. At present, video editing on a personal computer has become possible, and moving images can be easily created.
Thus, when the opportunity to obtain a moving image increases, a “moving image summary” for selecting a moving image that the user really wants to watch from among a large number of moving images becomes necessary. There is a method of extracting the screen (still image) having the scene change while sequentially reproducing the target moving image, and editing the still image having the scene change to create the summary. This method is cumbersome and time-consuming, and is particularly problematic for long moving images.
Therefore, several tools are provided for extracting and displaying only frames (still images) in a moving image that have undergone a scene change.
[0003]
As such a conventional method of detecting and displaying a scene change of a moving image, for example, there is a method described below.
(1) A method for detecting a change in a scene based on a change in a code amount of an image (2) A method for detecting a change in a scene based on a change in a feature amount of a hue and a brightness of an image again
As the method (1) described above, for example, Patent Document 1 “Image data encoding method, image data encoding device using the method, image data restoration method, image data restoration device using the method, scene change Detection Method, Scene Change Detection Apparatus Using the Method ".
In the invention of Patent Document 1, a scene change frame included in a series of moving image data is automatically extracted (detected), and the moving image data is identified as a scene change frame (change frame) and other normal frames. . Then, as a change frame detection condition, a difference in code amount between consecutive frames or a difference in change rate of code amount between consecutive frames is used. Then, only the frame of the scene change is automatically extracted and displayed.
[0005]
As the method (2), for example, there is an invention disclosed in Patent Document 2 “Video feature processing method”. The invention of Patent Literature 2 is an invention for automatically extracting a feature amount from a video and automatically classifying the video.
Then, in order to achieve the above object, a histogram is measured from information including any one of hue, saturation, and lightness obtained by converting a block of video, or a combination thereof, and an extreme value or the like is extracted from the histogram as a representative value. As the feature value. Thus, the feature amount can be automatically extracted. In addition, the above image cluster is developed in a feature space having the number of feature amounts as a dimension, a distance between a plurality of image clusters is calculated from the position, and clusters of the image clusters within a certain distance are clustered. This makes it possible to automatically classify the cluster of these images.
[0006]
However, any of the methods (1) and (2) is a mechanical process of detecting a change in a scene by a physical feature amount such as an image code amount and color information. It was hard to say that the scene change was detected from the viewpoint of (sensitivity). Therefore, it has been desired to provide a device that can create an abstract in accordance with human subjectivity (sensitivity).
[0007]
[Patent Document 1]
JP-A-6-133305 [Patent Document 2]
JP-A-6-251147
[Problems to be solved by the invention]
The present invention has been made to solve such a problem, and an object of the present invention is to use a scene change screen that is mechanically extracted using physical features when creating a summary of a moving image. Rather, it is an object of the present invention to provide a moving picture summarizing apparatus, a moving picture summarizing method, and a program that can create a summarization based on still images selected according to human subjectivity (sensitivity).
[0009]
[Means for Solving the Problems]
The present invention has been made in order to solve the above-mentioned problem, and a moving picture summary creating apparatus of the present invention is a moving picture summary creating apparatus that extracts a still image from a moving picture to create a summary, and is an object of the present invention. A still image generation unit for extracting a still image from a moving image based on a predetermined rule, an attention area calculation unit for calculating a pattern of an attention area of the extracted still image, and an attention area of the extracted still image And a summary still image selecting means for selecting a still image forming a summary based on a predetermined rule based on the above pattern.
With such a configuration, the still image generation unit cuts out a still image at a specified time interval, for example, from a moving image to be summarized. Then, the attention area calculation means calculates a pattern of the attention area of the cut-out still image. Then, based on the pattern of the region of interest of the cut-out still image, the summary image selecting means selects a still image with a high degree of attention as a still image constituting a summary. This makes it possible to create an abstract that is in line with human subjectivity (sensitivity).
[0010]
In the moving image summary creating apparatus of the present invention, when selecting a still image forming a summary, the summary still image selecting means selects a still image having a high degree of interest or more than a predetermined threshold. It is characterized.
With such a configuration, a still image whose attention level is equal to or greater than a predetermined threshold is selected as a still image forming the summary. As a result, a still image with a high degree of attention can be selected based on a simple criterion in accordance with human subjectivity (sensitivity).
[0011]
Further, the moving image summary creating apparatus of the present invention is characterized in that when selecting the still images constituting the summary, the summary selecting means selects the top N still images having a high degree of attention. I do.
With such a configuration, the still images are selected in descending order of interest as the still images forming the summary. This makes it possible to create a summary that more closely matches the human subjectivity (sensitivity).
[0012]
Further, the moving picture summary creating apparatus of the present invention is characterized by further comprising moving picture linking means for associating a still picture selected as constituting a summary with a position in the moving picture of the still picture.
According to such a configuration, the moving image linking means adds the moving image link information that associates the position of the still image in the moving image to the still image selected and extracted as forming the summary. Then, when a still image constituting the summary is selected, the moving image subsequent to the still image is reproduced by referring to the moving image link information of the still image. As a result, a scene in the summary can be selected, and a moving image can be reproduced therefrom.
[0013]
Further, the moving picture summarizing method of the present invention is a moving picture summarizing method in a summarizing apparatus for extracting a still image from a moving picture to create a summarizing, wherein Based on a predetermined rule based on a still image generation procedure for cutting out an image, an attention area calculation procedure for calculating a pattern of the attention area of the cut out still image, and a pattern of the attention area of the cut out still image. And a summary still image selection procedure for selecting a still image forming the summary.
In such a procedure, a still image is cut out at a specified time interval, for example, from the moving image to be summarized in the still image generation procedure. Then, the pattern of the attention area of the cut-out still image is calculated by the attention area calculation procedure. Then, a still image with a high degree of attention is selected as a still image forming a summary based on the pattern of the attention area of the cut-out still image by the summary image selection procedure. This makes it possible to create an abstract that is in line with human subjectivity (sensitivity).
[0014]
Further, the computer program of the present invention provides a computer in a moving image summary creating apparatus that extracts a still image from a moving image to create a summary, based on a predetermined rule, from a target moving image. An image generation procedure, an attention area calculation procedure for calculating a pattern of an attention area of the cut-out still image, and a still image forming a summary by a predetermined rule based on the attention area pattern of the cut-out still image. And a summary still image selection procedure for selecting an image.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings.
[0016]
FIG. 1 is a diagram showing an example of the configuration of the means included in the moving picture digest creation apparatus 100 of the present invention. Each means provided in the moving picture digest creation apparatus 100 shown in FIG. 1 performs the following processing.
The summary generation condition input unit 101 is a unit for designating a moving image to be summarized and a length (a target time for generating a moving image summary). It is also a means for designating selection conditions (rules) for a still image in the abstract still image selection means 105.
The moving image reading unit 102 is a unit for reading a moving image for which a summary is to be created.
The still image generation unit 103 is a unit for generating (cutting out) a still image from an input moving image. The conditions (rules) for cutting out still images will be described later.
[0017]
The attention area calculation means 104 is means for calculating the attention area. The image is divided into a plurality of regions, and the degree of attention is divided into two or more levels for each of the divided regions. For the calculation of the attention area, for example, the “attraction degree” disclosed in “Japanese Patent Application Laid-Open No. 2001-126070 (attention area extraction apparatus and automatic composition determination apparatus using the same)” can be used. In addition, about this attraction degree, the outline | summary is demonstrated in the item of "supplementary explanation about attraction degree" mentioned later.
[0018]
The summary still image selection unit 105 is a unit for extracting a still image with a high degree of attention (attraction) based on the condition (rule) specified by the summary generation condition input unit 101. The selected still image is recorded in the database 111. Note that a method of creating a summary of a moving image in the summary still image selection unit 105 will be described later.
The still image selection unit 106 is a unit for selecting a still image from a summary to reproduce a moving image from a scene desired by the user. For example, when a certain still image is selected, a moving image following the still image is reproduced.
The moving image link unit 107 associates information (frames, playback positions, and the like) relating to a portion of a moving image from which a still image (still image used for summarization) is cut off as “moving image link information” with the still image, and a database 111 Means for recording the information.
The moving image reproducing means 108 is a means for reproducing a moving image from a specified location.
The printing unit 109 is a unit for printing the selected still image.
The display unit 110 is a unit for displaying a moving image and a still image.
The database 111 is a database that records a moving image 112 and a summary 113 for which a summary is to be created, a still image 114 that forms the summary 113, and moving image link information 115 of the still image 114. Although the database 111 illustrated in FIG. 1 shows an example in which only one piece of information obtained by combining the moving picture 112 and the summary 113 is recorded, actually, a plurality of pieces of information are provided with identification codes. Is recorded.
[0019]
FIG. 2 is a diagram illustrating an example in which a pattern of the degree of interest is calculated by the attention area calculation unit 104. FIG. 2 is a diagram illustrating an example in which the pattern of the degree of interest is calculated using “attraction degree” as the degree of interest. is there. FIG. 2A is a diagram schematically illustrating an example of an image 10 in which a person 11 is arranged at the left end, and FIG. 2B is a diagram illustrating an attraction degree corresponding to the image of FIG. It is the figure which showed the pattern typically. In the attractiveness pattern screen 20 shown in FIG. 2B, the attractiveness is calculated to be highest in the area A, and the areas B and C are calculated to be lower in this order.
[0020]
FIG. 3 is a diagram illustrating another example of calculating the pattern of the degree of attraction. FIG. 3A is a diagram schematically illustrating an example of an image 30 in which a flower 31 is arranged at the right end, and FIG. 3B schematically illustrates a pattern of the degree of attraction corresponding to FIG. FIG. In the pattern screen 40 of the degree of attraction shown in FIG. 3B, the degree of attraction is calculated to be highest in the area A, and the areas B and C are calculated to be lower in this order.
[0021]
FIG. 4 is a diagram for explaining a method of cutting out a still image from a moving image. In the example illustrated in FIG. 4, an example is shown in which all the frames of “frame-1 to frame-n” included in the period T of the moving image playback time are cut out as “still image-1 to still image-n”. ing. In addition, the still image can be extracted at a predetermined frame interval (for example, every 10 frames) or at a predetermined time interval (for example, every 0.1 second).
[0022]
Next, a method of selecting (extracting) a still image forming a summary by the summary still image selecting unit 105 will be described. The selection of the still images constituting the summary can be performed as follows.
(1) FIG. 5 shows the sum of the attraction degrees obtained from the pattern of the attraction degrees of the cut out still images, and shows the change in the total attraction degrees for each of the cut out still images (frames). , Select the top N still images with the high degree of attraction.
For example, in FIG. 5, when two still images are selected in descending order of the degree of attraction, still image-1 and still image-n indicated by arrows are selected.
(2) FIG. 6 shows an example in which a still image with the highest degree of attraction is selected every fixed time (3 frames in the example shown in FIG. 6) as a still image used for summarizing a moving image. I have. In the example shown in FIG. 6, still image-1, still image-5, still image 9, and still image-n indicated by arrows are selected.
(3) FIG. 7 shows an example in which a still image having a degree of attraction of not less than a predetermined threshold value is selected. In the example shown in FIG. 7, still image-1, still image-5, and still image-n indicated by arrows are selected.
If necessary, the length of the reproduction time of the summary to be created (for example, 10 seconds at a reproduction speed of 30 frames / second, etc.) can be specified to select a still image forming the summary.
[0023]
FIG. 8 is a diagram showing a method of reproducing a moving image by selecting a still image. A method of selecting a still image forming a summary of the moving image and reproducing the moving image from the still image will be described. FIG. In the example shown in FIG. 8, the still images selected as the summary are still image-1 and still image-n, and each still image includes information on a frame and a reproduction start time (reproduction start position in a moving image). It is provided and recorded as “moving image link information”. Therefore, for example, when the still image-1 is selected, the moving image can be reproduced from the reproduction start time-a. Similarly, when the still image-n is selected, the moving image can be reproduced from the reproduction start time-b.
[0024]
FIG. 9 is a flowchart for creating a summary of a moving image. The creation of a summary of a moving image is performed according to the following procedure.
(1) Step S1: The moving image specified by the summary generation condition input unit 101 is read by the moving image reading unit 102.
(2) Step S2: A frame is selected at a specified time interval, and a still image is cut out by the still image generating means 103.
(3) Step S3: The attention area calculation means 104 calculates the degree of attraction.
(4) Step S4: The attraction degrees calculated for each of the divided areas of the still image are summed up and set as the attraction degree of the still image.
(5) Step S5: It is determined whether or not the processing of the target moving image has been completed.
(6) Step S6: A still image with a high degree of interest is selected (extracted) by the abstract still image selecting means 105 according to the condition specified by the abstract generating condition input means 101, and is set as a moving picture abstract.
(7) Step S7: The selected still image is displayed or printed as a summary of the moving image.
(8) Step S8: The displayed still image is selected by the still image selecting unit 106, the reproduction position is determined with reference to the “moving image link information” of the still image, and the moving image is reproduced by the moving image reproducing unit 108. .
[0025]
FIG. 10 is a block diagram showing an example of the configuration of a moving picture summary creating apparatus, and shows only parts directly related to the present invention. In FIG. 10, reference numeral 100 denotes a moving picture summarizing apparatus, 1 denotes a communication network such as the Internet, 116 denotes a communication interface for connecting the moving picture summarizing apparatus 100 and the communication network 1, and 117 denotes the whole moving picture summarizing apparatus. And a control program unit 120 for controlling the overall operation of the computer. It should be noted that a PC (personal computer) may be used as the moving picture digest creation apparatus 100.
[0026]
Further, the processing program unit 120 includes the following processing units.
The summary generation condition input processing unit 121 is a processing unit for designating a moving image to be summarized and a length (a target time for creating a moving image summary). It is also a processing unit for designating a condition for creating a summary of a moving image in the summary still image selection processing unit 125 (a condition for selecting a still image forming a summary).
The moving image reading processing unit 122 is a processing unit for reading a moving image for which a summary is to be created.
The still image generation processing unit 123 is a processing unit for generating (cutting out) a still image from an input moving image.
[0027]
The attention area calculation processing section 124 is a processing section for calculating the attention area. The image is divided into a plurality of regions, and the degree of attention is divided into two or more levels for each of the divided regions. For the calculation of the attention area, for example, the “attraction degree” disclosed in “Japanese Patent Application Laid-Open No. 2001-126070 (attention area extraction apparatus and automatic composition determination apparatus using the same)” can be used.
[0028]
The summary still image selection processing unit 125 is a processing unit for selecting (extracting) a still image having a large attention area (high attraction) based on the condition specified by the summary generation condition input processing unit 121. is there. The selected still image is recorded in the database 111.
The still image selection processing unit 126 is a processing unit for selecting a still image from a summary in order to reproduce a moving image from a scene desired by the user. For example, when a certain still image is selected, a moving image following the still image is reproduced.
The moving image link processing unit 127 associates information (a frame, a reproduction position, and the like) relating to a portion of a moving image in which a still image (still image used for summarization) is cut off as “moving image link information” with the still image, and This is a processing unit for recording the data in the memory 111.
The moving image reproduction processing unit 128 is a processing unit for reproducing a moving image from a specified location.
The print processing unit 129 is a processing unit for printing the selected still image.
The display processing unit 130 is a processing unit for displaying a moving image and a still image.
[0029]
Note that the processing program unit 120 may be realized by dedicated hardware, and the processing program unit includes a general-purpose information processing device such as a memory and a CPU (central processing unit). A program (not shown) for realizing the function of the unit may be loaded into a memory and executed to realize the function. Further, it is assumed that an input device, a display device, a printing device, and the like (all are not shown) are connected as peripheral devices to the moving image summary creating device 100. Here, the input device refers to an input device such as a keyboard and a mouse. The display device refers to a CRT (Cathode Ray Tube), a liquid crystal display device, or the like. The printing device refers to a laser printer, an inkjet printer, or the like.
[0030]
In addition, a program for realizing the function of the moving picture digest creation apparatus 100 shown in FIG. 10 is recorded on a computer-readable recording medium, and the program recorded on this recording medium is read and executed by a computer system. Thus, the processing necessary for the moving picture digest creation apparatus 100 of the present invention may be performed. Here, the “computer system” includes an OS and hardware such as peripheral devices.
The “computer system” includes a homepage providing environment (or a display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a general medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built in a computer system.
Further, a “computer-readable recording medium” refers to a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, which dynamically holds the program for a short time. This includes transmission media (transmission media or transmission waves), and those in which programs are held for a certain period of time, such as volatile memories in computer systems serving as servers and clients in that case.
Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be a program for realizing the above-mentioned functions in combination with a program already recorded in a computer system, that is, a so-called difference file (difference file). Program).
[0031]
The embodiment of the present invention has been described above. However, the moving picture summary creating apparatus of the present invention is not limited to the above illustrated example, and various modifications may be made without departing from the scope of the present invention. Obviously you can get it.
[0032]
[Supplementary explanation on the degree of attraction]
In the embodiment of the present invention, “attraction level” is used as a reference for extracting a region of interest. One example of a method of calculating the “attraction level” is disclosed in Japanese Patent Laid-Open No. 2001-126070 (attention region extraction device). And an automatic composition determination device using the same).
Here, only the outline will be described.
To extract a region of interest, the degree of attraction is evaluated according to the physical characteristics of the original image. Here, the degree of attraction means a parameter suitable for human subjectivity. The region of interest is extracted by extracting a region that stands out from the evaluation result as a region of interest. In other words, when the attention area is evaluated, the evaluation is performed in accordance with the human subjectivity according to the physical characteristics, so that the attention area suitable for the human subjectivity can be extracted.
For example, when the physical characteristics include the degree of heterogeneity of color, the degree of attraction can be evaluated based on the difference in color of each region.
In addition, since the physical characteristics further include, in addition to the color heterogeneity, the shape heterogeneity, the area heterogeneity, and the texture (pattern) heterogeneity, the physical heterogeneity is determined based on at least one of the four heterogeneities. By evaluating the degree of eye-catching, the degree of eye-catching can be accurately evaluated according to the characteristics of the original image.
In the case where three color components (hue, saturation, and lightness) are also evaluated, a region close to a prominent color (red) according to human subjectivity can be evaluated as the most prominent region.
Furthermore, by evaluating the spatial frequency and the area of each region in the original image, the evaluation of the most prominent region can be more accurately determined.
[0033]
The evaluation of the degree of attraction is performed according to the following procedure.
(1) First, the original image is divided into regions. In this case, the original image is divided into a drawing area and a picture area. As a method of this area division, a boundary detection method based on “edge flow” described in “Edge Flow: A Framework of Boundary Detection and Image Segmentation” by WYMa and BS Manjunath et al. In 1997 IEEE is applied.
(2) Next, the divided figure region is extracted, and the degree of attraction of the region is evaluated.
The evaluation of the degree of attraction is roughly performed as follows.
First, determine the degree of heterogeneity attraction in each area. In this case, the heterogeneity of color, the heterogeneity of texture, the heterogeneity of shape, and the heterogeneity of area are obtained, and weighting factors are assigned to each of them to perform linear combination, thereby obtaining the heterogeneity attractiveness of each region.
Next, the feature attractiveness in each area is obtained. In this case, the attraction of color, the attraction of spatial frequency, and the attraction of area are obtained, and weighting factors are assigned to each of them to perform linear combination to obtain the characteristic attraction of each region.
Next, the heterogeneous attractiveness and the feature attractiveness of each area are added to obtain a feature integrated value, and the feature integrated value is evaluated by a predetermined beta function to calculate the attractiveness.
(3) Also, a pattern diagram in which the degree of attraction is evaluated is generated from the original image.
[Brief description of the drawings]
FIG. 1 is a diagram showing an example of a configuration of a means included in a moving picture summary creation device.
FIG. 2 is a diagram showing an example of calculating an attractiveness degree pattern.
FIG. 3 is a diagram showing another example of calculation of the pattern of the degree of attraction.
FIG. 4 is a diagram showing a method of cutting out a still image from a moving image.
FIG. 5 is a diagram illustrating an example of selecting top N still images with a high degree of attraction;
FIG. 6 is a diagram showing an example in which a still image with the highest degree of attraction is selected at regular intervals.
FIG. 7 is a diagram illustrating an example of selecting a still image whose attractiveness is equal to or greater than a predetermined threshold.
FIG. 8 is a view showing a method of selecting a still image and reproducing a moving image.
FIG. 9 is a diagram showing a processing procedure of the moving picture digest creation apparatus.
FIG. 10 is a block diagram showing an example of the configuration of a moving picture digest creation apparatus.
[Explanation of symbols]
DESCRIPTION OF REFERENCE NUMERALS 100 Summarizing device for moving image 101 Summarizing condition input means 102 Moving image reading means 103 Still image generating means 104 Attention area calculating means 105 Summarizing still image selecting means 106 Still image selecting means 107 Moving image linking means 108 Moving picture reproducing means 109 printing means, 110 display means

Claims (6)

A video summarization creating apparatus that extracts a still image from a video and creates a summary,
A still image generation unit for extracting a still image from a target moving image based on a predetermined rule;
Attention area calculation means for calculating a pattern of the attention area of the cut-out still image,
A moving image summary creating apparatus, comprising: a summary still image selecting means for selecting a still image forming a summary according to a predetermined rule based on the pattern of the region of interest of the extracted still image. The moving image according to claim 1, wherein when selecting a still image constituting the summary, the summary still image selecting unit selects a still image whose attention level is equal to or higher than a predetermined threshold. Summarization device. 2. The moving image summary creating apparatus according to claim 1, wherein when the still image forming the summary is selected by the summary selecting unit, the top N still images having a high degree of attention are selected. 3. . The moving picture summary according to any one of claims 1 to 3, further comprising moving picture linking means for associating the position of the still picture selected in the moving picture with the still picture selected to form the summary. Creation device. A method for creating a summary of a moving image in a summary creating device that extracts a still image from a moving image and creates a summary,
A still image generation procedure for cutting out a still image from a target moving image based on a predetermined rule;
An attention area calculation procedure for calculating a pattern of the attention area of the cut-out still image;
A summary still image selecting step for selecting a still image forming a summary based on a predetermined rule based on the pattern of the region of interest of the cut-out still image. A computer in the video summary creation device that extracts still images from video and creates summaries,
A still image generation procedure for cutting out a still image from a target moving image based on a predetermined rule;
An attention area calculation procedure for calculating a pattern of the attention area of the cut-out still image;
A program for executing a summary still image selection procedure for selecting a still image forming a summary according to a predetermined rule based on the pattern of the region of interest of the extracted still image.

Images