JP2002269569A - Animation database - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide animation database capable of performing a process at high speed and identifying, searching, and classifying scene and animation by extracting feature amount after dividing frame image. SOLUTION: This animation database is for analyzing feature amount of frames and identifying animation changing point. Simple cuts are eliminated from the animation with using difference of feature amount between frames, and animation frames eliminating the cuts are divided in a fixed direction. Gathering of feature amount, extracted from each domain, is defined as a divided image feature amount, the divided image feature amount corresponding to plural division directions is calculated, and the divided image characteristic amount corresponding to the plural division directions is analyzed to identify a changing point of animation.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to structuring, identification, and the like of a moving image using feature amounts of the moving image extracted from the moving image information.
It is related to technology related to search and classification.

[0002]

2. Description of the Related Art Japanese Patent No. 2914170 discloses a video change point detection method for detecting pan, wipe, and dissolve as a change point of a moving image.

[0003]

However, since functions such as search, classification, and identification are not provided, scenes and moving images cannot be classified, searched, and identified at high speed and with high accuracy.

The present invention has been made to solve the above-mentioned problems of the prior art, and performs high-speed processing by extracting a feature amount after dividing a frame image, and at the same time, scenes and moving images. It is an object of the present invention to provide a moving image database capable of identifying, searching, and classifying images.

[0005]

According to the first aspect of the present invention,
A moving image database for analyzing a feature amount of a frame and identifying a change point of the moving image, wherein a moving image in which a simple cut is removed from a moving image using a degree of difference of a feature amount between frames, and the cut is removed. After dividing the frame of the image in a certain direction, a set of feature amounts extracted from individual regions is used as a divided image feature amount, and a divided image feature amount corresponding to a plurality of division directions is calculated. The method is characterized in that a feature point of a moving image is identified by analyzing a feature value of a divided image corresponding to a dividing direction.

According to a second aspect of the present invention, in the first aspect, the frame is divided into grid-like regions.

[0007] The invention described in claim 3 is claim 1 or 2.
In the described invention, when the difference between the divided image feature amounts corresponding to one division direction is significantly smaller than the difference between the divided image feature amounts corresponding to the other division directions, the change point is set to the camera. In addition to identifying the workpiece, the direction of the division corresponding to the divided image feature amount is identified as the direction of the camera workpiece.

[0008] The invention described in claim 4 is claim 1 or 2.
In the described invention, when the difference between the divided image feature amounts corresponding to one division direction is significantly greater than the difference between the divided image feature amounts corresponding to the other division directions, the change point is wiped. And a division direction corresponding to the divided image feature amount is identified as a wipe direction.

The invention according to claim 5 is the first or second invention.
In the described invention, when the degree of difference between the divided image feature amounts changes almost uniformly in all regions, the change point is identified as dissolve.

According to a sixth aspect of the present invention, there is provided a moving image database for classifying a moving image by analyzing a characteristic amount of a frame constituting the moving image in a specific time section, wherein the characteristic amount is a moving image. Nearest from the frames that compose
It is characterized by using the feature amounts of a plurality of scene representative frames extracted by Neighbor clustering.

According to a seventh aspect of the present invention, there is provided a moving image database for classifying a moving image by analyzing a characteristic amount of a frame constituting the moving image in a specific time section. A feature of the present invention is to use a statistic of a difference in a feature amount between adjacent frames in a constituent frame.

According to an eighth aspect of the present invention, there is provided a moving image database for classifying a moving image by analyzing a characteristic amount of a frame constituting the moving image in a specific time section. It is characterized in that a statistic of the degree of difference between the feature amounts of each of the constituent frames and the first frame is used.

According to a ninth aspect of the present invention, in any one of the sixth to eighth aspects, a scene of a moving image is used as a specific time section.

The invention according to claim 10 is the invention according to claims 6 to 8
In the invention described in any one of the above, a fixed time interval is used as the specific time section.

According to an eleventh aspect of the present invention, there is provided a moving image database for classifying a moving image by analyzing a characteristic amount of a frame forming the moving image, wherein the characteristic amount is a time of a scene forming the moving image. It is characterized by using length statistics.

According to a twelfth aspect of the present invention, in the sixth aspect of the present invention, a statistic of a feature amount difference between adjacent scene representative frames is used as a moving image feature amount.

According to a thirteenth aspect of the present invention, in the invention according to the sixth aspect, a statistic of a difference between feature amounts between one scene representative frame and another scene representative frame is used as a moving image feature amount. And

The invention according to claim 14 is the invention according to claim 6, characterized in that scene representative frames are arranged in chronological order to form a digest of a moving image.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a moving picture database according to the present invention will be described below with reference to the drawings. The moving image database includes "moving image analysis", "moving image feature amount extraction", "moving image, moving image attribute, moving image feature amount management", "moving image classification, search, identification", "moving image list display". It consists of each function. FIG. 1 is a functional configuration diagram showing an embodiment of a moving image database according to the present invention. Reference numerals 1 to 5 indicate “moving image analysis”, “moving image feature amount extraction”, “moving image, moving image Functional blocks of “attribute and moving image feature amount management”, “moving image classification, search, identification”, and “moving image list display”.

A moving image analysis process using the moving image database according to the present invention will be described. The moving image analysis process sequentially performs the following procedure on the frames of the moving image in chronological order. (1) Let frame i be the preceding frame and frame i + d be the subsequent frame. Here, d is an integer of 1 or more,
When d is increased, the coarse inspection is performed, that is, the accuracy is reduced. (2) Displacement point detection processing is performed. If it is not the displacement point (7)
Move to the processing of. (3) Perform cut detection processing. If it is cut, the process moves to (7). (4) Perform camera work detection processing. If it is a camera work, the process proceeds to the process (7). (5) Perform a wipe detection process. If it is a wipe, the process moves to (7). (6) Dissolve detection processing is performed. If it is a dissolve, the process proceeds to (7). (7) Add d to i and return to (1). Below are the displacement points,
The respective detection processes of cut, camera work, wipe, and dissolve will be described.

The displacement point detection processing will be described. In the displacement point detection process, a displacement point is detected in the following procedure using the wide area moving image feature amount. Here, the wide area moving image feature amount includes
A general image feature amount such as a color histogram can be used. First, it extracts the wide-area video image characteristic amount F _i frames i, the frame i + global motion picture feature amount of d F _{i + d.}
Next, the distance (degree of difference) between F _i and F _{i + d} is calculated. If the calculated distance exceeds the threshold value Tg, it is determined as a displacement point.

Next, the cut detection processing will be described.
Also in the cut detection processing, a cut point is detected by the following procedure using the above-described wide area moving image feature amount. First, a wide area image feature F _{i of} frame i and a wide area image feature F i + d of frame _{i + d} are extracted. Next, the distance (degree of difference) between F _i and F _{i + d} is calculated. If the calculated distance exceeds the threshold Tc, it is simply a cut point. Note that Tg <T
c.

Next, the camera work detection processing will be described. In the camera work detection process, the camera work direction is detected using the divided image feature amount. The divided image feature amount is not a feature amount of the entire frame image but a set of feature amounts obtained by dividing the frame image into several regions and extracting the individual regions. 2A and 2B are diagrams illustrating an example of division of a region, wherein FIG. 2A illustrates vertical panning, FIG. 2B illustrates horizontal panning,
(C) and (d) are characteristic amounts for identifying the camera work directions of panning in the oblique direction, (e) zooming in and out, and (f) rotating. A general color histogram or the like can be used as a feature amount of each area, but in order to realize high-speed processing, an average color of the area is used.

The camera work direction is detected in the following procedure. First, the divided image feature amount of the frame i and the frame i
The + d divided image feature amount is extracted. Next, the distance between the individual divided image feature amounts is calculated. Only when the calculated distance between one of the divided image feature amounts is smaller than the threshold value Tc1 set in advance for each feature amount and the other distance is longer than Tc2 (Tc2> Tc1), the camera work direction is determined. It is determined that the direction corresponds to the divided image feature amount, and the camera work direction type is added to the frame i + d. That is, for example, in the case of panning in the left-right direction, the distance between the divided image feature amounts of the panning in the left-right direction of frame i and frame i + d is smaller than the distance of the other divided image feature amounts.

In the moving image analysis using the divided image feature amount described above, when one distance is remarkably smaller than other distances, it can be determined that the camera work is performed, and
The direction of the camera work can be identified corresponding to the minimum distance.

It is also possible to divide the area into a lattice and extract the divided image feature from the feature (grid-like feature) extracted from each of the grid-like areas. FIG. 3 (a)
FIG. 4 is a diagram showing an example of a region divided in a grid pattern. FIG. 3B is a diagram showing an example in which the oblique panning feature amount is calculated from the grid-like feature amount.

By extracting the divided image feature using the lattice feature, the divided image feature can be calculated at a higher speed.

Next, the wipe detection processing will be described.
In the wipe, the scene is switched at the border of the line on the normal screen, so that the change of the image on the line on the time axis becomes large. Therefore, when only one region is significantly large in the distance of the divided image feature amount, the wipe direction is determined to be the direction corresponding to the divided image feature amount, and the wipe and the direction are added to the frame. .

Next, the dissolve detection processing will be described. If the variance of the distance of the divided image feature amount or the lattice-shaped feature amount in the entire region is smaller than a certain threshold, that is, if the variance of the entire region is almost uniformly changed, the dissolve is determined.

In the processing described above, a section other than the camera work, which is divided by cutting, wiping, and dissolving, is defined as a scene. However, for the sake of simplicity, the subsequent processing may be performed simply by dividing the scene at a fixed time.

Next, the scene representative frame extraction processing will be described. Since a scene is a set of similar frames, any one frame in the scene can be used as a scene representative frame. However, in actuality, if the scene is completely different at the beginning and end and changes gradually, there may be one scene, so it is necessary to extract a plurality of scene representative frames. Therefore, one of the clustering algorithms, Nears
A plurality of scene representative frames are extracted by t Neighbor (nearest neighbor) clustering.

Next, Nearest Neighbor clustering will be described. Nearest Neig
The hbor clustering extracts a plurality of scene representative frames by the following procedure. (1) Let the first frame be the first representative frame. (2) i = 2, j = 1. (3) The feature amount of the j-th representative frame is compared with the feature amount of the i-th frame. If the distance exceeds a predetermined threshold, the frame is set as the (j + 1) th representative frame, and 1 is assigned to j.
Add. (4) Add 1 to i, and return to (3) if the frame of the scene has not reached the end.

Nearest Neighbor explained above
By the bor clustering, a plurality of appropriate frames representing the scene can be extracted.

Next, a description will be given of a moving image feature extraction process using the moving image database according to the present invention. In the moving image feature amount extraction processing, moving images and scenes are classified by clustering similar feature amounts, and searching for moving images,
A moving image feature amount for performing processes such as classification and identification is extracted.

The scene feature will be described. The scene feature amount includes a scene representative frame image feature amount and a motion feature amount. The scene representative frame image feature is an image feature extracted from the scene representative frame. Here, a general image feature amount such as a color histogram or an average color extracted from an area divided into a grid is used as the image feature amount.

The motion feature value is a statistical value of a distance in the image feature value space of the image feature value of each frame constituting the scene. As an example of the motion feature amount, an image feature amount (for example, an average color) extracted from each region of each frame divided into a lattice shape
Find the distance in area units between adjacent frames with respect to
A statistic such as an average and a variance of a distance in a region unit between all adjacent frames in the scene is set as a motion feature amount of the scene. Further, as another example of the motion feature amount, a distance of an area unit between adjacent frames with respect to an image feature amount (for example, an average color) extracted from each region of each frame divided in a lattice shape is obtained and extracted from the first frame. Statistics such as the average and variance of the image feature and the distance of each area between the frames in the scene are defined as the motion feature of the scene.

Since the above-described motion feature is extracted for each region, the motion of each position in the moving image can be indicated by the motion feature. Further, in the scene feature amount and the moving image feature amount, classification of a moving image and a scene can be performed by clustering similar feature amounts. Therefore, highly accurate classification, search, and identification of scenes and moving images can be performed. Furthermore, by associating a specific feature with a meaningful moving image or scene recognizable by a human, it becomes possible to automatically identify a moving image or scene similar to the feature, for example, a drama or variety. .

Here, since one moving image is composed of a plurality of scenes, a set of the above-mentioned scene feature amounts becomes one moving image feature amount. In addition, the “scene length feature amount” and the “representative scene motion feature amount” can be used as the moving image feature amount. The scene length feature amount is obtained by using a statistic such as a time length average and a variance of the scene as a moving image feature amount. The representative scene motion feature amount is calculated by calculating a distance of a feature amount between all the scene representative frames adjacent to the scene representative frame arranged in time series in an area unit, and calculating a statistic such as an average and a variance of the distance. This is a feature amount.

Further, the distance between the first scene representative frames of the scene representative frames arranged in time series and the feature amount between each scene representative frame is calculated in units of regions, and statistics such as the average and variance of the distances are also calculated. It can be a moving image feature amount.

Next, a moving image list display process using the moving image database according to the present invention will be described. The digest of a moving image is indicated by the scene representative frame sequence, and a plurality of moving images can be displayed on one screen if the number of images is small. As described above, if similar moving images are classified, each classification becomes a representative moving image in the moving image database, and if the digest of the representative moving image is displayed on the screen, it is easy to grasp the contents of the database.

[0041]

According to the first to fourth aspects of the present invention, it is possible to process the camerawork of a frame, the presence / absence of a wipe, and the identification of the direction at a high speed.

According to the fifth aspect of the present invention, it is possible to quickly determine whether or not a frame is dissolved.

According to the present invention, a plurality of appropriate frames representing a scene can be extracted.

According to the sixth, seventh, eighth, or eleventh aspect of the present invention, it is possible to extract an appropriate motion feature amount of a scene.

According to the fourteenth aspect, it is possible to generate a digest of an appropriate moving image which is easy for the user to grasp.

[Brief description of the drawings]

FIG. 1 is a functional configuration diagram showing an embodiment of a moving image database according to the present invention.

FIG. 2 is a diagram illustrating an example of division of a region of a frame image.

3A is a diagram illustrating an example in which a region of a frame image is divided into a grid shape, and FIG. 3B is a diagram illustrating an example in which a diagonal pan feature amount is calculated from a grid-like feature amount.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Functional block of moving image analysis 2 Function block of moving image feature amount extraction 3 Function block of moving image, moving image attribute, moving image feature amount management 4 Function block of moving image classification, search and identification 5 Function of moving image list display block

Claims (14)

[Claims] 1. A moving image database for analyzing a feature amount of a frame to identify a change point of a moving image, wherein a simple cut is removed from the moving image by using a difference of the feature amount between frames. After dividing a frame of a moving image excluding cuts in a certain direction, a set of feature amounts extracted from individual regions is used as a divided image feature amount, and a divided image feature amount corresponding to a plurality of division directions is calculated. A moving image database characterized by analyzing divided image feature amounts corresponding to the plurality of division directions and identifying a change point of the moving image. 2. The moving image database according to claim 1, wherein the frame is divided into a grid-like area. 3. When the degree of difference between the divided image feature amounts corresponding to one division direction is significantly smaller than the degree of difference between the divided image feature amounts corresponding to the other division directions, the change point is determined by the camera. 3. The moving image database according to claim 1, wherein the moving image database is identified as a workpiece and a direction of division corresponding to the divided image feature amount is identified as a direction of a camera work. 4. A change point is wiped when the difference between the divided image feature amounts corresponding to one division direction is significantly greater than the difference between the divided image feature amounts corresponding to the other division directions. 3. The moving image database according to claim 1, further comprising: identifying a division direction corresponding to the divided image feature amount as a wipe direction. 5. The moving image database according to claim 1, wherein a change point is identified as dissolve when the degree of difference between the divided image feature amounts changes substantially uniformly in all regions. 6. A moving image database for classifying a moving image by analyzing a characteristic amount of a frame constituting the moving image in a specific time section, wherein the characteristic amount is selected from frames included in the moving image. Near
A moving image database using the feature amounts of a plurality of scene representative frames extracted by est Neighbor clustering. 7. A moving picture database for classifying moving pictures by analyzing feature quantities of frames constituting a moving picture in a specific time section, wherein the feature quantities are close to each other in the frames constituting the moving picture. A moving image database using a statistic of a degree of difference between feature amounts between frames. 8. A moving image database for classifying a moving image by analyzing a feature amount of a frame constituting a moving image in a specific time section, wherein the feature amount is determined by each frame constituting the moving image and A moving image database using a statistic of a degree of difference between a feature amount and one frame. 9. The moving image database according to claim 6, wherein a scene of the moving image is used as the specific time section. 10. The moving image database according to claim 6, wherein a fixed time interval is used as the specific time section. 11. A moving image database for classifying a moving image by analyzing a characteristic amount of a frame forming the moving image, wherein the characteristic amount uses a statistic of a time length of a scene forming the moving image. A moving image database characterized in that: 12. The moving image database according to claim 6, wherein a statistic of the degree of difference between the characteristic amounts between adjacent scene representative frames is used as the moving image characteristic amount. 13. The moving image database according to claim 6, wherein a statistic of a feature amount difference between one scene representative frame and another scene representative frame is used as a moving image feature amount. 14. The moving image database according to claim 6, wherein scene representative frames are arranged in chronological order to form a digest of the moving image.