JPH0944639A - Method and device for classifying video blocks - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically classify images into sets of similar shots without finely dividing the image for the unit of a shot and without damaging time order property. SOLUTION: First of all, the feature amounts of respective shots are calculated and the shot is used as the block in an initial state (step 111). The degree of similarity between adjacent blocks is calculated (step 112), the adjacent block or step showing the maximum degree of similarity among the calculated degrees of similarity is merged with one block (step 113) and the feature amount of the merged block is calculated (step 114). The steps 112-114 are repeated until the classification is sufficiently performed, and the number of blocks is gradually decreased.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to video data processing, and more particularly to a method and apparatus for classifying image data sequences forming a video and structuring the blocks into scenes or scenes.

[0002]

2. Description of the Related Art Video data generally has an enormous amount of data, but the only way to know the content is to watch all the videos in chronological order. Considering that a set of temporally continuous image data is a video, if each image data forming the video is classified according to a certain scale, the outline of the video can be grasped, or so-called skipping can be performed. It is expected that it will be very useful for understanding the contents of the video in a short time. Hereinafter, classification of an image means dividing an image into a plurality of blocks with respect to a time axis by detecting hierarchical structural elements that are inherent in the image and bundling structural elements that belong to the same classification. If attention is paid to bundling structural elements belonging to the same classification, it can also be referred to as video structuring.

Generally, images are hierarchically classified in units of "frame", "take", "shot", "scene", and "story". This hierarchical classification includes everything from the physical level in hardware (lower level) to the semantic level of human creative work (upper level).

A "frame" is a physical unit corresponding to each frame of film at the time of shooting an image,
"Take" is a set of frames and is a unit indicating a temporally continuous video section shot by the same camera. The "take" is set to ON (0N) / OFF (0F) of the camera at the time of shooting.
F) is reflected.

A "shot" is a unit indicating a video section taken by the same camera as a take, but it means a video section selected from the takes at the time of editing the video, which is more human-like than the take. It is a semantic unit that reflects the intention. Furthermore, when shots are combined by editing work, it becomes a "story" of the video. That is, a take is a unit of video material, and a shot is a unit of video work.

On the other hand, a "scene" refers to a set of shots that can be regarded as semantically identical scenes in a video work, and is a unit for dividing the video work into several blocks. The classification of scenes requires a lot of human creative work and is classified differently by various interpretations. For example, if the characters are the same, it may be interpreted as the same scene, and no matter how far apart they are,
The scenes may be the same if the stage and place where the video was shot match. However, here, from the viewpoint of classifying scenes in order to help grasp the content of the work, the following conditions are set for the scenes. Condition 1: A scene is a set of shots having similar image characteristics (hereinafter, referred to as feature similarity condition). Condition 2: A scene is a set of shots that are temporally adjacent and continuous (hereinafter referred to as a temporal continuous condition).

The most extreme example of the scene satisfying the above conditions is an image created by continuously switching the images of the same subject photographed by a plurality of different cameras from different viewpoints. That is, this is an example in which one scene is composed of a plurality of similar and continuous shots. On the other hand, for example, the images often seen in news programs such as studios and live broadcasts on the spot are similar to each other, although the scenes in the studios are similar to each other and the broadcasts to the live are similar. Since there is little similarity between the video footage and the live video, they are not the same scene. Since similar images are not continuously composed, in such a case, a studio scene A, a scene scene A, a studio scene B, a scene scene B, a studio scene C, a scene scene C, ... , It is assumed that each completely different scene continues.

In order to perform an ideal classification, it is necessary to consider the story etc., but at present, this work can be performed only manually, and the work amount becomes enormous.
It is unrealistic except in special cases. Therefore, some sort of automation is required for classification of image data.

There are some conventional techniques for classifying image data so as to be easily used by the user based on the above-described image classification unit to structure the image.

For example, from the sum of the differences in brightness at corresponding positions (x, y) between consecutive image frames,
"Video cut point detection method" that detects the switching of video cut points by calculating the rate of change in consecutive frames
And the like. This can be regarded as a shot-based classification technique.

As a technique for classifying scenes, (1) from the standpoint that the same scene is composed of similar color information, the color information of the image data string is converted into a feature space, and clustering is performed on the feature space. However, in "video feature processing method" (Japanese Patent Laid-Open No. 6-251147) for classifying video, (2) generally, in a video work, sound information, which is an element that constitutes video data together with an image, is not a shot unit but a scene unit. "Video browsing interface using sound information" that classifies scenes based on this sound information, utilizing the fact that they are added to each other by semantic work (Technical Report of the Institute of Television Engineers of Japan, Vol. 19, No. 7, 1995/12),
(3) Obtain a representative screen for each shot, calculate the similarity based on the moment invariant of brightness between representative images and color information, display the representative screen and shots with high similarity, and display the low one "Content-based Browsing of" is created by creating an interface that does not display the result and displaying a similar representative screen to the user.
Video Sequences "(ACM, Multimedia 94, P.97-) and so on.

[0012]

When classifying video for the purpose of grasping the content of the video, it is desirable to classify the video into blocks that reflect the meaning, that is, scene units.

Among the above-mentioned conventional techniques, the "video cut point detection method" aims at detection of a cut point in a video and enables classification in shot units. But,
For example, a two-hour video work is generally composed of thousands of shots, and there is a problem in that shots are too subdivided for grasping the content of the video. Therefore, there is a need for a technique of classifying not by shots but by scenes.

On the other hand, other conventional techniques except the "video cut point detection method" aim to classify into more meaningful units in order to solve the point of being too subdivided by the shot unit classification. It is a thing. Among them, in the "video feature processing method", based on the assumption that the same scene is composed of similar color information, a feature space based on color information is used to classify images having similar color combinations. It is carried out. This "video feature processing method" satisfies the above-mentioned feature similarity condition (condition 1), but does not consider the temporal continuous condition (condition 2) of being temporally continuous. Therefore, according to this method, a group of similar images is extracted, and a temporally discontinuous and similar image is extracted as one image.
It may be detected as one block. That is, using the example of the news program video described above,
Images in a studio that are discontinuous in time are regarded as one similar block without being classified as individual scenes. After all, this method cannot stably extract scenes that are temporally continuous.

"Video browsing using sound expression
By using sound information such as BGM, the “interface” can merge the subdivided shots into a series of the same scenes that are temporally continuous, and can perform classification that satisfies the temporal continuous condition described above. it can. However,
Scenes and sound information do not necessarily correspond to each other, such as when the same BGM is continuous across scenes or when sound information is inserted in the middle of a scene, or even in a video work where sound information does not exist at all. If not, there is a problem that the image cannot be stably classified into scenes. Further, there is a problem that the feature similarity condition is not satisfied.

[Content-based Browsing of Video Sequ
In “ences”, classification is performed in advance for each shot, a representative image is selected for each shot, and the similarity is determined by using both the comparison of the moment invariants of these representative images and the comparison of color information. . By using the similarity by shape and color, by displaying only the representative screen that is similar when the representative screen is selected, it is possible to realize a simple display and search of information similar to the image being watched and related. There is. Therefore, similar to the “video feature processing method”, the feature similarity condition is satisfied, but the temporal continuous condition is not satisfied.

In order to perform the classification so that the content of the video can be accurately grasped, it is necessary to execute the scene classification satisfying both the characteristic similarity condition and the temporal continuous condition. There is no technology that satisfies these two conditions at the same time.

An object of the present invention is to classify an image into a set of similar shots without subdividing the image into shot units and without impairing the temporal order, that is, the feature similarity condition and the temporal similarity. An object of the present invention is to provide a video block classification method and device that can satisfy the continuous condition at the same time and stably classify video.

[0019]

A video block classification method and apparatus according to the present invention has the following configuration in order to classify video while satisfying a feature similarity condition and a temporal continuation condition.

That is, the video block classification method of the present invention is a video block classification method for classifying a video composed of an image data string into a plurality of video blocks, and a video inputting a video classified into a large number of blocks in advance. An input step, a feature amount calculation step of calculating a feature amount of the block from the image data string of each block, a similarity degree calculation step of calculating a similarity degree between adjacent blocks based on the feature amount,
A block merging step of merging adjacent blocks showing the maximum similarity among the calculated similarities into one block, and repeating the feature amount calculating step, the similarity calculating step and the block merging step. Then, a plurality of classified video blocks are obtained.

In the video block classification method of the present invention,
The video input step may include a step of classifying the video input in frame units into shots that are one type of block. Further, as the feature amount of each block, it is preferable to use a histogram obtained by extracting information including any one of color, hue, saturation, and lightness or a combination thereof from a frame included in the block. ,
When the number of frames included in a block is two or more, it is preferable to use a histogram obtained by performing a histogram AND operation on all histograms of video frames included in the block as a feature amount. Also, the similarity is
It is preferable to determine it based on the cumulative cumulative AND of the histograms of the adjacent blocks. further,
When merging adjacent blocks in the block merging step, it is preferable to perform a histogram AND operation on the histograms of the adjacent blocks, and use the histogram obtained as a result of the calculation as the feature amount of the merged block.

A video block classification device of the present invention is a video block classification device for classifying a video composed of an image data sequence into a plurality of video blocks, and an image data sequence memory for storing an input image data sequence. , An image information conversion unit that reads frame data from the image data string memory and calculates a frame feature amount for each frame, a shot unit classification unit that reads frames from the image data string memory, and classifies them into shot units, and a frame feature amount A shot feature amount calculation unit that calculates the feature amount of each shot based on, and for a block composed of one or a plurality of shots, the similarity between adjacent blocks is calculated using the feature amount for the block, By merging adjacent blocks showing the highest similarity among the calculated similarities into one block It generates a block including the number of shots, having a similarity assessment and an image classifying unit repeatedly executes the merging of similarity calculation and the block.

In the video block classification device of the present invention, the frame feature amount and the block feature amount are represented by a histogram, and the feature amount of each shot is calculated by calculating the histogram logical product of the frame feature amount for each shot, Similarity is calculated based on the histogram cumulative logical product of the histograms of the blocks to be processed, and the similarity evaluation and image classification processing unit performs histogram logical product operation on the histograms of the adjacent blocks when merging the adjacent blocks. It is preferable to use the histogram obtained as a result as the feature amount of the blocks after merging.

In the end, the video block classification method and apparatus of the present invention repeat the process of merging adjacent image blocks having similar feature amounts, typically, similar color information, from the one having the highest degree of similarity. are doing. Therefore, it is possible to extract the image blocks that satisfy the condition of the scene such as the feature similarity condition and the temporal continuity condition and are a set of similar images and that are temporally continuous. Further, the classified video blocks are displayed / provided to the user in various forms, and the interface for receiving the input from the user is enhanced, so that the user can understand the video contents.

[0025]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, the present invention will be described in more detail by explaining the embodiments of the present invention with reference to the drawings.

In this embodiment, for an image divided into a large number of blocks on a shot-by-shot basis, for each block, a histogram representing the features of the image of that block is obtained, and based on the histogram, two blocks temporally adjacent to each other are obtained. Is calculated, and two adjacent blocks having the maximum similarity are merged into one block. By repeating this process, the number of blocks is gradually reduced. It is determined whether sufficient classification has been performed, and as a result, the original video is classified into a scene when sufficient classification has been performed.

The histogram is obtained, for example, by extracting characteristic quantities such as color, hue, saturation, and lightness from image data and arranging the characteristic quantities in the order of components. The frequency distribution obtained by orthogonally transforming the spatial distribution such as color and brightness in the image data may be used as the histogram as it is. The histogram representing the image of the block is obtained by obtaining the histogram of each frame belonging to the block and then calculating the histogram logical product of the histograms of all the frames. As will be apparent from the description given below, the operation of the histogram logical product finds the minimum value of the value of the component in the histogram that is the target of the logical product for each component in the histogram, and arranges the minimum value for each component. Performed by generating a histogram. Therefore, when merging blocks,
The histogram of the blocks after merging can be obtained by calculating the histogram logical product of the histograms of the blocks to be merged to obtain the histogram. As the similarity between adjacent blocks, the area of the histogram obtained by calculating the histogram logical product of the histograms of the adjacent blocks, that is, the histogram cumulative logical product described later is used.

FIG. 1 is a block diagram showing the configuration of a video block classifying apparatus according to an embodiment of the present invention. This video block classifying device classifies the input image data sequence 10 into scenes and displays it as a video 16 classified into the scenes on the video scene display unit 17, which is input from the user through the user interface unit 18. By request,
It is possible to adjust the degree of classification (such as the number of scenes to be finally classified).

The image sample rate, the image data format, and the image size in the input image data sequence 10 are arbitrary. For example, the input image data string 10 is NTS
The C standard video signal may be sampled at 30 frames / second, or may be sampled at a sampling rate coarser than that. Further, the input image data string 10 may be an analog signal such as NTSC or a digital signal, and may be data directly input from a video camera or the like and stored in a storage device such as a hard disk or a CD-ROM. It may be a saved image file. In the example shown in FIG. 1, the input image data sequence 10 is an NTSC video signal composed of t + 1 frames I ₀ , I ₁ , ..., I _t .

An image data string memory 11 for storing the input image data string 10 in units of frames is provided. The image data string memory 11 may store not only the input image data string but also the data processed to some extent, the shot, the photographer's name,
Additional information such as a location at the time of shooting, information obtained as a result of the processing described below in the present embodiment, and the like may be stored at the same time. Here, it is assumed that the original signal of the input image data string is stored in the image data string memory 11.

A shot unit classification unit 12 and an image information conversion unit 13 to which an input image data string read from the image data string memory 11 is input are provided. Each shot classification unit 12 executes the shot unit classification process, each frame I _0, I _1, ..., shot S ₀ of the input image data (n + 1) based on data I _t (provided that n <t), I ₁ , ..., S _n
It is classified into. The shot unit classification processing may use shot information added in advance to the video signal, or may classify the shots by using an existing cut point detection technique. Alternatively, a person may classify the shots in advance. In the present embodiment, the existing cut point detection technique is used for classification into shots. It should be noted that since a block configured by one or a plurality of continuous frames is generally called a block, both a shot and a scene are one type of block.

The image information conversion unit 13 converts the image information of each frame into information such as color, hue, saturation, lightness, etc., and generates frame feature quantities H ₀ , H ₁ , ..., H _t. is there. The frame feature amount is represented as a histogram based on information such as color, hue, saturation, and lightness. Color, hue, saturation,
You may convert into information other than information, such as brightness. Here, all of the image frames I ₀ , I ₁ , ..., I _t are each converted into RGB information, and the frame feature amounts H ₀ , H ₁ , ..., H _t are output as an RGB histogram.

A shot characteristic amount calculating section 14 is provided, and the shot characteristic amount calculating section 14 includes information on shots S ₀ , S ₁ , ..., S _n classified by the shot unit classifying section 12 and an image information converting section. 13, the frame feature values H ₀ , H ₁ ,
, H _t , the histogram AND of all frames in each shot S ₀ , S ₁ , ..., S _n is calculated, and the feature amount of each shot, that is, the shot feature amount S
H ₀ , SH ₁ , ..., SH _n are calculated. The calculated shot feature quantities SH ₀ , SH ₁ , ..., SH _n are input to the similarity evaluation and image classification processing unit 15. The similarity evaluation and image classification processing unit 15 determines the feature amount of each block represented as a histogram (for the feature amount of a block, the shot feature amount S
Based on H ₀ , SH ₁ , ..., SH _n, and the feature amount of the block in which the shots are merged), the adjacent blocks (the blocks here also include the shots) The similarity is calculated and evaluated, and the adjacent combination with the maximum similarity is merged to form a new block. Similarity evaluation and image classification processing unit 15
When a new block is generated by merging blocks or shots, calculates the feature amount (histogram) of the merged block based on the histogram logical product of the feature amounts of the merged blocks or shots. Actually, according to the instruction from the user interface 18,
The similarity evaluation and image classification processing unit 15 performs this evaluation, merging,
The process of calculating the characteristic amount is repeatedly executed, and the video 16 classified into scenes is output to the video scene display unit 17. It should be noted that the user inputs a request for displaying the scene 16 displayed in the video scene display unit 17 in detail in the time direction from the user interface unit 18 by inputting a request from the user interface unit 18. It can be reflected in the image.

The shot unit classification unit 12, the image information conversion unit 13, the shot feature amount calculation unit 14, and the similarity evaluation and image classification processing unit 15 may realize the processing by software using a CPU having a calculation capability. However, it may be realized by a combination of a plurality of CPUs and software, a part thereof may be realized by dedicated hardware, or the whole may be realized by using dedicated hardware. Here, the processing is realized by software using a CPU having an arithmetic capability.

Next, the calculation of the shot characteristic amount will be described with reference to FIG. In the image processing conversion unit 13, color data (RGB data) is input from the data of each frame (step 101), and histograms H ₀ , H ₁ , ..., H _t of the frame feature amount are generated for each frame ( Step 1
02), and inputs it to the shot feature amount calculation unit 14. on the other hand,
Information regarding which frame belongs to each shot S ₀ , S ₁ , ..., S _n is also input from the shot unit classification unit 12 to the shot feature amount calculation unit 14. Therefore, the shot feature amount calculation unit 14 calculates, for each shot S ₀ , S ₁ , ..., S _n , a histogram logical product of frame feature amounts (histograms) included in the shot (step 103), and calculates a histogram logical product. .., SH _n are output as shot feature quantities SH ₀ , SH ₁ , ..., SH _n for each shot (step 104).

Here, details of the calculation of the histogram logical product will be specifically described. As described above, the calculation of the histogram logical product is not limited to the calculation of the shot feature amount,
It is used to calculate the degree of similarity and the feature amount of the merged blocks.

First, H_1, H_2, H_3, ... Are each histograms, and H_1 (j), H_2 (j), H_3 (j), ... Are each histograms H_1, H_2, H_3 in the component j.
The value of ... Further, the histogram obtained as a result of the calculation of the histogram logical product is set to H_new, and the value for the component j in this new histogram H_new is H_new.
(j).

When the component j of each histogram exists from 0 to m and the histogram for calculating the histogram logical product exists from 1 to k, the operation of the histogram logical product is

[0039]

## EQU1 ## H_new (j) = min {H_1 (j), H_2
(j), ..., H_k (j)} (where 0 ≦ j ≦ m).

FIG. 3 shows how, when histograms corresponding to two adjacent blocks (blocks A and B) are given as shown in FIGS. Is a diagram illustrating whether a histogram logical product between B and B is calculated. The figure (c) shows the histogram logical product between the blocks A and B. That is, for each characteristic component, the lower value among the values in each block is adopted in the histogram of the result of the histogram AND operation. For example, as for the component K, as shown by the solid arrow in the figure, the value of the block B is smaller, so the value of the component K in the histogram AND is equal to the value of the block B. Similarly, for the component L, the value of the block A is adopted.

Next, the similarity evaluation and image classification processing unit 15
The processing in step 4 will be described with reference to FIG. The process of the similarity evaluation and image classification processing unit 15 is simply a process of merging blocks including shots, and in the process of merging, feature amounts of adjacent image blocks are compared to calculate a similarity. Then, the video is merged from the small blocks to the large blocks based on the similarity, and finally the video is classified into blocks corresponding to a scene, for example.

At the initial stage of the similarity evaluation and merge processing in the image classification processing section 15, each block should be composed of one shot. Therefore, first, the shot from the shot feature quantity calculating unit _{14 S 0, S 1, ...} , shot feature amount SH _0, SH ₁ of S _n, ..., and block to be processed in these enter the SH _n here Yes (Step 11
1). Next, the similarity between two adjacent blocks is calculated based on the cumulative cumulative AND of the histograms (step 1
12). Histograms indicating the feature amounts of two adjacent blocks (including shots) are H_1 and H_2, respectively, and the values of the component j in the histograms H_1 and H_2 are H_1 (j) and H_2 (j), respectively. In addition, the cumulative histogram AND (that is, the degree of similarity) is V. Assuming that the component j of the histogram is 0 to m, the histogram cumulative logical product V is

[0043]

[Equation 2] (Where 0 ≦ j ≦ m).

FIG. 5 shows how, when histograms corresponding to two adjacent blocks (blocks A and B) are given as shown in FIGS. , B is a diagram illustrating whether the histogram cumulative logical product is calculated. Similar to the calculation of the histogram logical product, the values in the histogram are compared for each component, the lower value is set as the value for that component, and the cumulative value is obtained. That is, the histogram cumulative logical product V is the area of the shaded area in FIG. 6C showing the histogram logical product between the two. Since the area is obtained by the cumulative sum of each component, it is called a histogram cumulative logical product here.

When the area of the histogram logical product between two adjacent blocks is large, that is, when the histogram cumulative logical product is large, it is reflected that the values for the same component are large between the two adjacent blocks. It can be said that the images are similar at each component level in the feature amount. Therefore, it is possible to evaluate the degree of similarity between adjacent image blocks by the histogram cumulative logical product.

As described above, the higher the value of the similarity based on the cumulative cumulative AND of the histograms, the higher the degree of adjacency between two adjacent blocks. Therefore,
The maximum similarity among the similarities obtained between the adjacent blocks is searched, and the two adjacent blocks showing the maximum similarity are merged into one block (step 113). Then, the feature amount of the image block newly generated by the merging is calculated (step 114). Here 2 before the merge
The histogram logical product of the blocks is used as the feature amount of the blocks after the merge. After that, it is determined whether or not the video classification has proceeded sufficiently (step 115), and if the video classification is sufficient, the video classified into the scene 16 is output to the video scene display unit 17 and the processing ends. , If not enough,
Returning to step 112, the similarity between adjacent blocks is calculated for the merged blocks. Note that the similarity between unmerged blocks does not change,
When Step 112 is executed for the second time and thereafter, only the similarity between the block generated immediately before and the block adjacent to this block needs to be calculated.

Whether or not the classification is sufficiently performed can be determined by referring to the similarity between blocks. In this embodiment, the histogram that is the feature amount of a block is calculated by the logical product of the histograms that are the feature amounts of smaller blocks that form the block. Therefore, the block obtained after the merging is repeated is composed of a large number of frames or shots, while the value of the histogram, which is the feature amount, is small. Therefore, the value of the logical product of the histograms showing the similarity between the huge blocks is very small because the values of both histograms are very small.
Only a very small value. Using this fact, when the value of the cumulative cumulative AND of the histogram is 0, it is regarded as a completely dissimilar state, that is, the image is sufficiently classified, and when there is a value of the cumulative cumulative AND of the histogram which is not zero, It is possible that the classification process is to be continued because the classification is not sufficiently performed. In addition, since this embodiment is realized only by simple iterative work, the classification level of the scene can be adjusted in various ways. For example, it is possible to change the classification level according to a request input by the user via the user interface unit 18 and change the number of blocks finally classified, and the classification can be performed based on the input information from the user. You can also determine if it was done well.

The process in which shots are merged and classified into larger blocks in this embodiment will be described in more detail below with reference to FIG.

(A) shows five shots A to E classified by the shot unit classification unit 12. From these shots A to E, classification is performed in scene units, which is a classification unit suitable for the user. The shot feature amount (histogram) for each of the shots A to E calculated by the shot feature amount calculation unit 14 is shown in (b). The shot feature amount is obtained by calculating the histogram logical product from the histogram (feature amount) of each frame included in the shot.

(C) shows the process of calculating the degree of similarity between adjacent shots by showing the logical product of histograms between adjacent shots. Symbol in the figure

[0051]

[Outside 1] Indicates the operation of histogram AND. In the actual evaluation of the similarity, the area of the histogram logical product, that is, the histogram cumulative logical product is used. As a result of the evaluation of the degree of similarity, the combination of shot D and shot E showed the maximum degree of similarity, so as shown in (d), shot D and shot E are merged to obtain a merged block called DE.

In this embodiment, since the processing is performed until the video is sufficiently classified into the scenes, the processing is further performed on the shot group including the merged block DE. The feature quantity for each shot and merged block at this point is
It is shown in (e). Then, as shown in (f), the similarity between adjacent shots or blocks is calculated by histogram cumulative AND. actually

[0053]

(Equation 3) Uses the previous calculated value as it is,

[0054]

(Equation 4) Only newly calculate. As a result, the similarity between the shot C and the block DE is maximized, so that the merged block CDE is generated as shown in (g). The same process is repeated thereafter, and for example, the process of calculating the feature amount, calculating the degree of similarity, and determining the merged block may be repeatedly executed until the scene is completely classified.

The embodiment of the present invention has been described above. Here, one evaluation is performed once in the similarity evaluation.
Only merged once. Therefore, for example, when there is a request to view a video block in more detail via the user interface unit 18 or, for example, "I want a video in which the video is classified into n pieces", the request is instantly issued. It is possible to correspond to. That is, in the example of FIG. 6, when there is a request to classify the video composed of the five shots A to E of (a) into four, and to view the scene, the state of four blocks is set when the scene is obtained. It suffices to present the state shown in FIG. 3D, that is, the video blocks classified as A, B, and CDE.

In the above embodiment, an example has been described in which the similarity is calculated based on the cumulative AND of the histograms and adjacent blocks having the maximum similarity are merged.
Various other methods can be considered for the similarity evaluation and the merge rule. Also, the case where the block in the initial state is a shot has been described, but in the case of classifying images within one shot or one take, the above-described processing can be performed with the block in the initial state as a frame. Is.

Further, as an application of the present invention, a database browsing interface, various video processing such as production of a video context, and application to a user interface can be cited.

[0058]

As described above, according to the present invention, it is possible to repeatedly perform the processing of calculating the similarity between adjacent shots or image blocks, evaluating, and merging images classified into shots or block units. As a result, similar and temporally continuous image blocks can be concentrated and collected, and there is an effect that video classification that satisfies the feature similarity condition and the temporally continuous condition can be realized. Therefore, it becomes possible to classify the images classified into a plurality of subdivided shot units into scene units.

Further, since the present invention realizes image classification by simple iterative processing, it is possible to adjust the classification level in various ways, and it is possible to provide images classified into scenes seamlessly.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a video block classification device according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a process of calculating a shot feature amount.

FIG. 3 is a schematic diagram showing a method of calculating a histogram logical product.

FIG. 4 is a flowchart illustrating processing performed by a similarity evaluation and image classification processing unit.

FIG. 5 is a schematic diagram illustrating a method of calculating a histogram cumulative logical product.

FIG. 6 is a schematic diagram showing a process in which shots are collected into blocks.

[Explanation of symbols]

10 input image data sequence 11 image data sequence memory 12 shot unit classification unit 13 image information conversion unit 14 shot feature amount calculation unit 15 similarity evaluation and image classification processing unit 16 video classified into scenes 17 video scene display unit 18 user interface part 101～104,111～116 step A~E shot _{H 0, H 1, ...,} H t frame feature value _{I 0, I 1, ...,} I t frame _{S 0, S 1, ...,} S n shot SH ₀ , SH ₁ , ..., SH _n Shot feature quantity

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Yukinobu Taniguchi 1-1-6 Uchisaiwaicho, Chiyoda-ku, Tokyo Nihon Telegraph and Telephone Corporation

Claims (7)

[Claims] 1. A video block classification method for classifying a video composed of an image data string into a plurality of video blocks, the video input step of inputting a video classified into a large number of blocks in advance, and the image of each block. The feature amount calculation step of calculating the feature amount of the block from the data string, the similarity degree calculation step of calculating the similarity degree between adjacent blocks based on the feature amount, and the maximum similarity degree among the calculated similarity degrees. And a block merging step of merging adjacent blocks into one block, the plurality of videos classified by performing the feature amount calculating step, the similarity calculating step, and the block merging step repeatedly. Video block classification method for obtaining blocks. 2. The video block classification method according to claim 1, wherein the video input step includes a step of classifying a video input in frame units into shots which are one type of block. 3. A feature amount of each block is a histogram obtained by extracting information consisting of any one of color, hue, saturation, lightness or a combination thereof from a frame included in the block, The video block classification method according to claim 1 or 2, wherein when the number of included frames is 2 or more, it is represented as a histogram obtained by performing a histogram AND operation on all histograms of the frames included in the block. 4. The video block classification method according to claim 3, wherein the similarity is determined based on histogram cumulative logical product of histograms of adjacent blocks. 5. When merging adjacent blocks in the block merging step, a histogram AND operation is performed on the histograms of the adjacent blocks, and the histogram obtained as a result of the calculation is used as the feature amount of the blocks after merging. The video block classification method according to Item 3 or 4. 6. A video block classification device for classifying a video composed of an image data sequence into a plurality of video blocks, the image data sequence memory storing the input image data sequence, and the frame from the image data sequence memory. Read the data of
An image information conversion unit that calculates the frame feature amount for each frame, a shot unit classification unit that reads the frames from the image data string memory and classifies the shot units, and a shot that calculates the feature amount of each shot based on the frame feature amount. For the feature amount calculation unit and the block composed of one or more shots,
The similarity between adjacent blocks is calculated using the feature amount for the block, and the adjacent block showing the maximum similarity among the calculated similarities is merged into one block to obtain a plurality of shots. A video block classifying device having a similarity evaluation and image classification processing unit that generates configured blocks, and repeatedly executes similarity calculation and block merging. 7. A frame feature amount and a block feature amount are each represented by a histogram, and the feature amount of each shot is calculated by calculating a histogram AND of the frame feature amount for each shot, and the histogram of adjacent blocks is calculated. The similarity is calculated on the basis of the cumulative cumulative AND of the histograms, and the similarity evaluation and image classification processing unit performs a histogram logical AND operation on the histograms of the adjacent blocks when merging the adjacent blocks to obtain the result of the operation. 7. The video block classifying apparatus according to claim 6, wherein the histogram is used as a feature amount of the blocks after being merged.