JPH11252501A - Device for detecting special effect of moving image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a method to detect correctly only a wipe while distinguishing it from a moving camera and a motion of an object in a video image. SOLUTION: A processing unit that is provided with an image buffer means that stores an object moving image by (m) preceding frames at all times, a means that calculates a characteristic amount including a histogram of input frames, a means that obtains 1st correlation between the characteristic amount and a characteristic amount of a preceding frame by (m), and a means that obtains a 2nd correlation of a statistic between obtained an entire color or luminance before and after a change in a pixel whose color or luminance is changed by m-preceding frames has a means that discriminates it as a change of scenes due to a wipe when the 1st correlation and the 2nd correlation of images received in time series in the unit of frames exceed a predetermined permissible range respectively and extracts a period including the above or its vicinity. A scene change by special video effects including a wipe in addition to a conventional dissolve is detected by this method while reducing the effect of a moving camera and a large motion of an object, than important scenes received before and after the special effect are extracted with high accuracy.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a quick-view method for quickly grasping a moving image such as a video or a movie in a short time, and more particularly to a method for cutting a moving image stored on a video tape or a video disk (one unit). Wipe (continuous cuts A and B) between the continuous moving image sections photographed by the camera of (1), the area occupied by the image (A) on the screen at the transition between the cuts and the image (B) occupy The present invention relates to a moving image special effect detection device that detects a special video effect whose area gradually increases.

[0002]

2. Description of the Related Art In recent years, satellite broadcasting, cable television, and the like, in addition to ordinary television broadcasting, have become widespread, and the number of broadcasting channels has been increased. In the future, if a broadband communication infrastructure called an information highway is prepared, it will be easier to distribute broadcasts, and more broadcasters will participate than in the current situation, which will accelerate the increase in the number of channels.

[0003] From such a large amount of broadcast information,
Distinguishing and selecting useful and useless information for an individual viewer is a very laborious and time-consuming task. For this reason, research on technologies for efficiently creating summary information (digests) for quickly grasping the contents of video has been advanced. The most basic and indispensable process for creating a digest is to select important scenes from the video. Various methods have already been proposed to select such important situations.
For example, Japanese Patent Application Laid-Open No. 9-65287 by the inventors describes a method that focuses on a combination of various event points in a video, such as a transition between cuts and the presence or absence of a person or subtitles. In particular, special effects such as dissolves and wipes are given by the creator of the image with an expectation of some effects, and therefore, there is a high possibility that there is an important scene near the given effect. In fact, in sports programs, it is widely used at the time of switching from replay of scenes, and replayed scenes are clearly important scenes such as score scenes and good play scenes.

[0004]

There are two types of special effects, namely, a dissolve and a wipe, but both have a common property that the cut gradually changes over time, and sampling is performed with a certain time interval. By comparing two frame images (one frame image constituting a moving image), it is possible to detect the difference based on a large difference. However, such a method not only makes it difficult to distinguish between dissolves and wipes, but also detects changes in the image due to camera movements such as panning and zooming. There is. Therefore, regarding the former dissolve, a detection method has been disclosed by the inventors and others, in which excessive detection is suppressed by using a feature unique to the dissolve. However, regarding the latter wipe, ARUN HAMPAPUR, RAMESH JAIN AND TERRY E
WEYMOUTH, “Production Model Based Digital VideoSeg
mentation ”, Multimedia Tools and Applications, Vo
l.1, pp. 9-46 (1995) proposes a wipe-specific detection method for switching between ideal scenes with no motion, but a stable detection method considering motion Does not exist.

An object of the present invention is to realize a technique for correctly detecting only a wipe, in distinction from the movement of a camera or the movement of an object in an image.

[0006]

In order to achieve the above object, a target moving image is input to a processing device in a time series on a frame basis, and the processing device always stores frame images for the past m frames. Image buffer means for performing
Means for calculating a feature amount including a histogram of an input frame; means for calculating a first correlation between the feature amount and the feature amount in a frame past m frames; Means for calculating a second correlation between a statistic obtained from the entire color or luminance of the pixel before the change and a statistic obtained from the entire color or luminance of the pixel after the change, When the first and second correlations are out of the predetermined allowable ranges, it is determined that there is a change in the scene due to the wipe, and there is provided means for extracting a section including the above-mentioned time point or its vicinity. .

[0007] If necessary, the second correlation is set to m
A means for storing frames, a means for determining whether or not a scene change due to a wipe is made based on a sum of correlations for m frames, a cut point is detected, and if a cut point has occurred in the past m frames, the cut point is obtained by that time. Means for returning the correlation to the initial value, means for not changing the scene by wiping if the number of pixels whose color or luminance has changed monotonically during the past m frames does not change, In the case of changing to a short frame interval and returning to the original value, the change is not regarded as a change. In addition to the above-described first correlation, the feature amount between the feature amount in the input frame and the feature amount in the m + δ frames past frame is added. And a means for re-determining the higher correlation as the first correlation is provided.

According to the present invention, in addition to the conventional dissolve, a change in scene due to a special video effect including a wipe can be detected with a small effect of a camera motion or a large motion of an object. It is possible to accurately extract important scenes that are successively played.

[0009]

FIG. 1 is an example of a schematic block diagram of a system configuration for realizing the present invention. 1 is CR
A display device such as T, which displays an output screen of the computer 4. Commands to the computer 4 can be performed using an input device 5 such as a keyboard or a pointing device. The moving image reproducing device 10 is a tuner device for receiving a broadcast program such as terrestrial broadcasting, satellite broadcasting, and cable television, or a device for reproducing moving images recorded on an optical disk, a video tape, or the like.

A video signal output from a moving image reproducing apparatus is sequentially converted into digital image data by three A / D converters and sent to a computer. Inside the computer, the digital image data enters the memory 9 via the interface 8 and is processed by the CPU 7 according to the program stored in the memory 9. When a number (frame number) is assigned to each frame of the moving image handled by the moving image 10 in order from the beginning of the moving image, the frame number is transmitted to the moving image reproducing apparatus through the control line 2,
The moving image of the scene can be called and reproduced.
Further, various kinds of information can be stored in the external information storage device 6 as necessary for processing. The memory 9 stores various data created by the processing described below, and is referred to as needed.

Hereinafter, a method for detecting a wipe which is one of the cut changes due to the special video effect will be described in detail.

FIG. 2 is an example of a flowchart of a moving image wipe detection program executed on the system shown in FIG. The program is stored in the memory 9 and the CP
First, as initialization processing, U7 sets various variables necessary for the execution of the program to initial values (200), and enters a main loop (202 to 260). The main loop is repeated until the end of processing is instructed.

In the main loop, first, in a process 202, a frame image f (n) output from the moving image reproducing apparatus 10 is captured. Here, n is n input from the start of processing.
This is a counter for indicating the meaning of the フ レ ー ム th frame, and is incremented by one at 260.

In the processes 204 to 258, a characteristic characteristic of the wipe is detected. Here, the wipe is
As shown in B, before and after the change of cut, like B
This is a cut change having a section in which part of the frame images A and C of the preceding and following cuts are mixed with each other. A and C in B
The area ratio of A at the start of the wipe is 100% and C is 0%.
From the state described above, the ratio reverses over time, and when the A finally becomes 0% and the C becomes 100%, the wipe is completed. In the case of the wipe in this example, it can be seen that it is difficult to distinguish from the so-called pan, in which the image capturing camera moves sideways in scene A, and the screen changes, and the state of the movement. Therefore, the following three points are considered as characteristics unique to the wipe that are not affected by movement.

1) An image portion newly appearing in the wipe section has low similarity to the previous image.

2) The color difference is large at the boundary between the image part newly appearing in the wipe section and the image up to that point.

3) The similarity of the frame images at the start and end of the wipe section is low.

Regarding 1), most landscapes in the natural world are smoothly continuous in color, and the image part newly entering the screen by the movement of the camera is:
Very similar in color to the previous image. Even in scenes where linear artifacts such as signboards and buildings newly enter, the area ratio of these to the image part newly entering the screen is often small, and the colors are statistically very similar It will be. However, the image part newly entered by the wipe is originally a part of a different scene inserted, and thus the similarity of the image is clearly low.

Regarding 2), the boundary between the image part newly inserted by the wipe and the image to be inserted (in the frame image B in FIG. 3, a vertical bar separating the image of a person and a car) In the vicinity), the difference in color (or luminance) between pixels adjacent to each other across the boundary becomes very large.
Such an artificial clear tomographic slice is unlikely to exist in a natural image for the same reason as in 1), and thus is a major feature indicating that a wipe has occurred. By focusing on these two points, it is possible to distinguish them from changes due to camera movement.

The last 3) is a feature for distinguishing a large movement of a subject in a video from a wipe. The subject has a clear boundary with its background, and if it moves greatly, the background hidden behind the subject may newly appear. Fulfills the characteristics of However, since the movement of the subject is in the same scene, the statistical color of the entire screen does not show a large difference before and after the movement,
Thereby, the movement of the subject and the wipe can be distinguished. However, before and after the movement, there is a change in the composition and background such as the movement of the subject, so that it is necessary to compare the entire screen by comparing statistical feature amounts independent of the position of the image.

Although there are various patterns in the wipe depending on the position where the image is inserted and the style of transition, the above three features are universally established in any case.

Next, the wipe detection algorithm will be described more specifically with reference to FIG. First, processing 204
Then, the histogram H (n) of the frame image input in 202 is calculated. Here, n represents the frame number as described above, and means that a histogram is created corresponding to each frame. The histogram is 1
This is the frequency of appearance of pixels having the same color in the entire frame image.

For example, when a color histogram of 64 colors of 2 bits each of RGB is created, the RGB color value of each pixel of the frame is reduced to 6 bits 64 colors expressed only by the upper 2 bits of each of RGB, and each of the 64 colors is reduced. For, the number of pixels indicating the color is counted by degeneration. In this case, the color histogram can be represented by an array H (n, i), and i takes a value from 0 to 63. For example, the frequency H (n, 0) when i is 0 indicates the number of pixels in the frame where the upper two bits of the RGB color value are all 0 for each of RGB.

As described above, since the histogram does not reflect the distribution state of colors on the screen, the histogram has a characteristic that it takes a stable value even if there is movement in the screen or a partial change. The histogram calculated here is, as described later, the above 3)
Used to detect the features of

In subsequent steps 206 to 210, first, a buffer B (n) for storing the distribution state of the newly entered image area is initialized to zero. B (n) is a two-dimensional array having the same size w × h as the input frame image f (n), and the cell corresponding to the coordinates (x, y) on the frame image is B (n, x, y). ). B (n,
x, y) takes a value of 0 or 1, and when 1, indicates that the pixel at the coordinates (x, y) corresponding to the cell constitutes the newly entered image area.

In steps 212 to 220, each pixel of the latest frame image f (n) is checked one by one, and the same coordinates (x, y) in the frame image f (n-1) input immediately before are checked. The difference in color (or luminance) from the located pixel is equal to or greater than the threshold th1 and is close to 4 on f (n), that is, (x−δ, y), (x + δ, y),
If there is one or more pixels whose color (or luminance) difference is equal to or larger than the threshold th2 among the pixels located at (x, y-δ) and (x, y + δ), the coordinates (x, y) Is assigned to B (n, x, y) corresponding to.

At this time, when the threshold value is determined not only by the luminance but also by the color difference, for example, it is determined whether or not even one luminance difference exceeds th1 (or th2) for each of the RGB components. I do. Also, the condition that there is a difference between the pixels located in the vicinity of 4, that is, the condition that the pixel is an edge is that the number of changed pixels that are not related to the edge is suppressed. The characteristic of 2), that is, the relative ratio of the pixels constituting the tomogram (these naturally become edges) that occur at the region boundary between the previous image and the inserted image to the whole is increased, and the characteristics of the wipe are more likely to become apparent. That's why. In this case, since the edge is simply obtained, it is not necessary to be limited to the vicinity of four,
It does not matter even if it is checked in the vicinity.

After the bit map B (n) consisting of 0 and 1 is created as described above, the B
(N) is ORed by a bit operation, and a bit map B0 representing the result is obtained (222 to 23).
6). This B0 represents the entire image area that may have been inserted during m frames. Here, m indicates a search range in the processing for detecting the wipe, and when a value of approximately 8 to 15 is obtained, a good experimental result is obtained. The buffer capacity for storing various information related to each frame image may be m frames.

In the above processing, B0 was not obtained simply from the change between the two frame images separated by m frames. This is because the color of a region far away moves and the number of pixels largely changing due to the movement increases. When compared between consecutive frames as in the present embodiment, since the width of motion is small, there are many changes within a region having the same color, and the number of changed pixels due to motion is statistically suppressed. In the case of a wipe, there are many pixels that change greatly even at short time intervals due to the principle that different images are inserted.

Next, the histogram H obtained in 204
(N) and a frame image f (n−
The difference D1 with the histogram H (nm) in m) is calculated (238). The degree of difference D1 can be determined by a calculation formula such as a chi-square test, for example.

Thus, it is checked whether or not there is a possibility that there is a wipe in the section of the past m frames. As described in the feature 3), since the wipe is a change of the cut, the fact that the wipe causes a large change in color is used. At this time, if the degree of difference D1 is greater than the threshold th3 (240), it is determined that there is a high possibility of a wipe, and the processing shifts to processing for checking whether or not the above-mentioned feature 1) appears.

First, in the process 242, for the latest frame image f (n), only the pixels satisfying B0 (x, y) = 1 are sampled to create a histogram Hc. The creation method is basically the same as that of H (n), except that the pixels for creating the histogram are limited by B0. In the following 244, the frame image f (n-
For m), similarly, only the pixels satisfying B0 (x, y) = 1 are sampled to create the histogram Hc '. Then, the difference D2 (n) between Hc and Hc 'is calculated (246). As described above, B0 is a bitmap representing an area newly appearing during the past m frames, so that D2 (n) is an image area newly entered,
The degree of difference from the previous image area is shown. D2
When (n) is equal to or larger than the threshold th4 (248), the past m
The sum of D2 for the frames is obtained to obtain an evaluation value eval (2
50-254). If eval is greater than the threshold th5 (256), it is determined that the eval is in the wipe section and the detection process 258 is performed.
I do.

The wipe detection process 258 is an application-like process in which the detection of a wipe is used as a trigger to specify and store a replay scene. here,
The sum of D2 for the past m frames is obtained because the value of D2 often takes a large value continuously in the wipe section, and conversely, in the case of a change due to motion, it is often fragmentary. This is because the sum can be more easily distinguished. If it is desired to increase the sensitivity even if excessive detection is permitted, steps 250 to 254 are unnecessary. FIG. 4 is a schematic diagram showing the basic concept of the algorithm described above.

Although the wipe can be detected by the above algorithm, some conditions can be added in order to further improve the detection accuracy.

Since the wipe is a video effect in which the screen gradually changes and finally the entire screen is switched, if the state is followed over a certain period of time, the boundary between the previous image and the inserted image becomes The trajectory moves as the wipe progresses so as to cover the entire screen. In the wipe section, the change area indicated by B0 includes the trajectory of the boundary line for a certain time. The trajectory is only partially at the beginning of the wipe,
As the wipe spreads over the entire screen as the wipe progresses,
The area of the change region indicated by B0 tends to monotonously increase in a wipe section. By making this a condition,
Wiping can be further limited. This can be done by storing the total number of 1s in B0 for a fixed time and checking whether the total number monotonically increases.

In addition, the algorithm shown in FIG. 2 has a problem in that, even if there are cut points in the past m frames, the algorithm is erroneously detected as a wipe. To make a clear distinction between cut points and wipes,
The cut point is independently detected by another method, and when the cut point is detected, various variables for wipe detection are reset so as not to be detected as a wipe. For the cut detection in this case, a conventional method can be used as it is.

As a problematic image change similar to the cut point, instantaneous noise such as flash is generated. The flash is a phenomenon in which the entire screen or a part thereof becomes white for a very short time of a maximum of several frames. The flash over the entire screen is performed once when the screen becomes white and once when the original image is returned. Because the number of cut points is continuous at extremely short time intervals,
It can be detected from its features. When a flash is detected in this manner, the flash is reset as in the case of the above-described cut point, and is not detected as a wipe.

However, with such a method alone,
If a flash occurs on only part of the screen, it cannot be handled. In addition, the portion in which the flash has occurred is directly reflected in B (n) as a change area, which strongly affects the accuracy of wipe detection.

Therefore, when B (n) is obtained, in addition to the conditional statements of the processes 216 and 218 in FIG. 2, a pixel (x, y) having a high possibility of being a flash is set as a change area. Not accounted, that is, B (n, x, y)
= 0 is added. Pixels that are likely to be flash are the color (or luminance) at f (n−δ) and the color (or luminance) at f (n).
Is not less than the threshold th6, and the color (or luminance) at f (n−δ) and the color (or luminance) at f (n + δ) are not more than the threshold th7.

Another problem of the partial flash is that D1 in FIG. 2 becomes large and the processing 242 to 258 is performed unnecessarily on the assumption that there is a high possibility of a wipe, thereby increasing the risk of excessive detection. There is. Thus, H (n) and the frame f (nm) m frames before
In addition to the difference D1 between the histogram H (nm) and the histogram H (nm-δ) before m + δ frames.
Alternatively, the processing 240 may be performed by determining the degree of difference D1 ′ between H (n) and H (n) and setting the smaller one as D1. As a result, the correct dissimilarity D1 is obtained while avoiding the image disturbed by the flash.

By detecting a wipe by the method described above, a highlight scene of a sports broadcast can be easily found. At the time of switching from the replay scene, in many cases, the image is once in a still image state, and then the wipe is applied.Therefore, the still image state is also detected and the wipe is performed immediately after that. May be determined in combination with a switch from the replay scene. The detection of the still image state can be easily detected when a small number of states continue even if the change area B0 has all 0s or 1s for a while. As a result, the highlight scene can be detected with higher accuracy.

[0042]

According to the present invention, a special effect can be accurately detected without being affected by the movement of a camera or a subject, so that an important scene focused on the special effect can be selected, and the digest video can be automatically displayed. There is an effect that can be created with.
Further, even in a video handling system based on automatic indexing by detecting a cut point, it is possible to detect a wipe point, which has been difficult so far, so that a highly accurate index can be provided.

[Brief description of the drawings]

FIG. 1 is a block diagram of a system configuration for realizing an embodiment of the present invention.

FIG. 2 is a flowchart of a program for detecting a wipe.

FIG. 3 is an explanatory diagram illustrating the concept of a wipe.

FIG. 4 is an explanatory diagram showing a basic concept of an algorithm of the present invention.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 display, 2 control signal line, 3 A / D converter, 4 computer, 5 input device, 6 external information storage device, 7 CPU, 8 connection interface, 9 memory, 10 moving image Image reproducing device, 11 ... keyboard.

Claims (9)

[Claims] In a method for detecting a transition of a scene of a special video effect including a wipe from a moving image composed of a plurality of continuous images, a target moving image is input to a processing device in a time series in frame units. In the above processing apparatus, image buffer means for always storing frame images of the past m frames, means for calculating a feature amount including a histogram of an input frame, Means for obtaining a first correlation with the feature amount, statistics of pixels having significant changes in color or luminance during the past m frames, statistics obtained from the entire color or luminance of the pixel before the change, Means for obtaining a second correlation between the pixel and the statistic obtained from the entire luminance of the pixel after the change, wherein the first and second correlations are out of a predetermined allowable range. In time, and determined that the turn of scenes by wiping, the special effect detection device of the moving image and extracts the interval or its vicinity. 2. The apparatus according to claim 1, further comprising means for storing the second correlation for m frames, and means for determining whether or not a scene change due to a wipe is made based on the sum of the correlations for m frames. A special effect detecting device for a moving image. 3. The apparatus according to claim 1, further comprising means for detecting a cut point and, when the cut point has occurred in the past m frames, returning the correlation determined so far to an initial value. A special effect detection device for moving images. 4. The apparatus according to claim 1, wherein a pixel having a significant change in color or luminance during the past m frames includes a pixel having a large difference in color or luminance near the latest frame. A special effect detection device for a moving image, which is also a pixel to be processed. 5. The apparatus according to claim 1, wherein the number of pixels whose color or luminance has changed during the past m frames is:
Only if it increases monotonically within a certain section,
A special effect detecting apparatus for a moving image, comprising: means for determining that the section is a transition of a scene due to a wipe. 6. The apparatus according to claim 1, wherein a pixel having a significant change in color or luminance during the past m frames is defined as a pixel whose color or luminance changes during a short frame interval. A special effect detecting device for a moving image, wherein the pixel is not a pixel that returns to the original value. 7. The apparatus according to claim 1, wherein, in addition to the first correlation, a correlation between a characteristic amount in an input frame and a characteristic amount in an m + δ frames past frame is also determined, and a higher correlation is determined. A special effect detecting apparatus for a moving image, further comprising means for making a first correlation. 8. The apparatus according to claim 1, wherein when it is determined that a scene change due to a wipe has occurred, a frame image at the moment or in the vicinity is extracted and stored in the storage device as data representative of the nearby scene. A special effect detecting device for a moving image, characterized by comprising means for recording. 9. The apparatus according to claim 1, further comprising means for, when it is determined that a scene change has occurred due to a wipe, recording the address information of the moment or a nearby frame in a storage device. A special effect detection device for moving images.