JPH0846864A - Detection of video cutting point - Google Patents

Abstract

PURPOSE:To provide the method of detection of a video cutting point capable of reducing a detection error due to the motion of an object and hand shake in cut and having superior detection performance. CONSTITUTION:A video data string is constituted (101) in a space and time image that is the three-dimensional arrangement of image plane coordinates (x), (y) and a time (t), and the gradient vectors (DELTAxI, DELTAyI, DELTAtI) of the space and time image at a certain time (t) are calculated (102). The gradient vector is made nearly parallel with the axis (t) since the change of luminance and color is small in the directions of (x), (y) and large in the direction of axis (t) at a cutting point due to the steep change of a pattern. Therefore, a value representing the size of the number of gradient vectors set nearly parallel with the axis (t) is found from the count values count1, count2 of a picture element which satisfy a conditional equation by using equation of diff=count2/count1 (103-105). When diff exceeds a thresh-old value theta, this time is judged as the cutting point (106). Such operation is successively performed with respect to the time (t) (107).

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for detecting a cut point (point at which a scene changes) from a video, that is, a sequence of a plurality of image data.

[0002]

2. Description of the Related Art The point at which a scene changes in a video is called a cut point. The image cut point detection method is also called scene change detection, and various methods have been proposed.

As a typical method, a difference in luminance value between corresponding pixels of two temporally adjacent images I _t and I _t-1 is calculated, and the sum of absolute values thereof is defined as D (t), When D (t) is larger than a given threshold, some consider t as a cut point (Otsuji, Tonomura, Ohiwa: Moving image browsing using luminance information. IEICE Technical Report, IE9
0-103, 1911. ).

Pixel change area, luminance histogram difference, block-wise color correlation, χ ² test amount, etc. are used as D (t) instead of inter-frame difference (Otsuji, Tonomura: Study of automatic video cut detection method. Technical Report of the Society of John, Vo
l. 16, No. 43, pp. 7-12).

There is also a method of thresholding the result of the action of various temporal filters on D (t), instead of directly thresholding D (t). This method is characterized in that erroneous detection is unlikely to occur even if there is a moving object or flash light that moves rapidly in the image (K. Otsuji an.
d Y. Yonomura: ProjectionDe
tecting Filter for Video
Cut Detection. Proc. of AC
M Multimedia 93, 1993, pp. Two
51-257).

[0006]

In the conventional technique described above, when the subject moves or the entire screen moves due to camera shake, the amount of change described above increases, and the detection performance deteriorates. There was a problem to do.

This problem will be described with reference to FIGS. 7 and 8. Consider an image data string as shown in FIG. In this example, a white (brightness value = 255) quadrangle (height H, width W) is moved to the left by Δx on a black (brightness value = 0) background, and the color of the quadrangle at time t _c. Changes to gray (brightness value = 127). Here, the time t _c is the cut point. Variation D (t) is adjacent Here temporally two images I _t, is calculated as the sum of the absolute value of the difference between the luminance values in corresponding pixels of I _t-1. Therefore, time t (≠
The amount of change D (t) in t _c ) is obtained by multiplying the area of the displaced portion of the quadrangle in FIG. 7B by the difference 255 between the brightness values of black and white, so that D (t) = 255 * It becomes 2 * Δx * H. At the cut point t = t _c , since the difference between the brightness values of white and gray is 255-127 = 128, D (t _c )
= 128 * W * H. In the graphs of FIGS. 8A and 8B, the horizontal axis represents time t and the vertical axis represents the amount of change D (t).
When the moving speed Δx of the quadrangle is small, as shown in FIG. 8A, D (t) is sufficiently smaller than D (t _c ), so that the cut point can be correctly detected. However, the moving speed Δ
As x increases, as shown in FIG. 8B, D (t) rises overall, so that D (t _c ) and D (t _c )
It becomes difficult to distinguish (t). For this reason, in the conventional method, the place where the movement of the subject is strong may be mistaken for the cut point. The same situation occurs due to camera shake, which is one of the causes of the deterioration in the performance of cut point detection.

Further, in the conventional method of preventing detection error by applying various time filters instead of directly thresholding D (t), the continuity of the motion of the subject is assumed, so that the subject suddenly appears. In the case of discontinuous movement, a detection error still occurred.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a cut point detection method with good detection performance capable of reducing detection errors caused by movement of an object and camera shake of a cut in order to solve the problems of the above-mentioned conventional techniques. To provide.

[0010]

To achieve the above object, in the present invention, in a video cut point detection method for detecting a cut point from an image data string forming a video, image plane coordinates x, y and time t. Is regarded as a variable and a gradient vector is calculated from the image data sequence, and a step of outputting a time point at which a large number of gradient vectors close to the t-axis are output as a cut point.

Further, in the above method, the procedure of calculating a gradient vector from an image data string is performed by applying a smoothing filter to the image data string to calculate the gradient vector of the smoothed spatiotemporal image. Is preferable in that the influence of noise is eliminated and the detection performance is stabilized.

Further, in the above method, it is preferable to configure the smoothing filter with a digital filter in order to reduce the amount of calculation and facilitate detection.

[0013]

The gradient vector of a spatiotemporal image, in which the image data string forming the image is formed as a three-dimensional array of image plane coordinates x, y and time t, is x, y at the cut point in the case of a luminance change. Since the pattern is small in the direction and suddenly changes in the t-axis direction, it is close to parallel to the t-axis because it is large. On the other hand, in the shot between the cut points, each degree formed by the gradient vector and the t-axis is determined by the movement of the subject (if the subject is stationary, it is perpendicular to the t-axis), but in any case, , The gradient vector and the t-axis are rarely parallel. The image cut point detection method of the present invention utilizes the property that the gradient vector becomes nearly parallel at the cut point in this way, and sequentially calculates the gradient vector at each time, and t
The detection performance of the cut point is improved by detecting the time when there are many gradient vectors near the axis as the cut point.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below with reference to the drawings.

First, the principle of the present invention will be described with reference to FIG. According to the present invention, in the video cut point detection method of detecting the cut points from the image data sequence forming the video, the image plane coordinates x, y and the time t are regarded as variables to form a three-dimensional array. Such an image data string considered as a three-dimensional volume is called a spatiotemporal image. Gradient vector (Δ _x I, Δ _y I, Δ _t I) from the image data sequence of this spatiotemporal image
Is sequentially calculated at time t, and the time t in which the number of the gradient vectors close to the t-axis is large is output as a cut point. The present invention utilizes the property that the gradient vector becomes close to parallel to the t-axis at the cut point.

[0016] The present invention is the gradient vector of the space-time image _{(Δ x I, Δ y I} , Δ t I) as well as the magnitude of delta _t I of the difference between the spatial direction Δ _x I, Δ _y It is different from the conventional technique in that I is taken into consideration. _Δt I is a quantification of whether the luminance or the color changes before and after the time t when focusing on a certain pixel. In the conventional technique, only the difference in the time direction, that is, | _Δt I | not look, the difference Δ _x I spatial direction, for Δ _y I did not take into account.

The property that the gradient vector of the spatiotemporal image becomes nearly parallel to the t-axis at the cut point will be described with reference to FIG. FIG. 6 shows a state in which the spatiotemporal image is cut along a plane parallel to the xt plane. In a continuous video section divided by cuts, that is, in a shot, the movement of the subject appears as a band-shaped flow. Since the pattern suddenly changes at the cut point, an edge perpendicular to the t-axis appears. Since the gradient vector (indicated by an arrow in the drawing) intersects the edge almost perpendicularly, it is almost parallel to the t-axis (not all are exactly parallel) at the cut point, as shown at 72. Taking the pixel 70 at the cut point as an example, the brightness change in the x direction near the pixel 70 is small.
Δ _x I | becomes smaller. Since the change in luminance in the t direction is large, | _Δt I | becomes large, and the gradient vector in the pixel 70 is almost parallel to the t axis. Within the shot, the degrees formed by the gradient vector and the t-axis are determined by the movement of the subject (if the subject is stationary, the angle is perpendicular to the t-axis), but in any case, as shown by 71, The t-axes are rarely parallel. According to the present invention, since a time point that is mostly parallel to the t-axis among the gradient vectors is detected as a cut point, the detection performance is improved,
Even if there is movement or camera shake of the subject, it is less likely to mistake it as a cut point.

Next, an embodiment of the present invention based on such a principle will be described. FIG. 1 is a process flow chart showing the present embodiment.

Here, the image data string is digitized, and the brightness value I (x,
Let us consider it as a three-dimensional array (spatiotemporal image) that defines y, t). Although only the luminance value is considered here as an example, color information such as RGB values may be considered.

In step 101, a smoothed spatiotemporal image is constructed by operating a filter having a smoothing action on the spatiotemporal image. By smoothing, it becomes possible to stably obtain the gradient vector (without smoothing, the gradient cannot be stably obtained due to noise). As a filter having a smoothing action, for example, a Gaussian filter

[0021]

[Equation 1]

Can be used. Convolution of the Gaussian filter G and the spatiotemporal image I (convoluti)
on) is the smoothed spatiotemporal image I '(= G *
I). σ _x , σ _y , and σ _t are smoothing scales in the x, y, and t directions, respectively. The larger σ is, the smoother the spatiotemporal image after filtering is, and the noise is removed, but the fine features are lost. In order to reduce the amount of calculation, the following digital filter can be used.

[0023]

[Equation 2]

However, W, H, and T represent smoothing scales.

In step 102, at time t, the gradient (Δ _x
I, Δ _y I, Δ _t I) are calculated. For a Gaussian filter,

[0026]

(Equation 3)

Since the following holds, instead of convolving the smoothing filter and then differentiating, the differential filter ∂G / ∂x may be convoluted. Also, other filters with differentiating effects, such as

[0028]

[Equation 4]

May be used.

[0030] At step 103, the gradient vector is _{| Δ x I | <θ 1} and | Δ y I | counts the number of pixels <theta satisfies region ₁ (1) (referred to as region A), it count1. │Δ _x I │ takes a large value when there are vertical edges in the vicinity, and │Δ _y I
Since | has a large value when there is a horizontal edge in the vicinity, it can be said that there is no edge in the vicinity of the region A that satisfies the expression (1). In the image of FIG. 2A, the area A
What was calculated is shown by the diagonal lines in FIG. Qualitatively, the area A is the area excluding the vicinity of the edge of the subject. (1)
Instead of equation _{| Δ x I | + | Δ} y I | <θ 1 _{or, √ (| Δ x I |} 2 + | Δ y I | 2) < may use an expression such theta _1.

In step 104, the number of pixels in a region (referred to as region B) in which the gradient vector satisfies both the condition (1) and the condition | Δ _t I |> θ ₂ (2) is counted as count2. To do. When the area that satisfies only the condition (2) is called area C, the common part of area C and area A becomes area B. Conventionally, the area of the region C divided by the total number of pixels is defined as diff ', and diff'
There was a method to consider when a certain threshold was exceeded as a cut point. In this method, as shown in FIG. 4, when the movement of the subject is strong, the diff 'increases even at the cut point, so that there is a problem that many detection errors occur due to the movement of the subject or camera shake. This is because if there is motion or camera shake of the subject, the inter-frame difference increases (| Δ _t I | increases) near the edge of the subject. Therefore, in one embodiment of the present invention, not the area (diff ') of the area C but the area of the area B is counted. Condition (1)
The condition (2) is satisfied that the gradient vector is parallel to the t-axis when viewed geometrically. As an example, consider the case where the scene switches from the screen of FIG. 3A to the image of FIG. In FIG. 3 (c), the area A is shaded and the area B is black. At the cut point, the cut point is detected by utilizing the property that the area B occupies a large portion of the area A.

In step 105, diff = count
Calculate t2 / count1. That is, the ratio of the area of the region B to the area of the region A is defined as diff. 0 ≦ diff
Since ≦ 1 and diff takes a value close to 1 at the cut point, it is determined in step 106 that there is a cut point when diff> θ ₃ is true, and otherwise it is determined that there is no cut point. FIG. 5 shows a typical example of the diff graph. Even if there is movement of the subject or camera shake, diff is
Unlike the above, since it does not rise, the cut point can be stably detected by the threshold processing.

At step 106, t is advanced to step 1
Return to 02.

[0034]

As described above, according to the video cut point detection method of the present invention, a video is considered as a spatiotemporal image having image plane coordinates and time as variables, and the gradient vector of the spatiotemporal image is calculated. Since the cut point is detected by considering not only the difference in the time direction but also the difference in the spatial direction,
There is an effect that the detection performance of the cut point is improved and the detection error caused by the movement of the subject and the camera shake in the spatiotemporal image can be reduced.

Further, in the above, when the smoothing filter is made to act on the spatiotemporal image, in particular, the influence of noise is eliminated and the detection of the cut point is stabilized.

Further, in the above, when the smoothing filter is composed of a digital filter, the amount of calculation can be reduced especially.

[Brief description of drawings]

FIG. 1 is a process flow chart of an embodiment of the present invention.

2A and 2B are explanatory views of a region (region A) that satisfies the condition (1) in the above-described embodiment.

3 (a), (b), and (c) are a region (region A) satisfying the condition (1) in the above-described embodiment and the condition (1),
Explanatory drawing of the area | region (area | region B) which satisfy | fills (2).

FIG. 4 is a diagram showing a typical time change of an area (diff ′) of a region (region C) satisfying only condition (2) in the above-mentioned embodiment.

FIG. 5 is a diagram showing a typical time change of the area (diff) of the region B.

FIG. 6 is an explanatory diagram of a cutting plane and a gradient vector of a spatiotemporal image for explaining the principle of the present invention.

7A and 7B are diagrams for explaining the problems of the conventional method, in which FIG. 7A is an input image sequence, and FIG. 7B is an explanatory diagram of a method of calculating a variation D (t).

8A and 8B are diagrams for explaining problems of the conventional method, in which FIG. 8A is a graph of a change amount D (t) when the movement amount Δ is small, and FIG. 8B is a graph when the movement amount Δ is large. The graph of change amount D (t).

[Explanation of symbols]

 101-107 ... Step

Claims (3)

[Claims] 1. A video cut point detection method for detecting a cut point from an image data string forming a video, wherein a gradient vector is calculated from the image data string by regarding the image plane coordinates x, y and time t as variables. A step of outputting, as a cut point, a time at which a large number of the gradient vectors that are close to the t-axis are output, as a cut point detection method. 2. A procedure of calculating a gradient vector from an image data string is a step of operating a smoothing filter on the image data string to calculate a gradient vector of a smoothed spatiotemporal image. The image cut point detection method according to claim 1. 3. The video cut point detection method according to claim 2, wherein the smoothing filter is a digital filter.