JPH0993530A - Video image change point detector and video image change point detecting method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate a frame number at the head of a succeeding scene by executing detection of a scene change finished below a threshold frame time and detection of a scene change requiring the threshold frame time or over separately after setting a threshold frame time (Ks) to be a natural number being 2 or over. SOLUTION: A short time length type detection means 6 is a processing to detect a scene change finished less than a Ks frame time to detect a cut and a video image moving model, a picture element synthesis model, and a picture element replacement model finished less than the Ks frame time. On the other hand, other model detection means 7 is a processing to detect a scene change finished over the Ks frame time, that is, a processing to detect other processing than the short time model such as to detect a video image moving model, a picture element synthesis model, and a picture element replacement model finished over the Ks frame time. Then a scene head confirmation means 8 processes a frame image including the head object of a succeeding scene obtained by the two procedures as above to confirm a head frame image of the scene.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video fast-viewing method and a video editing method capable of cueing a video or a video of a movie according to content, and particularly to a portion where the content is changed from the video stored in a video tape or a video disc. The present invention relates to a video change point detection method for detecting.

[0002]

2. Description of the Related Art In recent years, researches on methods for assisting retrieval, editing, processing, and quick viewing of images by applying a computer have become popular. As an example, there is a video fast-viewing method and a video editing method that can cue a video or a video of a movie by content. In order to realize these methods, it is necessary to divide the video by content, and the part where the content has changed (hereinafter referred to as the scene change) is detected and the start frame number of the scene (hereinafter referred to as the video change point). It is important to have a technique for estimating

As a scene change detection method, the authors have used a common color ratio method (Yamada, Fujioka, Kanamori, Matsushima, "Study of Scene Change Detection Method Focusing on Common Color for Each Partial Area", Technical Report of the Television Society of Japan, Vol.17, No.55, pp1-
6), Video change model method (Yamada, Fujioka, Kanamori, Matsushima, Sakauchi, "Scene change detection method of video including editing effect", Multimedia and video processing symposium '94, pp2
1-26) is proposed. In these methods, scene changes are first classified into the following four.

Cut: As shown in FIG. 25, the scene is switched in one frame time. Moving image type: As shown in FIG. 26, “Scene change (hereinafter referred to as pull-out) in which the previous scene moves and disappears”,
The previous scene or the next scene will be enlarged / reduced / transformed / moved.

Pixel combination type: Corresponding pixels between the previous scene and the next scene, such as "Scene change (hereinafter referred to as" dissolve ") in which different gains are applied to the previous scene and the next scene". Is synthesized.

Pixel replacement type: Pixel of a part of the video of the previous scene such as "scene change in which the pixel of the previous scene is sequentially replaced by the pixel of the next scene from the left (hereinafter, referred to as wipe)" shown in FIG. Are replaced with corresponding pixels in the next scene, and the replaced area increases.

The common color ratio method is a method for detecting a cut, and the video change model method is a method for detecting a video moving type, a pixel synthesizing type, and a pixel replacing type. The procedure of the common color ratio method will be briefly described below, and then the procedure of the image change model method will be briefly described.

In the common color ratio method, each frame image of the target video is divided into a plurality of partial areas, and a cut is detected using color information for each partial area. First, as shown in FIG. 29, a frame image I _n (n is 1 representing a frame number)
The above natural number) is divided into 16 partial regions R _{j, n} (j is an integer of 1 or more and 16 or less, n is a natural number of 1 or more). Next, as shown in the example of FIG. 30, the partial areas at the same position are set as “corresponding partial areas”, and consecutive frame images I
16 corresponding partial regions R _{j, n−1} , R of _n−1 , I _n
Compare _{j, n} using the intersection of the color histograms,
Find the similarity of the corresponding partial areas. The 16 "similarities of corresponding partial areas" are averaged to obtain the frame image I
The similarity S (n) between _n-1 and I _n (n is a natural number of 1 or more). Within the same scene, the similarity between successive frame images takes a value equal to or greater than a preset threshold value θhigh. As shown in FIG. 31, the similarity S (n) of the frame images before and after cutting is set in advance as compared with the similarities S (n-1) and S (n + 1) of the frame images in the same scene. When the conditional expression S (n-1) -S (n)> θcut S (n + 1) -S (n)> θcut S (n-1) ≧ θhigh is satisfied at the same time. Then, it is determined that “a cut is made between the frame images I _n−1 and I _n ”.

Assuming that the similarity between images before and after a scene change takes a value less than a preset threshold value θlow, as shown in FIG. 32, S (n-1) -S (n)> θcut S If (n + 1) −S (n)> θcut S (n) <θlow is satisfied at the same time, it may be determined that “a cut is between frame images I _n−1 and I _n ”.

On the other hand, in the video change model method, time-series frame images are sequentially processed to detect a scene change that is slow in time according to the procedure of the flowchart shown in FIG.

In step 3001 of FIG. 33, the frame number Nf to be judged is set. Step 3002 is a process of determining whether or not the frame number Nf to be determined is a “candidate for the end point of the scene change similar to the video moving type”.

[0012] When a moving image type scene change or movement of a subject occurs, enlargement / reduction / deformation / deformation on the screen is performed.
Movement such as movement occurs. When moving the subject, see FIG.
A change in luminance occurs due to the movement of the subject itself, as in the shaded area in the middle. In addition, in the video change type scene change,
As indicated by the shaded area in FIG. 35, “luminance change due to movement of entire previous scene or next scene” is added to luminance change due to movement of the subject.

The pixel change area RIC _n (n is a natural number of 2 or more) and the pixel change area IC (n) (n is a natural number of 2 or more) are defined as follows, and the pixel change area is defined as "the movement of the screen. Total amount "
Consider

Pixel change region RIC _n : frame image I _n-1 ,
A group of “pixels in which the absolute value of the brightness difference between pixels at the same position is equal to or greater than a preset threshold θw1” between I _n Pixel change area IC (n): Pixel included in pixel change region RIC _n Value divided by the total number of pixels in the frame image. Further, as shown in FIG. 36, the pixel change area is greater than or equal to the threshold value θfull in the section from the judgment start point No1 during the scene change to the scene change end point Nn. Suppose it decreases. Then, the time change satisfying FIG. 36 is detected, and
When the frame number corresponding to Nn is equal to the frame number Nf to be determined, Nf is regarded as "a candidate for the end point of the scene change similar to the video moving type". However, in FIG.
, ICs (n) (n is a natural number of 2 or more) is “a value obtained by smoothing the pixel change area IC (n) of the time-series frame image in the time direction”, and ICs (n−2), ICs (n-1), IC (n), IC
(n + 1) and IC (n + 2) are five median values in total.

The procedure 3003 is a process for determining whether or not the frame number Nf to be determined is a "candidate for the end point of the scene change similar to the pixel combination type".

It is assumed that the temporal change of the total sum of edge intensities in the frame image (hereinafter referred to as edge intensity coefficient) is convex downward during the pixel composition type. Based on this assumption,
The pixel combination type is classified into three types shown in FIGS. 37 to 39, Eps (n) (n is a natural number of 1 or more)
Is a “value obtained by smoothing the edge strength coefficient of a time series of frame images in the time direction”, and Ke and θedge are preset threshold values. Also, an expression for normalizing the difference in edge strength coefficient with the edge strength coefficient

[0017]

[Equation 1]

By the change rate of the edge strength coefficient Epr (n)
(N is a natural number of 1 or more) is calculated. In addition, "continuous Ke
The “natural number N” is, for example, a natural number of Ns + 1 or more and Ns + Ke or less. Further, Ke is set to 4, for example.

Then, a time change similar to any one of FIGS. 37 to 39 is detected, and the scene change portion in the drawing is regarded as a scene change candidate. When the "end point Nn of the scene change candidates" is equal to the frame number Nf to be determined, Nf is regarded as "a candidate for the end point of the scene change similar to the pixel combination type".

The procedure 3004 is a process for judging whether or not the frame number Nf to be judged is a "candidate for the end point of the scene change similar to the pixel replacement type".

The replaced area SIC (Nw0, Nf) (Nw0 is a natural number of 1 or more) is defined as follows, and is regarded as the ratio of pixels replaced from the previous scene to the next scene.

Replaced area SIC (Nw0, Nf): 2 frames
Image I_Nw0, I_NfChange region RIC of frame image between
_{Nw0 + 1}~ RIC_NfThe number of pixels in the union of
A value normalized by the total number of pixels of the image. Furthermore, as shown in FIG.
I_NfFrame image I with the previous threshold value Kw or less_Nw0~ I
_NfAnd the replaced area SIC (Nw0, Nf) is close to 100%.
This frame image I_Nw0~ I_Nf"Pixel replacement type
Similar scene change candidates ”. However,
Continuous frame images I in the middle of this candidate _N-1, I_N(N
Is a natural number greater than or equal to Nw0 + 1 and less than or equal to Nf)
Area of open area Absolute value of the difference between IC (N-1) and IC (N) ｜ IC (N) −IC
Conditional expression that assumes that (N-1) | is less than the threshold θgap

[0023]

[Equation 2]

If the condition is not satisfied, it is regarded as a rapid change due to the movement of the subject, and the frame number Nf to be judged is excluded from the "candidates for the end point of the scene change". Note that | x | represents the absolute value of x. Also the threshold
Kw is set to 35, for example.

Step 3005 is a process of determining whether the frame image I _Nf to be determined is the head candidate of the next scene by using the following steady state assumption.

Steady state assumption: threshold from the beginning of the scene
With Kc frame images, the movement of the subject and the camera movements such as pan and zoom are in a steady state. That is,
The difference between the maximum value and the minimum value of the pixel change area ICs (n) of the time-series images is less than or equal to the threshold value θup.

When the threshold Kc frame images starting from the “scene change end point candidate Nf” detected in steps 3002 to 3004 do not satisfy the steady state assumption, the frame image I _Nf is Not at the beginning of the scene
It is considered to be in the middle of discontinuous movement in the same scene. If not, the frame image I _Nf is regarded as the head candidate of the next scene.

In step 3006, the following constant velocity tracking assumption is used to determine whether the scene change candidate is a constant velocity tracking portion.

Assuming constant velocity tracking: When a subject is tracked at a constant velocity, the pixel change area ICs (n) of the time-series image is equal to or greater than the threshold value θstill and the variation amount is equal to or less than the threshold value θdiff. Becomes

When the scene change candidates detected in steps 3003 and 3004 satisfy the constant-velocity tracking assumption, the frame image I _Nf is regarded not in the head of the next scene but in the middle of the constant-speed tracking portion.

In step 3007, when the leading candidate of the next scene obtained in step 3005 is included in the constant velocity tracking portion obtained in step 3006, this candidate is excluded from the leading candidates of the next scene.

In step 3008, as shown in FIG. 41, the similarity between the head candidate I _Nf of the next scene found in step 3007 and the head frame image I _Np of the previous scene is found, and the similarity is determined. If the value becomes equal to or less than the value θchange, the head candidate of the next scene is regarded as the head frame image of the next scene.

In addition to the above-mentioned common color ratio method, there is a method (hereinafter referred to as a pixel change area method) described in Japanese Patent Application Laid-Open No. 5-37853 as a method for detecting the cut. In this method, when the above-mentioned pixel change area simultaneously satisfies IC (n) -IC (n-1)> θic IC (n) -IC (n + 1)> θic, the “frame image I _n-1 , The cut is between I _n ”. That is, when the pixel change area IC (n) of the frame image changes as shown in FIG. 42, the portion immediately before the frame image I _n is regarded as a cut.

[0034]

However, in the above scene change detection method, the moving image type, the pixel synthesizing type,
Among the pixel replacement types, there has been a problem that a detection failure often occurs even though it is completed in a short time.

For example, when an image moving type that ends in 2 frame times occurs, the pixel change area IC (N) is changed for 2 frame times like IC (n) and IC (n + 1) in the example shown in FIG. Project. Therefore, such a scene change cannot be detected by the pixel change area method. Also, the pixel change area IC (N)
When the change in pixel change area ICs (N) after the smoothing in the time direction is examined in the case of change as shown in FIG. 43, the protrusion disappears as shown in FIG. Therefore, such a scene change cannot be detected even by the image change model method.

The present invention solves the above-mentioned problems of the prior art. By detecting various scene changes including a scene change that ends in a short time, the leading frame number (video change point) of the next scene can be determined. It is an object of the present invention to provide a method for detecting an image change point to be estimated.

[0037]

In order to achieve this object, the threshold value Ks is 2 in the image change point detecting method of the present invention.
After setting the threshold value Ks so that it becomes the above natural number,
Scene change that ends in less than Ks frame time (hereinafter,
The image change point is estimated by separately executing the detection of the short-time long type) and the detection of the scene change that requires more than Ks frame time.

In the short-time long-type detection method, a time-series frame image is processed, and a change in the degree of similarity of the frame image during the Ks frame time is examined to obtain a scene head candidate and then the scene The frame image including the head candidate of is processed to determine the head frame image of the scene.

[0039]

[Function] Scene changes are classified into short-time long type and non-short-time long type, and the scene change of each type is detected by different means, so that the video changes rapidly and the image of one frame time interval High-precision detection of short-time long images, which has the feature that the similarity changes rapidly, and those other than short-time long images, in which the image changes slowly at one frame time interval and the similarity changes smoothly. it can.

Further, by processing the time-series frame images,
By examining the change in the similarity of frame images during the Ks frame time, it is possible to detect a short and long type scene change in which the image changes rapidly in the Ks frame time. In addition, by processing the frame image that includes the candidate for the beginning of the scene, it is possible to check whether the contents change before and after the scene change and whether the video is in a steady state at the beginning of the scene. As described above, it is possible to suppress excessive detection in a portion where the video changes rapidly in the same scene.

Since the video change point is the head frame number of the scene, the video change point can be detected with high accuracy by detecting the scene change with high accuracy.

[0042]

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is an overall system diagram of an image quick-viewing apparatus according to an embodiment of the present invention. In FIG. 1, 1, 2
Is an input / output device for a video to be processed, 1 is a video disk device, and 2 is a VTR. Further, 3 is a frame memory that captures a frame image of a video signal input from the video disk device 1 or VTR 2, and 4 processes the time-series frame images captured from the frame memory 3 to detect a video change point. This is a video change point detecting device for executing the above, and is composed of a control means 5, a short time length type detection means 6, another type detection means 7 and a scene head determination means 8. 9
Is a computer for reducing the frame images of the image change points detected by the image change point detection device 4, and 10 is a display for displaying a list of the images reduced by the computer 9. An external storage device 11 stores the image and the frame number of the video change point detected by the video change point detection device 4.

The overall operation of the image fast-viewing device configured as described above will be described with reference to the flowchart shown in FIG.

In step 201, the control means 5 in the video change point detection device 4 controls the video disk device 1, the VTR 2 and the frame memory 3 to make the frame image I to be processed I.
_{Import Nt} . However, the subscript Nt of the frame image to be processed
Represents the frame number of the captured image and is a natural number of 1 or more.

In step 202, the short time length type detecting means 6, the other type detecting means 7 and the scene start determining means 8 in the image change point detecting device 4 detect the image change point.

In step 203, if the image change point is detected in step 202, the process proceeds to step 204, and if not, the process proceeds to step 205.

In step 204, the computer 9 displays the image at the video change point (the first frame image of the scene) on the display 1.
Display reduced to 0. For example, as shown in FIG. 3, the first frame image of the newly detected scene is displayed side by side with the first frame image of the scene detected so far.

When the frame image is displayed on the display 10, it may be possible to communicate with the video disk device 1 or the VTR 2 to read the frame number of the captured frame image and display it together with the image.

In step 205, it is determined whether or not the image to be processed has ended. If the image has ended, the process ends. If the frame images of all the detected video change points are stored in the external storage device 11 at the end of the processing, the information of the video change points can be taken out at any time and can be utilized for video editing or the like. . Also,
The frame number instead of the frame image is stored in the external storage device 1
Even if it is stored in 1, the same effect can be obtained.

If the video has not ended, step 20.
Proceed to 6. In step 206, the frame image I _{Nt + 1} next to the current frame image to be processed is set as a new frame I _Nt to be processed, and then the process returns to step 201.

In this embodiment, the case where all the time-series frame images are processed has been described, but the frame images in the time direction may be thinned out. When thinning out frame images in the time direction, a natural number may be given as an image number to the sampled frame images and substituted for the subscript Nt of the frame image to be processed.

The video change point detection processing of the video change point detection device 4, which is a specific operation of step 202 in FIG. 2, will be described below.

FIG. 4 is a flow chart of an embodiment of the image change point detection processing of the image change point detection device 4 in FIG. 1. In step 401, a short type long scene change is detected, and in step 402, a short time. A scene change other than the long type is detected, and the first frame image of the next scene is determined in step 403.

Each procedure will be described in detail below. Step 40
1 is a process in which the short-time long-form detecting unit 6 detects a scene change (hereinafter, referred to as short-time long-form) that ends within the Ks frame time. In this embodiment, Ks is set to 4. Note that Ks may be a value other than 4 as long as it is a natural number of 2 or more.

Further, in the time-series frame images I _n (n is a natural number) of the video to be processed, the next scene candidate is the frame image I _Nf (Nf is a natural number of n or less) and later. a good, refers to the frame image between the determination target frame image I _Nf and the previous threshold Ks side, was set in advance of the immediately preceding image I _{Nf- human s} is a scene change candidate, the scene change candidate from the previous scene It refers to the previous frame image, and S '(n1, n2) is two frame images I _n1 and I _n2.
(N1 and n2 are natural numbers) indicates the similarity. As described in the conventional example, continuous frame images I _n-1 and I _n
25 is a cut as shown in FIG. 25, the similarity S (n) and the pixel change area IC (n) of consecutive frame images are
As shown in FIGS. 31, 32, and 42, S (n-1) -S (n)> θcut (1) S (n + 1) -S (n)> θcut (2) IC (n)- IC (n-1)> θic (3) IC (n) −IC (n + 1)> θic (4) is satisfied.

In the cut, the previous scene changes to the next scene at the same time in all the partial areas shown in FIG. on the other hand,
In the “video moving type that ends in a short time” as shown in FIG. 5, as shown in FIG. 6, the content of the partial area changes from the video of the previous scene to the video of the next scene at different frame times for each partial area. To do. Taking the case where the following still image assumption holds, the author's analysis results regarding the moving image type and the pixel replacement type that end in a short time will be described with reference to FIG.

Assumption of still image: Both the previous scene and the next scene are still images. The number of partial areas does not necessarily have to be 16.

[0059] two frame images I _N1, I _N2 of 16 corresponding partial region _{R j, N1, R j,} N2 (j is 1 to 16 integer, N1, N2 is a natural number of 1 or more) of The similarity between
(j, N1, N2) (j is an integer of 1 or more and 16 or less, N1, N2 are natural numbers of 1 or more). In addition, in proportion to the "appearance and disappearance of a new color" occurring between the "corresponding partial areas" R _{j, N1} and R _{j, N2} as shown in FIG. 29, the feature amount {1.
0-Sp (j, N1, N2)} is reduced such that the partial regions R _{j, N1} ,
The similarity Sp (j, N1, N2) of R _{j, N2} is defined. Note that Sp (j, N
The specific calculation procedure of (1, N2) will be described in the part for explaining the procedure 1301 in FIG. Since "the appearance and disappearance of new colors" occur in the part where the previous scene changes to the next scene,
From FIG. 6, “appearance and disappearance of new color” occur at different frame times for each partial area. Therefore, in the frame images I _{Ns to} I _Nn in the middle of the video moving scene change,

[0060]

(Equation 3)

Is satisfied. Two frame images I _N1 , I
_N2 of the degree of similarity S 'the (N1, N2)

[0062]

[Equation 4]

Defining as follows, from equations (5) and (6),

[0064]

(Equation 5)

Is satisfied. On the other hand, in the frame image I _Ns at the start of the scene change and the frame image I _{Nn + 1} immediately after the end of the scene change, the similarity S (Ns) and S (Nn + 1) between the previous frame image and , S (Ns) is S '(Ns-1, N
s) and S (Nn + 1) is equal to S '(Nn, Nn + 1))
It is the “similarity of consecutive frame images in the same scene”. Therefore, the similarity S (Ns), S (Nn +
Conditional expression S (Ns) = 1.0 (8) S (Nn + 1) = 1.0 (9) where 1) becomes maximum 1.0 holds.

Scene change frame time length (Nn-N
s) is an odd number, and is the absolute value | S (Ni) -S (Ni of the temporal change amount of the similarity S (Ni) of the frame image I _Ni (Ni is a natural number of Ns + 1 or more and Nn or less) in the middle of a scene change. FIG. 7 shows an example of the temporal change of the similarity S (n) of the frame images when −1) | has a constant value. Since the expressions (8) and (9) are established, in FIG. 7, the absolute value of the temporal change amount of the similarity S (Ni) of the frame images | S (Ni) -S (Ni-1) | (where Ni is
A natural number greater than or equal to Ns + 1 and less than or equal to Nn) is

[0067]

(Equation 6)

Satisfies. When the frame time length (Nn-Ns) of the scene change is an odd number, for example, as shown in FIGS. 8 and 9, the temporal change of the similarity S (Ni) does not always match that of FIG. 7, but (8), ( Since equation 9) holds,

[0069]

(Equation 7)

There is always Ni (where Ni is a natural number not less than Ns + 1 and not more than Nn) satisfying the above condition. Since the similarity S (n) of consecutive frame images is equal to S '(n-1, n), (7), (11)
From the formula,

[0071]

(Equation 8)

Holds. Similarly, when the frame time length (Nn-Ns) of the scene change is an even number, the similarity S (n) decreases by a constant value and then increases by a constant value, or the similarity S As shown in the examples of FIGS. 11 and 12, in which the time variation S (n) -S (n-1) of the similarity changes when (n) increases or decreases,

[0073]

[Equation 9]

There is always Ni (where Ni is a natural number not less than Ns + 1 and not more than Nn) that satisfies the above condition. Based on Eqs. (12) and (13), the frame number Ni (Ni is Ns + 1
For a natural number greater than or equal to Nn and less than or equal to Nn, the similarity S (Ni) with the previous frame image (where S (Ni) is S '(Ni-1, Ni)
The conditional expression S (Ni + 1) −S (Ni)> θss (14) S (Ni) −S (Ni + 1)> θss ( It is assumed that at least one of 15) is established.

When the expression (14) is satisfied, the "average value of the areas of newly appearing color and disappearing color in the partial area" equal to the similarity S of the frame images is "continuous frame images I _Ni , I _Ni". Compared to "between ₊₁ ", "between consecutive frame images I _Ni-1 and I _Ni " is smaller. Since a new color appears or disappears at the portion where the previous scene changes to the next scene, the pixel change area IC that represents the total number of pixels that changed from the previous scene to the next scene is expressed when Equation (14) holds. , The latter (between consecutive frame images I _Ni-1 and I _Ni ) is larger than the former (between consecutive frame images I _Ni and I _{Ni + 1} ), and IC (Ni) −IC (Ni +1)> θ ics (16) holds. Similarly, when equation (15) is established, IC (Ni + 1) -IC (Ni)> θics (17) is established.

In the pixel replacement type such as the wipe shown in FIG. 28, the previous scene changes to the next scene at a different frame time for each partial area, as in the video moving type. Therefore, at least one of the expressions (14) and (15) is established.

Regarding the pixel combination type which is completed in a short time,
The analysis results of the authors when the still image assumption is satisfied will be described.

In the pixel composition type scene change such as the dissolve shown in FIG. 27, the scene change is Nreq.
At the end of the frame time, the luminance Y (Ns + i, p) of the pixel p after i frame time from the start of the pixel combination type is the pixel p of the frame images I _Ns and I _Nn at the start and end of the scene change. Using the luminances Y (Ns, p) and Y (Nn, p) of

[0079]

(Equation 10)

It can be expressed as Therefore, the pixel change area IC (Ni) (Ni is a natural number not less than Ns + 1 and not more than Nn) in the middle of the pixel combining type has a luminance difference (Nreq × θw1) of the corresponding pixels of the images I _Ns and I _Nn before and after the scene change. ) The above conditional expression

[0081]

[Equation 11]

This is the total number of pixels p satisfying the condition. However,
x | represents the absolute value of x, and θw1 is the threshold value used in the definition of the pixel change region described in the “conventional example”.

From the still image assumption, IC (Ns) = IC (Nn + 1) = 0.0 (20) holds. Further, “the ratio of the pixel p satisfying the expression (19) to the total number of pixels of the frame image” is a threshold value θics for the frame image I _{Ni in} the middle of the pixel combination type (Ni is a natural number of Ns + 1 or more and Nn or less). Assuming that the value exceeds, IC (Ni)> θics, Ns + 1 ≦ Ni ≦ Nn (21). Therefore, from the equations (20) and (21), IC (Ns + 1) -IC (Ns)> θics (22) IC (Nn) -IC (Nn + 1)> θics (23).

A pixel p satisfying the equation (19) is a partial region R.
_{j, Ni}(J is a natural number of 1 or more and 16 or less, Ni is Ns + 1 or more and Nn or less
When it is included in the lower natural number), the corresponding partial region R
_{j, Ni-1}, R _{j, Ni}The appearance and disappearance of new colors in the same
Like a partial area of a frame image in the
Corresponding to the case where "appearance or disappearance of" does not occur
Minute region similarity Sp (j, n-1, n) (j is 1 to 16
A natural number, n is a natural number of 2 or more) becomes small. Also,
Similarity of the corresponding partial areas, as defined by equation (6)
Is the average of the frame image similarity S (n) (where S (n) is S '
(equal to (n-1, n)). Therefore, in the middle of pixel combination type
Frame image I_Ns~ I_NnThen, the pixels that are the same scene
Compared to immediately before the start of the composite type and immediately after the end of the pixel composite type,
The similarity S (n) of consecutive frame images in a row is the threshold θs
The conditional expression S (Ns) -S (Ns + 1)> [theta] ss (24) S (Nn + 1) -S (Nn)> [theta] ss (25) that is a value smaller than s is satisfied. Replacing Ns + 1 with Ni, equation (24) becomes
Equation (15) is obtained. If Nn is replaced with Ni, (2
Equation (5) becomes equation (14).

As described in the conventional example, the scene change can be classified into a cut type, an image moving type, a pixel replacement type and a pixel synthesizing type. As described above, in the case of finishing in a short time, (14), (1
There is a frame image I _Ni that satisfies at least one of equations (5). When Expression (14) is established, Expression (16) is established at the same time. If equation (15) holds,
Equation (17) is simultaneously established.

Based on the above analysis results, the specific operation in step 401 will be described with reference to FIG.

In step 1301, the partial areas at the same position are set as "corresponding partial areas" as shown in FIG.
Similarity Sp (j, Nt-1, Nt) between 16 corresponding partial regions R _{j, Nt-1} , R _{j, Nt} of consecutive frame images I _Nt-1 , I _Nt
Is the process of calculating

First, the histogram H (c, j, Nt) of the color c is obtained for each partial region R _{j, Nt} , and the constituent color CV _{j, Nt of the} partial region is calculated using the preset threshold θh. (J is an integer of 1 or more and 16 or less) is calculated by CV _{j, Nt} = {c | H (c, j, Nt)> θh} (26).

As the color c, for example, 512 colors each having 8 gradations of red component, green component and blue component are used. Further, in the present embodiment, the case where H (c, j, Nt) ≦ θh (27) is satisfied is considered as noise, not as “color c exists in the partial area”.

Next, a common color CC _{j, Nt} (j is 1 or more) which is a common part of the constituent colors CV _{j, Nt-1} , CV _{j, Nt} of the corresponding partial areas of the frame images I _Nt-1 and I _Nt. 16 or less)

[0091]

(Equation 12)

Calculate by Furthermore, the common color C in each of the corresponding partial regions R _{j, m} (m is Nt-1, Nt)
Area of the region with C _{j, m}

[0093]

(Equation 13)

And the area of the region having the constituent color CV _{j, m}

[0095]

[Equation 14]

Using and, the similarity Sp of the corresponding partial area is
(j, Nt-1, Nt)

[0097]

(Equation 15)

Calculate by The similarity calculated by the equation (31) has a feature that it decreases in proportion to the area of a newly appearing color or a disappearing color between the corresponding partial regions R _{j, Nt −1} and R _{j, Nt} .

In step 1302, the similarity S (Nt) of the frame image is calculated using the following equation.

[0100]

(Equation 16)

In step 1303, the pixel change area IC (Nt) is calculated. Steps 1304 and 1306 are processes for determining whether or not the frame image I _Nf to be determined is the leading image candidate of the next scene.

In the frame image of threshold Kc sheets from the beginning of the scene, the movement of the subject and the camera operation should be in a steady state. Therefore, the first frame image I of the scene
_{From Nn} to Kc frame images, the similarity S of the frame images
It is assumed that the conditional expression S (n ′)> θ high (33) is satisfied in which (n ′) (where n ′ is a natural number of Nn + 1 or more and Nn + Kc−1 or less) is equal to or more than the threshold value θ high. In this embodiment, the threshold value Kc is 6
Set to. At this time, n'is (Nn + 1) or more (Nn + 5)
It becomes the following natural number.

In step 1304, after substituting (Nt-5) into the frame number Nf to be judged, whether or not the expression (33) is satisfied for the natural number n'of (Nf + 1) or more and (Nf + 5) or less is determined. Find out. Further, with respect to the natural number Ni that satisfies the conditional expression Nf−Ks <Ni ≦ Nf (34) for “the interval between the images I _Ni−1 and I _Ni is in the middle of the short-time long form”, (14), ( It is checked whether the equation 16) is satisfied. Then, as shown in FIGS. 14 and 15, when there is a natural number Ni that simultaneously satisfies the expressions (14), (16), (33), and (34), the determination target frame image _{INf is set} to the next scene. It is regarded as the first image candidate.
Examples of scene changes that satisfy both FIG. 14 and FIG. 15 are the scene changes shown in FIG. 8 and the scene changes shown in FIG. The frame number Nf of the determination target frame image does not necessarily have to be set to (Nt-5). Further, as described above, the threshold value Ks is set to 4 in this embodiment.

In step 1305, if it is determined in step 1304 that the frame image to be determined is a leading image candidate for the next scene, step 401 is terminated, and if not, step 1306 follows.

In step 1306, conditional expression Nf-Ks≤Ni <Nf for whether or not the expression (33) is satisfied and "the interval between the images I _Ni and I _{Ni + 1} is in the middle of a short time length type". It is checked whether or not the equations (15) and (17) hold for a natural number Ni that satisfies (35). As shown in FIGS. 16 and 17, when there is a natural number Ni that simultaneously satisfies the expressions (15), (17), (33), and (35), the judgment target frame image I _Nf is set as the start image of the next scene. Consider it a candidate. FIG.
Examples of scene changes that simultaneously satisfy 6 and FIG. 17 include the scene changes shown in FIG. 9 and the scene changes shown in FIG.

After the procedure 1306 is completed, the procedure 401 is completed. In the procedure 1304 of this embodiment, (14),
Whether or not the equations (16), (33), and (34) are satisfied at the same time is checked. However, if the decrease in the detection accuracy is allowed, one of the similarity S (n) and the pixel change area IC (n) is determined. May be used. For example, either one of the following procedures may be executed.

Execution example 1: (14), (33), (34)
When there is a natural number Ni that simultaneously satisfies the expression, the determination target frame image I _Nf is regarded as the first image candidate of the next scene. Execution example 2: There is a natural number Ni that simultaneously satisfies the expressions (16), (33), and (34). At this time, the determination target frame image I
Similarly, _Nf is regarded as the first image candidate of the next scene. In step 1306 of this embodiment, the similarity S
Either (n) or the pixel change area IC (n) may be used.

Further, in the present embodiment, the similarity Sp (j, Nt-1, Nt) of the corresponding partial area is calculated using the equation (31), but the newly appearing color and the disappearing color are calculated. A physical quantity having a characteristic that decreases in proportion to the area may be used as the similarity. For example, to be proportional to the average area of newly appearing and disappearing colors,

[0109]

[Equation 17]

The same effect as that of this embodiment can be obtained. Furthermore, the execution order of the steps 1301 to 1306 may be changed as long as it is within the executable range. For example,
You may exchange the execution content of the procedure 1304 and the procedure 1306. The procedure 1303 may be executed before the procedure 1301.

In this embodiment, the leading image candidate of the next scene is obtained by using the expression (33), but the judgment target frame image I
_As long as it is an equation for processing the degree of similarity of an image selected from frame images after _Nf (hereinafter referred to as a next scene candidate), an equation in which the method of calculating the degree of similarity and the method of selecting the image are changed may be used. For example, conditional expression S ′ (Nf, N) according to the similarity between the determination target frame image I _Nf and another frame image I _N of the next scene candidate (N is a natural number of Nf + 1 or more and Nf + 5 or less)> θ high (37) may be used.

Further, instead of using the equation (33), the frame image I _{n ′} of the next scene candidate (n ′ is Nf + 1 or more and Nf + Kc−
Similarity S (n ') (where S (n') is S'with a natural number less than or equal to 1)
The conditional expression | S (n ′ + 1) −S (n ′) | <θ up (38) in which (equal to (n′-1, n ′)) is within a certain range may be used. In this case, FIG. 18, as shown in FIG. 19, the condition similarity between the first frame image I _Nf and the next frame image I _{Nf + 1} of the next scene candidate is greater than or equal to the threshold value θhigh formula S (Nf + 1) > Θ high (39) may be used in combination. Similarly, the conditional expression S (Ni ') <θlow in which the similarity S (Ni ′) is less than θlow in the frame image I _{Ni ′} during the scene change (where Ni ′ is a natural number of Nf-Ks + 1 or more and Nf or less) (40) may be used in combination.

Further, in the present embodiment, when the expression (34) is satisfied, it is considered that "between the images I _Ni-1 and I _Ni is in the middle of a short-time long type scene change candidate". As shown in, when the conditional expression S (Ni) <θhigh (41) in which the similarity S (Ni) is less than θhigh is satisfied, “a short-time long scene is generated between the images I _Ni−1 and I _Ni. It is in the middle of a change candidate ”. Similarly, instead of using the expression (35), as shown in FIG. 21, when the conditional expression S (Ni + 1) <θhigh (42) in which the similarity S (Ni) is less than θhigh is satisfied, the “image Between I _Ni and I _{Ni + 1} is in the middle of a short-time long-type scene change candidate. ”

The procedure 402 is a process in which the other-type detecting means 7 detects a scene change that ends after the Ks frame time. As described above, Ks is set to 4 in this embodiment.

A scene change that ends after the Ks frame time can be detected using the image change model method described in the conventional method. For example, steps 3002 to 3007 in FIG. 33 may be executed.

FIG. 22 shows the relationship between the steps 401 and 402. In FIG. 22, the table shows the time required for a scene change and the classification result of the scene changes using the two scene synthesizing method. Step 401
Is a process for detecting a short time long type, and detects a cut and a video moving type, a pixel synthesizing type, and a pixel substituting type which end in less than the Ks frame time. The procedure 402 is a process for detecting other than the short-time long type, and for example, detects the video moving type, the pixel synthesizing type, and the pixel replacing type which end in the Ks frame time or more by using the steps 3002, 3003, and 3004, respectively.

In step 403, the scene head determining means 8
This is a process of processing the frame image including the head candidate of the next scene obtained in steps 401 and 402 to determine the head frame image of the scene. A specific operation in step 403 will be described with reference to FIG.

At step 2101, if "at step 401 it is judged that the judgment target frame image I _Nf is the leading image candidate of the next scene", the process proceeds to step 2103. Otherwise, proceed to step 2102.

In step 2102, if “in step 402 it is determined that the determination target frame image I _Nf is the leading image candidate of the next scene”, the process proceeds to step 2103. If not, it is considered that "the judgment target frame image _INf is not the start image of the next scene", and the procedure 403 is ended, and at the same time, the step 202 is ended.

In step 2103, as shown in FIG. 41, the similarity S '(Nf, Np) between the head image candidate I _Nf of the next scene and the head frame image I _Np of the current scene is obtained. When the condition S ′ (Nf, Np) ≦ θ change (43) that the similarity S ′ (Nf, Np) is less than or equal to the threshold θchange is satisfied, the first image candidate I _Nf of the next scene is set to the first image candidate of the next scene. It is regarded as a frame image, and the video change point (start frame number Nf of the next scene) is substituted for the start frame number Np of the scene. If the expression (43) is satisfied in step 2103, it is considered in step 203 that “the image change point is detected in step 202”.

Simultaneously with the end of step 2103, step 40
3. The procedure 202 ends. The procedure 2102 for detecting other than the short-time long type is the same as the procedure 21 for detecting the short-time long type.
It may be executed before 01. Further, the determination target frame image used in step 2101 and the determination target frame image used in step 2102 may be images of different frame times. Furthermore, if there is only a short and long scene change in the video to be processed, the procedure 210
There is no need to perform 2.

In step 2103, the procedure shown in FIG. 41 is performed.
, The first image candidate I of the next scene _NfAnd of the current scene
First frame image I_NpNext scene using the similarity between
I confirmed the top frame image of_NfNext scene after
Frame image I selected from candidates_Nfi(Nfi is more than Nf
And the frame of the previous scene before the scene change candidate
Image I_Nsi(Nsi is a natural number greater than Np and less than Ns)
You may use similarity S '(Nfi, Nsi). For example, short type
Immediately after the scene change candidate of I_NfAnd the scene Chen
Image I just before the candidate_{Nf-hi s}The similarity S '(Nf, Nf-Ks) between
If the threshold is less than θchange, the first image of the next scene
Image candidate I_NfIs regarded as the first frame image of the next scene
Good. The scene check is performed as in the example shown in FIGS.
As shown in Fig. 24, when the time length of the candidate candidate changes.
At the end of the previous frame image I_NsAnd short time long
Image I immediately after a scene change candidate_NfS '(N
f, Ns) is below the threshold θchange, the next scene
First image candidate I_NfTo the first frame image of the next scene
You may regard it. Also, you may change the method of calculating the similarity.
Yes. As described above, the video change point detection method of the present embodiment
According to, the similarity of frame images over Ks frame time
By investigating the change in
Can detect long and long scene changes where the image changes
You. Also, the content changes before and after the scene change, and
Check to see if the video is in a steady state at the beginning of the scene.
And, in the same scene like the scene where the flash is hit
Suppresses excessive detection in areas where the image changes rapidly
Can be

Further, by detecting a scene change other than the short-time long type by using a means different from the short-time long type, the video changes slowly, and the similarity of images at one frame time interval changes smoothly. It is possible to detect scene changes other than short-time long type with high accuracy.

Since the video change point is the head frame number of the scene, the video change point can be detected with high accuracy by detecting the scene change with high accuracy.

[0125]

As described above, according to the video change point detecting method of the present invention, the threshold value Ks is set so that the threshold value Ks becomes a natural number of 2 or more, and then, the short period is completed within the Ks frame time. By executing the detection of the time-length type scene change and the detection of the scene change that requires more than Ks frame time separately, the video changes rapidly and the similarity of the images at one frame time interval changes rapidly. A short and long type with features,
It is possible to detect, with high accuracy, other than the short-time long type in which the image changes slowly and the similarity of images at one frame time interval changes smoothly.

Further, by examining the change in the similarity of frame images during the Ks frame time, it is possible to detect a short-time long type scene change in which the video abruptly changes in the Ks frame time. Furthermore, by processing the frame image including the beginning candidate of the scene, the contents change before and after the scene change,
In addition, since it is possible to check whether or not the image is in the steady state at the beginning of the scene, it is possible to suppress excessive detection in a portion where the image rapidly changes in the same scene, such as a flash-flashed scene.

Since the video change point is the head frame number of the scene, the video change point can be detected with high accuracy by detecting the scene change with high accuracy.

[Brief description of drawings]

FIG. 1 is an overall system diagram of a video viewing device according to an embodiment of the present invention.

FIG. 2 is a flowchart of the operation of the image quick viewing apparatus according to the embodiment.

FIG. 3 is a diagram showing an example of a list display of head frame images of a scene in the embodiment.

FIG. 4 is a flowchart of image change point detection processing in the same embodiment.

FIG. 5 is a diagram showing an example of a video-moving type scene change that ends in 4 frame times in the embodiment.

FIG. 6 is a diagram showing an example of the timing of switching from the video portion of the previous scene to the video portion of the next scene in the video moving type scene change in the embodiment.

FIG. 7 is a diagram showing a first example of a temporal change in the similarity of frame images when the frame time length of a scene change is an odd number in the embodiment.

FIG. 8 is a diagram showing a second example of the temporal change in the similarity of frame images when the frame time length of a scene change is an odd number in the embodiment.

FIG. 9 is a diagram showing a third example of the temporal change in the similarity of frame images when the frame time length of the scene change is an odd number in the embodiment.

FIG. 10 is a diagram showing a first example of a temporal change in the similarity of frame images when the frame time length of a scene change is an even number in the embodiment.

FIG. 11 is a diagram showing a second example of the temporal change in the similarity of frame images when the frame time length of a scene change is an even number in the embodiment.

FIG. 12 is a diagram showing a third example of temporal changes in the similarity of frame images when the frame time length of a scene change is an even number in the same example.

FIG. 13 is a flowchart of a short-time long type scene change detection process in the same embodiment.

FIG. 14 is a frame number Nf to be determined in the embodiment.
Showing temporal change of similarity in the first case in which is regarded as the first candidate of the next scene

FIG. 15 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing temporal change of pixel change area in the first case in which is regarded as the top candidate of the next scene

FIG. 16 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing temporal change of similarity in the second case in which is regarded as the top candidate of the next scene

FIG. 17 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing temporal changes in pixel change area in the second case in which is regarded as the top candidate of the next scene

FIG. 18 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing temporal change of similarity in the first case in which is regarded as the first candidate of the next scene

FIG. 19 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing temporal change of similarity in the second case in which is regarded as the top candidate of the next scene

FIG. 20 is a diagram showing a frame number Nf to be determined in the embodiment.
Figure showing scene change candidates for the first case, where is considered as the beginning candidate for the next scene

FIG. 21 is a diagram showing a judgment target frame number Nf in the embodiment.
Showing scene change candidates for the second case, where is regarded as the beginning candidate of the next scene

FIG. 22 is a diagram showing a relationship between a procedure 401 for detecting a short-time long type and a procedure 402 for detecting other than the short-time long type in the embodiment.

FIG. 23 is a flowchart of a process of confirming a leading frame image of a next scene in the embodiment.

FIG. 24 is a diagram showing an image used for threshold value determination in the process of determining the first frame image of the next scene in the embodiment.

FIG. 25 is a diagram showing an example of conventional cutting.

FIG. 26 is a diagram showing an example of a conventional video moving scene change.

FIG. 27 is a diagram showing an example of a conventional pixel combination type scene change.

FIG. 28 is a diagram showing an example of a conventional pixel replacement type scene change.

FIG. 29 is a diagram showing an example of division of a conventional frame image into partial areas.

FIG. 30 is a diagram showing an example of a conventional corresponding partial area.

FIG. 31 is a diagram showing a first cut determination condition in the conventional common color ratio method.

FIG. 32 is a diagram showing a second cut determination condition in the conventional common color ratio method.

FIG. 33 is a flowchart of scene change detection processing in the conventional video change model method.

FIG. 34 is a diagram showing an example of a “region in which a luminance change occurs” when a conventional object moves.

FIG. 35 is a diagram showing an example of a “region where luminance change occurs” in the middle of a conventional video moving type scene change.

FIG. 36 is a diagram showing moving image type determination conditions in the conventional image change model method.

FIG. 37 is a diagram showing a first temporal change pattern of the edge strength coefficient in the middle of the pixel combination type, which is classified by the conventional image change model method.

FIG. 38 is a diagram showing a second temporal change pattern of the edge strength coefficient in the middle of the pixel combination type, which is classified by the conventional image change model method.

FIG. 39 is a diagram showing a third temporal change pattern of the edge intensity coefficient in the middle of the pixel combination type, which is classified by the conventional image change model method.

FIG. 40 is a diagram showing pixel combination type determination conditions in the conventional image change model method.

FIG. 41 is a diagram showing an example of determination processing of a leading candidate of the next scene in the conventional video change model method.

FIG. 42 is a diagram showing cut determination conditions in the conventional pixel changing area method.

FIG. 43 is a diagram showing an example of a temporal change of a pixel change area in a conventional video moving type which ends in two frame times.

FIG. 44 is a diagram showing an example of a temporal change of a pixel change area after smoothing in a time direction in a conventional moving image type that ends in two frame times.

[Explanation of symbols]

 1 Video Disc Device 2 VTR 3 Frame Memory 4 Video Change Point Detection Device 5 Control Means 6 Short Time Long Type Detection Means 7 Other Type Detection Means 8 Scene Start Determining Means 9 Calculator 10 Display 11 External Storage Device

Claims (4)

[Claims] 1. A preset threshold Ks (Ks is 2
A video change point detecting device comprising: a short time long type detecting means for detecting a scene change ending within the above natural number) frame time; and another type detecting means for detecting a scene change requiring the Ks frame time or more. 2. A preset threshold value Ks (Ks is 2
The above natural number) A short-time long-type scene change that is a scene change that ends in less than the frame time is detected, and the Ks
A video change point detection method characterized by detecting a long time type scene change that is a scene change requiring more than a frame time, and detecting a video change point representing the start frame number of the next scene. 3. A determination target frame image I _Nf (Nf is a natural number less than or equal to n) among the time-series frame images I ₁ to I _n (n is a natural number) of the video to be processed and preset immediately before that. The time-series images I _Ni-1 , I _Ni , I _{Ni + 1} (where Ni is Nf−Ks or more Nf) including the scene change candidate that is a frame image between the image I _Nf-s before the threshold value Ks sheets. Conditional expression │S '(Ni, Ni + 1) −S' (Ni-1, Ni) ｜ where the absolute value of the temporal change in the similarity between the following natural numbers is greater than or equal to a preset threshold θss ≧ θss (where S ′ (n1, n2) is two frame images I _n1 and I _n2
Two or more frames selected from the next scene candidates that have Ni satisfying (similarity between n1 and n2 is a natural number) and that are frame images after the determination target frame image _INf. All the similarities between the images are equal to or greater than a preset threshold value θhigh, and one or more frame images selected from the previous scene that is the frame image before the scene change candidate and the next scene candidate are selected. The case where the similarity with one or more frame images is less than or equal to a preset threshold value θchange is detected,
The video change point detection method according to claim 2, wherein the short-time long type scene change is detected by regarding the determination target frame image I _Nf as the end point of the short-time long type scene change. 4. Between two frame images whose similarity is less than a preset threshold value θhigh among time-series frame images I ₁ to I _n (n is a natural number) of a video to be processed. Conditional expression S '(Ni-1, Ni) <θhigh S' (Ni, Ni + 1) <θhigh for including a scene change candidate, where S '(n1, n2) are two frame images I _n1 , I _n2
(N1 and n2 are natural numbers), and at least one of the similarities among the time series images I _Ni-1 , I _Ni , and I _{Ni + 1} (Ni is a natural number less than or equal to n) The conditional expression that the absolute value of the amount of change over time becomes equal to or greater than the preset threshold value θss, | S '(Ni, Ni + 1) −S' (Ni-1, Ni) | ≧ θss holds, and one All the similarities between the two or more frame images selected from the next scene candidates that are the frame images after the consecutive scene change candidates described above are equal to or greater than a preset threshold value θhigh, and one The case where the similarity between the frame image selected from the previous scene which is the frame image before the successive scene change candidates and the frame image selected from the next scene candidate is equal to or less than a preset threshold value θchange is detected. , The first image of the next scene candidate 3. The video change point detection method according to claim 2, wherein a short-time long type scene change is detected by regarding it as an end point of the engine.