JP3194837B2 - Representative screen extraction method and apparatus - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for extracting a representative screen, and more particularly to a method and an apparatus for extracting a small number of representative screens representing the contents from a sequence of a plurality of image data.

[0002]

2. Description of the Related Art Generally, video data has an enormous data amount, but the only way to know its contents is to look at the video in chronological order. If a representative screen that efficiently expresses the video content is extracted from the video data, it is useful for grasping the outline of the video. Ideally, it is necessary to select a representative screen in consideration of the story and the like, but at the present time, the work can only be performed manually and the amount of work is enormous, which is impractical.

[0003] A conventional technique relating to automation of representative screen extraction will be described below with reference to a few application examples.

[0004] A first application example relates to display of a list of videos. FIG. 7 shows a schematic diagram of the video list display. If you want to know the contents of a videotape or find the necessary part of the videotape, conventionally, you have to use the fast forward and rewind functions of the VCR.
There was a problem that it took time and effort. If a representative screen is automatically extracted from a video tape and displayed on a display or a medium such as paper, a list of video contents can be obtained, and a rough grasp of video contents can be obtained in a short time. As a prior art relating to this, for example,
No. 74273, “Index image creation device”,
4-111483 "Video Printer", Japanese Patent Application No. 5-19
No. 5644, "Video image printing method and apparatus". Among them, the index image creating apparatus of Japanese Patent Publication No. Hei 5-74273 employs a method of extracting a cut point or image data immediately before the cut point as a representative screen.
A cut point is a joint between shots (a video section continuously captured by a camera), and if it is detected, a representative screen can be selected one by one for each shot. Specifically, a sequence of difference values between images is calculated to determine the presence or absence of an image change. Japanese Patent Application Laid-Open No. 64-11
The video printer of No. 3483 is characterized in that an image after a fixed time from a cut point is extracted as a representative screen and printed on paper. This is based on the rule of thumb that the main screen often appears in the middle of a shot. Further, in the video image printing method and apparatus of Japanese Patent Application No. 5-195644, a reproduced video signal is subjected to image processing, and an image meeting a specific condition is defined as an event representing a large change in the video which is important for grasping the video content. I'm pulling out.

[0005] A second application relates to quick viewing of video. It is possible to automatically generate a quick-view video by cutting out and merging each video section delimited by cut points (Otsuji, Tonomura "Subjective evaluation of high-speed moving image branding", Electronic Information The Communication Society Spring Conference, SD9-3, 1993 "). In this method, a cut point is detected, and an image immediately or immediately before the cut point is used as a representative screen.

The prior art of the video cut point detection method will be described below. As a typical method of detecting a video cut point, a difference between luminance values of corresponding pixels of two temporally adjacent images (the image at time t and the image at time t−1) is calculated, and the absolute value is calculated. The sum of the values (inter-frame difference) is D (t)
And when D (t) is greater than a given threshold,
There is a method in which the time t is regarded as a cut point (Otsuji, Totomura, Oba, "Moving picture browsing using luminance information", IEICE Technical Report, IE90-103, 1991).
In addition, instead of the inter-frame difference, a pixel change area, a luminance histogram difference, a color correlation for each block, or the like may be used as D (t) (Otsuji, Totomura: "Study of automatic video cut detection method", Technical Report of the Institute of Television Engineers of Japan, Vo
l. 16, No. 43, pp. 7-12). Also, D
Instead of thresholding (t) as it is, there is also a method of thresholding the result of applying various time filters to D (t) (K. Otsuji and Y. Tonomura: “Proj
ection Defecting Filter for Video Cut Detectio
n ", Proc. of ACM Multimedia 93, 1993,
pp. 251-257). This method is characterized in that erroneous detection is unlikely to occur even if there is a strongly moving object or flash light in the video.

[0007]

In the above prior art, a representative screen is selected based on a cut point.
The image just after the cut point or a certain time after that is used as the representative screen. However, there is a problem in that the image immediately after the cut point is not focused on the camera, or is blurred due to rapid movement of the subject, and may not be appropriate in terms of image quality as a representative screen. In the application of the list display, it is desirable not to extract a representative screen having poor image quality.

Further, the conventional video cut detection method has a problem that a temporally slow scene change cannot be detected. This is because the amount representing the rate of change of the scene is calculated only from two frames that are temporally adjacent to each other, and a long-time scene is hardly reflected. Regarding this, for example, Japanese Patent Application No. 5-31766.
No. 3 “Video cut point detection method and device” solves the problem by calculating the distance between a plurality of sets of image data between temporally distant images in addition to adjacent images. However, there is still a problem that a scene change that changes very slowly in a long time may not be detected. In other words, a scene change that can be perceived as completely changed by a human being may not be detected, so that there is a problem that a representative screen corresponding to the scene is leaked.

[0009] In addition, even when a pattern is changed by a camera operation such as panning ( shaking the camera horizontally) or tilting (shaking vertically), it is sometimes desired to extract an image after the camera operation as a representative screen. The way did not.

An object of the present invention is to firstly detect an editing effect such as fade and wipe or a temporally slow scene change due to a camera operation, and secondly, any video including temporal noise such as flash light. Thirdly, it is an object of the present invention to provide a representative screen extracting method and apparatus capable of extracting a representative screen which can be adapted to a video and which is appropriate in terms of image quality.

[0011]

In order to achieve the above object, according to the present invention, image data at a certain time is regarded as reference image data, and the distance between the reference image data and the image data at the time t is determined. sequentially calculated while changing the time t, the first condition that the distance is greater than a predetermined threshold, the change of the image data in the temporal vicinity of the time t is smaller than a predetermined threshold value, i.e., the scene is stable The image at the time t is extracted as a representative screen when both of the second condition that the image data is satisfied and the second condition are satisfied.

Further, the present invention is characterized in that when determining whether or not a scene is stable, an increase or decrease in a distance between image data obtained in advance is examined. Sand
That is, when the change in the distance decreases near the time point of time t,
It is assumed that the second condition is satisfied. Furthermore, the present invention is characterized in that when the first or second condition is not satisfied, an image cut point is detected, and when it is determined that there is a cut, the image data at time t is extracted as a representative screen. I do.

[0013]

According to the present invention, the distance between the reference image data and the image data at time t is calculated. This distance is used to evaluate the difference between the pictures in the image. Since the reference image data is fixed and changes in the pattern from there are observed,
It becomes possible to detect a very slow scene change that could not be detected by the prior art, and as a result, leakage of representative screen detection is reduced. In addition, by providing a procedure for determining whether the scene is stable or not, it is possible to prevent the representative screen from being extracted differently when the picture temporarily changes due to flash light, Also,
When a scene change due to a camera operation or a slow scene change occurs, it is possible to prevent the representative screen from being excessively extracted during the change, and further caused by subject movement or camera blur. It is possible to prevent a representative screen having poor image quality from being extracted.

Further, when determining the stability of the scene, an increase or decrease in the distance between the previously obtained reference image data and the image data at each time is examined, so that an extra image is required to check the scene stability. There is no need for processing, and the amount of calculation can be reduced. Furthermore, by combining with a cut detection procedure, a cut point having a similar pattern can be detected as a representative screen, and omission of extraction of the representative screen can be reduced.

[0015]

An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of a representative screen extracting apparatus according to the present invention. In FIG. 1, reference numeral 10 denotes an input image data sequence, and the sampling rate, image format, and image size of an image may be arbitrary. That is, the NTSC standard video signal is transmitted at 30 frames / sec.
, Or may be sampled at a coarser sampling rate. The input image data sequence 10 may be an analog signal such as NTSC, a digital signal, a hard disk, a CD-
It may be an image file stored in a storage device such as a ROM. 11 is a reference image buffer memory,
This is a memory for storing image data at a certain point in the input image data sequence 10 for reference. The reference image buffer memory 11 stores data obtained secondarily by performing a certain process on the image.
There may be . For example, it may store a luminance histogram or a color histogram for one image, or may store edge information. Also,
They may be averaged over time. Further, an image reduced in order to shorten the processing time in the calculation processing unit 13 may be stored. Here, these are collectively referred to as image data. Reference numeral 12 denotes a buffer memory for temporarily storing image data of the input image data sequence 10 near time t. The buffer memory 12 may be, for example, a frame buffer for temporarily storing image data sequentially transmitted from an image transmission source, or may be configured using a shift buffer array so as to store a plurality of image data. You may. Further, similarly to the reference image buffer memory 11, a luminance histogram,
A color histogram may be stored. A calculation processing unit 13 performs a representative screen extraction process using the reference image data of the reference image buffer memory 11 and the input image data of the buffer memory 12. This calculation processing unit 13
It is constituted by a so-called CPU having a built-in memory such as a RAM or a ROM. 14 is an output line for representative screen frame number information,
Reference numeral 15 denotes an output line for representative screen information.

FIG. 2 is a processing flowchart of an embodiment of a representative screen extracting method according to the present invention .
Of the calculation processing unit 13. First, variables s and t representing time t are initialized to 0 (step 201). At this time,
The reference image buffer memory 11 stores image data at time s = 0, that is, time t = 0, in the input image data sequence 10 as reference image data. The input image data sequence 10 after time t = 0 is sequentially stored in the buffer memory 12, and the image data sequence near time t is always stored. Here, it is assumed that a frame number is added to each image data. Next, t is advanced by 1 (step 2).
02), calculates the distance d _s between the image data I _t of the reference image data I _s and the time t at time s (t) (step 20
3). Here, the distance d _s (t) takes a positive value, becomes closer to 0 as the patterns of the two images are similar, and takes a larger value as the patterns of the two images are different. The calculation of the distance will be described later. Next, the distance d _s (t) is compared with a threshold T (step 204). Here, when d _s (t) <T, the change of the picture is regarded as small, and the process returns to step 202.

If d _s (t) ≧ T, it is considered that the picture has changed between the two images. In this case, a procedure for determining whether or not the scene is stable is called (step 20).
5) checking whether or not it is stable (step 206);
If it is not stable, the process returns to step 202. If it is determined that the image is stable, the image at time t is set as the representative screen (step 20).
7) Substitute t for s (step 208), and step 2
Return to 02. Substituting t for s means updating the reference image data to the image data at time t. The procedure for determining whether the scene is stable will be described later.

Here, in step 207, specifically, the calculation processing unit 13 sends the frame number information of the image data at the time t to the line 14, and at the same time, sends the frame number information to the buffer memory 12 through the control line 16. Give to
The image data of the corresponding frame number is read from the buffer memory 12 and transferred to the reference image buffer memory 11. Thus, the reference image data in the reference image buffer memory 11 is updated to the image data at the time t. The image data at the time t read from the buffer memory 12 is sent to the line 15 as a representative screen. This line 1
The representative screen No. 5 is stored in a secondary storage medium such as a hard disk, for example, with the frame number of the line 14 added thereto. Note that the calculation processing unit 13 may send out the image data at the time t obtained as the representative screen from the image data fetched from the buffer memory 12 for processing to the line 15 and write the image data into the reference image buffer memory 11. .

Next, the relationship between the temporal change of the distance d _s (t) and the processing flow will be described with reference to FIG. First, the image data at time s is considered as reference image data (step 20).
1). While increasing t (step 202), the distance d _s (t) between the reference image data and the image data at time t is sequentially calculated (step 203). Since the pattern of the image data differs from that of the reference image data over time, the distance d _s (t) gradually increases as shown in FIG. This distance d _s (t ₂ ) is compared with a threshold value T (step 204). At time t ₁ , since d _s (t) <T, it is assumed that the pattern has not changed sufficiently. At time t _2,
When switching from a building scene to a car scene, the distance d
_s (t) rapidly increases, and d _s (t ₂ ) ≧ T is satisfied. In this case, it is determined whether the scene is stable (steps 205 and 206). However, in the vicinity of time t ₂ , d
_Since the increase or decrease of _s (t) is large, it is determined in step 206 that the scene is not yet stable, and the process returns to step 202 without extracting the representative screen. At time t ₃ , d _s (t) starts to decrease, so it is regarded as stable, and a representative screen is extracted (step 207). Then, it sets the image data of the time t ₃ to the next reference image data. Specifically, s = t ₃
(Step 208), and returns to the original processing.

In the processing flow of FIG.
It is conceivable to omit the procedure for checking the stability of 5,206 scenes and simply consider that there is a representative screen when the distance d _s (t)> T, but this is not practical because of the following problems. (1) In the case of a slow scene change such as wipe or fade, the distance d _s (t) may exceed the threshold value T during the scene change as shown in FIG. For this reason, a representative screen may be extracted redundantly in a group of scene changes, or a cross-fade (an image editing effect in which two images overlap and a scene is switched from one scene to another scene) may occur. A representative screen in which two images overlap (which is not desirable in terms of image quality) may be extracted. (2) In a video shot of a scene in which a flash is being fired (such a scene is often seen in a news video), due to a sudden increase in luminance due to flash light, as shown in FIG. The distance d _s (t) may suddenly exceed the threshold T. Therefore, the representative screen is repeatedly extracted each time the flash is fired. In the present invention, the problems (1) and (2) are overcome by checking the stability of the scene.

Next, the procedure for calculating the distance d _s between the reference image data I _S and the image data I _t (t), second, describing the third implementation.

The first implementation uses a luminance histogram. That is, a luminance histogram with respect to the reference image I _s at time _s H s (n), the luminance histogram for the image data I _t at time _{t H s (n), n} = 1,2, ..., and N, the distance d _s (t) is calculated by [Equation 1]. Here, N is the number of steps in the histogram.

[0024]

(Equation 1)

The second implementation uses a color histogram. That is, the color histograms for the reference image at time s and the image at time t are respectively represented by H _s ′ (n _r , _ng ,
n _b ), H _t ′ (n _r , n _g , n _b ), n _r , n _g , n _b , 1, 2,.
When expressed as N, the distance d _s (t) is calculated by [Equation 2].

[0026]

(Equation 2)

In the embodiment described above, the distance d _s (t) is calculated from the feature amount based on the histogram, but the present invention is not limited to this. The distance d _s (t) may be calculated from the color information averaged in the block.

Next, a few examples of the procedure for evaluating the stability of a scene will be described.

The first implementation uses inter-frame differences. That is, the image data at time t and I _t, the coordinates (x, y) the luminance value I _t at (x, y) and represent the difference between the frames _{D (t) = Σ x,} y | I t (x , Y) −I _t−1 (x, y) |. When smaller than the threshold value theta with all the time width W there is a series of inter-frame difference, that _{is, D (tk) <θ,} k = 0,1, ..., when the W-1, the image in the vicinity of the time t It is determined that the data change is small, that is, the scene is stable.

In the second embodiment, the distance d _s (t) is also used for determining the stability of the scene. That is, when observing the distance d _s (t), if the scene is unstable, d _s (t) monotonically increases as shown in FIG. 4, or as shown in FIG.
May show an hour peak. So, for example, d
_{When s} (t-1)> d _s (t) is satisfied , the change of the image data is small.
That is, the scene may be considered to be stable, and similarly, when d _s (t−1)> d _s (t) and d _s (t−2)> d _s (t) are satisfied , May be considered stable.

FIG. 6 is a processing flow of another embodiment of the representative screen extracting method of the present invention. The method of FIG. 2 can detect a slow scene change, but cannot detect a cut point having a similar pattern. In such a case, FIG.
It is effective to use the processing flow in combination with the video cut point detection processing. In FIG.
Reference numeral 606 corresponds to steps 201 to 206 in FIG. If d _s (t) <T in step 604, or
If the stability of the scene is not detected in step 606, the process proceeds to step 610 in FIG. In step 610, a cut detection procedure is called to check whether there is a cut point in the image near time t (step 6).
11). If it is determined that there is a cut point, the flow advances to step 607 to use the image at time t as the representative screen. If no cut point is detected, the flow returns to step 602. As a result, it is possible to reduce the omission of extraction of the representative screen when there are cut points having similar patterns. The method of detecting the cut point may be any of the conventional methods mentioned above.

[0032]

As described above, according to the present invention,
It can detect editing effects such as fades and wipes, and slowly changing scenes over time due to camera operation, and can be applied to any video such as video that includes temporal noise such as flash light. The effect is that the screen can be extracted.

Further, when determining whether or not the scene is stable, it is necessary to perform extra image processing in order to check the stability of the scene by checking the increase or decrease in the distance between the previously determined image data. Therefore, there is an effect that the amount of calculation can be reduced.

Further, in combination with the cut point detection processing, when a cut is determined, a representative screen is detected, whereby
It is possible to prevent a cut point having a similar pattern from being overlooked, and to reduce the omission of extraction of the representative screen.

The present invention can be applied to, for example, a video list display or a quick view, but can also be applied to an interface of a video database. When a large amount of video is stored in the video database, it is customary to assign a keyword indicating the content of the video.However, in the past, there was a problem that a keyword alone could not be used to extract the desired scene as desired. Was. In this case, a video index or a video content list display interface is useful for helping to confirm whether or not the video that has been selected as a candidate in the keyword search is really what you want. According to the present invention, the index creation can be automated.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of a representative screen extracting device of the present invention.

FIG. 2 is a processing flowchart of a representative screen extracting method according to an embodiment of the present invention.

FIG. 3 is a diagram for explaining a relationship between a temporal change of a distance and a processing flow.

FIG. 4 is a schematic diagram illustrating a temporal change in distance in the case of a slow scene change.

FIG. 5 is a schematic diagram for explaining a temporal change in distance due to flash light.

FIG. 6 is a processing flowchart of another embodiment of the representative screen extracting method of the present invention.

FIG. 7 is a schematic diagram of a video list display.

[Explanation of symbols]

 Reference Signs List 10 Input image data string 11 Reference image buffer memory 12 Buffer memory 13 Calculation processing unit 14 Representative screen frame number information 15 Representative screen information

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int. Cl. ⁷ , DB name) G11B 20/00 H04N 5/783 H04N 5/91-5/956

Claims (4)

(57) [Claims] 1. A representative screen extracting method for extracting a representative screen from an image data sequence, wherein image data at a certain time is used as reference image data, and a distance between the reference image data and the image data at the time t is determined. sequentially calculated while changing the time t, the distance is the first that is greater than a predetermined threshold conditions and, in the image data in the temporal vicinity of the time t change the second of less than a predetermined threshold condition A representative screen extracting method, wherein image data at time t that satisfies both is extracted as a representative screen. 2. A method for extracting a representative screen from a sequence of image data.
In the front screen extraction method, image data at a certain time is used as reference image data.
Between the reference image data and the image data at time t.
The distance is sequentially calculated while changing the time t, and the distance is larger than a predetermined threshold.
The first condition and the change in the distance near the time t.
At time t that satisfies both the second condition that
A representative screen extracting method characterized by extracting image data as a representative screen . 3. The representative screen extracting method according to claim 1, wherein an image cut point is detected when the first or second condition is not satisfied, and when it is determined that there is a cut point. A representative screen extracting method, wherein the image data at the time t is extracted as a representative screen. 4. A representative screen extracting apparatus for extracting a representative screen from an image data sequence, comprising: a reference image buffer memory for storing image data at time s as reference image data; and an image data near time t. 4. A buffer memory, the reference image buffer memory, and image data read from the buffer memory, a representative screen is extracted according to the representative screen extracting method according to claim 1, and the extracted representative screen is stored in a reference image buffer. A representative screen extraction device, comprising: a calculation processing unit configured to be set in a memory.