JPH0832924A - Method for extracting key screen and device therefor - Google Patents

Abstract

PURPOSE:To reduce misdetection of a key screen and also to reduce the excessive extraction of a key image and the deterioration of picture quality by calculating the distance from a reference image to evaluate the difference of patterns and then detecting the change of scenes. CONSTITUTION:The image data obtained at a time (s) of the time t=0 of an input image data string 10 are stored in a reference image buffer memory 11 as the reference image data. Meanwhile the data strings 10 following the time t=0 are successively stored in a buffer memory 12. Then the distance between the time (s) and a time (t) near the time (s) is calculated by a prescribed means and the change and the stability of a pattern and decided. When it is judged that the pattern changes and is stable, a calculation processing part 13 outputs the frame number information on the image data of the time (t) to a line 14 and also gives this information to the memory 12. Then the part 13 transfers the image data on the corresponding frame number to the memory 11 to update the reference image data to the image data of the time (t) and also to store the updated image data in the memory via a line 15 as a key screen.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting a representative screen, and more particularly to a method and apparatus for extracting a small number of representative screens representing the contents of a sequence of a plurality of image data.

[0002]

2. Description of the Related Art Generally, the amount of video data is enormous, but the only way to know the contents is to watch the video in chronological order. Extracting a representative screen that efficiently expresses the video content from the video data is useful for grasping the outline of the video. Ideally, it is necessary to select a representative screen in consideration of the story, but at present, the work can only be done manually, and the amount of work becomes enormous, which is unrealistic.

[0003] A conventional technique related to the automation of representative screen extraction will be described below with reference to a few application examples.

The first application example relates to a video list display. FIG. 7 shows a schematic diagram of the video list display. When you want to know the contents of the video tape, or when you want to find the necessary part in the video tape, conventionally, you have to use the fast-forward and rewind functions of the VCR,
There was a problem that it took time and effort. If a representative screen is automatically extracted from a video tape and displayed on a medium such as a display or paper, a list of video contents can be displayed, and a rough understanding of the video contents can be achieved in a short time. As a conventional technique related to this, for example, Japanese Patent Publication No. 5-
No. 74273, "Index image creating device", Japanese Patent Laid-Open No.
No. 4-11483 "Video Printer", Japanese Patent Application No. 5-19
5644 "Video image printing method and apparatus" and the like. Among them, the index image creating apparatus of Japanese Examined Patent Publication No. 5-74273 adopts a method of extracting the cut point or the image data immediately before the cut point as a representative screen.
The cut point is a joint between shots (video sections continuously shot by a camera), and if it is detected, a representative screen can be selected one by one for each shot. Specifically, the difference value sequence between images is calculated to determine the presence or absence of image change. Also, JP-A-64-11
The 3483 video printer is characterized in that an image after a certain time from the cut point is extracted as a representative screen and printed on paper. This is based on the rule of thumb that major screens often appear in the middle of a shot. Further, in the video image printing method and apparatus of Japanese Patent Application No. 5-195644, the reproduced video signal is subjected to image processing, and an image which meets a specific condition is treated as an event indicating a large change in the video which is important for grasping the video content. It's pulled out.

The second application is related to video viewing. It becomes possible to automatically generate a quick-view video by cutting out each video segment that is distinguished by cut points little by little and combining them (Otsuji, Tonomura “Subjective evaluation of high-speed moving image branding”, electronic information Communication Society Spring Meeting, SD9-3, 1993 "). Also in this method, the cut point is detected and the image immediately after or immediately before is used as the representative screen.

The prior art of the image cut point detection method will be described below. As a typical method of detecting the video cut point, the difference between the luminance values of the corresponding pixels of two temporally adjacent images (the image at time t and the image at time t−1) is calculated, and the absolute value is calculated. The sum of the values (difference between frames) is D (t)
And when D (t) is greater than a given threshold,
There is a method in which the time t is regarded as a cut point (Otsuji, Tonomura, Ohba, “Browsing video using luminance information”, IEICE technical report, IE90-103, 1991).
In addition, instead of the inter-frame difference, a pixel change area, a luminance histogram difference, a block-by-block color correlation, etc. may be used as D (t) (Otsuji, Tonomura: “Consideration of automatic video cut detection method”, Television Society Technical Report, Vo
l. 16, No. 43, pp. 7-12). Also, D
There is also a method in which (t) is not subjected to the threshold processing as it is, but the result obtained by applying various time filters to D (t) is subjected to the threshold processing (K. Otsuji and Y. Tonomura: “Proj.
ection Detecting Filter for Video Cut Detectio
n ”, Proc. of ACM Multimedia 93, 1993,
pp. 251-257). This method has a feature that erroneous detection is unlikely to occur even if there is a moving object or flash light in the image.

[0007]

In the above prior art, the representative screen is selected based on the cut points.
The image just after the cut point or after a certain time from that point is used as the representative screen. However, there is a problem in that the image immediately after the cut point may not be in focus on the camera or may be blurred due to strong movement of the subject, and thus may not be appropriate in terms of image quality as a representative screen. In the application of list display, it is desirable not to extract a representative screen with poor image quality.

Further, the conventional image cut detection method has a problem that it is not possible to detect a scene change which is slow in time. This is because the amount indicating the rate of change of the scene is calculated from only two frames that are temporally adjacent to each other, and a long-time scene is hardly reflected. Regarding this, for example, Japanese Patent Application No. 5-31766.
No. 3, "Video Cut Point Detection Method and Apparatus" solves this problem by calculating the distance between a plurality of sets of image data between images that are temporally separated in addition to adjacent images. However, there is still a problem that a scene change that changes very slowly in a long time may not be detected. In other words, even if a scene change that can be perceived by human beings as a complete scene change is not detected, there is a problem in that the representative screen corresponding to that scene is missed.

In addition, even if a pattern is changed by a camera operation such as panning (swinging the camera horizontally) or tilting (swing the camera vertically), it is sometimes desired to extract the image after the camera operation as a representative screen. The method couldn't do it.

An object of the present invention is to detect an editing effect such as a fade and a wipe or a scene change caused by a camera operation with time, and secondly an image including temporal noise such as flash light. A third object of the present invention is to provide a representative screen extracting method and apparatus capable of adapting to any video and thirdly extracting a representative screen suitable for image quality.

[0011]

In order to achieve the above object, the present invention uses image data at a certain time as reference image data and determines the distance between the reference image data and the image data at time t. Sequential calculation is performed while changing the time t, and both the first condition that the distance is larger than a predetermined threshold value and the second condition that the scene is stable near the time t are both satisfied. At times, the image at the time t is extracted as a representative screen.

Further, the present invention is characterized in that, when determining whether or not the scene is stable, the increase / decrease of the distance between the image data which is obtained in advance is examined. further,
The present invention is characterized in that image cut point detection is performed when the first or second condition is not satisfied, and image data at time t is extracted as a representative screen when it is determined that there is a cut.

[0013]

In the present invention, the distance between the reference image data and the image data at time t is calculated. This distance is used to evaluate the difference in the pattern of the image. Since the reference image data is fixed and the change in the pattern from that point is observed,
It becomes possible to detect a very slow scene change that cannot be detected by the conventional technique, and as a result, there is less omission in representative screen detection. In addition to that, by providing a procedure to determine whether the scene is stable, it is possible to prevent the representative screen from being extracted differently when the pattern is temporarily changed by the flash light, Also,
When a scene change due to camera operation or a slow scene change occurs, it is possible to prevent excessive extraction of the representative screen during the change, which is caused by the movement of the subject or the blur of the camera. It is possible not to extract a representative screen with poor image quality.

In addition, when determining the stability of the scene, it is possible to check the increase or decrease in the distance between the reference image data and the image data at each time, which has been obtained in advance, so that an unnecessary image for checking the scene stability can be obtained. The amount of calculation can be reduced without the need for processing. Further, by combining with the cut detection procedure, it is possible to detect a cut point having a similar pattern as a representative screen, and it is possible to reduce extraction omission of the representative screen.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of an embodiment of a representative screen extracting device according to the present invention. In FIG. 1, 1
0 is an input image data string, and the image sampling rate, image format, and image size may be arbitrary. That is, the NTSC standard video signal may be sampled at 30 frames / sec or may be sampled at a coarser sampling rate. The input image data string 10 may be an analog signal such as NTSC, a digital signal, a hard disk, a CD-ROM.
Etc., the image file stored in the storage device may be used. A reference image buffer memory 11 is a memory for storing image data at a certain point in the input image data string 10 for reference. The reference image buffer memory 11 may be one that stores data obtained by secondarily applying a certain process to the image. For example, a brightness histogram or a color histogram for one image may be stored, or edge information may be stored. Also, they may be averaged over time. Furthermore, a reduced image may be stored to reduce the processing time in the calculation processing unit 13. Here, these are collectively referred to as image data. Reference numeral 12 is a buffer memory for temporarily storing the image data near the time t in the input image data string 10. The buffer memory 12 may be, for example, a frame buffer for temporarily storing image data sequentially sent from an image transmission source, or configured by using a shift buffer array so that a plurality of image data can be stored. May be. Further, similarly to the reference image buffer memory 11, a luminance histogram and a color histogram may be stored. A calculation processing unit 13 performs a representative screen extraction process using the reference image data of the reference image buffer memory 11 and the input image data of the buffer memory 12. The calculation processing unit 13 is composed of a so-called CPU having a built-in memory such as a RAM or a ROM. Reference numeral 14 is a representative screen frame number information output line, and 15 is a representative screen information output line.

FIG. 2 is a processing flowchart of an embodiment of the representative screen extracting method of the present invention. The calculation processing section 13 of FIG. 1 is responsible for this processing. First, variables s and t representing time t
Is initialized to 0 (step 201). At this time, the image data at time s = 0, that is, at time t = 0 in the input image data string 10 is stored in the reference image buffer memory 11 as reference image data. The buffer memory 12 sequentially stores the input image data sequence 10 after time t = 0, and always stores the image data sequence near time t.
Here, it is assumed that a frame number is added to each image data. Then, t is advanced by 1 (step 20
2) Calculate the distance d _s (t) between the reference image data I _s at time _s and the image data I _t at time t (step 20).
3). Here, the distance d _s (t) takes a positive value and becomes closer to 0 as the patterns of the two paste images are more similar, and takes a larger value as they are different. The calculation of the distance will be described later. Next, the distance d _s (t) is compared with the threshold T (step 204). Here, if d _s (t) <T,
The change in the pattern is considered small, and the process returns to step 202.

When d _s (t) ≧ T, it is considered that the pattern has changed between the two images. In this case, subsequently, a procedure for determining whether or not the scene is stable is called (step 20).
5) check if it is stable (step 206),
If it is not stable, the process returns to step 202. If it is determined to be stable, the image at time t is set as the representative screen (step 20
7), substituting t for s (step 208), step 2
Return to 02. Substituting t into s means updating the reference image data to the image data at time t. The procedure for determining whether or not the scene is stable will be described later.

Here, in step 207, specifically, the calculation processing section 13 sends the frame number information of the image data at the time t to the line 14, and at the same time, the frame number information is sent to the buffer memory 12 through the control line 16. Give to
The image data of the corresponding frame number is read from the buffer memory 12 and transferred to the reference image buffer memory 11. As a result, the reference image data in the reference image buffer memory 11 is updated to the image data at time t. Further, the image data at time t read from the buffer memory 12 is sent to the line 15 as a representative screen. This line 1
The representative screen of No. 5 is stored in a secondary storage medium such as a hard disk with the frame number of the line 14 added thereto. Note that the calculation processing unit 13 may send the image data at the time t obtained as the representative screen out of the image data fetched from the buffer memory 12 for processing to the line 15 and write it in the reference image buffer memory 11. .

Next, the relationship between the temporal change of the distance d _s (t) and the processing flow will be described with reference to FIG. First, consider the image data at time s as reference image data (step 20).
1). While increasing t (step 202), the distance d _s (t) between the reference image data and the image data at time t is sequentially calculated (step 203). Since the pattern of the image data is different from that of the reference image data as time passes, the distance d _s (t) gradually increases as shown in FIG. This distance d _s (t ₂ ) is compared with the threshold value T (step 204), but at the time t ₁ , d _s (t) <T, so it is considered that the pattern has not changed sufficiently. At time t ₂ ,
When switching from the building scene to the car scene, the distance d
_s (t) rapidly increases, and d _s (t ₂ ) ≧ T is satisfied. In this case, it is subsequently determined whether the scene is stable (steps 205 and 206). However, in the vicinity of time t ₂ , d
_Since the increase / decrease in _s (t) is large, it is determined in step 206 that the scene is not yet stable, and the flow returns to step 202 without extracting the representative screen. Since d _s (t) starts to decrease at time t ₃ , it is considered stable, and the representative screen is extracted (step 207). Then, the image data at time t ₃ is set as the next reference image data. Specifically, s = t ₃
(Step 208), the process returns to the original process.

In the process flow of FIG. 2, step 20
It is possible to omit the procedure for inspecting the stability of the scenes of 5,206 and simply consider that there is a representative screen when the distance d _s (t)> T, but it is not practical because of the following problems. (1) In the case of a slow scene change such as wipe or fade, as shown in FIG. 4, the distance d _s (t) may exceed the threshold T during the scene change. For this reason, the representative screens may be duplicated and extracted during a group of scene changes, or 2 in the middle of a crossfade (video editing effect in which two images overlap and switch from one scene to another). A representative screen in which images are overlapped (not preferable in terms of image quality) may be extracted. (2) In a video shot of a scene in which a flash is fired (though such a scene is often seen in news footage), as shown in FIG. 5, due to a sudden increase in brightness due to flash light, The distance d _s (t) may suddenly exceed the threshold value T. Therefore, the representative screen is repeatedly extracted each time the flash is fired. In the present invention, by checking the stability of the scene,
It overcomes the problems of (1) and (2) above.

Next, a few examples of the procedure for calculating the distance d _s (t) between the reference image data I _S and the image data I _t will be described.

The first implementation uses a luminance histogram. That is, the luminance histogram for the reference image I _{s at} time s is H _s (n), the luminance histogram for the image data I _{t at} time t is H _s (n), n = 1, 2, ..., N, and the distance d _{s is set.} Calculate (t) by [Equation 1]. However, N is the number of steps in the histogram.

[0024]

[Equation 1]

The second implementation uses a color histogram. That is, the color histograms for the reference image at time s and the image at time t are H _s ′ (n _r , n _g ,
n _b ), H _t ′ (n _r , n _g , n _b ), n _r , n _g , n _b , 1, 2, ...
When represented as N, the distance d _s (t) is calculated by [Equation 2].

[0026]

[Equation 2]

In the implementation example described above, the distance d _s (t) is calculated from the feature amount based on the histogram, but the present invention is not limited to this. The distance d _s (t) may be calculated from the color information averaged by the blocks.

Next, a few examples of the procedure for evaluating the stability of the scene will be described.

The first implementation uses the interframe difference. That is, the image data at time t is I _t , the brightness value at coordinates (x, y) is represented by I _t (x, y), and the inter-frame difference is D (t) = Σ _{x, y} | I _t (x , Y) -It _-1 (x, y) | When the sequence of inter-frame differences is less than a certain threshold θ in a certain time width W, that is, when D _(tk) <θ, k = 0, 1, ..., W-1, the scene near the time t Is stable.

The second implementation also uses the distance d _s (t) to determine the stability of the scene. That is, when observing the distance d _s (t), when the scene is unstable, d _s (t) monotonically increases as shown in FIG. 4, or as shown in FIG.
May show peak for one hour. So, for example, d
_The scene may be considered to be stable when _s (t-1)> _ds (t), or _ds (t-1)> _ds (t) and _ds (t-2).
A scene may be considered stable when> d _s (t) is satisfied.

FIG. 6 is a processing flow of another embodiment of the representative screen extracting method of the present invention. The method of FIG. 2 can detect a slow scene change, but may not be able to detect cut points having similar patterns. In such a case,
It is effective to combine the processing flow of 1) with the video cut point detection processing. In FIG. 6, steps 601 to
Reference numeral 606 corresponds to steps 201 to 206 of FIG. 2, respectively. If d _s (t) <T in step 604, or
If the stability of the scene is not detected in step 606, the process proceeds to step 610 in FIG. In step 610, the cut detection procedure is called and the image near the time t is inspected for a cut point (step 6).
11). If it is determined that there is a cut point, the flow advances to step 607 to set the image at time t as the representative screen, and if no cut point is detected, the flow returns to step 602. As a result, it is possible to reduce the omission of extraction of the representative screen when there are cut points with similar patterns. The method of detecting the cut point may be any of the conventional methods mentioned above.

[0032]

As described above, according to the present invention,
It is possible to detect editing effects such as fades and wipes and scene changes that occur slowly with camera operation, and it can be applied to all images including temporal noise such as flash light. The effect is that the screen can be extracted.

When determining whether or not the scene is stable, it is necessary to perform extra image processing in order to check the scene stability by checking the increase / decrease of the distance between the image data which is obtained in advance. This has the effect of reducing the amount of calculation.

Furthermore, in combination with the cut point detection processing, by detecting the representative screen when it is determined that the cut is made,
It is possible to prevent the cut points having similar patterns from being overlooked and reduce the omission of the representative screen.

The present invention can be applied to, for example, a video list display or a quick view, but can also be applied to an image database interface. When a large amount of video is stored in the video database, it is customary to add a keyword that represents the video content, but conventionally there is the problem that the keyword alone cannot be used to derive the desired scene. It was In this case, a video index or video content list display interface is useful in order to confirm whether the videos selected as candidates in the keyword search are really what one wants. By using the present invention, this index creation can be automated.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of a representative screen extraction device of the present invention.

FIG. 2 is a processing flow chart of an embodiment of a representative screen extracting method of the present invention.

FIG. 3 is a diagram for explaining a relationship between a time change of a distance and a processing flow.

FIG. 4 is a schematic diagram for explaining a temporal change in distance in the case of a slow scene change.

FIG. 5 is a schematic diagram for explaining a temporal change in distance due to flash light.

FIG. 6 is a processing flow chart of another embodiment of the representative screen extracting method of the present invention.

FIG. 7 is a schematic diagram of displaying a list of videos.

[Explanation of symbols]

 10 input image data string 11 reference image buffer memory 12 buffer memory 13 calculation processing unit 14 representative screen frame number information 15 representative screen information

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication H04N 5/937

Claims (4)

[Claims] 1. A representative screen extracting method for extracting a representative screen from an image data string, wherein image data at a certain time is used as reference image data, and a distance between the reference image data and the image data at time t is calculated. Time is sequentially calculated while changing t, and both the first condition that the distance is larger than a predetermined threshold value and the second condition that the scene is stable near the time t are both. A representative screen extracting method, characterized in that image data at a time t to be satisfied is extracted as a representative screen. 2. The representative screen extracting method according to claim 1, wherein whether or not the scene is stable near time t is determined between the reference image data and the image data near time t. A representative screen extraction method characterized by being performed by checking increase and decrease in distance. 3. The representative screen extraction method according to claim 1, wherein image cut point detection is performed when the first or second condition is not satisfied, and when it is determined that there is a cut point. A representative screen extracting method characterized by extracting the image data at the time t as a representative screen. 4. A representative screen extracting device for extracting a representative screen from an image data string, a reference image buffer memory for storing image data at time s as reference image data, and image data in the vicinity of time t. A buffer memory, the reference image buffer memory, and image data read from the buffer memory, a representative screen is extracted according to the representative screen extraction method according to claim 1, and the extracted representative screen is used as a reference image buffer. A representative screen extraction device, comprising: a calculation processing unit set in a memory;