JP4007406B2 - Feature scene detection method for moving images - Google Patents

Description

The present invention relates to a method and apparatus for quickly seeing a moving image such as a video or a movie in a short time, and in particular, cuts from a moving image stored on a video tape or a video disk (captured by one camera). By detecting a dissolve (a special video effect in which B fades in at the same time as A fades out when there are continuous cuts A and B when there are continuous cuts A and B). The present invention relates to a method and an apparatus for detecting a feature scene of a moving image that identifies a scene representing the scene.

In recent years, in addition to normal TV broadcasting, satellite broadcasting and cable TV are becoming popular.
Multi-channel broadcasting is progressing. In the future, if a broadband communication infrastructure called an information highway is established, it will be easier to distribute broadcasts, and more broadcasters will enter the market, and the number of channels will be accelerated. It is very time consuming and time consuming to distinguish and select useful information and unnecessary information for individual viewers from such a large amount of broadcasted information. For this reason, research on techniques for efficiently creating summary information (digests) for quickly grasping video content is underway. The most basic and indispensable process for creating a digest is to select important scenes from the video. If the importance of a scene in a video can be automatically judged by a computer, the creation of a digest becomes very easy. For example, in Japanese Patent Laid-Open No. 4-294694, in a baseball broadcast, the correspondence between the movement result of a moving object in a video and a specific event (corresponding to the movement of the runner to the home position and the score, etc. A method for selecting a scene with high importance is shown.

JP-A-4-294694

However, the motion analysis of moving objects is not sufficient in accuracy and processing speed at the current state of the art of image recognition, and the correspondence between the obtained motion pattern and a specific event does not always correspond. There is a point. Even if an event can be detected correctly, it is extremely difficult to automatically determine which range before and after it should be extracted as an important scene. In addition, the digest itself is much shorter than viewing the entire video, but there is still a need to watch over a certain amount of time, and there is a need for a technology that allows a more concise overview. .

SUMMARY OF THE INVENTION An object of the present invention is to provide a method for determining whether an important scene in an image is important and specifying the range thereof easily and at high speed. Another object is to determine and classify what field (news, sports broadcast, etc.) the video belongs to, and provide it as information that helps the user to select the video.

In many cases, the broadcast video has various video effects that emphasize important scenes on the broadcasting station side. This property is particularly noticeable in the case of sports broadcasts. For example, when a score is entered, a replay is broadcast. Replay images are often taken from cameras with different viewpoints, and it is not possible to determine whether a replay image is simply the same or not. Special video effects such as these are used, and it is designed to allow viewers to clearly see that they are temporarily out of normal broadcasting. Furthermore, the same video effect is used when returning to normal broadcasting. Therefore, by detecting these special video effects,
It is possible to select important scenes.

Therefore, the target moving image is input to the processing device in a time series in units of frames, and in the processing device, the color or luminance of each pixel in the frame spans a plurality of consecutive frame groups. Check whether the color or brightness value of the first frame of the image has a monotonous trend toward the color or brightness value of the last frame, and change the number of pixels that satisfy the condition When the evaluation value is calculated and the evaluation value falls outside the predetermined allowable range, it is determined that there has been a scene change due to a special video effect such as a dissolve in the section spanning the continuous frames. Then, the section or the vicinity thereof is determined as a characteristic point in the moving image.

In addition, the target moving image is input to the processing device in time series in units of frames, or the target sound is input to the processing device in time series. In the processing device, a plurality of types including cut changes and color tones are input. A means for detecting a change in the image feature amount and a means for detecting a change in the speech feature amount including a speaker change as necessary are provided, and the detection means detects that a change has occurred, or a plurality of changes occur simultaneously or The type of program is discriminated based on a feature amount that is generated in a specific order.

According to the present invention, it is possible to obtain an important scene and its range at the same time, and to produce a digest video automatically. In general, scenes to be replayed are often important scenes. However, in the present invention, a replay scene being broadcast can be accurately detected by detecting a section of a special video effect including a dissolve.

Furthermore, the type of video is automatically determined by means of determining the type of program based on the feature amount consisting of multiple types of image feature amount changes, including cut changes and color tones, occurring simultaneously or in a specific order. Therefore, if the video is of a type that is not of interest to the viewer, it can be rejected without seeing the digest video, and the video can be efficiently selected. In addition, in the determination of the video type, since the determination is performed based on a simple change in image and sound and a combination thereof, the processing can be performed at high speed.

The scenes that are replayed by broadcasting are the parts that the expert has determined to be important. If such replay scenes can be detected, it is very easy to create a digest. According to the present invention, since a scene change due to a special video effect including a dissolve can be detected, it is possible to accurately extract an important scene that is flowed before and after the special effect. At the same time, the range of the scene can be obtained.

Furthermore, the type of video is automatically determined by means of discriminating the type of the program based on the feature amount consisting of multiple types of image feature amount changes including cut changes and color tones simultaneously or in a specific order. Therefore, if the video is of a type that is not of interest to the viewer, the video can be rejected without seeing the digest video, allowing efficient video selection. In addition, in the determination of the video type, since the determination is performed based on a simple change in image and sound and a combination thereof, the processing can be performed at high speed.

  Hereinafter, an embodiment of the present invention will be described in detail.

FIG. 1 is an example of a schematic block diagram of a system configuration for realizing the present invention. 1 is C
It is a display device such as RT, and displays the output screen of the computer 4. Commands to the computer 4 can be performed using an input device 5 such as a keyboard or a pointing device. A moving image reproduction device 10 is a tuner device for receiving broadcast programs such as terrestrial broadcasting, satellite broadcasting, and cable television, or a device for reproducing moving images recorded on an optical disk, a video tape, or the like. The video signal output from the moving image reproducing apparatus is sequentially converted into digital image data by the A / D converter 3 and sent to the computer.
Inside the computer, the digital image data enters the memory 9 via the interface 8 and is processed by the CPU 7 in accordance with a program stored in the memory 9. When each frame of a moving image handled by 10 is assigned a number (frame number) in order from the beginning of the moving image, the frame number is sent to the moving image reproducing device by the control line 2, thereby moving the moving image of the scene. Images can be recalled and played back. In addition, various information can be stored as needed.
Can be stored in the external information storage device. The memory 9 stores various data created by the processing described below and is referred to as necessary.

In the following, a method for detecting a dissolve, which is one of cut changes caused by special image effects, in selecting an important scene will be described in detail.

FIG. 2 is an example of a flowchart of a moving image dissolve detection program executed on the system shown in FIG. The program is stored in the memory 9, and the CPU 7 first sets various variables necessary for program execution as initial values as initialization processing (200).
. Next, 0 is substituted into each element of the m two-dimensional array B (x, y) that stores the luminance value of each pixel of the past frame image (202). When the frame image size is w × h, x is 0 to w-1,
y takes a value from 0 to h-1. In the process 204, the frame image output from the moving image playback apparatus 10 is captured (204). A process 206 sets a variable eval in which an evaluation value is entered to 0, and assigns an initial value 0 to the loop counter. Then, the following processes 208 to 228 are performed for all the pixels in the frame image.

In the processing from 208 to 228, the characteristic peculiar to the dissolve is detected. Here, as shown in FIG. 3, the dissolve is a cut change having a section in which the frame images A and C of the preceding and following cuts are mixed like B before and after the cut change. The mixing ratio of A and C in B is as follows. From the state where A is 100% and C is 0% at the start of the dissolution, the ratio reverses over time, and finally A is 0% and C is 100%. At this point, the dissolve is complete. In the case of a grayscale image, the luminance value of A is Ba, the luminance value of B is Bb, the luminance value of C is Bc, and the mixing ratio of C is α (0 ≦
When α ≦ 1), it can be approximated by the equation Bb = Ba × (1−α) + Bc × α. If this equation is modified, Bb = (Bc−Ba) × α + Ba, and in the case of a dissolve in which the mixing ratio α increases monotonically from 0, the value of Bb also monotonously increases or decreases from Ba to Bc. Therefore,
The resolution can be detected by always storing the luminance value of the pixel in the buffer for the past m frames and checking whether the luminance value monotonously increases or decreases in the section of the m frame length. If the value of m is set to about 8 to 15, good results can be obtained experimentally.

First, in process 208, the luminance value of the pixel represented by coordinates (x, y) is substituted into the m-th array Bm of the two-dimensional array B storing the luminance values of the past frames. Then, 1 is assigned to the loop counter i, and 0 is assigned to the variable num. Next, the brightness value B1 (x, y) stored in the first array
Are compared with the values of the m-th array Bm (x, y) (212), and then the luminance value Bi (x, y) stored in the i-th array is the next array Bi + 1 (x, It is compared whether it is larger than the value of y) (214,
216). When B1 (x, y) is larger than Bm (x, y), the value of num is incremented by 1 when Bi (x, y) is larger than Bi + 1 (x, y).
Conversely, when B1 (x, y) is smaller than Bm (x, y), Bi (x, y) is smaller than Bi + 1 (x, y).
The value of num is increased by 1 (218). In the subsequent process 220, by substituting Bi + 1 (x, y) for Bi (x, y), the m arrays B are sequentially shifted one by one, and always counted from the latest frame. The luminance values for m frames are stored as a buffer. Process 22
In 2, the loop counter i is incremented by one, and at the time of processing 212 until i becomes larger than m.
When B1 (x, y) is larger than Bm (x, y), the process 214 is performed. Otherwise, the process 216 is performed.
The processing is repeated after returning to (224). When num is larger than the threshold value th1 (226), the value of eval is increased by 1 because the pixel at the coordinate (x, y) has increased or decreased sufficiently monotonously (228). Natural moving images usually have irregular fluctuations due to noise and the like, and the speed of the dissolve is uneven when humans perform a dissolve operation, and is not constant. A margin is given by providing. In order to perform the above process for all the pixels in the frame image, the process returns to 208 and is repeated (230 to 236). As a result, the number of pixels satisfying the characteristics of the dissolve enters eval.
Finally, it is checked whether or not eval exceeds the threshold value th2 (238), and if it exceeds, it is determined that there is a dissolve, and a dissolve detection process (240) is executed. Finally, the processing returns to the processing 204, and the processing from 204 is repeated until the end of the video.

In the above method, even when there is a camera movement such as zooming and panning, eval appears high. This is because if the camera moves, the luminance of each pixel in the frame image changes accordingly, and there are not a few pixels whose luminance increases or decreases monotonously. For this reason, there are cases where it is difficult to distinguish between a dissolve and a camera movement. Therefore, in the following, a description will be given of a method for detecting a dissolve so that the dissolve can be understood more clearly.

In general, the resolution time is often 1 second (30 frames in the case of NTSC video) or more. Therefore, in the section where the dissolution is applied, the value of eval continues to be high for 22 frames when m = 8 and for 15 frames or more even when m = 15. On the other hand, in the case of camera movement, the value is not as high as in the case of a dissolve, and the high state does not always continue. Therefore, when the sum total of the eval values for the past n frames is taken, a significant difference appears between the sum value during the dissolve and the sum during the camera movement.
FIG. 4 shows a dissolve detection method to which the above concept is added.

First, as initialization processing, various variables necessary for program execution are set to initial values (400). Next, m two-dimensional arrays B (
While substituting 0 for each element of x, y), n variables E1 to En for storing eval values for the past n frames are all set to 0 (402). When the frame image size is w × h, x takes a value from 0 to w−1, and y takes a value from 0 to h−1. In process 404, the frame image output from the moving image playback apparatus 10 is captured (404). Thereafter, the processing from 206 to 236 shown in FIG. 2 is executed to obtain eval (406). Then, assign the value of eval to En. The sum from E1 to En is obtained in sum and shifted while substituting Ej + 1 values one after another into Ej so that the latest eval values are always stored in E1 to En (408 to 412). . Finally, sum is the threshold t
It is determined whether it is larger than h3 (414). If it is larger, a dissolve detection process 240 is performed. If not, nothing is done and the process returns to process 404 and is repeated.

In the dissolve detection process 240, a scene sandwiched between the dissolves is selected as an important scene.
When the dissolve detection method of FIG. 2 and FIG. 4 is executed, a graph showing the time transition of the evaluation value as shown in FIG. 5 can be obtained. The evaluation value does not show a large value for a moment in the dissolve interval, but has a feature of showing a triangular change that rapidly increases and decreases rapidly. And
The two vertices that form the base of the triangle almost correspond to the start and end points of the dissolve. When creating a digest, it is unsightly if a part with a special image effect such as a dissolve remains at the beginning or end. To do. For this reason, in addition to the first threshold value 500 used for determining whether or not it is a dissolve in the above-described dissolve detection method, a lower second threshold value 502 is used.
Then, when a dissolve is detected as the starting point of the important scene, a point 506 where the evaluation value is below the second threshold for the first time after the point 504 where the evaluation value exceeds the first threshold is set as the starting point of the important scene. . At this time, the start point may be delayed with a margin. In addition, when a dissolve is detected as an end point of an important scene, a point 508 that falls below the second threshold for the first time when the evaluation value goes back to the past from the point 510 where the evaluation value exceeds the first threshold is important. The end point of the scene. At this time, like the start point, the end point may be set for an earlier time with a margin. The time between dissolves can be used to determine whether the detected dissolve indicates the start point or the end point of an important scene. If normal broadcasting continues, there will be no dissolve, so the time interval between dissolves will be longer, and if it is an important scene, the interval will be relatively short. The important scenes obtained in this way can be played in order to make a digest.

In the above embodiment, a monotonous change in luminance was examined, but a similar change in color can also be used. The color is three-dimensional information unlike the luminance which is one-dimensional information. Therefore, it is not possible to examine monotonic changes based simply on increasing and decreasing values. Here, the monotonous change from A to B means that when mapping two colors to a three-dimensional color space, the distance from A is gradually increased while the distance from A is gradually increased. Can be viewed as a trend. Therefore, instead of the two-dimensional array B that stores the luminance values of the past frames in FIG. 2, a two-dimensional array B ′ that stores colors is used, and each color in B ′ increases in color difference from B′1. At the same time, if it is determined that the color difference from B′m is reduced, the same method as in the case of luminance can be used.

The fact that scenes using special video effects such as dissolve as described above can be regarded as important scenes is actually limited to some programs such as sports broadcasts. In addition, special video effects frequently appear in commercials inserted between sports programs, so there are many cases where excessive detection is performed under the condition of a section sandwiched between dissolves. Of course, there is no problem in practical use if the video is sufficiently shorter than the original video to detect more. However, if important scenes can be extracted with higher accuracy, the time taken for grasping the outline can be further saved. Therefore, a means for distinguishing what kind of video is the video for which a digest is to be created is used to select an important scene.

FIG. 6 and FIG. 7 illustrate events that occur in news programs and sports programs, respectively, along the time axis. Here, let us consider the point that the characteristics of images and sounds change greatly as events. In the figure, seven items of 1) composition, 2) color tone, 3) speaker, 4) subtitle, 5) dissolve, 6) replay, and 7) slow playback are given as examples. The appearance and combination of such events has characteristics depending on the type of program, and programs can be classified based on the characteristics. For example, in a news program, cuts with casters appearing multiple times appear in time, so the image with the same composition, more specifically, the skin color that is the face color occupies a large area near the center. There is a feature that an image appears multiple times. In addition, there is a feature that the speakers at that time are often the same person or that subtitles appear frequently in the entire program.
On the other hand, in the case of sports broadcasts, broadcasting is often performed while switching a plurality of cameras installed at fixed positions, and images with the same or very similar composition frequently appear. Especially in baseball and soccer, the color is green, which is the color of the lawn. Another feature is that replay and slow playback are frequently used. Furthermore, in the case of CM, there are few sound interruptions, B
It is characterized by frequent use of GM, vivid colors, many cuts, and short time length. In this way, the type of video can be estimated to some extent from the combination pattern of a plurality of events in the video. The events listed here are image recognition /
In the need of speech recognition technology, it is relatively easy to find and has high reliability. That is, there is no need to recognize the semantic content of a video such as a story.

FIG. 8 is an example of a block diagram of a system for discriminating video types. The input video is digitized by the image capturing unit 800 and the sound capturing unit 802 for each of the image signal and the audio signal. The digitized data is sent to the event detection unit 804, where 804
Event detection processing is performed by dedicated detection units 806 to 820 provided for each type. The detected events are counted by event type by the event counter 822. The co-occurrence counter unit 824 sets the counter corresponding to the combination of events to 1 only when a plurality of events appear simultaneously or in a prescribed order.
increase. The appearance frequency distribution of various events obtained by these counters is compared and collated by the comparison unit 828 to determine which type of program the appearance frequency distribution of the event is close to.

  Next, each block in FIG. 8 will be described in detail.

Of the event detection unit 804, the cut point detection unit 806 detects a cut transition. Regarding the method, for example, by the inventors, IPSJ Journal Vol.33, No.4,
Methods such as “automatic indexing method and object search method in color video image” and Japanese Patent Laid-Open No. 4-111181 can be used. The event-specific counter unit 822 counts the number of cut points.

The same composition detection unit 806 detects whether a picture having the same composition or a similar composition has appeared in the past within a predetermined time. For this, an image comparison method represented by template matching can be used. Specifically, for each pixel at the same coordinate position in the two frame images to be compared, the luminance difference or the color difference is obtained and the total for the entire screen is taken, and this is taken as the difference between the images. To do. If the degree of difference is smaller than a predetermined threshold, it can be determined that they are the same or similar. Here, it takes time to detect whether all the frame images in the video have the same composition or not, and it is useless considering the characteristics of moving images that have high image similarity between consecutive frame images. . Therefore, only the image of the cut point is examined in conjunction with the cut point detection. The counter for each event counts the number of frames having the same composition.

The color tone detection unit 810 detects whether a picture having the same color tone or a similar color tone appears in the past within a predetermined time. For this, for example, a chromaticity distribution for the entire frame screen can be used. This is a feature quantity that represents how many colors are used regardless of the composition. Specifically, for each of the two frame images to be compared, the color of the pixel representing the image is classified into about 64 colors, and how many of these colors exist in the frame image is counted. Then, the sum of the absolute values of the differences between the frequencies in the obtained frequency distribution is used as the color difference. If the degree of difference is smaller than a predetermined threshold, it can be determined that they are the same or similar. For the same reason as the composition, the color tone is efficient when only the cut point image is targeted. In the event counter unit, the number of frames having the same color tone is counted. Further, the color tone detection unit may check which color is most frequently used by using the frequency distribution obtained in the middle. Specifically, a counter for each color such as red, blue, and green is prepared in the event-specific counter section, and if there are many red colors, the red counter is increased, and if there are many greens, the green counter is increased. To.

The caption detection unit 812 detects whether captions appear in the video. As the method, for example, the method shown in Japanese Patent Application No. 5-330507 by the inventors can be used. The event counter 822 counts the number of subtitles.

The dissolve detection unit 814 detects special effects such as a dissolve in the video. The method is as described in the first half of the present invention. The event counter 822 counts the number of occurrences of the dissolve.

The replay detection unit 816 detects whether or not the same video appears in the past within a predetermined time. This can be done by comparing frame images by template matching or the like as in the same composition detection unit 808. However, if template matching is performed for each frame between moving images to be compared, it takes too much processing time.
Each frame is converted into a code of about several characters, and the collation of the code string is used as the collation of the moving image. A single code corresponding to one frame has a very small amount of information. However, since a moving image is composed of many frames, a single moving image includes a large number of codes, and a sequence of codes in the moving image. Has enough information to identify a piece of video.
A method for collating moving images based on this concept is disclosed in Japanese Patent Laid-Open No. 7-11456 by the inventors.
It is shown in No.7.

The slow playback detection unit 818 detects slow playback video. Slow playback is realized by continuously displaying frame images at longer intervals than normal playback (2 times at 1/2 slow, 4 times at 1/4 slow). The image digitized by the image capturing unit 800 has a feature that two or more identical images continue (two at 1/2 slow,
4 shots at 1/4 throw). Therefore, in order to determine whether or not the playback is slow, two consecutive frames are examined, and the image difference is examined by template matching. Then, the transition of the degree of difference for a certain period of time is examined, and if the degree of difference repeats a large value and a small value in a specific cycle, it is determined that it is slow reproduction. For example, in the case of 1/2 slow, since the same image continues two by two, the degree of difference alternately repeats a small value and a large value. In the case of 1/4, the small value is repeated three times and the large value is repeated once. However, in the case of a moving image, even if it is not slow reproduction, two consecutive frame images are similar, so it is necessary to determine whether the difference is large or small with a low threshold. The event-specific counter unit 822 counts the number of slow playback appearances.

The same speaker detection unit 820 detects whether or not the same speaker has spoken back in the past within a predetermined time. For example, the autocorrelation of speech can be obtained and checked by checking whether the frequency having the largest value matches. The event counter 822 counts the number of utterances of the same speaker.

The co-occurrence counter unit 824 performs counting when some of the above events appear simultaneously or in a specific order. There are as many counters as the number of combinations of events to be detected. For example, in the case where the same speaker is speaking at the same composition, the counter corresponding to the simultaneous occurrence of the composition event and the speaker event is incremented by one. Similarly, when there is a dissolve and slow playback is detected immediately after that, the counter corresponding to the continuous occurrence of the dissolve event and the slow playback event is incremented by one.

The comparison unit 828 refers to the clock 826 and compares what kind of program the tendency of the appearance frequency of events in the video for a certain time from time t1 to t2 is similar.
Prior to the comparison, first, typical events such as news programs and sports programs are examined, and values are assigned so that the events become higher as they become more important events that characterize the program. Create a rank list for each event. In the comparison, the normalized value of the appearance frequency value of each event is multiplied by the value described in this rank list, and the total sum of the values obtained for each event exceeds the threshold value. If it is, it is determined that the program corresponds to the rank list.

The events obtained in this way can be displayed as a list on the display 1 in FIG. 1 in a table format with one of them as a time axis as shown in FIG. 6 or FIG. Even if the computer cannot judge automatically by this list display, the user can guess the type of program by using this information as one clue and using information obtained from others, experience, knowledge, etc. There is a possibility. In addition, when a new type of program that is not taught to the computer is newly input, an important event or a combination of events may be selected and registered from this list display. This is because the pointing device 5 such as a mouse shown in FIG.
It is very convenient for the user if it is performed by direct and visual operations such as clicking the changing point of each event on the list and the display part of the section.

Note that the present invention can be realized by using a PC / WS, and can also be applied as one function such as a TV and a VTR.

It is a system block diagram for realizing an embodiment of the present invention. It is a flowchart of the program which detects a dissolve. It is a figure showing the concept of a dissolve. It is a flowchart of another program which performs a detection of a dissolve. It is a graph showing the time transition of the evaluation value when the program which performs a dissolve detection is executed. It is a typical event chart of a news program. It is a typical event chart of a sports broadcast. It is a block diagram of the system which classifies a video.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Display, 2 ... Control signal line, 3 ... A / D converter, 4 ... Computer, 5 ... Input device, 6 ... External information storage device, 7 ... CPU, 8 ... Connection interface, 9 ... Memory, 10
... moving picture playback device, 11 ... keyboard.

Claims (2)

The input receiving means receives the moving images constituting the program in time series in units of frames,
The processing equipment
In the detection unit, an event that is a point at which the feature of the moving image changes from the received moving image and an occurrence timing for each detected event are detected.
Based on the occurrence timing of each detected event type, the tendency of the appearance frequency of the event is extracted and compared with the event characteristics for each program type stored in the storage unit. Determine the type
A program digest creation method comprising: discriminating a type of the input program, extracting a feature scene of the moving image, and creating a digest of the program based on the feature scene.   2. The digest creation method according to claim 1, wherein the processing device has a feature that the feature amount of the frame of the moving image that has received the input monotonously approaches the feature amount of the newest frame in order from the oldest. A program digest creation method comprising: extracting a video section from the frame to a predetermined time as a feature scene and creating a digest of the program when the determination is true.

Images