JP2004509529A - How to use visual cues to highlight important information in video programs - Google Patents

Abstract

A way to highlight significant progress in video clips of sporting events, such as soccer games, is by inferring progress from clues provided by low-level features of the video clips. The method detects a sequence of frames in a video clip having preselected visual or audio cues. The number of frames in each sequence having a clue is compared to a predetermined threshold. If the number of frames is greater than or equal to a predetermined threshold, significant progress is declared in the frame immediately before the format sequence that satisfies the threshold.

Description

[0001]
The present invention relates to content-based video extraction and retrieval, and more particularly, to a method for automatically identifying important information or development in a video clip of a sporting event.
[0002]
Many video applications require search methods that allow searching large amounts of video material to find certain important clips. Such applications may include, for example, interactive TV and pay-as-you-go systems. Customers using interactive TV and pay-as-you-go systems want to see parts of the program before renting. Video browsers allow customers to find programs of interest.
[0003]
Most work in content-based video extraction and retrieval is based on low-level features such as color, texture, shape and camera movement. Low-level features are useful for some applications, but many other interesting applications require higher levels of meaningful information. Bridging the gap between low-level features and high-level meaningful information is not easy. In most cases, when a high level of meaningful information is needed, manual annotation using keywords is commonly used.
[0004]
One important application for video archiving and extraction is for sports such as soccer, football, and the like. Therefore, there is a need for a method that can automatically extract high-level information using low-level features.
[0005]
The present invention is particularly directed to a method for automatically identifying important developments in video clips of sporting events, such as soccer games. The method
Detecting a frame sequence of a video clip having preselected cues, indicating that the frame of the immediately preceding video clip may have significant progress;
Comparing a predetermined threshold with the number of frames in the frame sequence having the clue;
If the number of frames in the sequence is greater than or equal to the threshold, determining that there is significant progress in the frame immediately preceding the frame sequence.
[0006]
The method further includes obtaining pre-selected cues from low-level features in the images of each frame of the sequence. In such an embodiment, the preselected cues are based on changes in the camera's aim. More specifically, when significant progress occurs in a video clip, the camera is typically focused on the audience or player, so that the sequence of frames immediately following the frame with significant progress The image has little or no turf area.
[0007]
The advantages, characteristics, and various additional features of the present invention will become more apparent from consideration of the exemplary embodiments described in detail below in connection with the accompanying drawings.
[0008]
The method of the present invention extracts high-level information from video or multiple images using low-level features to achieve content-based (content-based) extraction and search facilitation. This is achieved by the present invention, which identifies a particular target domain and automatically extracts high-level information based on low-level features by using knowledge specific to that domain. One particularly useful application for the present invention is in highlighting important developmental segments in video clips of sporting events, including soccer games and football games. Such video clips typically include video, audio, and textual information (close-caption).
[0009]
The method of the present invention highlights important progress in a video clip by inferring the progress from one or more cues provided from low-level features and textual information of the video clip. More specifically, the method detects a sequence of frames in a video clip having preselected visual, audible, and / or text (close-caption) cues. The number of frames in each sequence having a clue is then compared to a predetermined threshold. If the number of frames in the sequence is greater than or equal to the threshold, significant progress is declared in the immediately preceding frame with a frame sequence that satisfies the threshold with cues.
[0010]
It has been found that significant progress in sporting event video clips is typically characterized by visual cues associated with changes in the focus of the camera's interest. For example, after significant progress has occurred in a sporting event, such as a soccer game, the video camera is typically focused on a player or a stadium crowd. When the camera is aimed at an athlete or stadium crowd, little or no turf on the stadium is visible in the camera's field of view.
[0011]
Using a change in the interest of the camera, the method of the present invention detects a sequence of frames in a video clip with an image that has little or no turf area in the stadium. The number of frames in each sequence is compared to a predetermined threshold. If the number of frames in the sequence is greater than or equal to the threshold, significant progress is declared in the immediately preceding frame with a frame sequence that satisfies the threshold with little or no turf area. The threshold is based on the assumption that if there are not enough frames in a sequence that has little or no turf area on the stadium, the camera must aim at the player or stadium. Consequently, the frame immediately preceding the sequence of frames is considered to include significant progress, such as when scoring in a soccer game.
[0012]
FIG. 1 is a flowchart outlining an exemplary embodiment of an algorithm for implementing the method of the present invention when applied to highlight segments of a significant event in a video clip of a soccer match. The algorithm detects a sequence of frames in a video clip with little or no turf area in step S1. If in step S2 the number of frames in the sequence is greater than a predetermined threshold, then in step S3 significant progress is declared in the previous set of frames in the video clip.
[0013]
In the detection step S1, the present algorithm detects a green area having a color similar to grass. The algorithm is trained to distinguish green from other colors in each frame so that turf regions within the frame can be identified. This is achieved using patches from one or more soccer games or from a turf region image training set that has been extracted from a soccer game in a video clip. The algorithm learns from the above patches how to convert the turf region to green values. Given an image in a frame of a video clip, training is used to determine whether a given pixel in the frame is turf.
[0014]
The algorithm is trained by calculating the red and green normalized colors (r, g) for each pixel in the training patch and obtaining a normalized histogram for the class turf. Here, r = R / (R + G + B) and g = G / (R + G + B). The color histogram of an image is obtained by dividing the color space, such as red, green, and blue, into discrete image colors (called bins) and traversing every pixel in the image to produce each discrete color appearance. It is obtained by counting the number of times.
[0015]
The normalized histogram can be considered as a probability distribution function for the class turf, p (pixel value | green). Said detection step S1 is realized by marking pixels in each frame having a value of p (pixel value | green) exceeding a predetermined threshold as grass pixels.
[0016]
Based on the pixel classifying described above, step S1 of the present algorithm looks for connected components with grass-like colors in the image of each frame, and if they are large enough, the camera will Assume that you are aiming at However, if the connected grass color component found in the image of the frame is small, it is assumed that the camera is aiming at the audience or players. If a small grass color component is only detected in step S2 for a short time, for example in only one to three or four frames, no significant progress is declared in step S3. However, if a small grass color component is detected for a relatively long time, for example in 200 to 300 frames, a significant progress is declared in step S3.
[0017]
The results obtained using this algorithm can be further refined using other cues, such as speech or closed captions, or other cues from the same format. Clues from the same or different formats confirm the identification of the significant event or activity detected, and more importantly, the detection of goals, goal attempts, penalties, injuries, brawls between players, etc. Significant events or activities can be classified into meaningful classes and used to rank them by importance.
[0018]
In one embodiment, the method of FIG. 1 is performed by computer readable code by a data processing device. The code may be stored in a memory in the data processing device, or may be read / downloaded from a recording medium such as a CD-ROM or a floppy disk. In other embodiments, a hardware circuit may be used in coordination with or in place of software instructions for performing the present invention. The present invention can also be executed by, for example, the computer 30 shown in FIG.
[0019]
Computer 30 has a network connection 31 for connecting to a data network such as a variable bandwidth network or the Internet, and a fax / modem connection for connecting to other remote information sources such as video or digital cameras (not shown). 32. The computer 30 includes a display for displaying information (including video data) to the user, a keyboard for inputting characters and user commands, a mouse for positioning a cursor on the display and inputting user commands, It may include a disk drive for reading from and writing to stored floppy (R) disks, and a CD-ROM drive for accessing information stored on a CD-ROM. Further, the computer 30 may include one or more retrofit peripheral devices 38 for inputting images and the like, and a printer for outputting images, characters, and the like.
[0020]
FIG. 3 shows the internal structure of a computer 30 that includes a memory 40, which may include computer-readable media such as random access memory (RAM), read-only memory (ROM), and a hard disk. The products stored in the memory 40 may include the operation system 41, the data 42, and the application 43. The operating system 41 may be a windowing type operating system such as UNIX (R), but the present invention can be used with other operating systems such as Microsoft Windows 95 (R) as well. .
[0021]
In addition to the method of FIG. 1, applications stored in the memory 40 include a video coder 44, a video decoder 45, and a frame grabber 46. Video coder 44 encodes the video data in a conventional manner, and video decoder 45 decodes the video data encoded in a conventional manner. Frame grabber 46 allows capturing and processing a single frame from a video signal stream.
[0022]
What is built into the computer 30 includes a central processing unit (CPU) 50, a communication interface 51, a memory interface 52, a CD-ROM drive interface 53, a video interface 54, and a bus 55. The CPU 50 constitutes a computer readable code, that is, a microprocessor or the like for executing the above-described application from the memory 50. Such an application may be stored in memory 40 (as described above), or may be stored on a floppy disk in disk drive 36 or on a CD-ROM in a CD-ROM drive. The CPU 50 accesses the application (or other data) recorded on the floppy (R) disk via the memory interface 52, and accesses the application (or other data) stored on the CD-ROM to the CD-ROM drive interface 53. Access via
[0023]
Input video data may be received through video interface 54 or communication interface 51. The input video data may be decoded by the video decoder 45. The output video data is encoded by the video coder 44 for transmission through the video interface 54 or the communication interface 51.
[0024]
Although the present invention described above has been described with reference to the above embodiments, various modifications and changes may be made without departing from the spirit of the invention. Accordingly, all such modifications and changes are considered to be within the scope of the following claims.
[Brief description of the drawings]
FIG.
5 is a flowchart outlining an algorithm for performing an exemplary embodiment of the method of the present invention.
FIG. 2
FIG. 2 is a block diagram of a computer for realizing the present invention.
FIG. 3
FIG. 2 is a block diagram showing an internal structure of a computer for realizing the present invention.

Claims (16)

A method of automatically identifying important events or activities in a video clip of a sporting event,
a) providing a video clip of a sporting event generated by the camera;
b) detecting a frame sequence of the video clip with preselected cues, indicating that the frame of the immediately preceding video clip may have significant progress;
c) comparing a predetermined threshold value with the number of frames in the frame sequence having the clue;
d) determining that there is significant progress in the immediately preceding frame of the frame sequence if the number of frames in the sequence is greater than or equal to the threshold. The method of claim 1, wherein the preselected cues are visible. The method of claim 1, wherein the preselected cues are based on changes in camera sighting. The method of claim 1, wherein each frame of the sequence comprises an image, and wherein the preselected cues are obtained from the image. 5. The method of claim 4, wherein the preselected cues are images having little or no turf area. The method of claim 1, wherein the sporting event displayed in the video clip is a soccer match. The method of claim 1, wherein the preselected cues are provided from low-level features of the video clip. The method of claim 1, wherein the preselected cues are provided from low-level visual features of the video clip. The method of claim 8, wherein the low-level visual features include color. The method of claim 1, wherein the preselected cues are provided from low-level audio features of the video clip. The method of claim 1, wherein the preselected cues are provided from textual information of the video clip. 12. The method of claim 11, further comprising using the textual information of the video clip to confirm the identification of a significant event or activity. 12. The method of claim 11, further comprising using the textual information of the video clip to classify the significant event or activity into a meaningful class. The method of claim 1, wherein the preselected cues are a plurality of preselected cues. The method of claim 1, wherein the preselected cues include low-level visual and audio features of the video clip and textual information of the video clip. A device that automatically identifies important events or activities in a video clip of a sporting event,
A memory for storing executable code;
Based on the code stored in the memory,
a) providing a video clip of a sporting event generated by the camera;
b) detecting a frame sequence of the video clip with preselected cues, indicating that the frame of the immediately preceding video clip may have significant progress;
c) comparing a predetermined threshold value with the number of frames in the frame sequence having the clue;
d) if the number of frames in the sequence is greater than or equal to the threshold, the processor performing a step of determining that there is significant progress in a frame immediately preceding the frame sequence.

Images