KR20000064909A - Navigation and navigation equipment through video material with multiple key-frame parallel representations - Google Patents

Abstract

Navigation through the video material by means of a plurality of key-frame parallel representations while allowing selective access of the displayed keyframes as a mapping representation of the accessed keyframes to control substantial access to the video material.

The method selects between a first mode of operation for arranging keyframes in a temporally arranged manner on a screen and a second mode of operation for arranging keyframes with multiple selectable granularities between adjacent keyframes as indicated. To be permitted within a single user interface organization.

Description

Navigation and navigation equipment through video material with multiple key-frame parallel representations

Keyframes may be derived from its acceptable video material at the user's through a derivation algorithm, or keyframes may be called, for example, such as a video provider where each video shoot begins with a keyframe. . The third way is that the frames are contiguous with each other at uniform time intervals as related to the standard video speed. The idea is that the keyframes are combined with reliability useful for selection or non-selection or editing of successive displays, in order to make them more accessible to users. It will of course be used to give a dynamic overview of the above.

A particular problem with the scheme for digital and compressed coding of video images is that storage on it on a large amount of media is generally not immediately accessible, especially linear storage densities such as displayable at storage sizes per frame. Is called. It has been proposed to add high capacity main storage media such as tape to the second storage medium with smaller capacity and increased accessibility. In such cases, the execution of trick modes such as fast forward and reverse as well as editing of video material for continuous display in excerpted, modified, and rearranged forms is shown in terms of the user interface. Not only are they recognized in terms of storage technology, but both pose significant difficulties.

The invention relates to a method according to the preamble of claim 1. The use of keyframes that are recorded, such as the display portions of a video display, ie for the next optional playback, has been proposed in other cases. A continuous video stream, meaning that the video is in an "on" state, includes a cartoon movie, a series of still images, or interactive continuous images. Characteristics can vary, such as films, news, or shopping lists, for example. The description of this technology is described in S.W. Smoliar and H.J. Zhang appears in a paper titled 'Content-Based Video Indexing and Retrieval' on pages 62-72 of IEEE Multimedia, Summer 1994.

1 is a block diagram of a TV-recorder combination.

2 is an exemplary structural diagram of video recording.

3 is a schematic diagram of a scrolling mosaic user interface.

4 is a schematic diagram of a scrolling list user interface.

5 is a more extensive graphical user interface.

6 is a display diagram of an explanatory caption.

7 is a state diagram of system operation.

As a result, among other things, the object of the present invention is not only to provide the user with a more natural sense of storage organization, but also to provide the user with a more natural sense of storage organization as well as eliminating the need for continuous access to the main storage medium. Introducing good flexibility into the organization. Therefore, at present, the invention according to one of its aspects is characterized according to the remainder of claim 1. Displaying frames in a temporally organized manner can affect rapid forward and rapid reverse in a simple manner if, for example, the frames continue at uniform time intervals with respect to one another, relative to the standard video speed. Moreover, easy changes of hierarchical levels with varying granularity when exactly fitting between frames allow for easy accessing and editing. The same is true if the keyframes or at least a small portion thereof is leaked from the start of filming or from other related work created by the original film editor. In this way, the clustering operation is automatically affected. Conveniently, the method highlights and simultaneously simultaneously enlarges the currently selected keyframe in a multi-size format with respect to other keyframes, and furthermore the navigation detects a harmful video staggering effect and if so eliminates the effect by vertical elimination. Reducing. On the other hand, video distortion in relatively small keyframes has been allowed, and if a particular keyframe is enlarged, extra steps must be taken to improve the picture. The inventors have recognized that, although not always achieving the quality level present under standard display conditions, this improvement gives a pleasant and beneficial improvement in picture quality.

The invention also relates to an apparatus arranged to perform the steps of the method as described. Furthermore, in the advantageous aspects of the invention are described in the dependent claims.

Various benefits are described below, particularly those that are exclusive to general customer and home use:

Keyframes must be displayed in such a way that they can be distinguished from one another by a user located at a typical TV viewing (viewing) distance.

The number of keyframes displayed at the same time should be sufficient to provide the user with an overview of the important part of the content of the digital video material.

Keyframes should be displayed in an undistorted form, such as to maintain aspect ratio.

Preferably, the remote control device of the TV set itself operates as a user control device.

Feedback information should be recognizable from typical viewing (viewing) distances.

Computer concepts such as "drag and drop" are generally not necessary.

• Facilities should be convenient to use only occasionally, rather than being used continuously.

The user interface should reflect the linear model of the familiar video representation.

(Special Example)

1 shows a TV-recorder combination according to the invention. Item 20 represents an adjacent control and power source associated with the TV-set display panel. Item 22 represents a connection with another form of signal distribution body, such as antenna or cable distribution. This item also includes the part of the digital signal from the received or derived signal of the digital video information, as appropriate. Item 34 represents the routing of video streams and related information between the various subsystems of FIG. 1. The routing is controlled by control box 28 via control signals on line 35. The latter (line 35) has been made as a single bidirectional interconnect, but may in fact consist of any number of unidirectional or bidirectional lines. The control box receives detection signals from display panel 20 on line 30 and also receives detection signals from them while controlling subsystems 38 and 40. Block 38 is a linear tape recorder with a very high storage capacity in a multi-gigabyte area. Block 40 is a magnetic disk recorder with high storage capacity, but nevertheless is only a small part of recorder 38; On the other hand, the access in the recorder 40 is very rapid through cross-track jumping. In addition, blocks 38 and 40 are formed of a two level storage organization and somewhat similar to a computer memory cache system, storing items of all video representations at least once. Item 24 represents a remote control device, communicates with display device 20 via radio 26, indirectly communicates with subsystem 28, and further communicates with subsystems 38 and 40. .

2 shows an exemplary structure of a video display. To achieve video material functionality, bar 60 includes the video itself in one of a series of compressed video materials, such as frames or MPEG encoding. Although the actual storage needs need not be made uniform beyond the playback time, the information is stored along the same bar as the video time lapse. Interspersed keyframes appear as dark vertical stripes, such as 68. Keyframes are used to become the entire typical video and display it at intervals up to the next keyframe. The keyframe may be selected by the video provider as the first frame of each new shot through adding a label or inclusion in the "content table". Alternatively, through some algorithm the receiver detects that the video content suddenly changes from one frame to the next. The present invention also naturally applies to related algorithms. As shown, their arrangement can be non-uniform. Another mechanism is that consecutive keyframes persist with each other at the aforementioned intervals, such as every two to three seconds. In this embodiment, only keyframes are represented by the display 62. Moreover, the keyframes are organized slightly hierarchically, and the indication 64 has only a limited set of height related keyframes. This hierarchy is multi-level, and the display 66 involves only a single keyframe for all of the video displays 60. The various levels of keyframes may be determined differently in one of the organizations described above, or may exist side by side.

The storage mapping in FIG. 1 can be influenced by the fact that the main body of the video representation is stored in the tape recorder 38, while at least keyframes are played in the disc recorder 40, as far as possible associated keyframes. Can be immediately played with short video and / or voice portion (audio) intervals. Such interval length corresponds to the time delay of the linear tape recorder 38, whereby real time access can be achieved. Naturally, the video display can be essentially film-like. Another usage is to include cartoons, still images, or other images so that certain storage intervals can be used by the current consumer. The possible action of the keyframe is to suppress it. This effectively combines the time interval before the keyframe in question and the timeframe after the keyframe. The reset feature does not combine that interval again. In addition, various kinds of keyframes may be suppressed, such as those separated from each other by fixed time intervals. Various other kinds of keyframes may be used as an indication, such as those introduced by a provider versus those generated by a logic algorithm at reception.

3 shows a design of a scrolling mosaic user interface. Every screen displays 20 keyframes starting from the top left to the bottom right: each keyframe has its number within the entire hierarchy of keyframes shown. In practice, keyframe 144 is highlighted by a right angle control cursor. The user activates the remote control via navigation control on the cursor device to move the cursor freely on top of the displayed keyframes as well as on top of the buttons indicated by the top and bottom bars. If the user moves the control cursor to the left in the top left corner, the display is skipped back by 20 keyframes. Moving to the right within the bottom right corner allows forward skipping above the 20 keyframes. Accessing the top bar of the screen controls access to other parts of the display, the latter being divided into five equal length parts: the black horizontal bar is covered by 20 keyframes shown here in full display. Indicates the total time spent.

The other functions are initialized by successively selecting one of the special keyframes and one of the bottom buttons. The "view program" controls the starting point in the cursor-accessed keyframe. The "view segment" is also controlled in the same way, but only a single segment is executed, ending at the next keyframe. The "view from x to y" controls the earliest starting point in time of the two cursor-accessed keyframes and ends at the latest in the two times. Other modes are viable with keyframe-selective functionality. Examples are fast-forward or late-forward, allow the user to check a particular interval for certain events and also allow the fast / slow reverse to achieve any video effects. During display, passing momentarily within the time belonging to a particular keyframe, the latter becomes active and validly displaying the video stream until the moment associated with the next keyframe is reached. As a result, the latter becomes the active frame. This feature allows the user to program the video recorder for the interval indication sequence straightly as excluding certain segments such as an advertisement, or vice versa, to draw attention to certain items in a slow-forward direction. During the display, the audio portion (audio) can be activated or suppressed via a control button not shown. In the other way, control causes the audio to be sent as one, while the video cursor is discontinuous and only steps from interval to interval with adequate brightness.

4 is an exemplary design of a scrolling list user interface. Here, at the bottom, the entire screen is represented by five keyframes, and the keyframe 145 highlighted by the orthogonal control cursor moves along the edge of the screen. Keyframe 145 is also displayed in very large size in the background. Although the button positions are different, the control interface is the same as in FIG. 3. Alternatively, the enlarged keyframe is suppressed within the multi-keyframe bar.

5 shows a very broad graphical user interface. First, left and right are play, stop, cut, paste, fast reverse, zoom +, zoom-, zoom forward ( row of control buttons for fast forward). The button row has little relationship to each other and has a sequence of nine keyframes, each belonging to different scenes or shots. Although staged within the hierarchical organization of keyframes, a good overview of scene dynamics is gathered. The inner-keyframe distance is, for example, 10 seconds, but larger and smaller spaces can be implemented. Especially with short distances between successive keyframes, properties such as fast forward are very well realized. On the other hand, while the video skips from one keyframe to the next, the same space size can be used for the playback of all audio. Now, the central keyframe is also represented in an enlarged manner. When executing closely spaced keyframes with sufficiently low granularity, the enlarged keyframe is displayed in a dynamic manner for such an effect of a fast forward (or backward) mode. Material of the next keyframe Upon reaching the sailboat shown here, the bottom row switches from one position to the left, so that the "sun" on the left is invisible and the new keyframe enters from the right. As mapped onto a display from a background storage medium, such a display (display) may be made at a particularly faster frame rate than standard video. Reverse organization allows for rapid reverse.

FIG. 6 shows a representation of descriptive captions in a general format as discussed with reference to FIG. 5. In the central area, the space 50 has been written in a substantial frame; Space 52 is used to display related or other relevant information related to leaked descriptive subtitles or video displays, such as translated from language to phrases for deaf or translated into another language than used for actual languages. come. At the bottom of the screen, there is no need for explanatory subtitles that only leak out in the range of seven keyframes. Their relevance can span more. Moreover, each keyframe has a time code 54 or other related data placed thereon. Two columns of control buttons 56, 58 have been written to application operations on the left side and to internal-program operators on the right side. The top of the screen shows the subject 60 of the actual video program.

The fundamental principle of an arrangement for having a dynamic representation of a video cursor that is executed correctly in the active keyframe area is that the static representation of the frame as a whole is such that when such dynamics allow the user to better understand the evolution of the event. It is not enough to suggest dynamics. Therefore, semantics is increased as follows. After the system becomes meaningless for keyframes, a certain amount of time, the cursor 'enclose' becomes 'recovery', which includes some associated audio and better effects, because it starts running digital material on a small scale. . If the next keyframe arrives during playback, the cursor will automatically 'skip' to the next keyframe displayed in the user interface until the user initiates (resumes) interaction with the system. In general, the organization described above is different and is to be read through information separate from the entire video string. Although only the audio part (audio) is run in a dynamic manner while skipping from one keyframe to the next, the user gets a better indication of the underlying video, especially with low storage needs.

In this regard, FIG. 7 is a state diagram of system operation. While displaying multiple keyframes, in state 100, the system waits for input from the user. Such input implies skipping between the displayed keyframes, skipping to another set of keyframes, and selecting a keyframe to indicate the associated interval. Any such input affects arrow 104 and starts a new time interval. If any such input is absent for n seconds (such as 20 seconds), then the arrow 108 is affected, thus reaching state 102. Within it, the system advances the dynamic video cursor frame. While no user input is received, arrow 110 is affected and the system continues as long as displayable video material is available. However, if user input is received, arrow 106 is affected, and the system alternatively stops at the substantial content of the dynamic video cursor frame or at the beginning of the substantial interval.

(Detection and filtering of keyframes affected by the "INTERLACING" effect)

Some of the keyframes used to distract the content of a video program have been extracted from sequences with high behavior. In the case of a video sequence encoded in the normal staggered coding mode, this creates a difficult zigzag effect, which is the case of a frame consisting of two regions that contribute to forming a complete frame, where even lines belong to one region and odd lines differ It belongs to one area. This problem is more apparent and difficult in small keyframes when the image is enlarged and the lines become thick blocks, where the effect is very visible.

First, keyframes affected by such a staggering effect must be detected. This effect can be observed on the rows of the image and changes in luminance cause high frequency values. This can be used with many subbands by separating the spatial frequency spectrum, and then only considering the high frequency components.

In practice, the effect we detect and degrade should be given by the change in luminance between even and odd lines, and therefore the largest sampling frequency of the picture. The only coefficient that must be calculated is the highest frequency component of the frequency transform (FFT or, more preferably, DCT) on the columns. When the image is affected by the zigzag effect due to staggering, this element has a high value.

However, this effect can also be visualized in motion corresponding to the object, in particular with elements in the horizontal direction. We therefore do not consider the sum of the coefficients, since this also yields high values in detailed and noticeable differences in the pattern, resulting in a false positive number.

Better results can be obtained by separating the image in some sub-parts, taking into account the largest value per area. For example, the two highest values of each add up to make the total less sensitive than the finer details.

As a result, since the low vertical resolution is not more difficult than the zigzag effect, the simplest way to filter this image is to consider only one area and then upsample it vertically by a factor of two. As mentioned in the next paragraph, an interpolation filter can be applied before showing the resulting picture.

Therefore, detection and correction affect as follows. The first step is to discard one region by removing half of the rows of either even or odd; Upsampling by a factor of 2 is then performed according to the columns to restore the original size of the keyframe followed by the interpolation filter. In this case, the filter performs simple linear interpolation.

(Upsampling and interpolation)

To make it easier to see at TV-viewing distances, the keyframes should be enlarged to a screen size that is almost full with upsampling followed by an interpolation filter. On the other hand, in general, keyframes have a low resolution, which must be magnified by a very high factor. This means that since the pixels are large blocks, the more processed the result will be. Therefore, the picture must be filtered, but we need to find an optional advantage because we need to create a good quality picture showing high resolution, and also find a fast process for the application to have a short response time.

The argument is that magnification must be performed on images in flight, and because it requires too much storage space, it means that images cannot be magnified and filtered at once to be stored and reused on the hard disk. Therefore, the upsampling and filtering process should be done as quickly as possible while simultaneously maintaining acceptable results. Ordinary everyday interpolation filters can be used (see any book on digital signal processing; related papers include digital interpolation of discontinuous images by HC Andrews, CL Patterson, published in IEEE Trans. Comput. 196, v25, 196-102). Can be heard)

Other techniques for improving picture quality can be used better. Among them, wavelet-based resolution and dimensional fission graphics approaches seem to lead to higher computational points, but with significant results in visible quality. In fact, the dimensional fission diagram approach compression technique is well known for independent resolution: details at higher resolutions can be reproduced or simulated by repeatedly applying the same decoding process. This case that can be stored is a dimensional cleavage figure compressed image, yielding a high compression factor. Similarly, by using the ripple transform, it is envisaged that high frequency elements on higher scales can obtain higher resolution images without blurring.

(Text search in video programs based on descriptive subtitles)

In current video delivery, descriptive subtitles are often delivered programmatically (often at vertical blanking intervals for analog systems or in separate simple streams in digital delivery). It is usually used for people who are unable to listen and are usually used for programs that are not classified or tuned into foreign languages. Such information is usually double printed on the screen, but may also be recorded on the storage medium. In this way, the programming language, and sometimes some sound description for the deaf, can also be used as search operations.

This kind of information extraction takes place in real time when the program is being recorded. If this technique is combined with a keyframe extraction routine, the picture and its associated text, such as a dialogue, can be linked, resulting in the portion of the program being extracted from the keyframe. In this way, with current text recovery techniques, we can implement text recovery based on special key words. A special tool of its application is to offer simple questions based on keywords and the possibility to implement their configuration as is now commonly used in "Web" search devices.

As an example, assume that a news program is recorded. If you want to recover news about France, when the word "France" is inserted, the system will automatically find this word in the body of the program. If the result is positive, the user will be given a description subtitle of the special part where the key word was found and a keyframe associated with that particular part of the program. The user can then start watching the program that starts the ROM at the particular pointed out portion. If keyframes are found as a result of the query, they will all be shown on the bottom of the screen as in Figure 5, so the user can analyze the related text one-to-one on a larger window. Of course similar keywords (Franch, Paris) can be used if the result is negative. Such a system can be usefully used in sports programs to extract reports that include a particular team or sport.

Many other applications are easy to check whether a movie is allowed for children's viewing, for example by checking whether words are used in conversations or included in a list of bad words.

A possible extension of such a system is

If the text cannot be used separately from the video, extracting the text from the screen, such as by OCR technology on still images,

• Use language recognition techniques to extract dialogue from the program.

In this case the system can always be independent of the service provided by the broadcaster, so even if no explanatory subtitles are provided, body recovery will always be possible on at least some special keywords and the system tries to recognize it. Can be trained.

The present invention is suitable for use as video keyframes.

Claims (8)

Through video material by parallel representation of one or more series of a plurality of key-frames allowing selective accessing of displayed keyframes by indicating a mapping of accessed keyframes to control substantial access to the video material. In navigation, The method arranges the keyframes in a single user interface organization with a first mode of operation for arranging the keyframes in a temporally arranged manner on the screen and with non-uniform and selectable granularity between adjacent keyframes as indicated. Navigation between the second modes of operation for 2. The navigation method of claim 1, wherein progressively playing back the speech portion intervals associated with the temporally focused keyframes during the temporally organized manner. 3. The navigation of claim 2, wherein successive speech segment intervals constitute a substantially continuous speech segment representation with respect to a sequence of discontinuously located keyframes. 2. The navigation method of claim 1, wherein the navigation of the voice portion intervals associated with the keyframes substantially accessed in the second mode of operation. 2. The method of claim 1, wherein the currently selected keyframe is enlarged and emphasized with respect to other keyframes in a multi-size format, and the navigation detects a harmful video staggering effect and if so reduces the effect by vertical elimination. Navigation further comprising. 2. The method of claim 1, wherein the currently selected keyframe is enlarged and emphasized with respect to other keyframes in a multi-size format, wherein the navigation further comprises applying an upsampling filter to the image prior to display. Navigation. The navigation system of claim 1, further displaying descriptive captions or other related information extracted about related keyframes or successive keyframes in relation to a materialized keyframe. An apparatus arranged to carry out the method as claimed in claim 1.