JP2000516006A - Method and apparatus for navigating video content by displaying a plurality of key frames in parallel - Google Patents

Abstract

(57) [Summary] The video content is guided by displaying a plurality of key frames in one or more rows in parallel, and the actual access to the video content is selectively performed by selectively accessing the displayed key frame. This is a method that can be controlled while representing the mapping of the accessed key frame. The method has a first mode of operation in which key frames are arranged in a time-ordered order on a screen within a single user interface organization, and a number of selectable granularities between displayed consecutive key frames. The feature is that selection can be made between the second operation mode in which key frames are arranged.

Description

DETAILED DESCRIPTION OF THE INVENTION Method for guiding video content by displaying a plurality of key frames in parallel apparatusBackground of the Invention   The invention provides video by displaying multiple keyframes in one or more rows in parallel. In addition to guiding the contents, the selective access of the displayed key frame The actual access to the video content is mapped to the keyframes accessed in this way. The present invention relates to a method capable of performing control while expressing the ringing. For later selective regeneration Use a representative portion of the recorded video presentation as a keyframe It has been proposed that A continuous video stream means that the video remains "on" Means that this stream contains an animation, a series of still images, An interactive sequence of images can be included. The character is various , For example, a movie, news, or shopping list. This technology is described in S.A. W. Smolalr and H.S. J. Zhang's dissertation “Content-Based Video Indexi ng and Retrieval ”, IEEE Multimedia, Summer 1994, pp.62-72. You.   The key frame is derived from the video material when it is received at the user's home. Can be retrieved by the algorithm, or, for example, by a video provider. Label keyframes so that video shots start at keyframes be able to. A third method is to average these frames relative to a standard video rate. Successive one at a time interval. The present invention uses keyframes to Dynamic overview of video presentations with useful facilities Together give the user access to video material, selection or deselection of the next display, Or, it is based on the recognition that editing needs to be more forgiving.   For current projects on digital and compression coding of video images A particular problem is that its storage on high-capacity media generally does not allow immediate access, especially The point that the linear storage density that can be expressed by the number of frames per unit storage size is not uniform is there. For high-capacity main storage media such as tapes It has been proposed to add an auxiliary storage medium having the same. In this case, high-speed sequential Perform trick modes such as fast and fast reverse and video for next display Editing material into summary, modified, or rearranged form This creates considerable difficulties, both from the standpoint of storage and storage technology.Summary of the Invention   Therefore, it is an object of the present invention to increase the user interface organization in particular. Introduces flexibility and gives users more control over memory organization and video material. To give a natural feeling. To this end, the features of the present invention are simply Place keyframes on screen within one user interface organization A first operation mode in which the keys are arranged in a spatially ordered sequence, and a continuous key frame to be displayed Arrange keyframes with multiple non-uniform selectable granularities in between Between the first and second operation modes. Keyframe Presented in a temporally-ordered order, for example, these frames are at standard video speed If they are continuous with each other at a uniform time interval associated with High-speed reverse feed can be easily executed. In addition, variable graphs between frames Easy access and access through simple and easy change of hierarchical level And edit it. Key frames or at least a part of them Generated from the beginning of the movie or generated by the original film editor The same is true when retrieving from other related events. in this way Then, the clustering process can be automatically executed.   In the method of the present invention, the currently selected key frame is replaced with another key frame. This frame is highlighted by enlarging it in multiple times the format And detect harmful video interlacing effects, and if so, Vertical decimation reduces this effect and also enhances the image before display. A sampling filter process is performed. Video distortion of relatively small keyframes is acceptable I know from experience that you can do this, but if you want to zoom in on a particular keyframe, Additional measures need to be taken to improve the image. The present inventor has Dings do not always achieve image quality under normal conditions, but provide a comfortable and beneficial improvement in image quality To make sure that   The invention also relates to an apparatus configured to perform the method described above. Book Other features of the invention are set out in the dependent claims.BRIEF DESCRIPTION OF THE FIGURES   These and other features and advantages of the present invention will be described with respect to preferred embodiments with reference to the drawings. This will be described in detail below. In the drawing,   FIG. 1 is a block diagram of a TV-recorder combination device,   FIG. 2 shows a typical structure of a video recording,   FIG. 3 shows a design example of a scrolling mosaic user interface,   FIG. 4 shows a design example of a scrolling list user interface,   FIG. 5 shows a more extensive graphical user interface design example.   FIG. 6 shows the display of the subtitle,   FIG. 7 shows a state diagram of the system operation.Detailed Description of the Preferred Embodiment   In particular, as it pertains to normal customer and private home use, Various advantages, which are not limited to those related to proper use, are: . -Key frames are distinguished from each other by users located at a typical TV viewing distance. Must be presented so that they can ・ The number of key frames to be presented at the same time Should be sufficient to give an overview of the significant part of the tree. ・ Key frames are displayed without distortion, for example by maintaining the aspect ratio Should. ・ TV set remote control device itself as user control device Preferably, it works. ・ Feedback information must be recognizable from the representative observation distance. ・ Computer concepts such as “drag” and “drop” are generally unnecessary To Various functions are implemented so that they are used only occasionally, not continuously. There must be. The user interface is a well-known linear model for video presentations. Should represent Dell.Description of specific embodiments   FIG. 1 is a block diagram showing a TV-recorder combination device according to the present invention. Element 20 Indicates a TV set display and associated immediate control and power supply. Element 22 is Have other types of signal distribution entities, such as antennas or cable distribution Shows the connection. This element contains digital video information or digital An entity for extracting a signal portion may be included as appropriate. Element 34 of FIG. Shows the routing of video streams and related information between various subsystems . Routing is controlled by a control signal via line 35 by control box 28 can do. Line 35 is shown as a single bidirectional interconnect line. However, it can actually consist of any number of unidirectional or bidirectional lines. Wear. The control box 28 receives the detection signal from the display 20 via the line 30. Receive the detection signals from the other subsystems 38 and 40, Control the subsystem. Block 38 is extremely high storage in the multi-gigabyte area This is a linear tape recorder having a capacity. Block 40 has high storage capacity This recorder is a part of the storage capacity of the recorder 38. But the access speed of this recorder is Access makes it much faster than recorder 38. Block 38 and Block 40 Together form a two-level storage organization similar to computer memory cache systems. And store all items of the video presentation at least once . Element 24 communicates with display device 20 via wireless path 26. A resource primarily indirectly communicating with subsystem 28 and other subsystems 38 and 40. 3 shows a mote control device.   FIG. 2 shows a typical configuration of a video presentation. Functional video content Bar 60, the video itself, in the form of frames or in MPEG code Included as a string of such compressed video content. Information along the bar video time The actual storage requirements must be uniform over the playback time. No need. The dotted keyframes are shown with black vertical stripes like 68 You. Each keyframe represents the entire video in the interval up to the next keyframe Or as a representative of it. Keyframes are for video professionals Binder labels or inserts “table of contents” (TOC) By doing so, it can be selected as the first frame of each new shot. Some In addition, the receiver uses one algorithm to convert the video content into one frame. It is possible to detect a sudden change from the program to the next frame. The present invention Assume that the algorithm is correct. As shown in the figure, the distribution of keyframes Can be non-uniform. Other mechanisms specify consecutive keyframes , For example, every 2-3 seconds. In this example , Display section 62 shows only key frames. In addition, keyframes are somewhat hierarchical Organized, display 64 shows only a limited set of high-level keyframes. This hierarchical structure can be multi-level, and the display 66 can be used for video presentation. Can be associated with only a single keyframe for the entire Wear. The different levels of keyframes must be defined in a different organization from the one described above Can be arranged.   Figure 1 shows the memory mapping, In addition to storing in the coder 38, at least the key frame -A short video and / or audio interval immediately following the frame The playback can be performed by the screcoder 40. Such an interval Length corresponds to the waiting time of the linear tape recorder 38. Seth can be achieved. Video presentations are as essential as movies Can be linear. In other uses, animate at a given storage interval. It can include alternative images, still images, or other images used by consumers. Certain keyframes can suppress this. This is the keyframe The previous time interval is effectively combined with the subsequent time interval. reset The function allows the intervals to be separated again. In addition, keys of various classes -Frames, e.g. classes of keys separated from one another at certain time intervals Lame can be suppressed. A variety of different classes of keyframes, e.g. To the provider for keyframes generated by the local algorithm upon transmission More introduced keyframes can be used for one presentation You.   FIG. 3 shows a design example of a scrolling mosaic user interface. Which The screen also presents 20 key frames starting from the upper left to the lower right. Each key The key frame has the number of the total rank of the key frame shown in the figure. In fact, -The frame 144 is highlighted by a rectangular control cursor. You The remote control is driven by the navigation control of the cursor device. The cursor on the displayed keyframe and on the bars displayed at the top and bottom. The user can freely move the displayed button. The user moves the control cursor Moving to the left in the upper left corner will cause the display to jump back 20 keyframes. Pump. Moving to the right in the lower right corner will cause the display to move forward 20 frames Jump. 5 equal lengths by accessing the top bar of the screen Control access to other parts of the presentation that are divided into pieces And a black horizontal bar shows 20 of the total presentations displayed here Indicates the total time covered by keyframes.   First select a specific keyframe, then select one of the bottom buttons , Other functions can be started. "view program" The button controls the start at the keyframe accessed by the cursor. "View segm The ent "(segment view) button is the same as the view program, but the next key Play only a single segment ending in the frame. "view from x to y" View to y) button is for two keyframes accessed with the cursor Controls start at earlier frame and stop at later frame I do. Other modes can be implemented with the keyframe selection function. example Fast-forwa allows users to check for the occurrence of a particular interval Achieve rd (fast forward) or slow-forward, or predetermined video effects Fast / slow reverse can be achieved. While displaying During the instantaneous passage associated with a particular keyframe, this keyframe Video stream until it reaches the moment associated with the next keyframe. Display effectively. When the next keyframe arrives, this frame Become a game. These features allow the user to interface the video recorder. For a global display sequence, a predetermined segment such as a commercial By removing, the low-speed progressive will focus on certain details. It becomes possible to program straight forward. While displaying, Activate or suppress audio with no control buttons Can be. Alternatively, you continue to control the audio, but the video cursor Cleat and interval to interval with appropriate highlighting instructions It is also possible to only step.   FIG. 4 shows a typical design example of a scrolling list user interface. . In this example, the entire screen has five keyframes displayed at the bottom, Key frame 145 is moved by a rectangular control cursor along its edge. It is highlighted. Keyframe 145 is background at high magnification It is also displayed inside. This control interface is identical to that of FIG. The position of the tongue is different. Enlarged keyframes are suppressed within the multi keyframe bar You can also press.   FIG. 5 shows a more extensive graphical user interface. First, left The right column is play (play), stop (stop), select (select), cut (cut) ), Paste (paste), fast reverse (fast reverse), zoom + (zoom +), zo om- (zoom), a column of control buttons for fast forward. You. The bottom row relates to different scenes or shots that have little correlation. It has a sequence of nine consecutive keyframes. In the keyframe hierarchy Dynamic, good overview from scene to scene by stepping Can be collected. The key frame interval can be, for example, 10 seconds, , Larger or smaller intervals may be used. Especially continuous keef When the interval between frames is small, a function such as high-speed sequential forwarding is realized. The other The same size interval can be used for full playback of all audio However, the video only jumps from one keyframe to the next You. In this case, the center key frame is further enlarged and displayed. Granular enough low When playing key frames at short intervals with Perform fast forward (or reverse) mode, which can be presented in a mix In Wear. In this example, when the material for the next keyframe, representing the sailboat, is reached, the bottom row is 1 Shifts the position to the left, the "sun" at the left disappears, and a new keyframe starts at the right appear. Such displays are especially useful for mapping from background storage media onto presentations. And can be done at a faster frame rate than standard video.   FIG. 6 shows the presentation status of the subtitle in the general format described with reference to FIG. Indicates an option. Space 50 in the central area is allocated to the actual frame and Source 52 has been retrieved from the video presentation or other relevant information, eg For example, speech-to-text converted for the deaf or actual spoken language Assigned to display subtitles related to translation into another language. Sa Titles should only be taken from the area associated with the seven keyframes at the bottom of the screen. You don't have to. Their relevance can be broadened further. In addition, each key The frames have a time code 54 or other relevant data placed thereon. control Two rows of buttons 56, 58 are located on the left for application operation and on the right Assigned for the program operator. The top of the screen is displayed Have the title 60 of the actual video program being played.   A video cursor that moves over time within the active keyframe area The reason for configuring to have a dynamic representation of A static presentation of the frame alone will divert the video presentation as a whole. Insufficient to be natural and give users a good understanding of event development Because. For this purpose, the semantics are improved as follows. the system, After a predetermined period of inactivity, a digital video with associated audio and other effects Cursor to “include” key to start displaying material in reduced size The frame becomes "alive". During playback, when the next keyframe is reached, The sol automatically changes to the next keyframe presented in the user interface “Jump”, which causes the user to (again) begin interacting with the system Continue until In general, the organization described here differs from all video strings. Allows scanning of other information. Any keyframe to next keyframe Even if you only play the audio dynamically with the jump to The user can get a good impression of the presented video at certain low storage requirements.   In this regard, FIG. 7 is a state diagram of the system operation. In state 100, The system waits for user input while displaying a number of key frames. This Such inputs can be jumps between multiple displayed keyframes, another set of keyframes. To jump to the game, including selecting keyframes to display the relevant intervals be able to. Any such input drives arrow 104 and a new time interface Start the ball. If there is no such input for n seconds (for example, 20 seconds), the arrow 108 It is driven and state 102 is reached. In this state, the system is dynamic Perform a video cursor frame. Arrow 11 unless user input is received. 0 is activated and the system continues to display as long as displayable video material is available You. However, when user input is received, arrow 106 is activated and the system Actual video cursor frame actual content location or actual interface Remains at the start of the globalDetection and filtering of keyframes affected by the "interlacing" effect Ning   A key frame used to scan through the content of a video program. Some may have been extracted from sequences with high motion. This This means that the video sequence is encoded in interlaced encoding mode as usual, The frame consists of two fields which are used to construct a complete frame, If the input line belongs to one field and the odd line belongs to the other field, Produces an annoying zigzag effect. The problem is in small keyframes. More noticeable and annoying, in which case the image is enlarged and This effect becomes more noticeable when the blocks become thicker.   First, look for keyframes affected by such interlacing effects. I need to get it out. This effect is observed on the rows of the image, where the frequency values where the luminance change is high Occurs. Taking advantage of this, the spatial frequency spectrum is divided into a number of subbands. And consider only high frequency components. In fact, the effect you are trying to detect is even It should show alternating luminance values between the lines and the odd lines, so this image It should have a high sampling frequency. Frequency transformation on columns (FFT or good More preferably, only the coefficient that is the highest frequency component of DCT needs to be calculated. image Is affected by the zigzag effect, this component has a high value.   However, this effect can be applied to objects with motion, especially objects with horizontal motion components. Appears to correspond. Therefore, the sum of the coefficients should not be considered. The reason is The sum of numbers yields high values in images with detailed and high-contrast patterns. This is because errors eventually occur. Divide the image into several small parts, Better results can be obtained by considering the maximum / area. For example , The sum affects the highly detailed image by adding the two highest values of each area No longer.   Finally, the lower vertical resolution is less annoying than the zigzag effect, so this image The easiest way to filter is to consider only one field and Upsampling is performed twice in the vertical direction. The supplement mentioned in the next section An inter-filter can be applied before displaying the resulting image.   Therefore, detection and correction are performed as follows. First step is half row, even number Discard one field by removing rows or odd rows, then keyframe Double upsampling to restore the original size of the Inter filter processing is performed. In this case, the interpolation filter performs a simple linear interpolation .Upsampling and interpolation   To make it easier to see from the TV viewing distance, the key frame is Must be increased to almost full screen size by subsequent upsampling. It is necessary. In general, keyframes have a lower resolution, so they are enlarged at a higher magnification There is a need. This will result in a larger block of pixels with further processing, Means that it is not easy to see. Therefore, it is necessary to filter the image. Need to generate good quality images and display at high resolution On the other hand, it is necessary to perform high-speed processing so as to have a short response time. Need to find out. The problem is that magnification must be done during the flight of the image. In other words, this means that the storage space required for storage on the hard disk is required. The image cannot be enlarged and filtered even once because it is too large. Means that Because of this, the upsampling and filtering process It must be as fast as possible while still maintaining acceptable results. General To Normal interpolation filter can be used (optional for digital signal processing) Book: For example, H.C.Andrews and C.L.Patterson's paper "Digital Interpolation of Di screte Image ", IEEE Trans. Comput. 196, v25, pp. 196-202).   Other techniques for improving image quality can also be used. In particular, the wavelet method And fractal methods lead to high computational burdens but show significant results in visual quality. In practice, fractal compression techniques are known, and by repeating the same decoding process. Details can be reconstructed or simulated at high resolution. this In this case, the stored image is a fractal compressed image that generates a high compression ratio. same In the same way, by using the wavelet transform, By predicting the components, a high-resolution image having no blurring effect can be obtained.Text Search for Video Programs Based on Subtitle   In current video transmission, subtitles are often transmitted along with programs. (Often inserted in the vertical blanking period in analog systems and In digital transmission, it is inserted in each elementary stream). This is usually Used for programs delivered in foreign languages or used for the hearing impaired . Such information is usually superimposed on the screen, but must be recorded on a storage medium. You can also. In this way, the audio of the program and sometimes for the hearing impaired The description can be used for the search process.   This type of information extraction must be performed in real time while recording the program. You. Combining this technique with the keyframe extraction routine allows the image to be I.e. binding to the dialog that occurs in the program part where the key frame was extracted Can be. In this way, using current text search technology, Can perform a text search based on Application specific tools , As currently used in the “Web” search engine, Simple queries can be performed.   As an example, it is assumed that a news program has been recorded. New about France If you want to search for a keyword, insert the keyword "France" and the system will Look for this word in the text of the program dynamically. If the result is yes, give the user a key Keyword for the part of the program where the word was found and the specific part of the subtitle A frame is presented. The user can then view the program from a particular point. Question If a large number of keyframes are detected as a result of the All of the games are displayed at the bottom of the screen as shown in FIG. Related texts can be analyzed one by one on a large window. result If no, you can use similar keywords (France, Paris) You. This system covers a specific team or a specific sport in a sports program. It is also useful for searching reports that bar.   Many other uses are possible, such as checking whether a movie is good for children to watch. Whether the words used in the examination and conversation are included in the list of "immoral languages" Can be used for inspection.   Possible extensions of such a system include: If the text is not available separately from the video, Extracting by, for example, OCR technology, Extracting conversations from programs using voice recognition technology; There is. In this case, the system will always be free from the services provided by the broadcaster. If the relationship works and no subtitles are provided, train the system Can always perform a text search based on at least some keywords Can be done.

────────────────────────────────────────────────── ─── Continuation of front page    (81) Designated countries EP (AT, BE, CH, DE, DK, ES, FI, FR, GB, GR, IE, IT, L U, MC, NL, PT, SE), JP, KR

Claims (1)

[Claims] 1. Propose video content by displaying multiple keyframes in columns or more in parallel   And the selective access of the displayed keyframes   Mapping of keyframes accessed in this way to actual access to content   In a method that can be controlled while expressing     Screen keyframes within a single user interface organization   A first operation mode arranged in a time-sequential order above, and a continuous key displayed   Keyframes with non-uniform selectable granularity between frames   A selection between the second operation mode and the second operation mode.   E Contents guidance method. 2. In the key frame located at the center in time in the order arranged in time   2. The method according to claim 1, wherein the related audio intervals are sequentially reproduced.   The described method. 3. Sequential audio intervals into discrete keyframe sequences   3. A method according to claim 2, wherein said audio representation comprises a substantially continuous audio representation.   Method. 4. In the second mode of operation, audio associated with the actually accessed key frame   2. The method according to claim 1, wherein the audio interval is reproduced. 5. Highlight the currently selected keyframe and   Enlarge the frame to a format that is many times larger than other keyframes,   In addition, it detects harmful video interlacing effects, which are detected.   If this is the case, reduce such effects by vertical decimation.   The method of claim 1, wherein: 6. Highlight the currently selected keyframe and   Enlarge the frame to a format that is many times larger than other keyframes,   In addition, the image may be up-sampled before being displayed.   The method of claim 1, wherein: 7. Sub extracted for the relevant keyframe or sequence of keyframes   The method of claim 1, wherein a title or other relevant information is displayed. 8. An apparatus configured to perform the method of claim 1.