WO2001056292A1 - Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams - Google Patents

Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams Download PDF

Info

Publication number
WO2001056292A1
WO2001056292A1 PCT/US2001/000830 US0100830W WO0156292A1 WO 2001056292 A1 WO2001056292 A1 WO 2001056292A1 US 0100830 W US0100830 W US 0100830W WO 0156292 A1 WO0156292 A1 WO 0156292A1
Authority
WO
WIPO (PCT)
Prior art keywords
encoded video
video streams
multiple encoded
display
simultaneous rendering
Prior art date
Application number
PCT/US2001/000830
Other languages
French (fr)
Inventor
Vincent Dureau
Rama Kalluri
Original Assignee
Open Tv, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US09/494,067 external-priority patent/US6931660B1/en
Priority claimed from US09/494,184 external-priority patent/US6539545B1/en
Application filed by Open Tv, Inc. filed Critical Open Tv, Inc.
Priority to ES01900989.3T priority Critical patent/ES2555953T3/en
Priority to DK01900989.3T priority patent/DK1262068T3/en
Priority to CA002397645A priority patent/CA2397645C/en
Priority to EP01900989.3A priority patent/EP1262068B8/en
Priority to JP2001554620A priority patent/JP4084941B2/en
Priority to BRPI0107872A priority patent/BRPI0107872B1/en
Priority to AU26388/01A priority patent/AU773588B2/en
Publication of WO2001056292A1 publication Critical patent/WO2001056292A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
    • H04N21/2365—Multiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H04N21/25808—Management of client data
    • H04N21/25825—Management of client data involving client display capabilities, e.g. screen resolution of a mobile phone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4347—Demultiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4622—Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8166—Monomedia components thereof involving executable data, e.g. software
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/16—Analogue secrecy systems; Analogue subscription systems
    • H04N7/173—Analogue secrecy systems; Analogue subscription systems with two-way working, e.g. subscriber sending a programme selection signal
    • H04N7/17309—Transmission or handling of upstream communications
    • H04N7/17336—Handling of requests in head-ends
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/70—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by syntax aspects related to video coding, e.g. related to compression standards
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
    • H04N5/445—Receiver circuitry for the reception of television signals according to analogue transmission standards for displaying additional information
    • H04N5/45—Picture in picture, e.g. displaying simultaneously another television channel in a region of the screen

Definitions

  • the present invention relates generally to interactive video delivery mediums such as interactive television. More particularly, the present invention relates to a system and method for the simultaneous transmission and rendition of multiple MPEG-encoded digital video signal streams in an interactive television application.
  • Interactive television is an interactive audio/video delivery medium which provides broadcast audiovisual content to a number of subscribers.
  • Interactive television provides broadcast video and audio to users and may also provide a return path for the user to interact with the content, e.g., to make selections or order desired products, etc.
  • a television broadcast by a television network such as a broadcast of a bicycle race
  • the television network may generate multiple video feeds to the network from various angles of the race or from the various bicyclists, for example.
  • the network may select one or more feeds from the multiple video feeds and broadcast the selected video feed(s) to the viewing audience at any given point in time. As is evident, each viewer does not have the option to individually select which video feeds are to be rendered simultaneously for viewing.
  • a point-to-point network can enable each viewer to select the video feeds to be rendered simultaneously from a set of available video feeds.
  • each viewer may send a request to the head-end server selecting which video feeds the viewer wishes to view. The server may then recompose the screen for each viewer on the head-end and then sent it to the specific viewer.
  • a point-to-point network or on-line environment requires a significant amount of bandwidth as well as a return path from the viewer site to the head-end server in order for the viewer to send its video selections to the head-end server.
  • such a point-to-point network or on-line environment also requires additional hardware in the head-end server for picture re-composition for each active client.
  • Another system which can enable each viewer to select the video feeds that the viewer wishes to view from the set of available video feeds is a system having as many decoders in a receiver at the viewer site as individual videos to be rendered simultaneously. For example, if six individual videos are to be rendered simultaneously, the receiver at the viewer site must provide six decoders. However, such a system would require significant processing power in the receiver and increase the cost of the receiver. In addition, the number of videos that can be rendered simultaneously would be limited to the number of decoders provided in the receiver.
  • MPEG compression is a set of methods for compression and decompression of full motion video images which uses interframe and intraframe compression techniques.
  • MPEG compression uses both motion compensation and discrete cosine transform (DCT) processes, among others, and can yield compression ratios of more than 200:1.
  • DCT discrete cosine transform
  • MPEG-1 The two predominant MPEG standards are referred to as MPEG-1 and MPEG- 2.
  • the MPEG-1 standard generally concerns inter-field data reduction using block- based motion compensation prediction (MCP), which typically uses temporal differential pulse code modulation (DPCM).
  • MCP block- based motion compensation prediction
  • DPCM temporal differential pulse code modulation
  • MPEG-2 is similar to the MPEG-1 standard but includes extensions to cover a wider range of applications.
  • MPEG refers to MPEG-1, MPEG-2, and/or any other suitable MPEG-standard compression and decompression techniques.
  • An MPEG stream includes three types of pictures or frames, referred to as the Intra (I) frame, the Predicted (P) frame, and the Bi-directional Interpolated (B) frame.
  • the I or Intra frames contain the video data for the entire frame of video and are typically placed every 10 to 15 frames. Intra frames provide entry points into the file for random access, and are generally only moderately compressed.
  • Predicted frames are encoded with reference to a past frame, i.e., a prior Intra frame or Predicted frame. Thus P frames only include changes relative to prior I or P frames.
  • Predicted frames receive a fairly high amount of compression and are used as references for future Predicted frames. Thus, both I and P frames are used as references for subsequent frames.
  • Bi-directional pictures include the greatest amount of compression and require both a past and a future reference in order to be encoded. Bi-directional frames are not used as references for other frames.
  • An MPEG encoder divides respective frames into a grid of 16 by 16 pixel squares called macroblocks. The respective frames are divided into macroblocks in order to perform motion estimation/compensation.
  • Each picture is comprised of a plurality of slices.
  • the MPEG standard defines a slice as a contiguous sequence of 2 or more macroblocks (16 x 16 pixel blocks) that begin and end on the same row of macroblocks. A slice begins with a slice start code and includes information indicating the horizontal and vertical location where the slice begins in the picture.
  • a system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams in an interactive television application are disclosed. It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines.
  • a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines.
  • the method for simultaneous rendering of multiple encoded video streams at a user site generally comprises modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning at least one of the multiple encoded video stream for display, interleaving the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and outputting the single composite video stream.
  • the method may further comprise receiving the multiple encoded video streams from a broadcast center or via cable, satellite, MMDS, or terrestrial networks.
  • the method further comprises receiving a list of encoded video streams available for simultaneous rendering and optionally receiving user input from a user at the user site, the user input including the multiple encoded video streams selected from the list of encoded video streams available for simultaneous rendering.
  • the user input may further include a display location for each of the multiple encoded video streams.
  • the modifying may utilize the user input display locations for each of the multiple encoded video streams for repositioning the at least one of the multiple encoded video stream for display. Alternatively, the modifying may utilize a default display location for each of the multiple encoded video streams for repositioning the at least one of the multiple encoded video stream for display.
  • the default display locations for the multiple encoded video streams may be static values stored in the interactive decoder or receiver, determined by layout rules (such as HTML layout engine style) as applied by the interactive decoder, and/or any other suitable methods for determining default display locations.
  • layout rules such as HTML layout engine style
  • the layout rules may be modified by the producer on the fly at the production center and be transmitted to the receiver for rendering.
  • the method may further include decoding the single composite interleaved video stream prior to the outputting.
  • the outputting may include outputting to a display for displaying the single composite interleaved video stream as a video display, the video display comprising a display of each of the modified multiple encoded video streams including a repositioned display of each of the at least one modified encoded video stream.
  • each of the multiple encoded video streams is preferably encoded using MPEG-1 or MPEG-2 compression technology.
  • the method generally comprises receiving multiple encoded video streams broadcast from a broadcast center for display, selecting at most one of the multiple encoded video streams for each display location, interleaving the selected video streams into a single composite interleaved video stream, and outputting the single composite video stream.
  • each of the multiple encoded video streams is configured for display in one of a plurality of display locations, where each display location covers a subportion of a display and where the multiple encoded video streams include a plurality of encoded video streams for at least one of the display locations.
  • a system for simultaneous rendering of multiple encoded video streams generally comprises an interactive decoder adapted to modify at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display, interleave the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and output the single composite video stream.
  • the system may further comprise a user interface in communication with the interactive decoder and adapted to receive user input from a user.
  • the user input generally includes the multiple encoded video streams selected from the list of encoded video streams available for simultaneous rendering, wherein the interactive decoder is further adapted to receive the user input from a user via the user interface.
  • system may comprise a display for displaying the single composite interleaved video stream as a video display, wherein the video display comprises a video of each of the modified multiple encoded video streams including a repositioned video of each of the at least one modified encoded video stream.
  • An interactive decoder for simultaneous rendering of multiple encoded video streams comprising means for modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display, means for interleaving the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and means for outputting the single composite video stream.
  • An example of an application for the system and method described herein is a personalized EPG (electronic program guide) mosaic.
  • a personalized EPG mosaic With the personalized EPG mosaic, a plurality of the end user's favorite shows is displayed, based upon a personalized list, for example. The end user may optionally select a subset from the plurality of the displayed screens or video feeds.
  • FIG. 1 is a schematic diagram illustrating an interactive television system including a remote network that provides programming content according to one embodiment of the present invention
  • FIG. 2 is a block diagram of the interactive decoder of FIG. 1;
  • FIG. 3 is a schematic diagram illustrating the operation of the interactive decoder of the present invention.
  • FIGS. 4 A and 4B are flow diagram illustrating the process of the interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering;
  • FIGS. 5A-5D illustrate by way of example the operation of the interactive television system of the present invention;
  • FIG. 6 is a table listing the MPEG-1 and MPEG-2 MBAI VLC (macroblock address increment variable length codeword) and the corresponding increment values;
  • FIG. 7 is a flow diagram illustrating the process of an MPEG-2 interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering;
  • FIG. 1 is a schematic diagram of an interactive television system 100 according to one embodiment of the present invention. It is noted that FIG. 1 is illustrative only and other interactive television system embodiments may be utilized as desired. As shown, the interactive television system 100 employs a broadcast center
  • the broadcast center 102 includes an audio/video source 104 for providing the audiovisual content.
  • the audiovisual content may comprise movies, sports, news, sitcoms or other audiovisual programming, as is normally seen on television.
  • the audiovisual content preferably also comprises still images which may be utilized for various purposes.
  • the audiovisual content preferably comprises a plurality of video streams and preferably configured or designed such that two or more of the video streams can be simultaneously displayed.
  • the broadcast center 102 provides a plurality of program channels each comprising audiovisual content.
  • One or more program channels may comprise audio- video-interactive (AVI) signals including audiovisual content and associated interactive applications.
  • the encoder 108 at the broadcast center 102 is operable to encode or compress the videos and to transmit the compressed video streams.
  • the audio/video source 104 may include video compression logic for compressing still video images into compressed still video images. As shown, the audiovisual content is optionally supplied by a remote network 120 or a live feed as desired.
  • the broadcast center 102 preferably also includes an application server 106 for creating and/or generating interactive application content.
  • the interactive application content comprises application code and data which is designed to be executed by a processor within a set top box or television to support an interactive television feature.
  • the application server 106 is preferably configured for generating or providing "OpenTV" interactive applications.
  • the application server may also provide "Java" applets or other interactive program content as desired. It is noted that the system and method for simultaneous transmission and rendering of multiple encoded digital video signal streams of the present invention need not be implemented in conjunction with such interactive application content.
  • An Encoder/Multiplexer 108 combines the interactive application content with the audiovisual content to produce an audio-video-interactive (AVI) signal.
  • the Encoder/Multiplexer 108 also synchronizes the interactive application content with the audiovisual content to ensure that the interactive application content is inserted in the proper position within the audiovisual content.
  • Certain channels may comprise non-interactive programming content. In other words, certain audiovisual content may not have associated interactive content.
  • the Encoder/Multiplexer 108 preferably multiplexes a plurality of signals, including AVI signals and optionally non- interactive signals, together for transmission.
  • the broadcast center 102 also includes an uplink satellite 110 for transmitting the broadcast signal for receipt by end-users or a subsequent distribution link.
  • FIG. 1 illustrates a satellite network example where a satellite is utilized to transmit the broadcast signal, other transmission methods may be utilized, including cable distribution through coaxial cable, fiber optic transmission, microwave transmission and/or any other suitable means.
  • the broadcast signal is received from broadcast center 102 via a satellite downlink 132, for example.
  • the broadcast signal is then provided to a plurality of end-users or subscribers.
  • the broadcast signal may include one or more non-interactive program signals and may include one or more AVI signals on respective channels.
  • Each end user at a subscriber site 130 preferably includes a set top box receiver or interactive decoder 140 as well as a television set 150.
  • the set top box or interactive decoder 140 is coupled to the television set 150.
  • the logic of the interactive decoder 140 may be comprised in the television 150 such that the television set 150 and the interactive decoder 140 comprise a single integrated unit rather than separate units.
  • the television 150 may comprise a general purpose programmable computer having a display screen or any other suitable viewing device.
  • the term "subscriber television” is intended to include the television set 150 or other viewing device, such as a computer, LCD (liquid crystal display) screen, etc., and may include associated decoder logic for decoding compressed video and/or executing interactive programs.
  • the subscriber television comprises the television set 150 coupled to the interactive decoder or set top box 140.
  • a user interface such as a television remote control 152 may be provided at the user site to facilitate user interaction with the television 150 and/or interactive decoder 140.
  • the user can select desired television channels for viewing or select from various interactive selections.
  • the user can select two or more video streams from among a plurality of simultaneously broadcast videos streams and/or still pictures for display.
  • the user can also optionally select the display locations of each of the multiple video streams on the television set 150.
  • the end user can also selectively configure a graphical user interface (GUI) on the display screen using the present invention.
  • GUI graphical user interface
  • the receiver and the interactive decoder 140 are shown as one unit, the receiver may include any suitable number of interactive decoders and optionally any additional components, such as a recomposition engine, as will be described below.
  • the system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams described herein are suitable for the general case where the number of streams rendered is equal to or greater than the number of decoders present in the receiver.
  • the number of streams rendered is typically less than or equal to the number of video streams are sent but can be greater than the number of video streams are sent such as where certain streams are displayed more than once.
  • the signals on the selected television channels are decoded by the interactive decoder 140 which provides an output to the television set 150.
  • the interactive decoder 140 preferably executes a real time operating system, such as OpenTV.
  • OpenTV a real time operating system
  • the description herein is in terms of a system that processes or decodes signals received via broadcast, it is to be understood that the processing may by performed on broadcast signals and/or signals stored in a component resident at the user site, i.e., resident memory, such as in a hard drive (or other memory) of the decoder (or other component).
  • the interactive decoder 140 may also execute an interactive application program conveyed within the selected AVI signal to enable an interactive television feature.
  • a return channel 156 optionally interconnects the interactive decoder 140 to a transaction server 160 which monitors certain selections by the user and responds accordingly.
  • the return channel 156 preferably utilizes a standard POTS (plain old telephone system) phone line and associated modems (not shown) for communication.
  • POTS plain old telephone system
  • Other return channel options such as coaxial cable, fiber optic cable, DSL (Digital Subscriber Line), ATM (Asynchronous Transfer Mode), or others, may be used, as desired.
  • the broadcast center 102 broadcasts program such as a bicycle race which includes various video feeds such as a video feed from a camera on each rider's bicycle. Each video feed is independently compressed and independently broadcast to all satellite downlinks 132.
  • the incoming streams broadcasted by the broadcast center 102 are MPEG streams but need not be fully compliant with the MPEG standard. For purposes of clarity, such non-compliant streams are also referred to as MPEG streams herein.
  • the incoming broadcast streams may be encoded MPEG styles but some of the elements of the streams, such as the sequence headers, may be added by the receiver to the combined output stream.
  • a list of available video streams is preferably made available by the interactive application of the broadcast center 102.
  • Each user may select the desired videos from the list of available video streams, thereby allowing each user to monitor the angles of the race that is of interest to the user, such as the points of view of the user's favorite racers.
  • the interactive decoder 140 receives as inputs the user's selection of which video feeds are to be simultaneously displayed on the user's television set 150 and, optionally, where each of the selected video feed is to be displayed on the user's television set 150.
  • the interactive decoder 140 ignores or discards the non-selected video signals, preprocesses the selected signals in order to reposition the selected video signals for display, and decodes the preprocessed signals.
  • the interactive decoder 140 then presents the user's selected videos as a large composite video which is a mosaic of small pictures on the screen of the user's television set 150.
  • Such a process of the interactive decoder 140 although interactive in that the user selects the desired videos and their respective display locations, does not require the utilization of the return channel 156 as the processing of the user's selections are completed by the interactive decoder 140 which directly outputs the decoded video signals to the television set 150.
  • the simultaneous display of multiple video streams may be used in a number of different interactive television applications. For example, the viewer may choose to display a web page or MPEG still from a commercial while simultaneously display other programs. As another example, an interactive commercial can allow the user to select to simultaneously view a plurality of products from a line of products or details of various features of a product.
  • FIG. 2 is a schematic diagram illustrating the set top box or interactive decoder 140.
  • the interactive decoder 140 includes a demultiplexer 302 for receiving input broadcast channel signals 304 over one or more channels.
  • the input broadcast signals preferably comprises a plurality of channels containing programming content, such as movies, sports, television shows, news, advertising, etc.
  • the broadcast signal preferably includes a plurality of compressed or encoded digital video streams.
  • the audiovisual component is preferably compressed and more preferably MPEG-1 or MPEG-2 compressed. As noted above, only the video (or audio visual) signals cooesponding to the video streams selected to be displayed are processed by the interactive decoder 140 while the non-selected video signals are discarded.
  • the demultiplexer 302 operates to separate the audio/video component from the interactive component of the input broadcast channel signals 304.
  • the interactive component comprises an interactive application and/or data which is compliant with the OpenTV standard promulgated by OpenTV Corp.
  • OpenTV OpenTV Standard
  • other suitable types of interactive applications may be included in the AVI signal.
  • the demultiplexer 302 provides the audio/video component to an audio/video decoding processor or decoder 304 via a picture buffer 306.
  • the decoding processor 304 operates to preprocess the plurality of selected video streams in order to reposition the video signals for display, interleaves the preprocessed signals, and decodes the interleaved compressed audiovisual signals to produce decompressed data.
  • the broadcast channel transports MPEG-2 compressed data.
  • the decoding processor 304 preferably includes an MPEG-2 preprocessor, interleave processor, and decoder.
  • the decoder 304 preferably includes one or more digital signal processors ("DSPs") and one or more associated memories. It is noted that the decoder may comprise various types of logic, including one or more DSPs, CPUs, or microcontrollers, discrete logic, or combinations thereof. Details of the decoding processor 304 will be described in more detail.
  • the decoding processor 304 provides an audio/video output which is preferably provided to the television set 150 for display.
  • the decoder outputs a single decompressed video stream comprising a mosaic of smaller repositioned pictures to the user's television set 150.
  • the interactive decoder 140 may further include a CPU 314 coupled to a memory 316.
  • the memory 316 is preferably a read only memory (ROM) for storing the interactive application operating system.
  • the operating system is preferably the OpenTV operating system from OpenTV Corp.
  • the demultiplexer 302 provides the interactive program component of the AVI signal to a interactive application memory 312 coupled to the CPU 314.
  • the interactive application memory is preferably a random access memory (RAM) for storing the interactive application, i.e., the OpenTV application.
  • the interactive application from the AVI signal is stored in the RAM 312 and the CPU 314 can read the interactive application from the RAM 312 and execute the application.
  • the CPU 314 reads the OpenTV operating system from the ROM 316, whereby the operating system controls the operation of the CPU 314.
  • the CPU 314 also reads the interactive application from the RAM 312, provided with the AVI signal, and executes the interactive application under control of the OpenTV operating system stored in the ROM316.
  • a list of video streams available to the user to be displayed simultaneously may be comprised in the interactive application.
  • the list of available video streams is preferably included as part of the interactive application rather than as part of the audiovisual component.
  • the video streams are included as part of the audio/video stream.
  • the list of video streams available may be transmitted in the signaling of the channel.
  • the signaling of the channel is the list of components that is sent by the broadcaster to the receivers in a uniform manner for all channels. This mechanism is, for example, utilized to allow the receiver to detect secondary audio channels.
  • the RAM 312 is coupled to the CPU 314.
  • the CPU 314 is in turn coupled to a graphic on-screen display 322.
  • the graphic on-screen display 322 is coupled to provide an output to the television set in conjunction with the audio/video output provided by decoder 304.
  • the graphic on-screen display 322 organizes data in a format which can be overlaid on top of the audio/video output of the decoder 304.
  • the CPU 314 executes an interactive application and generates on-screen graphics which are provided in conjunction with the audio/video stream output from the decoder 140.
  • the CPU 314 executing the application provides the appropriate output to the graphic on-screen display 322, and the graphic on-screen display 322 causes the respective images or selection buttons to be overlaid on top of and/or displayed with the audio/video output of the decoder 304.
  • the graphic on-screen display 322 may be 4-bit or 8-bit color memory.
  • the graphic on-screen display 322 may be used for various purposes in addition to a presentation of selection buttons or a list of video streams available for simultaneous display.
  • the user may provide user input 324 to the CPU 314.
  • the user input may be provided from various devices and is preferably provided via the remote control, buttons on the television set 150, or the interactive decoder 140.
  • the user input 324 provided to the CPU 314 enables a user to interact with the interactive application stored in RAM 312.
  • the user or viewer may select the plurality of video streams to be simultaneously displayed on the television set.
  • Other user inputs 324 may include selection to order a product or order information, provide answers to a television game show, select an insert picture or insert feed from a choice of insert pictures or feeds, wherein the selected insert picture is for display in a background picture, for example.
  • the user or viewer may select locations of each of the plurality of selected video streams to be simultaneously displayed on the television set.
  • the CPU 314 includes an output coupled to the picture buffer 306 such that the picture buffer 306.
  • the CPU 314 sends signals to the picture buffer 306 relating to which the video (or audio visual) signals are selected for simultaneous display by the user and the corresponding display location on the television set of each selected video stream.
  • the picture buffer 306 only receives the video streams selected to be displayed while the non-selected video signals are discarded.
  • the CPU 314 is preferably further coupled to the return channel 156 via a modem 326 for providing user interactivity.
  • the modem 326 may be used for bidirectional communication.
  • the return channel 156 is preferably coupled to a transaction server 160.
  • the transaction server 160 receives this order and processes the order for fulfillment.
  • the answers are provided on the return channel 156 to the transaction server 160.
  • FIG. 3 is a schematic diagram illustrating the operation of the interactive decoder 140.
  • four video streams in particular, video streams 1-4, are simultaneously displayed.
  • video streams 1-4 are displayed in the lower right, upper left, upper right, and lower left quadrant of the display of the television set, respectively.
  • the interactive decoder 140 shown in FIG. 3 is utilized for MPEG-2 compressed video streams.
  • the video stream signals 330 and a corresponding control signal 332 containing, for example, the display location information are input to a MBAI (macroblock address increment) modifier 334.
  • the video stream signals 330 represents smaller pictures.
  • the MBAI modifier 334 modifies the MBAI of the video stream signals 330, if necessary, according to the display location of that video stream contained in the cooesponding control signal 332.
  • the output of the MBAI modifiers 334 i.e., video streams with modified MBAI, are input into an interleave 336.
  • the interleave 336 interleaves the smaller modified video streams into a larger single video stream.
  • the single larger video stream is essentially a mosaic of the repositioned smaller pictures.
  • the video decoder 338 preferably a MPEG-2 video decoder, then decodes the single video stream and outputs to the user's television set.
  • the television set displays the decoded single video stream representing the mosaic of the repositioned smaller pictures according to the user-specified display locations, default display locations, or a combination of user-specified and default display locations.
  • the default display locations for the multiple encoded video streams may be static values stored in the interactive decoder or receiver, determined by layout rules (such as HTML layout engine style) and/or any other suitable methods for determining default display locations.
  • the default location may be computed from the layout rules by a recomposition engine which can reside in the receiver or the head-end, depending upon the configuration utilized.
  • a pixel refers to a fundamental picture element while a block refers to an 8 by 8 grouping of 64 pixels.
  • a macroblock (MB) is defined as a 2 by 2 grouping of 4 blocks or a 16 by 16 grouping of pixels.
  • a slice refers to a sequence of macroblocks which, for the purposes of the present invention, begins and ends on the same row of MBs.
  • a slice startcode (“SCC") is a byte-aligned unsigned sequence of 4 bytes indicating the start of a slice. The values for the slice startcode range between 0x00000101 and OxOOOOOlAF.
  • the last byte of the slice startcode is the slice startcode value which indicates the row of macroblocks on which the current slice belongs. For example, a value of 01 indicates that the pixels in the slice belong on rows 0 to 15 inclusive and OxAF indicates that the pixels in the slice belong on rows 2784 to 2799 inclusive.
  • a macroblock address increment (MBAI) is a variable length codeword (VLC) of length 1 to 11 bits.
  • the MBAI indicates an increment, in units of macroblock s, from the last macroblock of a slice on the same row and indicates the horizontal positioning of the current slice, i.e., the position of the slice from the left edge of the display. For example, an MBAI value of 1 indicates that the current slice belongs on column 0 and extending to at least column 15, inclusive.
  • the MBAI VLC and/or the SCC may be collectively referred to as display position code(s) as they are codes which indicate the display positions of the particular slice.
  • FIGS. 4A and 4B are flow diagram illustrating the process 350 of the interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering.
  • the interactive decoder first receives the encoded video signals broadcast by the broadcast center as well as a long of available videos, preferably in the interactive component of the AVI signal in step 352.
  • the interactive decoder generates the control signals based upon the user's selections of the desired videos for simultaneous display and their corresponding specified and/or default display positions. It is to be understood that, although the user may select each of the videos that are to be displayed simultaneously, one or more videos may be displayed by default. In other words, one or more default videos and one or more user-selected videos may be displayed.
  • the interactive decoder receives the broadcast video signal streams. Alternatively, the interactive decoder may receive the list of the available videos in an interactive component of the AVI signal, such as in step 352, generate control signals in step 354, and selectively receive the user-selected broadcasted video streams.
  • the selective reception of the user-selected broadcasted video streams may be achieved based upon an identification or identifier of the packets into which each video stream is divided, such as by the server at the broadcast center. It is noted that the broadcast center divides each MPEG video stream into video packets identified with an identifier and multiplexes the packets in the time domain.
  • the interactive decoder determines whether each of received video streams correspond to one of the selected videos. If so, the interactive decoder processes the selected video streams and outputs the processed output video signals to a display for simultaneous display of the selected videos at step 360. In other words, in step 360, the interactive decoder combines the MPEG encoded video streams prior to decoding. Alternatively, for video streams which do not correspond to the videos selected by the user, the interactive decoder discards or ignores the non-selected video streams.
  • FIG. 4B illustrates process or step 360 of the interactive decoder in more detail.
  • the interactive decoder preprocesses each slice of the selected video streams to relocate each video stream as necessary depending upon the selected or default display position for that video or video stream.
  • the relocation includes modifying the MBAI LVC and/or the slice startcode or SCC.
  • the SCC is a byte-aligned unsigned sequence of 4 bytes indicating the start of a slice, the last byte of which is the slice startcode value which indicates the row of macroblocks on which the current slice belongs.
  • the value of the SCC is modified. It is noted that because the SCC is not of a variable length, the value of the SCC may simply be modified without concern regarding bit-alignment within a byte, i.e., byte-alignment.
  • the interactive decoder interleaves the preprocessed slices of the selected smaller videos in an order such that a single video stream representing a larger composite video is generated. The interleaving process can be similar to that described in copending U.S.
  • the interactive decoder can first interleave the slices of the plurality of the selected video streams and then preprocess, i.e., patch the coordinates of the slices of the interleaved video stream that are to be repositioned.
  • preprocess i.e., patch the coordinates of the slices of the interleaved video stream that are to be repositioned.
  • the interactive decoder decodes the single interleaved video stream representing the larger composite video.
  • the interactive decoder outputs the decoded single interleaved video stream representing the larger composite video of the videos selected by the user to a display, such as a television set.
  • the interactive decoder optionally resizes the single interleaved composite video stream in step 368.
  • the individual sequences are encoded in a spatial resolution that is smaller than the display such that multiple pictures can be combined and displayed with or without resizing, depending upon the number of pictures that are combined.
  • the broadcast center preferably broadcasts each video feed in accordance with a predetermined size, e.g., one-quarter of a typical display size, such as the typical high-definition television display size. Thus, if the end user selects four pictures for simultaneous display, resizing by the decoder would be unnecessary.
  • the specific predetermined size broadcasted by the broadcast center may depend in part on the optimization of the desired resolution of the picture while minimizing processing requirements.
  • the video feed is divides into macroblocks and each macroblock is compressed into a compressed stream.
  • the interactive decoder decompresses each compressed stream to rebuild the picture.
  • the interactive decoder can then zoom the picture to the desired size.
  • the interactive decoder generally cannot resize each compressed slice but can resize an overall picture.
  • the interactive decoder preferably resizes the single interleaved composite video stream in order to display the resulting video in a desired size.
  • the interactive decoder thus preferably resizes the composite video stream into a desired size, preferably a size which does not distort the videos but maximizes the utilization of the viewing portion of the display.
  • the receiver preferably also inserts a fill-in sequence to cover portions of the display or screen that are not covered with the combined videos. For example, if the incoming streams each take one-quarter of the output display and three sequences are selected and combined, the receiver may cover the remainder quarter of the display screen with a fill-in video which can either be sent by the broadcast server or generated by the receiver, such as an empty sequence (e.g., a green rectangle) or a still I-frame.
  • a fill-in sequence to cover portions of the display or screen that are not covered with the combined videos. For example, if the incoming streams each take one-quarter of the output display and three sequences are selected and combined, the receiver may cover the remainder quarter of the display screen with a fill-in video which can either be sent by the broadcast server or generated by the receiver, such as an empty sequence (e.g., a green rectangle) or a still I-frame.
  • none of the streams is modified for positioning.
  • the head-end sends multiple optional streams for each location covering a subportion of the display and where the end user of the receiver, e.g., by default and/or based on user profile, selects an option for each position.
  • the multiple encoded video streams broadcast would be configured for display in one of the two or more display locations.
  • the multiple encoded video streams would include more than one encoded video stream for at least one of the display locations.
  • a commercial may include a number of sub-pictures each at fixed display location and the content of each display location would be selected by the end user and/or by the receiver based on user profile.
  • a commercial for a department store may comprise a left sub-picture that advertises the department store in general and a right sub-picture that advertises a particular product.
  • the left sub-picture may be selected based on the preferred language of the end user while the right sub-picture may be selected based on the profile of the end user household, such as the number and ages of the children in the household.
  • the left sub-pictures sent by the head-end may be an English and a Spanish advertisement for the department store in general and the right sub-pictures sent by the head-end may be a toy and a cloth advertisement.
  • the process 360 of combining multiple MPEG encoded streams prior to decoding is preferably executed at the user site, the process 360 may be implemented elsewhere, such as at the broadcast center, for example.
  • the configuration for such an implementation is preferably where a primary head-end sends the compressed videos to a secondary head-end and where the secondary headend recomposes videos and narrow cast the recombined stream to a single receiver or group of receivers.
  • the utilization of the secondary head-end is optional and provides savings in bandwidth.
  • the benefit of such a configuration is that hardware costs in the receivers are minimized or reduced. For example, such a configuration would be beneficial in the case where receivers are already deployed and cannot easily be upgraded with the necessary additional hardware or software. In addition, such a configuration would be beneficial in the case where it is cheaper to centralize the capability in secondary head-ends if the feature is only utilized by and/or available to a limited number of receivers at a given point in time.
  • FIGS. 5A-5D illustrate an example of the operation of the interactive television system of the present invention.
  • FIG. 5 A illustrates three separate pictures or videos 402, 404, 408.
  • Each of the separate videos contains a number of slices.
  • video 402 includes slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6.
  • video 404 includes slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6
  • video 406 includes slices 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6.
  • FIG. 5B illustrates the transmission of each of the videos 402, 404, 406 from the broadcast center to the interactive decoder as a video stream.
  • video 402 comprising slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 is transmitted as a video stream 412.
  • the video stream 412 comprises the slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 in series.
  • video 402 is transmitted as video stream 414 comprising the slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6
  • video 406 is transmitted as video stream 416 comprising the slices 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 in series.
  • the broadcast center preferably also transmits a list of the video streams available to the user to be displayed simultaneously.
  • the list of available video streams may be comprised in the interactive component of the AVI signal.
  • the user may select two or more of the videos 402, 404, 406 for simultaneous display.
  • the user selects to display video 402 on the left hand side and video 404 on the right hand side of the television set display.
  • a display configuration may be the default display configuration where the user does not specify the display locations of all of the videos.
  • the interactive decoder preprocesses the video streams 412, 414 corresponding to the videos 402, 404 and discards or ignores the video stream 416 corresponding to the video 406.
  • the interactive decoder generates control signals based upon the user's selections and optionally the default display configuration. Based upon the control signals, the interactive decoder preprocesses each slice of the selected video streams to relocate each video stream as necessary depending upon the selected or default display position for that video or video stream. The interactive decoder then interleaves the preprocessed slices of the selected smaller videos, namely videos 402 and 404.
  • FIG. 5C illustrates a portion of the resulting single interleaved video stream 420 representing a larger composite video as generated by the interactive decoder. Because each slice of the video stream 412 corresponding to video 402 is positioned at the left edge or border of the display, the slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 of the video stream 412 are not modified as they are not repositioned in the final video output. In other words, the slices in the resulting single interleaved video stream 420 cooesponding to video 402 are unmodified and are interleaved with modified video slices cooesponding to other video streams.
  • the slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6 of the video stream 414 are modified such that the video stream contains data indicating such destination position of the slices.
  • the modified slices 2.1 '-2.6' include patched coordinate 422 representing the new destination location of the slices.
  • slices 2.1-2.6 are modified and become slices 2.1 '-2.6' where slices 2.1 '-2.6' differ from slices 2.1-2.6 only in their vertical, i.e., column, positions, as given by the slice start code value.
  • the interactive decoder only modifies the horizontal position of each slice of the one video stream that is repositioned in the example shown and described herein, the interactive decoder may need to modify both the horizontal and vertical positions or only the vertical position of the one or more video stream that are to be repositioned.
  • FIG. 5D schematically illustrates a composite video 424 which would be displayed on the user's display, such as a television set.
  • the composite video 424 represents the resulting single interleaved video stream 420.
  • the composite video 424 can be generated from the single interleaved video stream 420.
  • FIG. 6 is a table listing the MPEG-1 and MPEG-2 MBAI VLC (macroblock address increment variable length codeword) and the cooesponding increment values.
  • the MBAI VLC has length of 1 to 11 bits and indicates the horizontal positioning of the slice.
  • the number of bits for increment values of 1, 2-3, 4-5, 6-7, 8-9, 10-15, 16-21, and 22-33 are 1, 3, 4, 5, 7, 8, 10, and 11, respectively.
  • a slice can be easily modified to have an MBAI VLC value having the same number of bits by simply modifying the bits to a different value.
  • a slice having an MBAI VLC value cooesponding to an increment value of 2 can be easily modified to have an MBAI VLC value cooesponding to an increment value of 3, or vice versa.
  • a slice having an MBAI VLC value cooesponding to an increment value in the range of 10-15, the range of 16-21, or the range of 22-33 can be easily modified to have another MBAI VLC value cooesponding to an increment value in the same range.
  • a slice can also be easily modified to have an MBAI VLC value having the same number of bits modulo 8.
  • the resulting MBAI VLC will have the same bit position within a byte as the original MBAI VLC, i.e., the bit alignment within a byte would be unchanged.
  • the MBAI VLC maintain the same bit alignment within a byte obviates the need to rotate all bits in the slice following the MBAI VLC bits, such as in a shift register. Avoiding such shifting or rotation of the remaining bits in the slice avoids a relatively expensive process.
  • the number bits modulo 8 for increment values of 2-3 and the number bits modulo 8 for increment values of 16-21 are both equal to 3.
  • the MBAI VLC of the slice can be easily modified to a value cooesponding to an increment value of between 22 and 33, or vice versa.
  • VLC has the desired or destination value of 0000 0100 001.
  • the reverse conversion process can also be easily implemented by removing a byte from the MBAI VLC and modifying the remaining 3 bits to the desired value.
  • all slices that begin at the left edge of the display are preferably encoded with a MBAI VLC value cooesponding to increment values of 2 or 3 by the broadcast center, for example, such that the slices can be easily modified to be repositioned to a display position cooesponding to an increment value of between 22 and 33.
  • the MBAI value is preferably modified from an increment value of 2 or 3 to an increment value of between 22 and 33, inclusive.
  • the above-described modification of the MBAI VLC can be applied to any other modification which results in a modified MBAI VLC having the same number of bits modulo 8.
  • a slice having an MBAI VLC value of 0000 0101 11 cooesponding to an increment value of 16 can be modified to a slice having an MBAI VLC value of 0000 0001 000 0000 111 cooesponding to an increment value of 41.
  • the MBAI VLC value of 0001 000 0000 111 corresponding to an increment value of 41 is obtained by appending the 0000 111 bits cooesponding to an increment value of 8 to the MB escape value of 0000 0001 000.
  • the original MBAI VLC has 10 bits and the modified MBAI VLC has 18 bits.
  • the original MBAI VLC and the modified MBAI VLC have the same number of bits modulo 8, i.e., 2.
  • the MBAI escape value it is noted that the largest MBAI VLC has 11 bits and the MBAI escape value can be repeated as necessary to add 33 to the increment. Thus, if the MBAI escape value is repeated twice, followed by 0000 1001, the resulting increment is 78.
  • FIG. 7 is a flow diagram illustrating an exemplary process of an MPEG-2 interactive decoder of the present invention in processing a plurality of video streams encoded in MPEG-2 for simultaneous transmission and rendering.
  • the MPEG-2 interactive decoder determines the destination MBAI VLC or the cooesponding increment value of the slices for each video stream to be repositioned.
  • the MPEG-2 interactive decoder modifies the MBAI VLC value of each slice of the video stream that is to be repositioned while maintaining the same bit- alignment of the MBAI VLC within a byte.
  • Examples of the modification of the MBAI VLC value performed by the MPEG-2 interactive decoder include: (1) adding a byte and modifying the original bits, if necessary; (2) removing a byte and modifying the remaining bits, if necessary; (3) modifying the original bits; (4) modifying the original bits, if necessary, and appending the modified bits to MB escape.
  • the system and method of the present invention may also be utilized in the context of MPEG-1 compression.
  • MPEG-1 compression provides an addition MBAI_stuffing code.
  • the MBAI_stuffing code has 11 bits, a prime number.
  • the MBAI code can be bit-aligned to any bit position within a byte by adding a selective number of the MBAI_stuffing code. It is noted that in the case of MPEG-1 encoded video streams, an additional requirement is that all slices start and end on the same row.
  • FIG. 8 is a table listing the number of MPEG-1 MBAI_stuffmg codes, the cooesponding number of bits, and the cooesponding number of bits modulo 8. As shown, adding 0, 1, 2, 3, 4, 5, 6, or 7 MBAI_stuffing codes results in a bit-alignment shift of 0, 3, 6, 1, 4, 7, 2, or 5 bits. Thus, the MBAI code can be bit-aligned to any bit position within a byte by adding between 0 and 7 MBAI_stuffing codes.
  • FIG. 9 is flow diagram illustrating a process 400 of an MPEG-1 interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering.
  • the interactive decoder determines the destination MBAI VLC and/or the cooesponding increment value of the slices for each video stream to be repositioned.
  • the interactive decoder determines the number of MBAI stuffing codes to be added to the MBAI such that, after the MBAI is modified, the resulting code is bit-aligned to the desired bit position within a byte.
  • the interactive decoder modifies the MBAI value of each slice of the video stream to be repositioned such that the video slice would be repositioned at the desired location.
  • the interactive decoder inserts the predetermined number of MBAI_stuffing codes into the slice so as to maintain the same bit-alignment of the MBAI within a byte.
  • an application for the system and method described herein is a personalized EPG (electronic program guide) mosaic where a plurality of the end user's favorite shows is displayed, based upon a personalized list, for example.
  • the end user may optionally select a subset of displays from personalized EPG mosaic and/or their respective display positions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Graphics (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Television Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Time-Division Multiplex Systems (AREA)

Abstract

A system and method for the simultaneous transmission and rendition of multiple digital video streams in an interactive television application are disclosed. Simultaneous transmission and rendition of multiple encoded digital video signal streams in an interactive television application generally comprises modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video streams for display (362), interleaving the multiple encoded video streams having the at least one modified encoded video stream into a single composite interleaved video stream (364), and outputting the single composite video stream. The method may further comprise receiving the multiple encoded video streams from a broadcast center. Each of the multiple encoded video streams is preferably encoded using MPEG-1 or MPEG-2 compression technology.

Description

INTERACTIVE TELEVISION SYSTEM AND METHOD
FOR SIMULTANEOUS TRANSMISSION AND
RENDERING OF MULTIPLE ENCODED VIDEO
STREAMS
CROSS REFERENCE TO RELATED APPLICATIONS
This application is related to co-pending U.S. Patent Application 09/494,067 (Attorney Docket No. OPTVP009), entitled "Interactive Television System and Method for Simultaneous Transmission and Rendering of Multiple MPEG-Encoded Video Streams", filed concurrently herewith, which is incorporated herein by reference in its entirety.
BACKGROUND OF THE INVENTION
1. Field of the Invention The present invention relates generally to interactive video delivery mediums such as interactive television. More particularly, the present invention relates to a system and method for the simultaneous transmission and rendition of multiple MPEG-encoded digital video signal streams in an interactive television application.
2. Description of Related Art
Interactive television is an interactive audio/video delivery medium which provides broadcast audiovisual content to a number of subscribers. Interactive television provides broadcast video and audio to users and may also provide a return path for the user to interact with the content, e.g., to make selections or order desired products, etc. In a television broadcast by a television network, such as a broadcast of a bicycle race, the television network may generate multiple video feeds to the network from various angles of the race or from the various bicyclists, for example. The network may select one or more feeds from the multiple video feeds and broadcast the selected video feed(s) to the viewing audience at any given point in time. As is evident, each viewer does not have the option to individually select which video feeds are to be rendered simultaneously for viewing.
A point-to-point network can enable each viewer to select the video feeds to be rendered simultaneously from a set of available video feeds. In the point-to-point network such as in an on-line environment, each viewer may send a request to the head-end server selecting which video feeds the viewer wishes to view. The server may then recompose the screen for each viewer on the head-end and then sent it to the specific viewer. However, such a point-to-point network or on-line environment requires a significant amount of bandwidth as well as a return path from the viewer site to the head-end server in order for the viewer to send its video selections to the head-end server. Further, such a point-to-point network or on-line environment also requires additional hardware in the head-end server for picture re-composition for each active client.
Another system which can enable each viewer to select the video feeds that the viewer wishes to view from the set of available video feeds is a system having as many decoders in a receiver at the viewer site as individual videos to be rendered simultaneously. For example, if six individual videos are to be rendered simultaneously, the receiver at the viewer site must provide six decoders. However, such a system would require significant processing power in the receiver and increase the cost of the receiver. In addition, the number of videos that can be rendered simultaneously would be limited to the number of decoders provided in the receiver.
Thus, it would be greatly desirable to provide a relatively simple and cost effective system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams in an interactive television application such that each viewer may select its own set of one or more video feeds from a number of video feeds. Ideally, such a system and method would not require a significant amount of bandwidth or a return path from the viewer site to the head-end server.
MPEG Background
Background on MPEG (Moving Pictures Experts Group) compression is presented here in order to facilitate discussion and understanding of the present invention. MPEG compression is a set of methods for compression and decompression of full motion video images which uses interframe and intraframe compression techniques. MPEG compression uses both motion compensation and discrete cosine transform (DCT) processes, among others, and can yield compression ratios of more than 200:1.
The two predominant MPEG standards are referred to as MPEG-1 and MPEG- 2. The MPEG-1 standard generally concerns inter-field data reduction using block- based motion compensation prediction (MCP), which typically uses temporal differential pulse code modulation (DPCM). The MPEG-2 standard is similar to the MPEG-1 standard but includes extensions to cover a wider range of applications. As used herein, the term "MPEG" refers to MPEG-1, MPEG-2, and/or any other suitable MPEG-standard compression and decompression techniques.
An MPEG stream includes three types of pictures or frames, referred to as the Intra (I) frame, the Predicted (P) frame, and the Bi-directional Interpolated (B) frame. The I or Intra frames contain the video data for the entire frame of video and are typically placed every 10 to 15 frames. Intra frames provide entry points into the file for random access, and are generally only moderately compressed. Predicted frames are encoded with reference to a past frame, i.e., a prior Intra frame or Predicted frame. Thus P frames only include changes relative to prior I or P frames. In general, Predicted frames receive a fairly high amount of compression and are used as references for future Predicted frames. Thus, both I and P frames are used as references for subsequent frames. Bi-directional pictures include the greatest amount of compression and require both a past and a future reference in order to be encoded. Bi-directional frames are not used as references for other frames. An MPEG encoder divides respective frames into a grid of 16 by 16 pixel squares called macroblocks. The respective frames are divided into macroblocks in order to perform motion estimation/compensation. Each picture is comprised of a plurality of slices. The MPEG standard defines a slice as a contiguous sequence of 2 or more macroblocks (16 x 16 pixel blocks) that begin and end on the same row of macroblocks. A slice begins with a slice start code and includes information indicating the horizontal and vertical location where the slice begins in the picture. SUMMARY OF THE INVENTION
A system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams in an interactive television application are disclosed. It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication lines. Several inventive embodiments of the present invention are described below. The method for simultaneous rendering of multiple encoded video streams at a user site generally comprises modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning at least one of the multiple encoded video stream for display, interleaving the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and outputting the single composite video stream. The method may further comprise receiving the multiple encoded video streams from a broadcast center or via cable, satellite, MMDS, or terrestrial networks.
In one preferred embodiment, the method further comprises receiving a list of encoded video streams available for simultaneous rendering and optionally receiving user input from a user at the user site, the user input including the multiple encoded video streams selected from the list of encoded video streams available for simultaneous rendering. The user input may further include a display location for each of the multiple encoded video streams. The modifying may utilize the user input display locations for each of the multiple encoded video streams for repositioning the at least one of the multiple encoded video stream for display. Alternatively, the modifying may utilize a default display location for each of the multiple encoded video streams for repositioning the at least one of the multiple encoded video stream for display. For example, the default display locations for the multiple encoded video streams may be static values stored in the interactive decoder or receiver, determined by layout rules (such as HTML layout engine style) as applied by the interactive decoder, and/or any other suitable methods for determining default display locations. The layout rules may be modified by the producer on the fly at the production center and be transmitted to the receiver for rendering.
The method may further include decoding the single composite interleaved video stream prior to the outputting. The outputting may include outputting to a display for displaying the single composite interleaved video stream as a video display, the video display comprising a display of each of the modified multiple encoded video streams including a repositioned display of each of the at least one modified encoded video stream. Further, each of the multiple encoded video streams is preferably encoded using MPEG-1 or MPEG-2 compression technology.
In another preferred embodiment, the method generally comprises receiving multiple encoded video streams broadcast from a broadcast center for display, selecting at most one of the multiple encoded video streams for each display location, interleaving the selected video streams into a single composite interleaved video stream, and outputting the single composite video stream. In particular, each of the multiple encoded video streams is configured for display in one of a plurality of display locations, where each display location covers a subportion of a display and where the multiple encoded video streams include a plurality of encoded video streams for at least one of the display locations.
A system for simultaneous rendering of multiple encoded video streams generally comprises an interactive decoder adapted to modify at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display, interleave the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and output the single composite video stream. The system may further comprise a user interface in communication with the interactive decoder and adapted to receive user input from a user. The user input generally includes the multiple encoded video streams selected from the list of encoded video streams available for simultaneous rendering, wherein the interactive decoder is further adapted to receive the user input from a user via the user interface. In addition, the system may comprise a display for displaying the single composite interleaved video stream as a video display, wherein the video display comprises a video of each of the modified multiple encoded video streams including a repositioned video of each of the at least one modified encoded video stream.
An interactive decoder for simultaneous rendering of multiple encoded video streams, comprising means for modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display, means for interleaving the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream, and means for outputting the single composite video stream.
An example of an application for the system and method described herein is a personalized EPG (electronic program guide) mosaic. With the personalized EPG mosaic, a plurality of the end user's favorite shows is displayed, based upon a personalized list, for example. The end user may optionally select a subset from the plurality of the displayed screens or video feeds.
These and other features and advantages of the present invention will be presented in more detail in the following detailed description and the accompanying figures which illustrate by way of example the principles of the invention.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which:
FIG. 1 is a schematic diagram illustrating an interactive television system including a remote network that provides programming content according to one embodiment of the present invention;
FIG. 2 is a block diagram of the interactive decoder of FIG. 1; FIG. 3 is a schematic diagram illustrating the operation of the interactive decoder of the present invention;
FIGS. 4 A and 4B are flow diagram illustrating the process of the interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering; FIGS. 5A-5D illustrate by way of example the operation of the interactive television system of the present invention;
FIG. 6 is a table listing the MPEG-1 and MPEG-2 MBAI VLC (macroblock address increment variable length codeword) and the corresponding increment values; FIG. 7 is a flow diagram illustrating the process of an MPEG-2 interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering;
FIG. 8 is a table listing the number of MPEG-1 MBAI_stuffing codes, the corresponding number of bits, and the corresponding number of bits modulo 8; and FIG. 9 is flow diagram illustrating the process of an MPEG-1 interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering.
DESCRIPTION OF SPECIFIC EMBODIMENTS A system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams in an interactive television application are disclosed. The following description is presented to enable any person skilled in the art to make and use the invention. Descriptions of specific embodiments and applications are provided only as examples and various modifications will be readily apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications and equivalents consistent with the principles and features disclosed herein. For purpose of clarity, details relating to technical material that is known in the technical fields related to the invention have not been described in detail so as not to unnecessarily obscure the present invention.
Interactive Television System In a preferred embodiment, the system and method of the present invention are comprised in an interactive television system. However, it is to be understood that the system and method of the present invention may be included in any of various types of video systems, including standard television, interactive television, the Internet, and other types of video or graphics delivery mediums. The term "video" as used herein is intended to include still images, graphics, and live action or motion video. FIG. 1 is a schematic diagram of an interactive television system 100 according to one embodiment of the present invention. It is noted that FIG. 1 is illustrative only and other interactive television system embodiments may be utilized as desired. As shown, the interactive television system 100 employs a broadcast center
102 for generating audiovisual content, including non-interactive audiovisual content and/or interactive audiovisual content. The broadcast center 102 includes an audio/video source 104 for providing the audiovisual content. The audiovisual content may comprise movies, sports, news, sitcoms or other audiovisual programming, as is normally seen on television. The audiovisual content preferably also comprises still images which may be utilized for various purposes. The audiovisual content preferably comprises a plurality of video streams and preferably configured or designed such that two or more of the video streams can be simultaneously displayed. The broadcast center 102 provides a plurality of program channels each comprising audiovisual content. One or more program channels may comprise audio- video-interactive (AVI) signals including audiovisual content and associated interactive applications. According to a preferred embodiment of the present invention, the encoder 108 at the broadcast center 102 is operable to encode or compress the videos and to transmit the compressed video streams.
The audio/video source 104 may include video compression logic for compressing still video images into compressed still video images. As shown, the audiovisual content is optionally supplied by a remote network 120 or a live feed as desired.
The broadcast center 102 preferably also includes an application server 106 for creating and/or generating interactive application content. The interactive application content comprises application code and data which is designed to be executed by a processor within a set top box or television to support an interactive television feature. The application server 106 is preferably configured for generating or providing "OpenTV" interactive applications. The application server may also provide "Java" applets or other interactive program content as desired. It is noted that the system and method for simultaneous transmission and rendering of multiple encoded digital video signal streams of the present invention need not be implemented in conjunction with such interactive application content.
An Encoder/Multiplexer 108 combines the interactive application content with the audiovisual content to produce an audio-video-interactive (AVI) signal. The Encoder/Multiplexer 108 also synchronizes the interactive application content with the audiovisual content to ensure that the interactive application content is inserted in the proper position within the audiovisual content. Certain channels may comprise non-interactive programming content. In other words, certain audiovisual content may not have associated interactive content. The Encoder/Multiplexer 108 preferably multiplexes a plurality of signals, including AVI signals and optionally non- interactive signals, together for transmission.
The broadcast center 102 also includes an uplink satellite 110 for transmitting the broadcast signal for receipt by end-users or a subsequent distribution link. Although FIG. 1 illustrates a satellite network example where a satellite is utilized to transmit the broadcast signal, other transmission methods may be utilized, including cable distribution through coaxial cable, fiber optic transmission, microwave transmission and/or any other suitable means.
The broadcast signal is received from broadcast center 102 via a satellite downlink 132, for example. The broadcast signal is then provided to a plurality of end-users or subscribers. As noted above, the broadcast signal may include one or more non-interactive program signals and may include one or more AVI signals on respective channels.
Each end user at a subscriber site 130 preferably includes a set top box receiver or interactive decoder 140 as well as a television set 150. The set top box or interactive decoder 140 is coupled to the television set 150. It is noted that the logic of the interactive decoder 140 may be comprised in the television 150 such that the television set 150 and the interactive decoder 140 comprise a single integrated unit rather than separate units. In addition, the television 150 may comprise a general purpose programmable computer having a display screen or any other suitable viewing device. In the present disclosure, the term "subscriber television" is intended to include the television set 150 or other viewing device, such as a computer, LCD (liquid crystal display) screen, etc., and may include associated decoder logic for decoding compressed video and/or executing interactive programs. For example, in one embodiment, the subscriber television comprises the television set 150 coupled to the interactive decoder or set top box 140.
A user interface such as a television remote control 152 may be provided at the user site to facilitate user interaction with the television 150 and/or interactive decoder 140. The user can select desired television channels for viewing or select from various interactive selections. According to one embodiment of the present invention, the user can select two or more video streams from among a plurality of simultaneously broadcast videos streams and/or still pictures for display. The user can also optionally select the display locations of each of the multiple video streams on the television set 150. The end user can also selectively configure a graphical user interface (GUI) on the display screen using the present invention.
Although the receiver and the interactive decoder 140 are shown as one unit, the receiver may include any suitable number of interactive decoders and optionally any additional components, such as a recomposition engine, as will be described below. The system and method for the simultaneous transmission and rendition of multiple encoded digital video signal streams described herein are suitable for the general case where the number of streams rendered is equal to or greater than the number of decoders present in the receiver. In addition, the number of streams rendered is typically less than or equal to the number of video streams are sent but can be greater than the number of video streams are sent such as where certain streams are displayed more than once.
The signals on the selected television channels are decoded by the interactive decoder 140 which provides an output to the television set 150. The interactive decoder 140 preferably executes a real time operating system, such as OpenTV. Although the description herein is in terms of a system that processes or decodes signals received via broadcast, it is to be understood that the processing may by performed on broadcast signals and/or signals stored in a component resident at the user site, i.e., resident memory, such as in a hard drive (or other memory) of the decoder (or other component).
In the case where the channel being viewed comprises an AVI signal, the interactive decoder 140 may also execute an interactive application program conveyed within the selected AVI signal to enable an interactive television feature.
A return channel 156 optionally interconnects the interactive decoder 140 to a transaction server 160 which monitors certain selections by the user and responds accordingly. The return channel 156 preferably utilizes a standard POTS (plain old telephone system) phone line and associated modems (not shown) for communication. Other return channel options such as coaxial cable, fiber optic cable, DSL (Digital Subscriber Line), ATM (Asynchronous Transfer Mode), or others, may be used, as desired.
In the preferred embodiment, the broadcast center 102 broadcasts program such as a bicycle race which includes various video feeds such as a video feed from a camera on each rider's bicycle. Each video feed is independently compressed and independently broadcast to all satellite downlinks 132. The incoming streams broadcasted by the broadcast center 102 are MPEG streams but need not be fully compliant with the MPEG standard. For purposes of clarity, such non-compliant streams are also referred to as MPEG streams herein. For example, the incoming broadcast streams may be encoded MPEG styles but some of the elements of the streams, such as the sequence headers, may be added by the receiver to the combined output stream.
In addition, a list of available video streams is preferably made available by the interactive application of the broadcast center 102. Each user may select the desired videos from the list of available video streams, thereby allowing each user to monitor the angles of the race that is of interest to the user, such as the points of view of the user's favorite racers.
The interactive decoder 140 receives as inputs the user's selection of which video feeds are to be simultaneously displayed on the user's television set 150 and, optionally, where each of the selected video feed is to be displayed on the user's television set 150. The interactive decoder 140 ignores or discards the non-selected video signals, preprocesses the selected signals in order to reposition the selected video signals for display, and decodes the preprocessed signals. The interactive decoder 140 then presents the user's selected videos as a large composite video which is a mosaic of small pictures on the screen of the user's television set 150. Such a process of the interactive decoder 140, although interactive in that the user selects the desired videos and their respective display locations, does not require the utilization of the return channel 156 as the processing of the user's selections are completed by the interactive decoder 140 which directly outputs the decoded video signals to the television set 150. The simultaneous display of multiple video streams may be used in a number of different interactive television applications. For example, the viewer may choose to display a web page or MPEG still from a commercial while simultaneously display other programs. As another example, an interactive commercial can allow the user to select to simultaneously view a plurality of products from a line of products or details of various features of a product.
Interactive Decoder 140
FIG. 2 is a schematic diagram illustrating the set top box or interactive decoder 140. As shown, the interactive decoder 140 includes a demultiplexer 302 for receiving input broadcast channel signals 304 over one or more channels. The input broadcast signals preferably comprises a plurality of channels containing programming content, such as movies, sports, television shows, news, advertising, etc. The broadcast signal preferably includes a plurality of compressed or encoded digital video streams. The audiovisual component is preferably compressed and more preferably MPEG-1 or MPEG-2 compressed. As noted above, only the video (or audio visual) signals cooesponding to the video streams selected to be displayed are processed by the interactive decoder 140 while the non-selected video signals are discarded. The demultiplexer 302 operates to separate the audio/video component from the interactive component of the input broadcast channel signals 304. In a preferred embodiment of the invention, the interactive component comprises an interactive application and/or data which is compliant with the OpenTV standard promulgated by OpenTV Corp. However, other suitable types of interactive applications may be included in the AVI signal.
The demultiplexer 302 provides the audio/video component to an audio/video decoding processor or decoder 304 via a picture buffer 306. The decoding processor 304 operates to preprocess the plurality of selected video streams in order to reposition the video signals for display, interleaves the preprocessed signals, and decodes the interleaved compressed audiovisual signals to produce decompressed data.
In a prefeoed embodiment, the broadcast channel transports MPEG-2 compressed data. Thus, in the preferred embodiment, the decoding processor 304 preferably includes an MPEG-2 preprocessor, interleave processor, and decoder. The decoder 304 preferably includes one or more digital signal processors ("DSPs") and one or more associated memories. It is noted that the decoder may comprise various types of logic, including one or more DSPs, CPUs, or microcontrollers, discrete logic, or combinations thereof. Details of the decoding processor 304 will be described in more detail.
The decoding processor 304 provides an audio/video output which is preferably provided to the television set 150 for display. When the decoding block 304 receives a plurality of user-selected video streams to be simultaneously displayed, the decoder outputs a single decompressed video stream comprising a mosaic of smaller repositioned pictures to the user's television set 150.
The interactive decoder 140 may further include a CPU 314 coupled to a memory 316. The memory 316 is preferably a read only memory (ROM) for storing the interactive application operating system. The operating system is preferably the OpenTV operating system from OpenTV Corp.
Where the input broadcast channel signals 304 include an AVI signal, the demultiplexer 302 provides the interactive program component of the AVI signal to a interactive application memory 312 coupled to the CPU 314. The interactive application memory is preferably a random access memory (RAM) for storing the interactive application, i.e., the OpenTV application. The interactive application from the AVI signal is stored in the RAM 312 and the CPU 314 can read the interactive application from the RAM 312 and execute the application. Thus, the CPU 314 reads the OpenTV operating system from the ROM 316, whereby the operating system controls the operation of the CPU 314. The CPU 314 also reads the interactive application from the RAM 312, provided with the AVI signal, and executes the interactive application under control of the OpenTV operating system stored in the ROM316. As noted above, in a preferred embodiment, a list of video streams available to the user to be displayed simultaneously may be comprised in the interactive application. Thus, the list of available video streams is preferably included as part of the interactive application rather than as part of the audiovisual component. In contrast, the video streams are included as part of the audio/video stream. Alternatively, the list of video streams available may be transmitted in the signaling of the channel. The signaling of the channel is the list of components that is sent by the broadcaster to the receivers in a uniform manner for all channels. This mechanism is, for example, utilized to allow the receiver to detect secondary audio channels. The RAM 312 is coupled to the CPU 314. The CPU 314 is in turn coupled to a graphic on-screen display 322. The graphic on-screen display 322 is coupled to provide an output to the television set in conjunction with the audio/video output provided by decoder 304. The graphic on-screen display 322 organizes data in a format which can be overlaid on top of the audio/video output of the decoder 304. Thus, the CPU 314 executes an interactive application and generates on-screen graphics which are provided in conjunction with the audio/video stream output from the decoder 140. For example, if the interactive application is written to cause one or more images or selection options to be displayed or overlaid on top of a television program or still image, the CPU 314 executing the application provides the appropriate output to the graphic on-screen display 322, and the graphic on-screen display 322 causes the respective images or selection buttons to be overlaid on top of and/or displayed with the audio/video output of the decoder 304.
The graphic on-screen display 322 may be 4-bit or 8-bit color memory. The graphic on-screen display 322 may be used for various purposes in addition to a presentation of selection buttons or a list of video streams available for simultaneous display.
The user may provide user input 324 to the CPU 314. The user input may be provided from various devices and is preferably provided via the remote control, buttons on the television set 150, or the interactive decoder 140. The user input 324 provided to the CPU 314 enables a user to interact with the interactive application stored in RAM 312. For example, the user or viewer may select the plurality of video streams to be simultaneously displayed on the television set. Other user inputs 324 may include selection to order a product or order information, provide answers to a television game show, select an insert picture or insert feed from a choice of insert pictures or feeds, wherein the selected insert picture is for display in a background picture, for example. Also, the user or viewer may select locations of each of the plurality of selected video streams to be simultaneously displayed on the television set.
The CPU 314 includes an output coupled to the picture buffer 306 such that the picture buffer 306. The CPU 314 sends signals to the picture buffer 306 relating to which the video (or audio visual) signals are selected for simultaneous display by the user and the corresponding display location on the television set of each selected video stream. Thus, the picture buffer 306 only receives the video streams selected to be displayed while the non-selected video signals are discarded.
The CPU 314 is preferably further coupled to the return channel 156 via a modem 326 for providing user interactivity. The modem 326 may be used for bidirectional communication. As shown in FIG. 1, the return channel 156 is preferably coupled to a transaction server 160. Thus, if the user selects an option to order information, or purchase a product, the transaction server 160 receives this order and processes the order for fulfillment. Also, if the user answers questions in a game show, the answers are provided on the return channel 156 to the transaction server 160. FIG. 3 is a schematic diagram illustrating the operation of the interactive decoder 140. In the example shown in FIG. 3, four video streams, in particular, video streams 1-4, are simultaneously displayed. In addition, video streams 1-4 are displayed in the lower right, upper left, upper right, and lower left quadrant of the display of the television set, respectively. Preferably, the interactive decoder 140 shown in FIG. 3 is utilized for MPEG-2 compressed video streams.
For each video stream selected to be displayed, the video stream signals 330 and a corresponding control signal 332 containing, for example, the display location information are input to a MBAI (macroblock address increment) modifier 334. The video stream signals 330 represents smaller pictures. The MBAI modifier 334 modifies the MBAI of the video stream signals 330, if necessary, according to the display location of that video stream contained in the cooesponding control signal 332. The output of the MBAI modifiers 334, i.e., video streams with modified MBAI, are input into an interleave 336.
The interleave 336 interleaves the smaller modified video streams into a larger single video stream. The single larger video stream is essentially a mosaic of the repositioned smaller pictures. The video decoder 338, preferably a MPEG-2 video decoder, then decodes the single video stream and outputs to the user's television set. The television set then displays the decoded single video stream representing the mosaic of the repositioned smaller pictures according to the user-specified display locations, default display locations, or a combination of user-specified and default display locations. For example, the default display locations for the multiple encoded video streams may be static values stored in the interactive decoder or receiver, determined by layout rules (such as HTML layout engine style) and/or any other suitable methods for determining default display locations. The default location may be computed from the layout rules by a recomposition engine which can reside in the receiver or the head-end, depending upon the configuration utilized. MPEG-2 Definitions of Certain Terms
For purposes of clarity and to facilitate the discussion set forth below, several terms are presented and defined according to MPEG-2 standard. For example, a pixel refers to a fundamental picture element while a block refers to an 8 by 8 grouping of 64 pixels. A macroblock (MB) is defined as a 2 by 2 grouping of 4 blocks or a 16 by 16 grouping of pixels. A slice refers to a sequence of macroblocks which, for the purposes of the present invention, begins and ends on the same row of MBs. A slice startcode ("SCC") is a byte-aligned unsigned sequence of 4 bytes indicating the start of a slice. The values for the slice startcode range between 0x00000101 and OxOOOOOlAF. The last byte of the slice startcode is the slice startcode value which indicates the row of macroblocks on which the current slice belongs. For example, a value of 01 indicates that the pixels in the slice belong on rows 0 to 15 inclusive and OxAF indicates that the pixels in the slice belong on rows 2784 to 2799 inclusive. A macroblock address increment (MBAI) is a variable length codeword (VLC) of length 1 to 11 bits. The MBAI indicates an increment, in units of macroblock s, from the last macroblock of a slice on the same row and indicates the horizontal positioning of the current slice, i.e., the position of the slice from the left edge of the display. For example, an MBAI value of 1 indicates that the current slice belongs on column 0 and extending to at least column 15, inclusive. The MBAI VLC and/or the SCC may be collectively referred to as display position code(s) as they are codes which indicate the display positions of the particular slice. Flow Chart Illustrating the Process of the Interactive Decoder 140
FIGS. 4A and 4B are flow diagram illustrating the process 350 of the interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering. As shown in FIG. 4A, the interactive decoder first receives the encoded video signals broadcast by the broadcast center as well as a long of available videos, preferably in the interactive component of the AVI signal in step 352.
At step 354, the interactive decoder generates the control signals based upon the user's selections of the desired videos for simultaneous display and their corresponding specified and/or default display positions. It is to be understood that, although the user may select each of the videos that are to be displayed simultaneously, one or more videos may be displayed by default. In other words, one or more default videos and one or more user-selected videos may be displayed. At step 356, the interactive decoder receives the broadcast video signal streams. Alternatively, the interactive decoder may receive the list of the available videos in an interactive component of the AVI signal, such as in step 352, generate control signals in step 354, and selectively receive the user-selected broadcasted video streams. The selective reception of the user-selected broadcasted video streams may be achieved based upon an identification or identifier of the packets into which each video stream is divided, such as by the server at the broadcast center. It is noted that the broadcast center divides each MPEG video stream into video packets identified with an identifier and multiplexes the packets in the time domain.
At step 358, the interactive decoder determines whether each of received video streams correspond to one of the selected videos. If so, the interactive decoder processes the selected video streams and outputs the processed output video signals to a display for simultaneous display of the selected videos at step 360. In other words, in step 360, the interactive decoder combines the MPEG encoded video streams prior to decoding. Alternatively, for video streams which do not correspond to the videos selected by the user, the interactive decoder discards or ignores the non-selected video streams.
FIG. 4B illustrates process or step 360 of the interactive decoder in more detail. In particular, at step 362, the interactive decoder preprocesses each slice of the selected video streams to relocate each video stream as necessary depending upon the selected or default display position for that video or video stream. The relocation, if necessary, includes modifying the MBAI LVC and/or the slice startcode or SCC. As noted above, the SCC is a byte-aligned unsigned sequence of 4 bytes indicating the start of a slice, the last byte of which is the slice startcode value which indicates the row of macroblocks on which the current slice belongs. Thus, to reposition or relocate the horizontal or row position of the slice, i.e., the row of macroblocks on which that the slice belongs, the value of the SCC is modified. It is noted that because the SCC is not of a variable length, the value of the SCC may simply be modified without concern regarding bit-alignment within a byte, i.e., byte-alignment. Next, at step 364, the interactive decoder interleaves the preprocessed slices of the selected smaller videos in an order such that a single video stream representing a larger composite video is generated. The interleaving process can be similar to that described in copending U.S. patent application Number 08/951,795, entitled "INTERACTIVE TELEVISION SYSTEM AND METHOD FOR INSERTING PICTURES WITHIN PICTURES USING MPEG-1 FRAMES", filed on October 16, 1997, and incorporated in its entirety herein by reference.
Alternatively, although not prefeoed, the interactive decoder can first interleave the slices of the plurality of the selected video streams and then preprocess, i.e., patch the coordinates of the slices of the interleaved video stream that are to be repositioned. However, because parsing of the interleaved video stream can be difficult, such a process is generally not preferred.
At step 366, the interactive decoder decodes the single interleaved video stream representing the larger composite video. Finally, at step 368, the interactive decoder outputs the decoded single interleaved video stream representing the larger composite video of the videos selected by the user to a display, such as a television set.
Further, the interactive decoder optionally resizes the single interleaved composite video stream in step 368. Preferably, the individual sequences are encoded in a spatial resolution that is smaller than the display such that multiple pictures can be combined and displayed with or without resizing, depending upon the number of pictures that are combined. In particular, the broadcast center preferably broadcasts each video feed in accordance with a predetermined size, e.g., one-quarter of a typical display size, such as the typical high-definition television display size. Thus, if the end user selects four pictures for simultaneous display, resizing by the decoder would be unnecessary.
The specific predetermined size broadcasted by the broadcast center may depend in part on the optimization of the desired resolution of the picture while minimizing processing requirements. In MPEG compression technology, the video feed is divides into macroblocks and each macroblock is compressed into a compressed stream. The interactive decoder decompresses each compressed stream to rebuild the picture. The interactive decoder can then zoom the picture to the desired size. In other words, the interactive decoder generally cannot resize each compressed slice but can resize an overall picture. Thus, the interactive decoder preferably resizes the single interleaved composite video stream in order to display the resulting video in a desired size.
For example, where each video feed from the broadcast center is one-quarter of the typical television display size and the user selects 6 video feeds for simultaneous viewing, the single interleaved composite video stream would be too large for the display. The interactive decoder thus preferably resizes the composite video stream into a desired size, preferably a size which does not distort the videos but maximizes the utilization of the viewing portion of the display.
The receiver preferably also inserts a fill-in sequence to cover portions of the display or screen that are not covered with the combined videos. For example, if the incoming streams each take one-quarter of the output display and three sequences are selected and combined, the receiver may cover the remainder quarter of the display screen with a fill-in video which can either be sent by the broadcast server or generated by the receiver, such as an empty sequence (e.g., a green rectangle) or a still I-frame.
In one embodiment, none of the streams is modified for positioning. Such a case may result where the head-end sends multiple optional streams for each location covering a subportion of the display and where the end user of the receiver, e.g., by default and/or based on user profile, selects an option for each position. In other words, the multiple encoded video streams broadcast would be configured for display in one of the two or more display locations. The multiple encoded video streams would include more than one encoded video stream for at least one of the display locations. For example, a commercial may include a number of sub-pictures each at fixed display location and the content of each display location would be selected by the end user and/or by the receiver based on user profile. In particular, a commercial for a department store may comprise a left sub-picture that advertises the department store in general and a right sub-picture that advertises a particular product. The left sub-picture may be selected based on the preferred language of the end user while the right sub-picture may be selected based on the profile of the end user household, such as the number and ages of the children in the household. In such a scenario, the left sub-pictures sent by the head-end may be an English and a Spanish advertisement for the department store in general and the right sub-pictures sent by the head-end may be a toy and a cloth advertisement.
It is to be noted that although the process 360 of combining multiple MPEG encoded streams prior to decoding is preferably executed at the user site, the process 360 may be implemented elsewhere, such as at the broadcast center, for example. The configuration for such an implementation is preferably where a primary head-end sends the compressed videos to a secondary head-end and where the secondary headend recomposes videos and narrow cast the recombined stream to a single receiver or group of receivers. The utilization of the secondary head-end is optional and provides savings in bandwidth. The benefit of such a configuration is that hardware costs in the receivers are minimized or reduced. For example, such a configuration would be beneficial in the case where receivers are already deployed and cannot easily be upgraded with the necessary additional hardware or software. In addition, such a configuration would be beneficial in the case where it is cheaper to centralize the capability in secondary head-ends if the feature is only utilized by and/or available to a limited number of receivers at a given point in time.
Example of Simultaneous Rendition of Multiple Video Streams
The process of the interactive decoder as shown and described with reference to the flow diagrams of FIGS. 4A and 4B will now be illustrated by way of example. In particular, FIGS. 5A-5D illustrate an example of the operation of the interactive television system of the present invention. FIG. 5 A illustrates three separate pictures or videos 402, 404, 408. Each of the separate videos contains a number of slices. For example, video 402 includes slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6. Similarly, video 404 includes slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6 and video 406 includes slices 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6.
FIG. 5B illustrates the transmission of each of the videos 402, 404, 406 from the broadcast center to the interactive decoder as a video stream. For example, video 402 comprising slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 is transmitted as a video stream 412. As shown, the video stream 412 comprises the slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 in series. Similarly, video 402 is transmitted as video stream 414 comprising the slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6 and video 406 is transmitted as video stream 416 comprising the slices 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 in series.
In addition to the video streams of the videos or pictures available to the user for simultaneous display, the broadcast center preferably also transmits a list of the video streams available to the user to be displayed simultaneously. As noted above, the list of available video streams may be comprised in the interactive component of the AVI signal.
At the user' site, the user may select two or more of the videos 402, 404, 406 for simultaneous display. In the example shown in FIGS. 5C and 5D, the user selects to display video 402 on the left hand side and video 404 on the right hand side of the television set display. Alternatively, such a display configuration may be the default display configuration where the user does not specify the display locations of all of the videos. Thus, the interactive decoder preprocesses the video streams 412, 414 corresponding to the videos 402, 404 and discards or ignores the video stream 416 corresponding to the video 406.
The interactive decoder generates control signals based upon the user's selections and optionally the default display configuration. Based upon the control signals, the interactive decoder preprocesses each slice of the selected video streams to relocate each video stream as necessary depending upon the selected or default display position for that video or video stream. The interactive decoder then interleaves the preprocessed slices of the selected smaller videos, namely videos 402 and 404.
FIG. 5C illustrates a portion of the resulting single interleaved video stream 420 representing a larger composite video as generated by the interactive decoder. Because each slice of the video stream 412 corresponding to video 402 is positioned at the left edge or border of the display, the slices 1.1, 1.2, 1.3, 1.4, 1.5, and 1.6 of the video stream 412 are not modified as they are not repositioned in the final video output. In other words, the slices in the resulting single interleaved video stream 420 cooesponding to video 402 are unmodified and are interleaved with modified video slices cooesponding to other video streams.
In contrast, because each slice of the video stream 414 cooesponding to video 404 needs to be repositioned such that the video's left edge is at the middle of the display, the slices 2.1, 2.2, 2.3, 2.4, 2.5, and 2.6 of the video stream 414 are modified such that the video stream contains data indicating such destination position of the slices. In one embodiment, the modified slices 2.1 '-2.6' include patched coordinate 422 representing the new destination location of the slices. Thus, slices 2.1-2.6 are modified and become slices 2.1 '-2.6' where slices 2.1 '-2.6' differ from slices 2.1-2.6 only in their vertical, i.e., column, positions, as given by the slice start code value. It is noted that although the interactive decoder only modifies the horizontal position of each slice of the one video stream that is repositioned in the example shown and described herein, the interactive decoder may need to modify both the horizontal and vertical positions or only the vertical position of the one or more video stream that are to be repositioned.
The unmodified slices 1.1-1.6 of the video stream 412 and the modified slices 2.1 '-2.6' of the video stream 414 are then interleaved to result in a single interleaved video stream 420 representing a larger composite video comprising both videos 412, 414 for simultaneous display. FIG. 5D schematically illustrates a composite video 424 which would be displayed on the user's display, such as a television set. The composite video 424 represents the resulting single interleaved video stream 420. In other words, the composite video 424 can be generated from the single interleaved video stream 420. Preferred Embodiment: Modifying the MPEG-2 MBAI VLC
FIG. 6 is a table listing the MPEG-1 and MPEG-2 MBAI VLC (macroblock address increment variable length codeword) and the cooesponding increment values. As noted above, the MBAI VLC has length of 1 to 11 bits and indicates the horizontal positioning of the slice. As shown in FIG. 6, the number of bits for increment values of 1, 2-3, 4-5, 6-7, 8-9, 10-15, 16-21, and 22-33 are 1, 3, 4, 5, 7, 8, 10, and 11, respectively.
It is noted that a slice can be easily modified to have an MBAI VLC value having the same number of bits by simply modifying the bits to a different value. For example, a slice having an MBAI VLC value cooesponding to an increment value of 2 can be easily modified to have an MBAI VLC value cooesponding to an increment value of 3, or vice versa. Similarly, a slice having an MBAI VLC value cooesponding to an increment value in the range of 10-15, the range of 16-21, or the range of 22-33 can be easily modified to have another MBAI VLC value cooesponding to an increment value in the same range.
In addition, a slice can also be easily modified to have an MBAI VLC value having the same number of bits modulo 8. By modifying the MBAI VLC to another MBAI VLC having the same number of bits modulo 8, the resulting MBAI VLC will have the same bit position within a byte as the original MBAI VLC, i.e., the bit alignment within a byte would be unchanged. In other words, by having the MBAI VLC maintain the same bit alignment within a byte obviates the need to rotate all bits in the slice following the MBAI VLC bits, such as in a shift register. Avoiding such shifting or rotation of the remaining bits in the slice avoids a relatively expensive process. In particular, the number bits modulo 8 for increment values of 2-3 and the number bits modulo 8 for increment values of 16-21 are both equal to 3. Thus, if an encoder encoded a slice having an MBAI VLC value cooesponding to an increment value of 2 or 3, the MBAI VLC of the slice can be easily modified to a value cooesponding to an increment value of between 22 and 33, or vice versa.
The modification of an MBAI VLC value of 011 cooesponding to an increment value of 2 to the MBAI VLC value of 0000 0100 001 cooesponding to an increment value of 24 will be presented as an example. In particular, a byte having value 0000 0100 is added to the original MBAI VLC and the 3 bits (011) of the original MBAI VLC are modified to a value of 001 such that the resulting MBAI
VLC has the desired or destination value of 0000 0100 001. As is evident, the reverse conversion process can also be easily implemented by removing a byte from the MBAI VLC and modifying the remaining 3 bits to the desired value. Generally, only the first MBAI of the slices in a given row needs to be modified. Thus, all slices that begin at the left edge of the display are preferably encoded with a MBAI VLC value cooesponding to increment values of 2 or 3 by the broadcast center, for example, such that the slices can be easily modified to be repositioned to a display position cooesponding to an increment value of between 22 and 33. In such an implementation, if the slices of a video stream are to be repositioned, the MBAI value is preferably modified from an increment value of 2 or 3 to an increment value of between 22 and 33, inclusive.
Although the modification between an increment value in the range of 2 to 3 and an increment value in the range of 22 to 33 is prefeoed and presented above, the above-described modification of the MBAI VLC can be applied to any other modification which results in a modified MBAI VLC having the same number of bits modulo 8. For example, a slice having an MBAI VLC value of 0000 0101 11 cooesponding to an increment value of 16 can be modified to a slice having an MBAI VLC value of 0000 0001 000 0000 111 cooesponding to an increment value of 41. The MBAI VLC value of 0001 000 0000 111 corresponding to an increment value of 41 is obtained by appending the 0000 111 bits cooesponding to an increment value of 8 to the MB escape value of 0000 0001 000. The original MBAI VLC has 10 bits and the modified MBAI VLC has 18 bits. Thus, the original MBAI VLC and the modified MBAI VLC have the same number of bits modulo 8, i.e., 2. With respect to the MBAI escape value, it is noted that the largest MBAI VLC has 11 bits and the MBAI escape value can be repeated as necessary to add 33 to the increment. Thus, if the MBAI escape value is repeated twice, followed by 0000 1001, the resulting increment is 78.
The above-described modification of the MBAI VLC value in all the slices of a video stream results in the repositioning of all the slices of the video stream.
Flow Chart Illustrating the Process of an MPEG-2 Interactive Decoder 140
FIG. 7 is a flow diagram illustrating an exemplary process of an MPEG-2 interactive decoder of the present invention in processing a plurality of video streams encoded in MPEG-2 for simultaneous transmission and rendering. In step 382, the MPEG-2 interactive decoder determines the destination MBAI VLC or the cooesponding increment value of the slices for each video stream to be repositioned. In step 384, the MPEG-2 interactive decoder modifies the MBAI VLC value of each slice of the video stream that is to be repositioned while maintaining the same bit- alignment of the MBAI VLC within a byte. Examples of the modification of the MBAI VLC value performed by the MPEG-2 interactive decoder include: (1) adding a byte and modifying the original bits, if necessary; (2) removing a byte and modifying the remaining bits, if necessary; (3) modifying the original bits; (4) modifying the original bits, if necessary, and appending the modified bits to MB escape.
Alternative Embodiment: Modifying the MPEG-1 MBAI
The system and method of the present invention may also be utilized in the context of MPEG-1 compression. In addition to the MBAI code, MPEG-1 compression provides an addition MBAI_stuffing code. The MBAI_stuffing code has 11 bits, a prime number. Thus, the MBAI code can be bit-aligned to any bit position within a byte by adding a selective number of the MBAI_stuffing code. It is noted that in the case of MPEG-1 encoded video streams, an additional requirement is that all slices start and end on the same row.
FIG. 8 is a table listing the number of MPEG-1 MBAI_stuffmg codes, the cooesponding number of bits, and the cooesponding number of bits modulo 8. As shown, adding 0, 1, 2, 3, 4, 5, 6, or 7 MBAI_stuffing codes results in a bit-alignment shift of 0, 3, 6, 1, 4, 7, 2, or 5 bits. Thus, the MBAI code can be bit-aligned to any bit position within a byte by adding between 0 and 7 MBAI_stuffing codes.
FIG. 9 is flow diagram illustrating a process 400 of an MPEG-1 interactive decoder of the present invention in processing a plurality of video streams for simultaneous transmission and rendering. At step 402, the interactive decoder determines the destination MBAI VLC and/or the cooesponding increment value of the slices for each video stream to be repositioned. At step 404, the interactive decoder determines the number of MBAI stuffing codes to be added to the MBAI such that, after the MBAI is modified, the resulting code is bit-aligned to the desired bit position within a byte. At step 406, the interactive decoder modifies the MBAI value of each slice of the video stream to be repositioned such that the video slice would be repositioned at the desired location. In addition, the interactive decoder inserts the predetermined number of MBAI_stuffing codes into the slice so as to maintain the same bit-alignment of the MBAI within a byte.
While the prefeoed embodiments of the present invention are described and illustrated herein, it will be appreciated that they are merely illustrative and that modifications can be made to these embodiments without departing from the spirit and scope of the invention. Thus, the invention is intended to be defined only in terms of the following claims. For example, an application for the system and method described herein is a personalized EPG (electronic program guide) mosaic where a plurality of the end user's favorite shows is displayed, based upon a personalized list, for example. The end user may optionally select a subset of displays from personalized EPG mosaic and/or their respective display positions.

Claims

CLAIMSWhat is claimed is:
1. A method for simultaneous rendering of multiple encoded video streams, comprising: modifying at least one of the multiple encoded video streams for repositioning the at least one of the multiple encoded video stream for display, at least one of said multiple encoded video streams being broadcast from a broadcast center, said modifying generates at least one modified encoded video stream; selecting from said at least one modified encoded video stream and multiple encoded video streams; interleaving said selected video streams into a single composite interleaved video stream; and outputting said single composite video stream.
2. The method for simultaneous rendering of multiple encoded video streams of claim 1, further comprising resizing the single composite interleaved video stream prior to said outputting.
3. The method for simultaneous rendering of multiple encoded video streams of claim 1, further comprising receiving the multiple encoded video streams from a broadcast center.
4. The method for simultaneous rendering of multiple encoded video streams of claim 1, further comprising receiving a list of encoded video streams available for simultaneous rendering.
5. The method for simultaneous rendering of multiple encoded video streams of claim 4, wherein said receiving the list of encoded video streams available for simultaneous rendering is via channel signaling.
6. The method for simultaneous rendering of multiple encoded video streams of claim 4, further comprising receiving user input from a user, said user input includes selection of the multiple encoded video streams selected from said list of encoded video streams available for simultaneous rendering.
7. The method for simultaneous rendering of multiple encoded video streams of claim 6, wherein said user input further includes a display location for each of at least one of the multiple encoded video streams.
8. The method for simultaneous rendering of multiple encoded video streams of claim 7, wherein said modifying utilizes each display location of the user input for the encoded video stream cooesponding to said display location.
9. The method for simultaneous rendering of multiple encoded video streams of claim 1, wherein said modifying utilizes a default display location for each of at least one of the multiple encoded video streams.
10. The method for simultaneous rendering of multiple encoded video streams of claim 9, wherein said modifying determines each default display location by computing each default display location from layout rules.
11. The method for simultaneous rendering of multiple encoded video streams of claim 1, further comprising decoding said single composite interleaved video stream prior to said outputting.
12. The method for simultaneous rendering of multiple encoded video streams of claim 1, wherein said outputting includes outputting to a display for displaying said single composite interleaved video stream as a video display, the video display comprising a display of the modified multiple encoded video streams.
13. The method for simultaneous rendering of multiple encoded video streams of claim 1, wherein each of the multiple encoded video streams is encoded using a compression technology selected from the group consisting of MPEG-1 and MPEG-2.
14. The method for simultaneous rendering of multiple encoded video streams of claim 1, wherein said selecting is performed based upon stored preferences.
15. The method for simultaneous rendering of multiple encoded video streams of claim 1, wherein at least one of said multiple encoded video streams is stored in resident memory.
16. A method for simultaneous rendering of multiple encoded video streams, comprising: receiving multiple encoded video streams broadcast from a broadcast center for display, each encoded video streams being configured for display in one of a plurality of display locations, each display location covering a subportion of a display, said multiple encoded video streams include a plurality of encoded video streams for at least one of said display locations; selecting at most one of said multiple encoded video streams for each of said plurality of display locations; interleaving said selected video streams into a single composite interleaved video stream; and outputting said single composite video stream.
17. A system for simultaneous rendering of multiple encoded video streams, comprising: an interactive decoder adapted to modify at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display and to generate modified multiple encoded video streams comprising the at least one modified encoded video stream and unmodified encoded video streams of the multiple encoded video streams, the interactive decoder further adapted to interleave the modified multiple encoded video streams comprising the at least one modified encoded video stream into a single composite interleaved video stream and to output said single composite video stream.
18. The system for simultaneous rendering of multiple encoded video streams of claim 17, wherein said interactive decoder is further adapted to receive the multiple encoded video streams from a broadcast center.
19. The system for simultaneous rendering of multiple encoded video streams of claim 17, wherein said interactive decoder is further adapted to receive a list of encoded video streams available for simultaneous rendering.
20. The system for simultaneous rendering of multiple encoded video streams of claim 19, further comprising a user interface in communication with said interactive decoder and adapted to receive user input from a user, said user input includes at least one of the multiple encoded video streams selected from said list of encoded video streams available for simultaneous rendering.
21. The system for simultaneous rendering of multiple encoded video streams of claim 20, wherein said user input further includes a display location for each of at least one of the multiple encoded video streams.
22. The system for simultaneous rendering of multiple encoded video streams of claim 21, wherein said interactive decoder is adapted to utilize each user input display location for the encoded video stream cooesponding to said display location for modifying the cooesponding encoded video stream for repositioning.
23. The system for simultaneous rendering of multiple encoded video streams of claim 17, wherein said interactive decoder is adapted to utilize a default display location for each of at least one of the multiple encoded video streams for repositioning.
24. The system for simultaneous rendering of multiple encoded video streams of claim 17, wherein said interactive decoder is adapted to decode said single composite interleaved video stream prior to outputting.
25. The system for simultaneous rendering of multiple encoded video streams of claim 17, further comprising a display in communication with said interactive decoder, wherein said interactive decoder is adapted to output to said display for displaying said single composite interleaved video stream as a video display, wherein the video display comprises a display of the modified multiple encoded video streams.
26. The system for simultaneous rendering of multiple encoded video streams of claim 17, further comprising a display for displaying said single composite interleaved video stream as a video display, wherein the video display comprises a video of each of the modified multiple encoded video streams including a repositioned video of each of the at least one modified encoded video stream.
27. The system for simultaneous rendering of multiple encoded video streams of claim 17, wherein each of the multiple encoded video streams is encoded using a compression technology selected from the group consisting of MPEG-1 and MPEG-2.
28. The system for simultaneous rendering of multiple encoded video streams of claim 17, further comprising a broadcast center for broadcasting the multiple encoded video streams to said interactive decoder.
29. An interactive decoder for simultaneous rendering of multiple encoded video streams, comprising: means for modifying at least one of the multiple encoded video streams broadcast from a broadcast center for repositioning the at least one of the multiple encoded video stream for display, said modifying means generates modified multiple encoded video streams comprising the at least one modified encoded video stream and unmodified encoded video streams of the multiple encoded video streams; means for interleaving the modified multiple encoded video streams into a single composite interleaved video stream; and means for outputting said single composite video stream.
30. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, further comprising means for receiving the multiple encoded video streams from a broadcast center.
31. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, further comprising means for receiving a list of encoded video streams available for simultaneous rendering.
32. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 31 , further comprising means for receiving user input from a user interface, said user input including selection of the multiple encoded video streams selected from said list of encoded video streams available for simultaneous rendering.
33. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 32, wherein said user input includes a display location for each of at least one of the multiple encoded video streams.
34. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 33, wherein said modifying means is adapted to utilize each display location of the user input to modify the encoded video stream cooesponding to said display location.
35. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, wherein said modifying means is adapted to utilize a default display location for each of at least one of the multiple encoded video streams.
36. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, further comprising means for decoding said single composite interleaved video stream.
37. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, wherein said outputting means is adapted to output to a display for displaying said single composite interleaved video stream as a video display.
38. The interactive decoder for simultaneous rendering of multiple encoded video streams of claim 29, further comprising means for displaying said single composite interleaved video stream as a video display.
PCT/US2001/000830 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams WO2001056292A1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
ES01900989.3T ES2555953T3 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and playback of multiple encoded video streams
DK01900989.3T DK1262068T3 (en) 2000-01-28 2001-01-10 Interactive television system and method for the simultaneous transmission and reproduction of multiple encoded video streams
CA002397645A CA2397645C (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams
EP01900989.3A EP1262068B8 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams
JP2001554620A JP4084941B2 (en) 2000-01-28 2001-01-10 Method for simultaneous communication rendering of multiple encoded video streams, simultaneous communication rendering system for multiple encoded video streams, and interactive decoder for simultaneous rendering of multiple encoded video streams
BRPI0107872A BRPI0107872B1 (en) 2000-01-28 2001-01-10 simultaneous rendering method of multiple encoded video streams, simultaneous rendering system of multiple encoded video streams, and interactive decoder for simultaneous rendering of multiple encoded video streams
AU26388/01A AU773588B2 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US09/494,067 2000-01-28
US09/494,184 2000-01-28
US09/494,067 US6931660B1 (en) 2000-01-28 2000-01-28 Interactive television system and method for simultaneous transmission and rendering of multiple MPEG-encoded video streams
US09/494,184 US6539545B1 (en) 2000-01-28 2000-01-28 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams

Publications (1)

Publication Number Publication Date
WO2001056292A1 true WO2001056292A1 (en) 2001-08-02

Family

ID=27051285

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/US2001/000830 WO2001056292A1 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams
PCT/US2001/000959 WO2001056293A1 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple mpeg-encoded video streams

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/US2001/000959 WO2001056293A1 (en) 2000-01-28 2001-01-10 Interactive television system and method for simultaneous transmission and rendering of multiple mpeg-encoded video streams

Country Status (11)

Country Link
EP (2) EP1262068B8 (en)
JP (2) JP4188599B2 (en)
CN (2) CN1193611C (en)
AU (2) AU777694B2 (en)
BR (2) BRPI0107872B1 (en)
CA (2) CA2397529A1 (en)
DK (1) DK1262068T3 (en)
ES (1) ES2555953T3 (en)
HK (1) HK1053564A1 (en)
PT (1) PT1262068E (en)
WO (2) WO2001056292A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004056098A1 (en) * 2002-12-16 2004-07-01 Koninklijke Philips Electronics N.V. Method for a mosaic program guide
FR2872988A1 (en) * 2004-07-12 2006-01-13 Canon Kk Mosaic video flow producing method for e.g. navigation in set of video data, involves constructing video flow forming mosaic flow by inserting extracted base segments of original video flows in same extraction order
US7937740B2 (en) 2002-08-16 2011-05-03 MediaIP, Inc. Method and apparatus for interactive programming using captioning
US8255966B2 (en) 2002-04-15 2012-08-28 Shusman Chad W Method and apparatus for internet-based interactive programming
US10856031B2 (en) 2003-04-15 2020-12-01 MedialP, Inc. Method and apparatus for generating interactive programming in a communication network
US11722697B2 (en) 2020-12-17 2023-08-08 Axis Ab Method and digital video camera for forming a combined image frame of a combined video stream
US12088863B2 (en) 2016-05-26 2024-09-10 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Broadcast streaming of panoramic video for interactive clients

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1579694A4 (en) * 2002-11-14 2009-10-28 Opentv Inc Positioning of images in a data stream
US20040111526A1 (en) * 2002-12-10 2004-06-10 Baldwin James Armand Compositing MPEG video streams for combined image display
US7519274B2 (en) * 2003-12-08 2009-04-14 Divx, Inc. File format for multiple track digital data
CN100515056C (en) * 2006-11-09 2009-07-15 华为技术有限公司 Method, system apparatus and set top box for displaying mutli-channel information in identical picture
ES2675950T3 (en) 2007-05-30 2018-07-13 Orange Generation of a customizable TV mosaic
GB2457253B (en) * 2008-02-07 2010-08-11 Ericsson Telefon Ab L M Controlling media distribution
WO2012011859A1 (en) * 2010-07-21 2012-01-26 Telefonaktiebolaget L M Ericsson (Publ) Picture coding and decoding
WO2012060459A1 (en) 2010-11-01 2012-05-10 日本電気株式会社 Dynamic image distribution system, dynamic image distribution method, and dynamic image distribution program
JP5685683B2 (en) * 2011-10-28 2015-03-18 株式会社Gnzo Byte alignment device and byte alignment method
US10409445B2 (en) 2012-01-09 2019-09-10 Activevideo Networks, Inc. Rendering of an interactive lean-backward user interface on a television
US9800945B2 (en) 2012-04-03 2017-10-24 Activevideo Networks, Inc. Class-based intelligent multiplexing over unmanaged networks
JP5812028B2 (en) * 2013-03-08 2015-11-11 ソニー株式会社 Video processing apparatus and video processing method
WO2014145921A1 (en) 2013-03-15 2014-09-18 Activevideo Networks, Inc. A multiple-mode system and method for providing user selectable video content
US9326047B2 (en) 2013-06-06 2016-04-26 Activevideo Networks, Inc. Overlay rendering of user interface onto source video
CN103501466A (en) * 2013-09-06 2014-01-08 上海骋娱传媒技术有限公司 Method and equipment used for video resource access control
CN103501465A (en) * 2013-09-06 2014-01-08 上海骋娱传媒技术有限公司 Method and equipment used for video resource access control
US9788029B2 (en) 2014-04-25 2017-10-10 Activevideo Networks, Inc. Intelligent multiplexing using class-based, multi-dimensioned decision logic for managed networks
FR3030978A1 (en) * 2014-12-23 2016-06-24 Orange GENERATING A MOSAIC OF VIDEOS
US10264293B2 (en) 2014-12-24 2019-04-16 Activevideo Networks, Inc. Systems and methods for interleaving video streams on a client device
US10523985B2 (en) 2014-12-24 2019-12-31 Activevideo Networks, Inc. Managing deep and shallow buffers in a thin-client device of a digital media distribution network
CN105451022A (en) * 2015-11-17 2016-03-30 深圳联友科技有限公司 Method of compressing multipath video streams into video stream and system thereof
JP6387511B2 (en) * 2016-06-17 2018-09-12 株式会社アクセル Image data processing method
WO2018105515A1 (en) 2016-12-09 2018-06-14 株式会社アクセル Image data processing method
JP6732337B2 (en) * 2016-12-09 2020-07-29 株式会社アクセル Image data processing method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5649032A (en) * 1994-11-14 1997-07-15 David Sarnoff Research Center, Inc. System for automatically aligning images to form a mosaic image
US5995146A (en) * 1997-01-24 1999-11-30 Pathway, Inc. Multiple video screen display system

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5481297A (en) * 1994-02-25 1996-01-02 At&T Corp. Multipoint digital video communication system
WO1995028795A2 (en) * 1994-04-15 1995-10-26 Philips Electronics N.V. Arrangement for decoding digital video signals
US5691768A (en) * 1995-07-07 1997-11-25 Lucent Technologies, Inc. Multiple resolution, multi-stream video system using a single standard decoder
JPH1023348A (en) * 1996-05-02 1998-01-23 Matsushita Electric Ind Co Ltd Television broadcast program transmitter and receiver
JP3662129B2 (en) * 1997-11-11 2005-06-22 松下電器産業株式会社 Multimedia information editing device

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5649032A (en) * 1994-11-14 1997-07-15 David Sarnoff Research Center, Inc. System for automatically aligning images to form a mosaic image
US5995146A (en) * 1997-01-24 1999-11-30 Pathway, Inc. Multiple video screen display system

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8661490B2 (en) 2002-04-15 2014-02-25 MediaIP, Inc. Method and apparatus for internet-based interactive programming
US8255966B2 (en) 2002-04-15 2012-08-28 Shusman Chad W Method and apparatus for internet-based interactive programming
USRE48579E1 (en) 2002-04-15 2021-06-01 Media Ip, Inc. Method and apparatus for internet-based interactive programming
US7937740B2 (en) 2002-08-16 2011-05-03 MediaIP, Inc. Method and apparatus for interactive programming using captioning
US8402504B2 (en) 2002-08-16 2013-03-19 Media Ip, Inc. Method and apparatus for interactive programming using captioning
US8826361B2 (en) 2002-08-16 2014-09-02 Media Ip, Inc. Method and apparatus for interactive programming using captioning
WO2004056098A1 (en) * 2002-12-16 2004-07-01 Koninklijke Philips Electronics N.V. Method for a mosaic program guide
US8385426B2 (en) 2002-12-16 2013-02-26 Entropic Communications, Inc. Method for a mosaic program guide
US10856031B2 (en) 2003-04-15 2020-12-01 MedialP, Inc. Method and apparatus for generating interactive programming in a communication network
US11076190B2 (en) 2003-04-15 2021-07-27 MedialP, Inc. Method and apparatus for generating interactive programming in a communication network
US11477506B2 (en) 2003-04-15 2022-10-18 MediaIP, LLC Method and apparatus for generating interactive programming in a communication network
US11575955B2 (en) 2003-04-15 2023-02-07 MediaIP, LLC Providing interactive video on demand
FR2872988A1 (en) * 2004-07-12 2006-01-13 Canon Kk Mosaic video flow producing method for e.g. navigation in set of video data, involves constructing video flow forming mosaic flow by inserting extracted base segments of original video flows in same extraction order
US12088863B2 (en) 2016-05-26 2024-09-10 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Broadcast streaming of panoramic video for interactive clients
US11722697B2 (en) 2020-12-17 2023-08-08 Axis Ab Method and digital video camera for forming a combined image frame of a combined video stream

Also Published As

Publication number Publication date
CA2397645C (en) 2005-08-16
BR0107875A (en) 2002-11-05
BRPI0107872B1 (en) 2015-10-27
CN1397135A (en) 2003-02-12
ES2555953T3 (en) 2016-01-11
AU2638801A (en) 2001-08-07
WO2001056293A1 (en) 2001-08-02
PT1262068E (en) 2015-12-28
CA2397645A1 (en) 2001-08-02
JP2003521177A (en) 2003-07-08
CN1173570C (en) 2004-10-27
JP4188599B2 (en) 2008-11-26
EP1281280A4 (en) 2003-06-04
CN1193611C (en) 2005-03-16
BRPI0107875B1 (en) 2015-09-08
EP1262068B1 (en) 2015-09-30
EP1262068B8 (en) 2015-11-04
JP4084941B2 (en) 2008-04-30
HK1053564A1 (en) 2003-10-24
EP1262068A1 (en) 2002-12-04
EP1281280A1 (en) 2003-02-05
EP1262068A4 (en) 2003-06-04
AU2938201A (en) 2001-08-07
CA2397529A1 (en) 2001-08-02
DK1262068T3 (en) 2016-01-11
AU773588B2 (en) 2004-05-27
JP2003521176A (en) 2003-07-08
CN1397136A (en) 2003-02-12
AU777694B2 (en) 2004-10-28
BR0107872A (en) 2002-11-05

Similar Documents

Publication Publication Date Title
US6539545B1 (en) Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams
US6931660B1 (en) Interactive television system and method for simultaneous transmission and rendering of multiple MPEG-encoded video streams
EP1262068B1 (en) Interactive television system and method for simultaneous transmission and rendering of multiple encoded video streams
US6606746B1 (en) Interactive television system and method for displaying a graphical user interface using insert pictures
US9591343B2 (en) Communicating primary content streams and secondary content streams
CA2307233C (en) Encoding system and method for scrolling encoded mpeg stills in an interactive television application
CA2388608C (en) Method and apparatus for transmitting video and graphics in a compressed form
CA2388606C (en) Picture-in-picture and multiple video streams using slice-based encoding
US7810116B2 (en) Apparatus and method for combining realtime and non-realtime encoded content
US5990972A (en) System and method for displaying a video menu
US20040111526A1 (en) Compositing MPEG video streams for combined image display
US20080267589A1 (en) Television bandwidth optimization system and method
US9094727B1 (en) Multi-functional user interface using slice-based encoding

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AL AM AT AU AZ BA BB BG BR BY CA CH CN CR CU CZ DE DK DM EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 26388/01

Country of ref document: AU

WWE Wipo information: entry into national phase

Ref document number: 2397645

Country of ref document: CA

WWE Wipo information: entry into national phase

Ref document number: 2001900989

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 018042449

Country of ref document: CN

ENP Entry into the national phase

Ref country code: JP

Ref document number: 2001 554620

Kind code of ref document: A

Format of ref document f/p: F

WWP Wipo information: published in national office

Ref document number: 2001900989

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

WWG Wipo information: grant in national office

Ref document number: 26388/01

Country of ref document: AU