US20110134218A1

US20110134218A1 - Method and system for utilizing mosaic mode to create 3d video

Info

Publication number: US20110134218A1
Application number: US12/963,035
Authority: US
Inventors: Darren Neuman; Jason Herrick; Qinghua Zhao; Christopher Payson
Original assignee: Broadcom Corp
Current assignee: Avago Technologies International Sales Pte Ltd
Priority date: 2009-12-08
Filing date: 2010-12-08
Publication date: 2011-06-09
Also published as: US20150341613A1; US9307223B2; US8947503B2; US20110134216A1; US9137513B2; EP2462748A1; EP2462748A4; US20110134212A1; CN102474632A; US20110134217A1; WO2011072016A1; US20110134211A1

Abstract

A method and system are provided in which a video feeder may receive video data from multiple sources. The video data from one or more of those sources may comprise three-dimensional (3D) video data. The video data from each source may be stored in corresponding different areas in memory during a capture time for a single picture. Each of the different areas in memory may correspond to a different window of multiple windows in an output video picture. The video data from each source may be stored in memory in a 2D format or in a 3D format, based on a format of the output video picture. When a 3D format is to be used, left-eye and right-eye information may be stored in different portions of memory. The video data may be read from memory to a single buffer during a feed time for a single picture.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS/INCORPORATION BY REFERENCE

This application makes reference to, claims priority to, and claims the benefit of:
U.S. Provisional Patent Application Ser. No. 61/267,729 (Attorney Docket No. 20428US01) filed on Dec. 8, 2009;
U.S. Provisional Patent Application Ser. No. 61/296,851 (Attorney Docket No. 22866US01) filed on Jan. 20, 2010; and
U.S. Provisional Patent Application Ser. No. 61/330,456 (Attorney Docket No. 23028US01) filed on May 3, 2010.
This application also makes reference to:
U.S. Provisional patent application Ser. No. ______ (Attorney Docket No. 20428U502) filed on Dec. 8, 2010;
U.S. Provisional patent application Ser. No. ______(Attorney Docket No. 23437U502) filed on Dec. 8, 2010;
U.S. Provisional patent application Ser. No. ______ (Attorney Docket No. 23438U502) filed on Dec. 8, 2010; and
U.S. Provisional patent application Ser. No. ______ (Attorney Docket No. 23440U502) filed on Dec. 8, 2010.
Each of the above referenced applications is hereby incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

Certain embodiments of the invention relate to the processing of three-dimensional (3D) video. More specifically, certain embodiments of the invention relate to a method and system for utilizing mosaic mode to create 3D video.

BACKGROUND OF THE INVENTION

The availability and access to 3D video content continues to grow. Such growth has brought about challenges regarding the handling of 3D video content from different types of sources and/or the reproduction of 3D video content on different types of displays.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with the present invention as set forth in the remainder of the present application with reference to the drawings.

BRIEF SUMMARY OF THE INVENTION

A system and/or method for utilizing a mosaic mode to create 3D video, as set forth more completely in the claims.
Various advantages, aspects and novel features of the present invention, as well as details of an illustrated embodiment thereof, will be more fully understood from the following description and drawings.

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates a system-on-chip that is operable to utilize mosaic mode to create multiple windows in an output video picture, in accordance with an embodiment of the invention.

FIG. 2 is a diagram that illustrates various packing schemes for 3D video data, in accordance with embodiments of the invention.

FIG. 3 is a block diagram that illustrates a processing network that is operable to handle 3D video data, in accordance with an embodiment of the invention.

FIG. 4 is a diagram that illustrates the flow of data in mosaic mode, in accordance with an embodiment of the invention.

FIG. 5 is a diagram that illustrates an exemplary output video picture with multiple windows that is generated utilizing mosaic mode, in accordance with an embodiment of the invention.

FIG. 6 is a diagram that illustrates the storage of decoded video data from multiple sources in corresponding buffers in mosaic mode, in accordance with an embodiment of the invention.

FIG. 7 is a diagram that illustrates the generation of an output video picture having a 2D output format in mosaic mode, in accordance with an embodiment of the invention.

FIGS. 8A-8C are diagrams that illustrate the generation of an output video picture having a 3D output format in mosaic mode, in accordance with embodiments of the invention.

FIG. 9 is a flow chart that illustrates steps for generating output video pictures with multiple windows utilizing mosaic mode, in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Certain embodiments of the invention may be found in a method and system for utilizing a mosaic mode to create 3D video. Various embodiments of the invention may relate to a processor comprising a video feeder, such as an MPEG video feeder, for example, wherein the processor may receive video data from multiple sources through the video feeder. The video data from one or more of those sources may comprise 3D video data. The video data from each source may be stored in a corresponding different area in memory during a capture time for a single picture. Each of the different areas in memory may correspond to a different window of multiple windows in an output video picture. The processor may store the video data from each source in memory in either a 2D format or a 3D format, based on a format of the output video picture. For example, when a 3D format is to be used, left-eye and right-eye information may be stored in different portions of memory. The video data may be read from the different areas in memory to a single buffer during a feed time for a single picture before being utilized to generate the output video picture.
By utilizing a mosaic mode to process video data as described herein, a user may be provided with multiple windows in an output video picture to concurrently display video from different sources, including 3D video. The multi-windowed output video picture may have a 2D output format or a 3D output format based on, for example, the characteristics of the device in which the output video picture is to be displayed, reproduced, and/or stored. That is, while the different sources may provide 2D video data and/or 3D video data, the video data in each of the windows in the output video picture may be in the same format.
FIG. 1 is a block diagram that illustrates a system-on-chip (SoC) that is operable utilize mosaic mode to create multiple windows in an output video picture, in accordance with an embodiment of the invention. Referring to FIG. 1, there is shown an SoC 100, a host processor module 120, and a memory module 130. The SoC 100 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to receive and/or process one or more signals that comprise video content, including 3D video content. Examples of signals comprising video content that may be received and processed by the SoC 100 include, but need not be limited to, composite, blanking, and sync (CVBS) signals, separate video (S-video) signals, high-definition multimedia interface (HDMI) signals, component signals, personal computer (PC) signals, source input format (SIF) signals, and red, green, blue (RGB) signals. Such signals may be received by the SoC 100 from one or more video sources communicatively coupled to the SoC 100. The SoC 100 may also be operable to receive and/or process graphics content from one or more sources of such content.
The SoC 100 may generate one or more output signals that may be provided to one or more output devices for display, reproduction, and/or storage. For example, output signals from the SoC 100 may be provided to display devices such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma display panels (PDPs), thin film transistor LCDs (TFT-LCDs), plasma, light emitting diode (LED), Organic LED (OLED), or other flatscreen display technology. The characteristics of the output signals, such as pixel rate, resolution, and/or whether the output format is a 2D output format or a 3D output format, for example, may be based on the type of output device to which those signals are to be provided. Moreover, the output signals may comprise one or more output video pictures, each of which may comprise multiple windows to concurrently display video from different sources.
The host processor module 120 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to control and/or configure the operation of the SoC 100. For example, parameters and/or other information, including but not limited to configuration data, may be provided to the SoC 100 by the host processor module 120 at various times during the operation of the SoC 100. The host processor module 120 may be operable to control and/or select a mode of operation for the SoC 100. For example, the host processor module 120 may enable a mosaic mode for the SoC 100.
The memory module 130 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to store information associated with the operation of the SoC 100. For example, the memory module 130 may store intermediate values that result during the processing of video data, including those values associated with the processing of video data during mosaic mode. Moreover, the memory module 130 may store graphics data that may be retrieved by the SoC 100 for mixing with video data. For example, the graphics data may comprise 2D graphics data and/or 3D graphics data for mixing with video data in the SoC 100.
The SoC 100 may comprise an interface module 102, a video processor module 104, and a core processor module 106. The SoC 100 may be implemented as a single integrated circuit comprising the components listed above. The interface module 102 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to receive multiple signals that comprise video content and/or graphics content. Similarly, the interface module 102 may be operable to communicate one or more signals comprising video content to output devices communicatively coupled to the SoC 100. For example, the SoC 100 may communicate one or more signals that comprise a sequence of output video pictures comprising multiple windows to concurrently display video from different sources. The format of the multi-windowed output video pictures may be based on, for example, the characteristics of the output devices.
The video processor module 104 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to process video data and/or graphics data. The video processor module 104 may be operable to support multiple formats for video data and/or graphics data, including multiple input formats and/or multiple output formats. The video processor module 104 may be operable to perform various types of operations on 2D video data and/or 3D video data. For example, when video data from several sources is received by the video processor module 104, and the video data from any one of those sources may comprise 2D video data or 3D video data, the video processor module 104 may generate output video comprising a sequence of output video pictures having multiple windows, wherein each of the windows in an output video picture corresponds to a particular source of video data. In this regard, the output video pictures that are generated by the video processor module 104 may be in a 2D output format or in a 3D output format in accordance with the device in which the output video pictures are to be displayed, reproduced, and/or stored. That is, even when a portion of the sources provide video data comprising 2D video data and another portion of the sources provide video data comprising 3D video data, the output video pictures generated by the video processor module 104 may be generated in either a 2D output format or a 3D output format. In some embodiments, when the video content comprises audio data, the video processor module 104, and/or another module in the SoC 100, may be operable to handle the audio data.
The core processor module 106 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to control and/or configure the operation of the SoC 100. For example, the core processor module 106 may be operable to control and/or configure operations of the SoC 100 that are associated with processing video content and/or and graphics content. In some embodiments of the invention, the core processor module 106 may comprise memory (not shown) that may be utilized in connection with the operations performed by the SoC 100. For example, the core processor module 106 may comprise memory that may be utilized during the processing of video data and/or graphics data by the video processor module 104. The core processor module 106 may be operable to control and/or enable mosaic mode in the SoC 100. Such control and/or enabling may be performed in coordination with the host processor module 120, for example.
In operation, mosaic mode may be enabled in the SoC 100 by the host processor module 120 and/or the core processor module 106, for example. In this mode, the SoC 100 may receive video data from more than one source through the interface module 102. The video data from any one of the sources may comprise 2D video data or 3D video data. The video processor module 104 may decode the video data and may store the decoded video data from each source in a different buffer in memory. In this regard, the video processor module 104 may comprise a video decoder (not shown), such as an MPEG decoder, for example. The memory may be dynamic random access memory (DRAM), which may be part of the memory module 130, for example, and/or part of the SoC 100, such as embedded memory in the core processor module 106, for example.
A video feeder (not shown) within the video processor module 104, such as an MPEG video feeder, for example, may be utilized to obtain the video data from the buffers and feed the video data for capture into memory. The video data from each of the buffers may be fed and captured separately into a corresponding area in memory. Moreover, the capture of the video data from the various buffers may occur during a single picture capture time, that is, may occur during the time it takes to capture a single picture into memory by the SoC 100.
The capture of the video data allows for the generation of the multiple windows in the output video picture by storing the video data from a particular source in an area of memory that correspond to a particular window in the output video picture. In other words, the windowing operation associated with mosaic mode occurs in connection with the capture of the video data to memory.
During capture, the video data may be stored in memory in a 2D format or in a 3D format, based on a format of the output video picture. When the video data is stored in memory in a 3D format, left-eye and right-eye information may be stored in different portions of memory. The video data may be read from memory to a single buffer during a video feed process for a single picture, that is, during the time it takes to feed a single picture from memory by the SoC 100. Once the video data has been placed in the single buffer, it may be subsequently processed to generate the output video picture with multiple windows for communication to an output device through the interface module 102.
FIG. 2 is a diagram that illustrates various packing schemes for 3D video data, in accordance with embodiments of the invention. Referring to FIG. 2 there is shown a first packing scheme or first format 200 for 3D video data. Also shown is a second packing scheme or second format 210 for 3D video data. Each of the first format 200 and the second format 210 illustrates the arrangement of the left-eye content (L) and the right-eye content (R) in a 3D picture. The left-eye content or information may also be referred to as a left 3D picture and the right-eye content or information may also be referred to as a right 3D picture. In this regard, a 3D picture may correspond to a 3D frame or a 3D field in a video sequence, whichever is appropriate. The L and R portions in the first format 200 are arranged in a side-by-side arrangement, which is typically referred to as a left-and-right (L/R) format. The L and R portions in the second format 210 are arranged in a top-and-bottom arrangement, which is typically referred to as an over-and-under (O/U) format. Another arrangement, one not shown in FIG. 2, may be one in which the L portion is in a first 3D picture and the R portion is in a second 3D picture. Such arrangement may be referred to as a sequential format because the 3D pictures are processed and/or handled sequentially.
Both the first format 200 and the second format 210 may be utilized as native formats by the SoC 100 to process 3D video data. The SoC 100 may also be operable to utilize the sequential format as a native format, which may be typically handled by the SoC 100 in a manner that is substantially similar to the handling of the second format 210. The SoC 100 may also support converting from the first format 200 to the second format 210 and converting from the second format 210 to the first format 200. Such conversion may be associated with various operations performed by the SoC 100, including but not limited to operations associated with mosaic mode. The SoC 100 may support additional native formats other than the first format 200, the second format 210, and the sequential format, for example.
FIG. 3 is a block diagram that illustrates a processing network that is operable to handle 3D video data, in accordance with an embodiment of the invention. Referring to FIG. 3, there is shown a processing network 300 that may be part of the video processor module 104 in the SoC 100, for example. The processing network 300 may comprise suitable logic, circuitry, code, and/or interfaces that may be operable to route and process video data. In this regard, the processing network 300 may comprise multiple devices, components, modules, blocks, circuits, or the like, that may be selectively interconnected to enable the routing and processing of video data in accordance with various modes of operation, including mosaic mode. The various devices, components, modules, blocks, circuits, or the like in the processing network 300 may be dynamically configured and/or dynamically interconnected during the operation of the SoC 100 through one or more signals generated by the core processor module 106 and/or by the host processor module 120. In this regard, the configuration and/or the selective interconnection of various portions of the processing network 300 may be performed on a picture-by-picture basis when such an approach is appropriate to handle varying characteristics of the video data.
In the embodiment of the invention described in FIG. 3, the processing network 300 may comprise an MPEG feeder (MFD) module 302, multiple video feeder (VFD) modules 304, an HDMI module 306, crossbar modules 310 a and 310 b, multiple scaler (SCL) modules 308, a motion-adaptive deinterlacer (MAD) module 312, a digital noise reduction (DNR) module 314, multiple capture (CAP) modules 320, compositor (CMP) modules 322 a and 322 b, and an MPEG decoder module 324. The references to a memory (not shown) in FIG. 3 may be associated with a DRAM utilized by the processing network 300 to handle storage of video data and/or graphics data during various operations. Such DRAM may be part of the memory module 130 described above with respect to FIG. 1. In some instances, the DRAM may be part of memory embedded in the SoC 100. The references to a video encoder (not shown) in FIG. 3 may be associated with hardware and/or software in the SoC 100 that may be utilized after the processing network 300 to further process video data for communication to an output device, such as a display device, for example.
Each of the crossbar modules 310 a and 310 b may comprise multiple input ports and multiple output ports. The crossbar modules 310 a and 310 b may be configured such that any one of the input ports may be connected to one or more of the output ports. The crossbar modules 310 a and 310 b may enable pass-through connections 316 between one or more output ports of the crossbar module 310 a and corresponding input ports of the crossbar module 310 b. Moreover, the crossbar modules 310 a and 310 b may enable feedback connections 318 between one or more output ports of the crossbar module 310 b and corresponding input ports of the crossbar module 310 a. The configuration of the crossbar modules 310 a and/or 310 b may result in one or more processing paths being configured within the processing network 300 in accordance with the manner and/or order in which video data is to be processed. For example, one or more processing paths may be configured in accordance with a mode of operation, such as mosaic mode, for example.
The MFD module 302 may be operable to read video data from memory and provide such video data to the crossbar module 310 a. The video data read by the MFD module 302 may have been stored in memory after being generated by the MPEG decoder module 324. During mosaic mode, the MFD module 302 may be utilized to feed video data from multiple sources to one of the CAP modules 320, for example.
The MPEG decoder module 324 may be operable to decode video data received through one or more bit streams. For example, the MPEG encoder 324 may receive up to N bit streams, BS1, . . . , BSN, corresponding to N different sources of video data. Each bit stream may comprise 2D video data or 3D video data, depending on the source. When the MPEG decoder module 324 decodes a single bit stream, the decoded video data may be stored in a single buffer or area in memory. When the MPEG decoder module 324 decodes more than one bit stream, the decoded video data from each bit stream may be stored in a separate buffer or area in memory. In this regard, the MPEG decoder module 324 may be utilized during mosaic mode to decode video data from several sources and to store the decoded video data in separate buffers.
The MPEG decoder module 324 may be operable to provide the decoded video data in one or more formats supported by the processing network 300. For example, the MPEG decoder module 324 may provide decoded video data in a 2D format and/or in a 3D format, such as an L/R format, an O/U format, and a sequential format. The decoded video data may be stored in memory in accordance with the format in which it is provided by the MPEG decoder module 324.
Since mosaic mode is utilized to generate an output video picture with multiple windows, the amount of video data that is displayed in one of the windows in the output video picture is a small portion of the amount of video data that is needed to display the entire output video picture. Therefore, the size of each buffer utilized in mosaic mode may be smaller than the size of a single buffer utilized to store decoded video data from a single source during a different mode of operation.
Each VFD module 304 may be operable to read video data from memory and provide such video data to the crossbar module 310. The video data read by the VFD module 304 may have been stored in memory in connection with one or more operations and/or processes associated with the processing network 300. The HDMI module 306 may be operable to provide a live feed of high-definition video data to the crossbar module 310 a. The HDMI module 306 may comprise a buffer (not shown) that may enable the HDMI module 306 to receive the live feed at one data rate and provide the live feed to the crossbar module 310 a at another data rate.
Each SCL module 308 may be operable to scale video data received from the crossbar module 310 a and provide the scaled video data to the crossbar module 310 b. The MAD module 312 may be operable to perform motion-adaptive deinterlacing operations on interlaced video data received from the crossbar module 310 a, including operations related to inverse telecine (IT), and provide progressive video data to the crossbar module 310 b. The DNR module 314 may be operable to perform artifact reduction operations on video data received from the crossbar module 310 a, including block noise reduction and mosquito noise reduction, for example, and provide the noise-reduced video data to the crossbar module 310 b. In some embodiments of the invention, the operations performed by the DNR module 314 may be utilized before the operations of the MAD module 312 and/or the operations of the SCL module 308.
Each CAP module 320 may be operable to capture video data from the crossbar module 310 b and store the captured video data in memory. One of the CAP modules 320 may be utilized during mosaic mode to capture video data stored in one or more buffers and fed to the CAP module 320 by the MFD module 302. In this regard, the video data in one of the buffers is fed and captured separately from the video data in another buffer. That is, instead of a single capture being turned on for all buffers, a separate capture is turned on for each buffer, with all the captures occurring during a single picture capture time. The video data stored in each buffer may be captured to different areas in memory that correspond to different windows in the output video picture.
Each of the CMP modules 322 a and 322 b may be operable to combine or mix video data received from the crossbar module 310 b. Moreover, each of the CMP modules 322 a and 322 b may be operable to combine or mix video data received from the crossbar module 310 b with graphics data. For example, the CMP module 322 a may be provided with a graphics feed, Gfxa, for mixing with video data received from the crossbar module 310 b. Similarly, the CMP module 322 b may be provided with a graphics feed, Gfxb, for mixing with video data received from the crossbar module 310 b.
FIG. 4 is a diagram that illustrates the flow of data in mosaic mode, in accordance with an embodiment of the invention. Referring to FIG. 4, there is shown a processing path 400 that may be configured by interconnecting several modules in the processing network 300 described above with respect to FIG. 3. The processing path 400 may be utilized to process 2D video data and/or 3D video data to generate output video pictures with multiple windows to concurrently display video data from different sources.
The processing path 400 illustrates a flow of data within the processing network 300 when operating in mosaic mode. For example, bit streams, BS1, . . . , BS2, are provided to the MPEG decoder module 324. Each bit stream may correspond to a different source of video data and to a different window in the output video picture. The video data in the bit streams may be decoded by the MPEG decoder module 324 and stored in separate buffers in a memory 410. The decoded video data associated with each bit stream may be in one of the multiple formats supported by the processing network 300. The memory 410 may correspond to the memory described above with respect to FIG. 3. The MFD module 302 may feed the video data stored in the buffers to the CAP module 320 for capture in the memory 410. The feed and capture process may comprise feeding and capturing the video data in each of the buffers separately. The feed and capture of the video data in all of the buffers may occur during a single picture capture time by the CAP module 320. The feed and capture of the video data may occur in a different portion of the memory 410 than the portion utilized for buffering the output from the MPEG decoder module 324. In another embodiment of the invention, different memories may be utilized for video data buffering and for video data capture.
The video data may be captured in such a manner that the captured video data from each source is stored within the memory 410 in an area of memory that corresponds to a window in the output video picture. When the output video picture is to have a 2D output format, the captured video data may be stored in the memory 410 in a 2D format. When the output video picture is to have a 3D output format, the captured video data may be stored in the memory 410 in a 3D format. In this regard, the left-eye information and the right-eye information in the video data may be stored in different portions of the memory 410, for example.
The VFD module 304 may read and feed the captured video data to a single buffer. In this regard, the operation of the VFD module 304 during mosaic mode may be substantially the same as in other modes of operation. When the captured video data is stored in a 2D format in the memory 410, the VFD module 304 may read and feed the captured video data to the single buffer in the appropriate order to enable the generation of the output video picture in a 2D output format. When the captured video data is stored in a 3D format in the memory 410, the VFD module 304 may read and feed the captured video data to the single buffer in the appropriate order to enable the generation of the output video picture in a 3D output format. The 3D output format may be a 3D L/R output format or a 3D O/U output format. The video data in the single buffer may be communicated to a compositor module, such as the CMP module 322 a, for example, which in turn provides an input to a video encoder to generate the output video picture in the appropriate output format.
The processing path 400 is provided by way of illustration and not of limitation. Other data flow paths may be implemented during mosaic mode that may comprise more or fewer of the various devices, components, modules, blocks, circuits, or the like, of the processing network 300 to enable the generation of output video pictures comprising multiple windows for concurrently displaying video data from different sources.
FIG. 5 is a diagram that illustrates an exemplary output video picture with multiple windows that is generated utilizing mosaic mode, in accordance with an embodiment of the invention. Referring to FIG. 5, there is shown an output video picture 500 with multiple windows, each of which is associated with a different source of video data. The output video picture 500 may be representative of the type of output video picture that may be generated by the SoC 100 when operating in mosaic mode. The output video picture 500 may comprise 12 windows associated with 12 different sources of video data. That is, the video data that is to be displayed in each of the windows in the output video picture 500 is generated by the SoC 100 from video data received from a different source. For example, the video data for “Window 1” is received by the SoC 100 from a source S1. Similarly, the video data for “Window 2,” . . . , “Window 12,” is received by the SoC 100 from sources S2, . . . , S12, respectively.
The output video picture 500 is provided by way of illustration and not of limitation. For example, the SoC 100 may generate output video pictures that may comprise more or fewer windows than those shown in the output video picture 500. The size, layout, and/or arrangement of the video windows need not follow the size, layout, and/or arrangement of the output video picture 500. For example, the windows in an output video picture may be of different sizes. In another example, the windows in an output video picture need not be in a grid pattern. Moreover, the SoC 100 may be operable to dynamically change the characteristics and/or the video data associated with any one window in the output video picture 500.
FIG. 6 is a diagram that illustrates the storage of decoded video data from multiple sources in corresponding buffers in mosaic mode, in accordance with an embodiment of the invention. Referring to FIG. 6, there is shown the MPEG decoder module 324 being utilized during mosaic mode. In this exemplary embodiment, the MPEG decoder module 324 may receive bit streams from sources 1, . . . , N. The MPEG decoder module 324 may decode the video data in the N bit streams and may respectively generate decoded video data 1, . . . , decoded video data N. The decoded video data 1 may be stored in a buffer 610 in the memory 410, for example. The buffer 610 may correspond to an area of the memory 410 in which to store the decoded video data from source 1. Similarly, each of the decoded video data 2, . . . , decoded video data N may be stored in corresponding buffers 610 in the memory 410. The characteristics of each of the buffers 610 may vary based on the number, size, layout, and/or configuration of the windows in the output video picture that is to be generated by the SoC 100 while operating in mosaic mode.
As noted above, the MPEG decoder module 324 may be operable to provide the decoded video data in one or more formats. For example, the MPEG decoder module 324 may provide decoded video data in a 2D format and/or in a 3D format, such as an L/R format, an O/U format, and a sequential format. The decoded video data may be stored in the buffers 610 in the memory 410 in accordance with the format in which it is provided by the MPEG decoder module 324.
FIG. 7 is a diagram that illustrates the generation of an output video picture having a 2D output format in mosaic mode, in accordance with an embodiment of the invention. Referring to FIG. 7, there are shown four sources of video data 710, which are arranged in the figure in the manner in which the video data from each of the sources is to be displayed in an output video picture 730. For example, the video data associated with a first source, 51, which comprises 2D video data, is to be displayed at the top-left window in the output video picture 730. The video data associated with a second source, S2, which comprises 3D video data, is to be displayed at the top-right window in the output video picture 730. The video data associated with a third source, S3, which also comprises 3D video data, is to be displayed at the bottom-left window in the output video picture 730. Moreover, the video data associated with a fourth source, S4, which also comprises 2D video data, is to be displayed at the bottom-right window in the output video picture 730.
The video data from the sources S1, S2, S3, and S4 is provided to the MPEG decoder module 324 through bit streams BS1, BS2, BS3, and BS4, respectively. The MPEG decoder module 324 may decode the video data and provide the decoded video data to the buffers 610 in the memory 410 for storage. As shown in FIG. 7, a first of the buffers 610, which is labeled B1, may store 2D video data, 2D1, from the first source, S1. A second of the buffers 610, which is labeled B2, may store 3D video data from the second source, S2. The 3D video data from the second source, S2, may comprise L2 video data and R2 video data, which correspond to the left-eye and the right-eye information, respectively. The third of the buffers 610, which is labeled B3, may store 3D video data from the third source, S3. The 3D video data from the third source, S3, may comprise L3 video data and R3 video data, which correspond to the left-eye information and right-eye information, respectively. Moreover, a fourth of the buffers 610, which is labeled B4, may store 2D video data, 2D4 , from the fourth source, S4.
As described above with respect to FIG. 4, the feed and capture of the video data in the buffers 610 may be performed by, for example, the MFD module 302 and one of the CAP modules 320. In this example, the capture may be turned on four times, one time for each of the buffers 610. The capture of the video data in all four buffers 610 to the memory 410 may be performed in a single picture capture time.
The windowing, that is, the arrangement of the video data in the memory 410 to construct or layout the windows in the output video picture 730, may be carried out by capturing the video data from a particular source in an area of the memory 410 that corresponds to the window in the output video picture 730 in which the video data from that particular source is to be displayed. For example, FIG. 7 shows a canvas 720 that corresponds to the area in the memory 410 in which the contents from the buffers 610 are to be stored. The canvas 720 has a 2D format and comprises a top-left memory area, a top-right memory area, a bottom-left memory area, and a bottom-right memory area, each of which corresponds to one of the windows in the output video picture 730.
The video data from each of the buffers 610 may be stored in a corresponding memory area in the canvas 720. For example, the 2D video data, 2D1, associated with the first source, S1, may be stored in the top-left memory area of the canvas 720. The L2 video data and the R2 video data associated with the second source, S2, are not in 2D format. In such instances, the L2 video data may be used and stored in the top-right memory area of the canvas 720 while the R2 video data may be discarded. Similarly, the L3 video data and the R3 video data associated with the third source, S3, are not in 2D format. As before, the L3 video data may be used and stored in the bottom-left memory area of the canvas 720 while the R3 video data may be discarded. Moreover, the 2D video data, 2D4, associated with the fourth source, S4, may be stored in the bottom-right memory area of the canvas 720.
Once the video data has been captured to the memory 410 as described above, the VFD module 304 may be utilized to read and feed the entire contents of the canvas 720 to a single buffer for further processing and to subsequently generate the output video picture 730. In this regard, a compositor module, such as the CMP module 222 a or the CMP module 222 b, for example, may be utilized to perform a masking operation in connection with the generation of the output video picture 730 in mosaic mode.
FIGS. 8A-8C are diagrams that illustrate the generation of an output video picture having a 3D output format in mosaic mode, in accordance with embodiments of the invention. Referring to FIG. 8A, there is shown the sources 710, the MPEG decoder module 324, and the buffers 610 as described above with respect to FIG. 7. Also shown are a left picture canvas 810 and a right picture canvas 820 in the memory 410. The left picture canvas 810 may correspond to an area in the memory 410 in which the left-eye information from the buffers 610 may to be stored. The right picture canvas 820 may correspond to an area in the memory 410 in which the right-eye information from the buffers 610 may be stored. Both the left picture canvas 810 and the right picture canvas 820 comprise a top-left memory area, a top-right memory area, a bottom-left memory area, and a bottom-right memory area, each of which corresponds to one of the windows in an output video picture 830. The output video picture 830 may have a 3D L/R output format.
The video data from the buffers 610 may be stored in the left picture canvas 810 and the right picture canvas 810 as appropriate. For example, the 2D video data, 2D1, associated with the first source, S1, which is not in a 3D format, may be stored in the top-left memory area of both the left picture canvas 810 and the right picture canvas 820. The L2 video data and the R2 video data associated with the second source, S2, may be stored in the top-right memory area of the left picture canvas 810 and in the top-right memory area of the right picture canvas 820, respectively. Similarly, the L3 video data and the R3 video data associated with the third source, S3, may be stored in the bottom-left memory area of the left picture canvas 810 and in the bottom-left memory area of the right picture canvas 820, respectively. Moreover, the 2D video data, 2D4, associated with the fourth source, S4, which is not in a 3D format, may be stored in the bottom-right memory area of both the left picture canvas 810 and the right picture canvas 820.
Once the video data has been captured to the memory 410 as described above, the VFD module 304 may be utilized to read and feed the entire contents of the left picture canvas 810 and of the right picture canvas 820 to a single buffer for further processing and to subsequently generate the output video picture 830. The output video picture 830 comprises a left picture 832L and a right picture 832R in a side-by-side configuration, each of which comprises four windows corresponding to the four sources of the video data. In this regard, a compositor module, such as the CMP module 222 a or the CMP module 222 b, for example, may be utilized to perform a masking operation in connection with the generation of the output video picture 830 in mosaic mode.
In one embodiment of the invention, the feeding and capturing of video data into the left picture canvas 810 may occur before feeding and capturing of video data into the right picture canvas 820. In another embodiment of the invention, the video data in the buffers 610 may be fed and captured sequentially to the memory 410.
For example, the video data from the first of the buffers 610, B1, may be fed and captured to the memory 410 first, the video data from the second of the buffers 610, B2, may be fed and captured to the memory 410 second, the video data from the third of the buffers 610, B3, may be fed and captured to the memory 410 next, and the video data from the fourth of the buffers 610, B4, may be fed and captured to the memory 410 last. In this regard, the left-eye information stored in one of the buffers 610 may be fed and captured to the memory 410 before the right-eye information stored in that same buffer may be fed and captured to the memory 410.
Referring to FIG. 8B, there is shown the sources 710, the MPEG decoder module 324, the buffers 610, the left picture canvas 810, and the right picture canvas 820 as described above with respect to FIG. 8A. In FIG. 8B, once the video data has been captured to the memory 410, the VFD module 304 may be utilized to read and feed the entire contents of the left picture canvas 810 and a right picture canvas 820 to a single buffer for further processing and to subsequently generate an output video picture 840. The output video picture 840 may have a 3D O/U output format. In this regard, the output video picture 840 may comprise a left picture 842L and a right picture 842R in a top-and-bottom configuration, each of which comprises four windows corresponding to the four sources of the video data. Moreover, a compositor module, such as the CMP module 222 a or the CMP module 222 b, for example, may be utilized to perform a masking operation in connection with the generation of the output video picture 840 in mosaic mode.
Referring to FIG. 8C, there is shown the sources 710, the MPEG decoder module 324, the buffers 610, the left picture canvas 810, and the right picture canvas 820 as described above with respect to FIG. 8A. In FIG. 8C, once the video data has been captured to the memory 410, the VFD module 304 may be utilized to read and feed the entire contents of the left picture canvas 810 and a right picture canvas 820 to a single buffer for further processing and to subsequently generate a first output video picture 850 and a second output video picture 855. The first output video picture 850 and the second output video picture 855 may correspond to a 3D sequential output format. In this regard, the first output video picture 850 may correspond to a left picture in the 3D sequential output format and may be generated based on the contents of the left picture canvas 810. The second output video picture 855 may correspond to a right picture in the 3D sequential output format and may be generated based on the contents of the right picture canvas 820. In some embodiments of the invention, the second output video picture 855 may be generated before the first output video picture 850. Moreover, a compositor module, such as the CMP module 222 a or the CMP module 222 b, for example, may be utilized to perform a masking operation in connection with the generation of the first output video picture 850 and the second output video picture 855 in mosaic mode.
The various embodiments of the invention described above with respect to FIGS. 7, 8A, 8B, and 8C are provided by way of illustration and not of limitation. One or more embodiments of the invention may be implemented in which the number of sources and/or the characteristics of the windows in the output video picture may be different from those described above. The various embodiments of the invention described herein may also be utilized when the video data from all of the sources may be in a 2D format or when the video data from all of the sources may be in a 3D format, for example.
FIG. 9 is a flow chart that illustrates steps for generating output video pictures with multiple windows utilizing mosaic mode, in accordance with an embodiment of the invention. Referring to FIG. 9, there is shown a flow chart 900 in which, at step 910, the MPEG decoder module 324 may receive video data from multiple sources. At step 920, the video data, after being decoded by the MPEG decoder module 324, may be stored in separate buffers based on the source of the video data.
At step 930, the video data stored in the various buffers may be fed and captured utilizing the MFD module 302 and one of the CAP modules 320. Multiple captures may be turned on based on the number of buffers from which video data is being captured. Moreover, the capture of the video data in all of the buffers may occur during a single picture capture time. The video data may be captured to memory according to the output format of the output video picture. For example, when the output video picture is to be in a 2D output format, the video data associated with a source, whether it is 2D video data or 3D video data, may be captured into memory utilizing a 2D canvas such as the 2D canvas 720. When the output video picture is to be in a 3D output format, the video data associated with a source, whether it is 2D video data or 3D video data, may be captured into memory utilizing a 3D canvas such as the left picture canvas 810 and the right picture canvas 820. In the case of having to capture video data into memory in a 3D format, the left-eye information and the right-eye information may be stored in different portions of the memory. Moreover, the windowing associated with the output video picture may be achieved through the capturing of the video data into portions of memory that correspond to a particular window in the output video picture.
At step 940, the VFD module 304 may be utilized to read and feed the captured video data stored in memory to a single buffer for further processing. The video feed may be performed in a single picture feed time. At step 950, the output video picture may be generated in the appropriate output format from the video data in the single buffer.
Various embodiments of the invention relate to a processor, such as the SoC 100 described above with respect to FIG. 1, for example, which comprises a video feeder that is operable to receive video data from a plurality of sources. The video feeder may be an MPEG video feeder such as the MFD module 302 described above with respect to FIG. 3, for example. The video data from one or more of the plurality of sources may comprise 3D video data. The SoC 100 may be operable to store the received video data from each of the plurality of sources into a corresponding area in a memory, wherein the storing of the received video data occurs during a capture time for a single picture. The SoC 10 may be operable to store the received video data from each source of the plurality of sources at different instants during the capture time for the single picture. The memory may be, for example, the memory module 130 described above with respect to FIG. 1 or the memory module 400 described above with respect to FIG. 4. Moreover, the area in the memory in which the received video data from a source of the plurality of sources is stored may correspond to a window of a plurality of windows in an output video picture, such as the output video pictures 500, 730, 830, and 840 described above.
The SoC 100 may be operable to store the received video data from each source of the plurality of sources in a 2D format or in a 3D format. The SoC 100 may be operable to store left-eye information associated with a current source of the plurality of sources before storage of right-eye information associated with the current source. The SoC 100 may subsequently store left-eye information associated with a next source of the plurality of sources before storage of right-eye information associated with the next source. Moreover, the SoC 100 may be operable to store left-eye information associated with two or more sources of the plurality of sources before storage of right-eye information associated with the two or more sources.
The SoC 100 may be operable to read the stored video data from the memory to a single buffer during a feed time for a single picture. The single buffer may be comprised within a device, component, module, block, circuit, or the like, that follows or is subsequent in operation to one of the VFD modules 304, such as the CMP modules 322 a and 322 b, and/or a video encoder, for example. When the stored video data is stored in the memory in a 3D format, the processor may be operable to read the stored video data from the memory to the single buffer in an L/R format or an O/U format.
In another embodiment of the invention, a non-transitory machine and/or computer readable storage and/or medium may be provided, having stored thereon a machine code and/or a computer program having at least one code section executable by a machine and/or a computer, thereby causing the machine and/or computer to perform the steps as described herein for utilizing mosaic mode to create 3D video.
Accordingly, the present invention may be realized in hardware, software, or a combination of hardware and software. The present invention may be realized in a centralized fashion in at least one computer system or in a distributed fashion where different elements may be spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted for carrying out the methods described herein is suited. A typical combination of hardware and software may be a general-purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein.
The present invention may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which when loaded in a computer system is able to carry out these methods. Computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form.
While the present invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the present invention without departing from its scope. Therefore, it is intended that the present invention not be limited to the particular embodiment disclosed, but that the present invention will include all embodiments falling within the scope of the appended claims.

Claims

1. A method, comprising:

receiving from a video feeder, video data from a plurality of sources, wherein the video data from one or more of the plurality of sources comprises three-dimensional (3D) video data; and

storing the received video data from each of the plurality of sources into a corresponding different area in a memory, wherein each of the corresponding different areas in the memory corresponds to a different window of a plurality of windows in an output picture video.

2. The method of claim 1, wherein the video feeder is an MPEG video feeder.

3. The method of claim 1, comprising storing the received video data during a capture time for a single picture.

4. The method of claim 3, comprising storing the received video data from each source of the plurality of sources at different instants during the capture time for the single picture.

5. The method of claim 1, comprising storing the received video data from each source of the plurality of sources in a 2D format.

6. The method of claim 1, comprising storing the received video data from each source of the plurality of sources in a 3D format.

7. The method of claim 6, comprising:

storing left-eye information associated with a current source of the plurality of sources before storing right-eye information associated with the current source; and

subsequently storing left-eye information associated with a next source of the plurality of sources before storing right-eye information associated with the next source.

8. The method of claim 6, comprising storing left-eye information associated with two or more sources of the plurality of sources before storing right-eye information associated with the two or more sources.

9. The method of claim 1, comprising reading the stored video data from the different areas in the memory to a single buffer during a feed time for a single picture.

10. The method of claim 9, comprising generating the output video picture in a left-and-right (L/R) format or an over-under (O/U) format when the video data is stored in the memory in a 3D format.

11. A system, comprising:

a processor comprising a video feeder, the processor being operable to:

receive video data from a plurality of sources through the video feeder, wherein the video data from one or more of the plurality of sources comprises three-dimensional (3D) video data; and

store the received video data from each of the plurality of sources into a corresponding different area in a memory, wherein each of the corresponding different areas in the memory corresponds to a different window of a plurality of windows in an output picture video.

12. The system of claim 11, wherein the video feeder is an MPEG video feeder.

13. The system of claim 11, wherein the processor is operable to store the received video data during a capture time for a single picture.

14. The system of claim 13, wherein the processor is operable to store the received video data from each source of the plurality of sources at different instants during the capture time for the single picture.

15. The system of claim 11, wherein the processor is operable to store the received video data from each source of the plurality of sources in a 2D format.

16. The system of claim 11, wherein the processor is operable to store the received video data from each source of the plurality of sources in a 3D format.

17. The system of claim 16, wherein the processor is operable to:

store left-eye information associated with a current source of the plurality of sources before storage of right-eye information associated with the current source; and

subsequently store left-eye information associated with a next source of the plurality of sources before storage of right-eye information associated with the next source.

18. The system of claim 16, wherein the processor is operable to store left-eye information associated with two or more sources of the plurality of sources before storage of right-eye information associated with the two or more sources.

19. The system of claim 11, wherein the processor is operable to read the stored video data from the different areas in the memory to a single buffer during a feed time for a single picture.

20. The system of claim 19, wherein the processor is operable to generate the output video picture in an L/R format or an O/U format when the video data is stored in the memory in a 3D format.