JP6085626B2 - Transfer of 3D image data - Google Patents

Description

  The present invention relates to a method for transmitting a 3D display signal for transferring 3D [3D] image data to a 3D display device, the 3D display signal having a sequence of frames constituting 3D image data according to a 3D video transfer format. The sequence of frames has units, each unit corresponding to a frame with video information, which is intended to be synthesized and displayed as a three-dimensional image.

  The invention further relates to a 3D source device, a 3D display signal and a 3D display device as described above.

  The present invention relates to the field of transferring 3D image data, eg 3D video, via a high-speed digital interface, eg HDMI, for display on a 3D display device.

  Devices for supplying 2D video data are known, for example video players or set top boxes such as DVD players that provide digital video signals. The source device is coupled to a display device such as a TV set or monitor. Image data is transferred from the source device via an appropriate interface (preferably a high speed digital interface such as HDMI). Currently, 3D expansion devices that supply three-dimensional (3D) image data have been proposed. Similarly, an apparatus for displaying 3D image data has been proposed. In order to transfer 3D video signals from a source device to a display device, a new high data rate digital interface standard that is compatible, for example, based on the existing HDMI standard, has been developed. Usually, transferring a 2D digital image signal to a display device requires transmitting video pixel data for each frame, and the frames are to be displayed sequentially. Such a frame represents a video frame of a progressive video signal (full frame) or (based on the well-known line interlace, one frame provides odd lines and the next frame is displayed sequentially) May represent a video frame of an interlaced video signal (providing even lines to be played).

  Document US 4,979,033 describes an example of a conventional video signal having an interlace format. Conventional signals include horizontal and vertical sync signals for displaying lines and frames of odd and even frames on a conventional television. Stereoscopic video systems and methods have been proposed that allow synchronization of stereoscopic video with a display using shutter glasses. Odd and even frames are used to transfer the left and right images of the stereoscopic video signal, respectively. The proposed 3D display device has a conventional envelope detector to detect conventional odd / even frames, but instead generates display signals for the left and right liquid crystal display units. In particular, in conventional interlaced analog video signals, equalization pulses that occur during different vertical blanking intervals in odd and even frames are counted to identify the left or right field, respectively. The system uses this information to synchronize the shutter glasses and the shutter glasses open and close alternately in synchrony with the stereoscopic video.

  There are many different ways in which stereoscopic images can be formatted, referred to as 3D image formats. Some formats are based on using 2D channels to further convey stereoscopic information. For example, the left and right views can be interlaced or placed side to side or up and down. These methods sacrifice resolution in order to convey stereoscopic information. The other option is to sacrifice color and this approach is called anagraph stereo.

  A new format has been developed to transmit 3D information to the display. MVD, as standardized in MPEG, allows M larger view cone graphics overlays (eg menus or subtitles in BED players or STBs) to be transmitted to the display. Requests transmission of {Video + Depth} for the view.

  It is an object of the present invention to provide a more flexible and reliable system for transferring 3D video signals to a display device.

  To this end, according to a first aspect of the present invention, in the method described in the opening paragraph, the 3D video format is an audio and auxiliary transmission carried using a series of packets including information frame packets. The data has a data island period as well as a video data period during which active video pixels are transmitted, outputs a 3D display signal, receives a 3D display signal in a 3D display device, and 3D image data on a 3D display Process the 3D display signal to generate the display control signal for rendering, the sequence of frames has units, the unit is the period from the vertical synchronization signal to the next vertical synchronization signal, each unit is Corresponding to a plurality of frames arranged according to a multiplexing scheme, the plurality of frames are combined and displayed as a three-dimensional image Each frame in the unit has a data structure for representing a sequence of digital image pixel data, each frame type represents a partial 3D data structure, and the method includes: In the 3D source device, including the 3D transfer information in the auxiliary information frame packet, the 3D transfer information includes the number of video frames in the unit to be configured as one 3D image in the 3D display signal. Having at least information about a multiplexing scheme, wherein the multiplexing scheme is selected from a group of multiplexing schemes including at least frame alternating multiplexing, wherein frame alternating means that the plurality of frames are sequentially within the video data period. The generation of the display control signal indicating the arrangement is performed depending on the 3D transfer information.

  For this purpose, according to a second aspect of the present invention, there is provided a 3D source device for transferring the 3D image data described in the opening paragraph to a 3D display device, the 3D source device comprising: Generating means for processing source image data to generate a 3D display signal having a sequence of frames constituting 3D image data according to a 3D video transfer format, wherein the 3D video format is an information frame Generating means having a data island period in which audio and auxiliary data are transmitted using a series of packets including packets and a video data period in which active video pixels are transmitted, and for outputting a 3D display signal Output interface means, each frame having a data structure for representing a sequence of digital image pixel data; Each frame type represents a partial 3D data structure, the sequence of frames has units, the unit is the period from the vertical synchronization signal to the next vertical synchronization signal, and each unit is a plurality arranged according to the multiplexing scheme A plurality of frames having output interface means having video information intended to be synthesized and displayed as a three-dimensional image, the output interface means depending on the 3D transfer information in the display device Adapted to transmit the 3D transfer information in an auxiliary information frame packet to generate a display control signal, the 3D transfer information in a unit to be configured as one 3D image in the 3D display signal At least information about the multiplexing scheme including the number of video frames, the multiplexing scheme at least Is selected from the group of multiplexing schemes including over arm alternately multiplexing, the frame alternately, indicating that the plurality of frames are arranged sequentially in the video data within the period.

  To this end, in yet another aspect of the present invention, the 3D display device data described in the opening paragraph, a 3D display for displaying 3D image data, an input interface for receiving a 3D display signal Means, the 3D display signal has frames constituting 3D image data according to the 3D video transfer format, and the 3D video format transmits audio and auxiliary data using a series of packets including information frame packets Input interface means having a data island period and a video data period during which active video pixels are transmitted, and processing means for generating a display control signal for rendering 3D image data on a 3D display, Each frame is a data representation of a sequence of digital image pixel data. Each frame type represents a partial 3D data structure, the sequence of frames has units, the unit is the period from the vertical synchronization signal to the next vertical synchronization signal, each unit is a multiplexing scheme Corresponding to a plurality of frames arranged in accordance with, the plurality of frames have processing means having video information intended to be synthesized and displayed as a three-dimensional image, and the 3D transfer information in the auxiliary information frame packet is , Having at least information about a multiplexing scheme including the number of video frames in a unit to be configured as one three-dimensional image in a 3D display signal, the multiplexing scheme including at least frame alternating multiplexing Selected from a group of schemes, frame alternating means that the plurality of frames are sequenced within the video data period. The processing means is arranged to generate a display control signal depending on the 3D transfer information.

  The present invention is further based on the following recognition. Unlike 2D video information, there are many possibilities for formatting 3D video data, such as stereo, image + depth (which may include occlusion and transparency), and multiple views. It is further envisioned that multiple 3D video data layers may be transmitted through the interface for composition before display. This many options lead to many video format options depending on the format of data available on the source device and the 3D video format accepted by the display. Most of these formats are characterized by a large amount of complex structured information that needs to be transferred for each 3D image to be displayed. According to the present invention, when data is transmitted as a unit and information about the unit is available in the 3D display signal, more data can be included in the unit so that the transmission system can be used in various 3D data formats. It is flexible by handling. Modern high-speed interfaces allow frames to be transmitted at a frequency much higher than the actual frequency of 3D images, typically 24 Hz, used by the cinematography industry. By using a unit of frames, a larger amount of data in a flexible format for each 3D image can be transmitted over the interface.

  In an embodiment, the group of multiplexing schemes includes field alternating multiplexing, line alternating multiplexing, side-by-side frame multiplexing (side-by-side frame multiplexing, wherein the plurality of frames are the video -Indicating that they are arranged side-by-side within the data period), at least one of 2D and depth frame multiplexing, 2D, depth, graphics and graphics depth frame multiplexing Have.

In general, transmission of 3D video data is
-Pixel repetition rate
-The number of frames in a unit of one 3D image frame
-Format (channel multiplexing mode)
Can be characterized by three parameters:

  In the preferred embodiment of the present invention, information across all these parameters is included in the 3D transfer information. For maximum flexibility, according to the present invention, these should be transmitted in three separate fields.

  In the embodiment of the present invention, HDMI is used as an interface, and the 3D transfer information is transmitted in an AVI information frame and / or an HDMI vendor-specific information frame. In the most preferred embodiment that allows for maximum flexibility, the 3D transfer information is transmitted in separate information frames.

  Further preferred embodiments of the method, 3D device and signal according to the invention are given in the appended claims, the disclosure of which is incorporated herein by reference.

  These and other aspects of the invention will be apparent from and will be further elucidated with reference to the embodiments described by way of example in the following description and the accompanying drawings.

  In the drawings, elements corresponding to elements already described have the same reference number.

A system that transfers three-dimensional (3D) image data. An example of 3D image data. The figure which shows the combination of a reproducing | regenerating apparatus and a display apparatus. FIG. 2 schematically shows possible units of a frame transmitted through a video interface for 3D image data corresponding to 2D + Stereo + DOT. FIG. 4 schematically shows further details of possible units of a frame transmitted through a video interface for 3D image data corresponding to 2D + Stereo + DOT. The figure which shows schematically the time output of the flame | frame on the video interface for 3D image data corresponding to 2D + Stereo + DOT. The figure which shows schematically the unit in which the frame arrangement | positioning for a stereo signal is possible. FIG. 5 shows horizontal and vertical blanking and signaling for 3D + DOT format @ 1920 pixels. FIG. 6 shows horizontal and vertical blanking and signaling for 3D + DOT format 720 pixels transmitted as 1920 progressive @ 30 Hz. FIG. 6 shows a table of AVI information frames extended by a frame type synchronization indicator for stereo 3D image data. The figure which shows the table | surface of 3D video format. The figure which shows a frame synchronization signal. FIG. 4 shows values for an auxiliary video layer.

  FIG. 1 shows a system for transferring three-dimensional (3D) image data, such as video, graphics or other visual information. The 3D source device 10 is coupled to the 3D display device 13 for transferring the 3D display signal 56. The 3D source device has an input unit 51 for receiving image information. For example, the input unit device may include an optical disc unit 58 for reading various types of image information from an optical record carrier 54 such as a DVD or Blu-Ray disc. In another aspect, the input unit can include a network interface device 59 for coupling to a network 55 (eg, the Internet or a broadcast network), such a device is commonly referred to as a set top box. Image data can be read from the remote media server 57. The source device can further be a satellite receiver, or a media server that directly provides the display signal, ie any suitable device that outputs a 3D display signal that is directly coupled to the display unit.

  The 3D source device has a processing unit 52 coupled to the input unit 51 for processing the image information to generate a 3D display signal 56 to be transferred via the output interface device 12 to the display device. The processing unit 52 is arranged to generate image data included in the 3D display signal 56 for display on the display device 13. The source device comprises a user control element 15 for controlling the display parameters (eg contrast or color parameters) of the image data. Such user control elements are well known, for controlling various functions (eg playback and recording functions) of the 3D source device and for setting the display parameters via eg a graphical user interface and / or menu. Remote control units with various buttons and / or cursor control functions.

  The source device has a transmission synchronization unit 11 for providing at least one frame type synchronization indicator in the 3D display signal, which indicator is included in the 3D display signal in the output interface unit 12, 12 is further arranged to transfer a 3D display signal having image data and a frame type synchronization indicator from the source device to the display device as a 3D display signal 56. The 3D display signal has a sequence of frames, which are organized as a group of frames, thereby constituting 3D image data according to a 3D video transfer format, in which the frame comprises at least two different frame types . Each frame has a data structure for representing a sequence of digital image pixel data and is typically arranged as a sequence of horizontal lines of pixels with a predetermined resolution. Each frame type represents a partial 3D data structure. For example, the 3D partial data structure in the 3D video transfer format frame type can be left and right images or 2D images and auxiliary depth, and / or like the occlusion or transparency information described below. Can be further 3D data. Note that the frame type is a combination frame type that indicates a combination of sub-frames of the above-described frame type (for example, four sub-frames with low resolution located in one full-resolution frame). You can also. In addition, multiple multi-view images can be encoded into a video stream of simultaneously displayed frames.

  The source device is adapted to include 3D transfer information having at least information on the number of video frames in the unit to be configured as one 3D image in the 3D display signal. This can be achieved by adding functions corresponding to the synchronization unit 11.

  The 3D display device 13 is for displaying 3D image data. This device has an input interface device 14 for receiving a 3D display signal 56 including 3D image data in a frame transferred from the source device 10 and a frame type synchronization indicator. Each frame has a data structure for representing a sequence of digital image pixel data, and each frame type represents a partial 3D data structure. The display device comprises a further user control element 16 for setting the display parameters of the display (eg contrast, color or depth parameters). The transferred image data is processed in the processing unit 18 according to a setting command from the user control element, and generates a display control signal for rendering the 3D image data on the 3D display based on each frame type. The device has a 3D display 17 (for example a dual LCD) that receives display control signals for displaying the processed image data. The display device 13 is a stereoscopic display called a 3D display having a display depth range indicated by an arrow 44. The display of 3D image data is performed depending on the respective frame, each providing a respective partial 3D image data structure.

  The display device further includes a detection unit 19 coupled to the processing unit 18 for reading the frame type synchronization indicator from the 3D display signal and detecting the respective frame type in the received 3D display signal. The processing unit 18 is arranged to generate display control signals based on various types of image data (eg 2D images and depth frames) as defined by the partial 3D data structure of the respective 3D video format. The Each frame is recognized and time synchronized as indicated by the respective frame type synchronization indicator.

  The display device detects 3D transfer information having at least information on the number of video frames in a unit configured as one 3D image in the 3D display signal, and generates a display control signal depending on the 3D transfer information It is adapted to use 3D transfer information. This can be achieved, for example, by adapting the detection unit 19 to detect 3D transfer information and by adapting the processing means (18) to generate a display control signal depending on the 3D transfer information, Can be achieved.

  Frame type synchronization indicator allows to detect which of the frames should be combined to be displayed at the same time, and each partial 3D data can be read and processed As shown, the frame type. The 3D display signal can be transferred through a suitable high-speed digital video interface (eg, the well-known HDMI interface) (see, for example, “High Definition Multimedia Interface Specification Version 1.3a,” Nov 10 2006).

  FIG. 1 further shows a record carrier 54 as a carrier for 3D image data. The record carrier is disc-shaped and has a track and a central hole. A track constituted by a sequence of physically detectable marks is arranged according to a spiral or concentric pattern of turns that constitutes a substantially parallel trajectory on the information layer. The record carrier is optically readable and is called an optical disc (eg CD, DVD or BD (Blu-ray Disc)). Information is represented on the information layer by optically detectable marks (eg, pits and lands) along the track. The track structure further includes position information (eg, header and address) to indicate the position of a unit of information (usually called an information block). The record carrier 54 is information representing digitally encoded image data, such as video encoded according to an MPEG2 or MPEG4 encoding system, for example, in a predetermined recording format such as DVD or BD format. Is carried.

  Note that the player can support playback of various formats, but cannot convert video formats and the display device can play a limited set of video formats There is. This means that there is a common divider for what can be played back. Note that depending on the disc or content, the format may change during playback / operation of the system. Requires real-time synchronization of the format, and real-time switching of the format is provided by a frame type synchronization indicator.

  The following sections provide an overview of 3D displays and depth perception by humans. 3D displays differ from 2D displays in that they can provide a more perceived depth perception. This is achieved because they provide more depth cues than 2D displays that can only show monocular depth cues and motion based cues.

  Monocular (or static) depth cues can be obtained from still images using one eye. Painters often use monocular cues to create a sense of depth in a picture. These cues include relative size, height relative to the horizon, occlusion, sense of distance, texture gradient and lighting / shading. An oculomotor cue is a depth cue derived from the strain in the observer's eye muscles. The eye has muscles to rotate the eye and stretch the eye lens. Tension and relaxation of the eye lens is called adaptation and is done when focusing on the image. The amount of lens muscle tension or relaxation provides a clue to how far away or near the object is. Eye rotation is done so that both eyes focus on the same object, which is called convergence. Finally, motion parallax is the effect that objects near the viewer appear to move faster than distant objects.

  Binocular parallax is a depth cue derived from both eyes looking at slightly different images. Monocular depth cues can and are used for any 2D image display type. Reproducing binocular parallax on a display requires that the display be able to separate the views for the left and right eyes so that each eye sees a slightly different image on the display. A display capable of reproducing binocular parallax is a special display called a 3D or stereoscopic display. A 3D display can display an image along a depth dimension that is actually perceived by the human eye and is referred to herein as a 3D display with a display depth range. Therefore, 3D displays provide different views for the left and right eyes.

  3D displays that can provide two different views have existed for many years. Most of them were based on using glasses that separate the left and right eye views. Now, with the development of display technology, new displays that can provide stereoscopic views without using glasses have entered the market. These displays are called autostereoscopic displays.

  The first approach is based on a liquid crystal display that allows the user to view stereoscopic video without using glasses. These are based on one of two technologies: lenticular screen and barrier display. In a lenticular display, the LCD is covered by a sheet of lenticular lenses. These lenses diffract light from the display so that the left and right eyes receive light from different pixels. This allows two different images to be displayed, one for the left eye view and one for the right eye view.

  An alternative to lenticular screens is a barrier display, which uses a parallax barrier behind the LCD and in front of the backlight to separate the light from the LCD pixels. The barrier is such that the left eye sees a different pixel from the right eye from a predetermined position in front of the screen. The barrier may be between the LCD and the viewer so that the display row pixels are viewed alternately by the left and right eyes. Problems with barrier displays are reduced brightness and resolution, and very narrow viewing angles. This makes the barrier display less attractive as a living room television compared to, for example, a lenticular screen with nine views and multiple viewing zones.

  A further approach is still based on using shutter glasses in combination with a high resolution beamer that can display frames at a high refresh rate (eg, 120 Hz). Since the left and right eye views are alternately displayed in the shutter glasses method, a high refresh rate is required. An observer wearing glasses perceives stereoscopic video at 60 Hz. The shutter glasses method allows for high quality video and a high level of depth.

  Both the autostereoscopic display and the shutter glasses method suffer from an adaptive-congestion mismatch. This limits the amount of depth and the time that can be comfortably observed with these devices. There are other display technologies such as holographic and volumetric displays, which do not suffer from this problem. It should be noted that the present invention can be used for any type of 3D display with a depth range.

  Image data for a 3D display is assumed to be available as electronic (usually digital) data. The present invention relates to such image data and manipulates the image data in the digital domain. The image data may already contain 3D information when transferred to the source, for example by using a dual camera, or a dedicated pre-processing system (re) generates 3D information from the 2D image May be involved in order to. The image data can be a still image such as a slide or can include a moving image such as a moving video. Other image data, usually referred to as graphical data, is available as a stored object or can be generated on the fly when requested by an application. For example, user control information such as menus, navigation items or text and help annotations can be added to other image data.

  There are many different ways in which stereoscopic images can be formatted, referred to as 3D image formats. Some formats are further based on using 2D channels to carry stereoscopic information. For example, the left and right views can be interlaced or arranged side-by-side or up and down. These methods sacrifice resolution in order to carry stereoscopic information. Another option is to sacrifice color and this approach is called anagraph stereo. Analog graph stereo uses spectral multiplexing based on displaying two separate overlaid images with complementary colors. By using eyeglasses with colored filters, each eye sees only an image of the same color as the filter in front of that eye. Thus, for example, the right eye sees only a red image and the left eye sees only a green image.

  Different 3D formats are based on two views using 2D images and auxiliary depth images (so-called depth maps), which convey information about the depth of objects in the 2D images. The format called image + depth differs in that it is a so-called “depth” or a combination of a parallax map and a 2D image. This is a tone image and the tone value of a pixel indicates the amount of parallax (or depth in the case of a depth map) of the corresponding pixel in the associated 2D image. The display device uses parallax, depth or image difference maps to calculate additional views using 2D images as input. This can be done in a variety of ways, and the simplest form is to shift the pixels left or right depending on the disparity value associated with those pixels. The paper "Depth image based rendering, compression and transmission for a new approach on 3D TV" by Christoph Fen gives an excellent overview of this technology (http://iphome.hhi.de/fehn/Publications/fehn_EI2004.pdf reference).

  FIG. 2 shows an example of 3D image data. The left part of the image data is usually a color 2D image 21, and the right part of the image data is a depth map 22. The 2D image information can be represented in any suitable image format. The depth map information can be an auxiliary data stream with a depth value for each pixel, possibly with a reduced resolution compared to a 2D image. In the depth map, the tone value indicates the depth of the associated pixel in the 2D image. White indicates close to the observer and black indicates a large depth away from the observer. The 3D display can calculate the additional views needed for stereo by using the depth values from the depth map and by calculating the required pixel transformation. Shielding can be solved using estimation or hole filling techniques. Additional frames can be included in the data stream, such as occlusion maps, disparity maps and / or transparency maps for transparent objects moving in front of the background, e.g. images and depth maps. It can be further added to the format.

  Furthermore, adding stereo to the video affects the format of the video as it is transmitted from the player device (eg, a Blu-ray Disc player) to a stereoscopic display. In the case of 2D, only the 2D video stream (decoded image data) is transmitted. For stereo video, this increases because a second stream containing a second view or depth map (for stereo) is needed. This may double the bit rate required on the electrical interface. A different approach is to format the stream so that the second view or depth map is interlaced or placed side-by-side with 2D video at the expense of resolution.

  FIG. 2 shows an example of 2D data and a depth map. The depth display parameter sent to the display allows the display to correctly interpret the depth information. An example of including further information in the video is described in ISO standard 23002-3 “Representation of auxiliary video and supplemental information” (see, for example, ISO / IEC JTC1 / SC29 / WG11 N8259 July 2007). Depending on the type of auxiliary stream, the further image data consists of 4 or 2 parameters. The frame type sync indicator may have a 3D video format indicator that indicates the respective 3D video transfer format in subsequent sections of the 3D display signal. This makes it possible to indicate or change the 3D video transfer format, or to reset the transfer sequence or to set or reset further synchronization parameters.

  In an embodiment, the frame type synchronization indicator includes a frame sequence indicator that indicates the frequency of at least one frame type. It should be noted that some frame types allow lower transmission frequencies (eg occlusion data) without significant degradation of perceived 3D images. Further, the order of each frame type can be shown as a sequence of each frame type to be repeated.

  In an embodiment, the frame type synchronization indicator and the 3D transfer information include a frame sequence number. Individual frames can also have a frame sequence number. For example, when all the frames constituting one 3D image are transmitted, the sequence number is regularly incremented, and the subsequent frames belong to the next 3D image. Thus, the numbers are different for each synchronization cycle or can only change for larger sections. Thus, when a jump is performed, a set of frames each having the same sequence number needs to be transferred before the image display can be resumed. The display detects the deviating frame sequence number and combines only the complete set of frames. This prevents the wrong frame combination from being used after jumping to a new position.

  When adding graphics on the video, additional data streams can be used to overlay additional layers in the display unit. Such layer data is contained in different frame types that are marked separately by adding respective frame type synchronization indicators in the 3D display signal as discussed in detail below. Here, the 3D video transfer format has a main video transferred via each frame type and at least one auxiliary video layer, and the frame type synchronization indicator includes a main frame type indicator and At least one of the auxiliary layer frame type indicators. The auxiliary video layer can be, for example, subtitles such as menus or other graphical information or any other on-screen data (OSD).

  Possible formats for the frame unit are described with reference to FIGS. This format is further described in EP Application No. 09150947.1 (Applicant Docket No. PH012841), the priority of which application is claimed and is incorporated herein by reference.

  The received compressed stream has 3D information that allows compositing and rendering on both stereoscopic and autostereoscopic displays, i.e., the compressed stream includes left and right video frames, and It has depth (D), transparency (T) and occlusion (O) information to enable rendering based on 2D + depth information. In the following, depth (D), transparency (T) and occlusion (O) information is abbreviated as DOT.

  The presence of both stereo and DOT as compressed streams allows for optimized composition and rendering by the display, depending on the display type and size, but composition is still controlled by the content creator.

The following components are transmitted through the display interface:
-Decoded video data (not mixed with PG and IG / BD-J)
-Presentation graphics (PG) data
-Interactive graphics (IG) or BD-Java (registered trademark) generated (BD-J) graphics data
-Decrypted video DOT
-Presentation Graphics (PG) DOT
-Interactive graphics (IG) or BD-Java (registered trademark) generation (BD-J) graphics

  4 and 5 schematically show units of frames transmitted over the video interface.

The output stage transmits a unit of six frames configured as follows through an interface (preferably HDMI).
Frame 1: The YUV components of Left (L) video and DOT video are combined into one 24 Hz RGB output frame (component) as shown at the top of FIG. YUV represents the reference luminance (Y) and saturation (UV) components by way of example in the field of video processing.
Frame 2: The right (R) video is transmitted unchanged at 24 Hz, preferably as shown at the bottom of FIG.
Frame 3: PC color (PG-C) is preferably transmitted at 24 Hz, unchanged, as an RGB component.
Frame 4: The transparency of PG-Color is copied to another graphics DOT output plane, as shown at the top of Figure 10, and the depth for the various planes as well as the 960x540 shielding and shielding depth ( OD) component.
Frame 5: BD-J / IG Color (C) is preferably transmitted unchanged at 24 Hz.
Frame 6: The transparency of BD-J / IG Color is copied to another graphics DOT output plane, as shown at the bottom of Figure 10, and with depth and 960x540 shielding and shielding depth (OD) components Can be combined.

  FIG. 6 schematically illustrates the temporal output of a frame on a video interface according to a preferred embodiment of the present invention. Here, the component is transmitted at 24 Hz and the component is time interleaved on the HDMI interface with an interface frequency of 144 Hz on the display.

The advantage of this 3D video format is
-Maximum resolution flexible 3D stereo + DOT format and 3D HDMI output, extended 3D video (variable baseline for display size dependency) and extended 3D for various 3D displays (stereo and autostereoscopic) Enables the possibility of graphics (less graphics restrictions, 3D TV OSD).
-No compromises on quality, authoring flexibility and minimum cost to player hardware. Compositing and rendering takes place in a 3D display.
-The higher video interface speeds required are defined in HDMI for the 4k2k format and can already be implemented with dual link HDMI. Dual link HDMI supports even higher frame rates (eg 30Hz).

  The 3D transport information indicator can have layer signaling parameters for the auxiliary video layer.

The parameter is
-Auxiliary layer type or format
-Position of auxiliary layer display relative to main video display
-Auxiliary layer display size
-Appearance, disappearance and / or duration of the auxiliary layer display
-At least one of auxiliary 3D display settings or 3D display parameters may be indicated.

  More detailed examples are described below.

  FIG. 3 shows a combination of a playback device and a display device. Player 10 reads the capabilities of display 13 and adjusts the video format and timing parameters to transmit the highest resolution video that the display can handle, both spatially and temporally. In practice, a standard called EDID is used. Extended display identification data (EDID) is a data structure provided by a display device to describe its capabilities to an image source (eg, a graphics card). It allows modern personal computers to know what type of monitor is connected. EDID is defined by a standard published by the Video Electronics Standards Association (VESA). See VESA Display Port Standard Version 1, Revision 1a, January 11, 2008, available at http://www.vesa.org/.

The EDID includes manufacturer name, product type, phosphor or filter type, timing supported by the display, display size, brightness data, and pixel mapping data (for digital displays only). The channel for transmitting EDID from the display to the graphics card is usually a so-called I ² C bus. The combination of EDID and I ² C is called Display Data Channel version 2 or DDC2. 2 distinguishes it from VESA's original DDC that used a different serial format. The EDID is often stored in the monitor in a memory device called serial PROM (programmable read-only memory) or EEPROM (electrically erasable PROM) compatible with the I ² C bus.

  The playback device sends an E-EDID request to the display through the DDC2 channel. The display responds by sending E-EDID information. The player determines the best format and begins transmitting over the video channel. In older types of displays, the display continuously sends E-EDID information on the DDC channel. No requests are sent. To further define the video format used on the interface, the Consumer Electronics Association (CEA) has established some further restrictions and extensions to E-EDID to make it more suitable for TV-type displays. I made it. In addition to specific E-EDID requirements, the HDMI standard (see above) supports identification codes and associated timing information for many different video formats. For example, the CEA861-D standard is adopted in the interface standard HDMI. HDMI supports CEA861-D and VESA E-EDID standards to define physical links and handle higher level signaling. The VESA E-EDID standard allows the display to indicate whether and in what format the stereoscopic video transmission is supported. It should be noted that such information regarding the capabilities of the display is passed back to the source device. Known VESA standards do not define any forward 3D information that controls 3D processing on the display.

  In an embodiment, the 3D transfer information in the 3D display signal is transferred asynchronously, eg as separate packets in the data stream, and identifies each frame associated with it. The packet can contain additional data for frames that are precisely synchronized with the video and can be inserted at an appropriate time in the blanking interval between successive video frames. In a practical embodiment, 3D transfer information is inserted into a packet in HDMI Data Island.

  An example of including 3D transfer information in Auxiliary Video Information (AVI) defined in HDMI in an audio video data (AV) stream is as follows. The AVI is transmitted as an Info Frame in the AV stream from the source device to the digital television (DTV) monitor. If the source device supports transmission of Auxiliary Video Information (AVI) and determines that the DTV monitor is capable of receiving that information, it will send the AVI to the DTV monitor once every VSYNC period. Send. This data is applied to the next full frame of video data.

  In the following section a brief description of HMDI signaling is given. In HDMI, a device having an HDMI output is known as a source, and a device having an HDMI input is known as a sink. An information frame (InfoFrame) is a data structure defined in CEA-861-D designed to carry various auxiliary data items for audio or video streams or source devices, from source to sink over HDMI. Communicated. The video field is a period from one VSYNC active edge to the next VSYNC active edge. The video format is well defined so that when it is received at the monitor, the monitor has enough information to correctly display the video to the user. Each format definition includes video format timing, image aspect ratio, and colorimetric space. Video format timing is a waveform associated with the video format. Note that a particular video format timing can be associated with multiple video formats (eg, 720X480p @ 4: 3 and 720X480p @ 16: 9).

  HDMI includes three separate communication channels, TMDS, DDC, and optional CEC. TMDS is used to carry all audio and video data, not just auxiliary data, including AVI and Audio InfoFrame describing active audio and video streams. The DDC channel is used by the HDMI source to determine the capabilities and characteristics of the sink by reading the E-EDID data structure.

  The HDMI source is expected to read the sink's E-EDID and provide only the audio and video formats supported by the sink. Furthermore, the HDMI sink is expected to detect information frames and properly process the received audio and video data.

  The CEC channel is used as an option for higher level user functions (eg, tasks associated with auto-configuration tasks or generally using infrared remote control).

  The HDMI link operates in one of three modes: a video data period, a data island period, and a control period. During the video data period, active pixels of the active video line are transmitted. During the data island period, audio and auxiliary data are transmitted using a series of packets. The control period is used when no video, audio or auxiliary data needs to be transmitted. The control period is required between any two periods that are not control periods.

Table 1 shows the packet types in the HDMI data island.

  The present inventors have confirmed that current information frame packets, AVI information frames, etc. are not suitable for handling the transmission of 3D video data.

In general, transmission of 3D video data can be characterized by three parameters.
− VIC (pixel repetition rate) from Table 8.7 of the HDMI specification (eg 1920x1080p @ 60Hz)
-The number of frames in a unit of one 3D image frame, N = 1 for monoscope
N = 2 for stereo and video + depth
N = 3 for video + depth + graphics
N = 4 for MVD @ M = 2 etc.
N = 6 for units defined with reference to FIGS.
-Format: Channel multiplexing-Frame alternation-Field alternation-Line alternation-Side-by-side-Checkerboard, etc.

  FIG. 8 shows horizontal and vertical blanking and signaling for 3D + DOT format @ 1920 pixels. The figure shows a multiplexing scheme for frame alternating multiplexing. In this example, the five frames indicated by Vactive / 5 constitute a 3D image in 3D + DOT format, and these frames are sequentially arranged in units between the vertical synchronization pulses VSYNC of the 3D signal indicated by Vfreq. . The vertical sync pulse indicates a video data period Vactive that starts after vertical blanking Vblank, during which frames are placed sequentially. Similarly, the horizontal blanking pulse HSYNC indicates a line period Hactive starting after the horizontal blanking Hblank. Therefore, the frame alternate multiplexing scheme indicates that the plurality of frames are sequentially arranged within the video data period.

  FIG. 9 shows horizontal and vertical blanking and signaling for 3D + DOT format 720 pixels transmitted as 1920 progressive @ 30 Hz. The figure shows a multiplexing scheme for side-by-side frame multiplexing. In this example, the five frames indicated by Hactive / 5 constitute a 3D image in 3D + DOT format, and these frames are side-by-side in the unit between the vertical synchronization pulses VSYNC of the 3D signal indicated by Vfreq.・ It is arranged on the side. The vertical sync pulse indicates a video data period Vactive that begins after vertical blanking Vblank, during which the frame is placed side by side. Similarly, the horizontal blanking pulse HSYNC indicates a line period Hactive starting after the horizontal blanking Hblank. Therefore, a side-by-side frame multiplexing scheme indicates that the plurality of frames are arranged sequentially within the video data period.

  For maximum flexibility, according to the present invention, the above parameters of the multiplexing scheme should be transmitted in three separate fields.

  In the embodiment of the present invention, these are transmitted through an AVI information frame and / or an HDMI vendor-specific information frame.

  In the following, a detailed embodiment in the case of the HDMI interface is shown.

  Table 2 describes the associated bytes of an information frame packet according to a preferred embodiment of the present invention.

  Here, HDMI_VIC0... HDMI_VIC7 describes a video format identification code. When transmitting any video format defined in this section, the HDMI source sets the HDMI_VIC field to the video code for that format.

HDMI_3D_FMT0 ... HDMI_3D_FMT describes a 3D format code. When transmitting any video format defined in this section, the HDMI source sets the HDMI_3D_Format field to the video code for that format.

  According to the invention, auxiliary video timing format values (identified by HDMI_VIC number) are defined for 3D (stereoscopic) transmission.

  The following video formats are used for 3D transmission. The left and right images for each of the viewer's eyes can be uniquely distinguished using the video format definition in this section, and no other auxiliary information packets are required. Table 3 shows the associated EDID and HDMI_VIC values described in the information frame.

Table 3 shows HDMI_VIC for 3D transmission.

  According to the present invention, the HDMI-only multiplexing format of the 3D channel is identified by the HDMI_3D_FMT number, an example of which is defined in Table 4. For 3D (stereoscopic) transmission, the following 3D formats are used for 3D transmission. The format of information multiplexing in 3D transmission channels can be uniquely distinguished by using the 3D format definition in this section, and no other auxiliary information packets are required. Table 4 shows the value of HDMI_3D_Format described in EDID and related information frames.

Table 4 is HDMI_3D_FMT for 3D transmission.

Table 5 is an HDMI_VIC for extended resolution transmission.

According to the present invention, the player device can send 3D metadata from the source to the sink: (among others)
-Content format
-Real-time 2D / 3D signaling
-Sync
-Recommended viewing angle
-Subtitle information

  According to the present invention, auxiliary 3D content metadata can be included in the transmitted 3D video data, and the metadata is preferably aligned in the SMPTE 3D master format.

Several options are described. The metadata can be included in one of the following:
-3D information frame type (CEA)
-AVI information frame
-Vendor-specific information frame (VSIF), CEC

  In the following section, a specific embodiment for transmitting stereo information or the case where a unit has two frames is introduced.

  It is proposed to use the black bar information in the AVI-Info frame to accommodate the frame type synchronization indicator, eg for left and right signaling and further information for proper rendering of 3D video on the display. The AVI-info frame is a data block transmitted at least every two fields. For this reason, it is the only information frame that can carry frame-based signaling, which is a requirement when used for synchronization of stereoscopic video signals. The advantage of this solution compared to other solutions that depend on relative signaling or rely on vendor-specific information frames is that it is compatible with current chipsets for HDMI and that Is to provide enough space (8 bytes) for frame-by-frame synchronization and signaling.

  In another embodiment, it is proposed to use the preamble bits defined in HDMI to indicate that the following video data is a left or right video frame. HDMI chapter 5.2.1.1 specifies that there is a preamble immediately before each Video Data Period or Data Island Period. This is a sequence of 8 identical control characters that indicate whether the upcoming data period is a Video Data Period or a Data Island. The values of CTL0, CTL1, CTL2, and CTL3 indicate the type of data period that follows. The remaining control signals (HSYNC and VSYNC) may change during this sequence. The preamble is currently 4 bits, CTL0, CTL1, CLT3 and CTL4. Currently, only 1000 and 1010 are used as values. For example, here the value 1100 or 1001 can be defined to indicate that the video data includes a left or right video frame, or a frame that includes image and / or depth information. Furthermore, the preamble bits can only indicate the 3D frame type or the first 3D frame of the sequence, while further distinction of the frame type follows the frame type synchronization sequence defined by the further data frame be able to. Further, HSYNC and VSYNC signaling can be adapted to convey at least part of the frame type synchronization, eg, whether the frame is a left video frame or a right video frame. HSYNC is arranged to precede the left frame video data, and VSYNC is arranged to precede the right frame of video information. The same principle can be applied to other frame types such as 2D images and depth information.

  FIG. 10 shows a table of AVI-info frames extended by the frame type synchronization indicator. AVI-info frames are defined by CEA and adopted by HDMI and other video transmission standards to provide frame signaling for color and saturation sampling, overscan and underscan, and aspect ratio. Additional information has been added to implement the frame type sync indicator as follows.

  The last bit F17 of data byte 1 and the last bit F47 of data byte 4 are reserved in a standard AVI-info frame. In the frame type sync indicator embodiment, these are used to indicate the presence of stereoscopic signaling in the black bar information. Black bar information is typically included in data bytes 6-13. Bytes 14-27 are usually reserved in HDMI and therefore may not be transmitted correctly on current hardware. Therefore, these fields are used to provide less important OSD location information. The syntax of the table is as follows: When F17 is set (= 1), up to 13 data bytes contain 3D parameter information. The default case is when F17 is not set (= 0), which means that 3D parameter information does not exist.

  Data bytes 12-19 indicate the position of the OSD / subtitle overlay. The further layer may be smaller than the main video layer and is arranged based on the byte 12-19 position data. This allows the 3D display to perform a specific rendering on the area of the screen indicated by the frame type sync indicator. The frame type sync indicator is used to indicate when subtitle / OSD information needs to appear and / or disappear, for example, in data bytes 20-27, called rendering parameters in FIG. Synchronization timing information can be further included.

  FIG. 11 shows a table of 3D video formats. The values in the left column each indicate a specific video format with a different frame type. The selected value is included in a frame synchronization indicator (eg, data byte 7 in the table of FIG. 10). Data byte 7 describes the stereoscopic video format that the source (player) is transmitting. The table in FIG. 11 lists some possible values. A value of 0 indicates that the associated frame is 2D, which is useful when transmitting 2D video segments between 3D titles. The display device (3D-TV) can adapt the internal image processing to this change of the 3D video format, for example in the case of the frame sequential format, switching off time upconversion.

  FIG. 12 shows a frame synchronization signal. The synchronization signal can be included in a frame synchronization indicator (eg, data byte 8 of FIG. 10). Data byte 8 carries the stereo synchronization signal and FIG. 12 shows the format of the synchronization signal. The synchronization signal indicates the content of the video frame along with the video format.

  The values of data bytes 9 and 10 in FIG. 10 depend on the video format. For example, for (auto) stereoscopic video, they indicate the maximum and minimum parallax of the video content. In another aspect, they can indicate an offset and scaling factor for “depth” information. If higher bit precision is required (ie 10 bit depth), additional registers can be used to store the lower bits.

  FIG. 13 shows values for the auxiliary video layer. The video format can be extended by allowing a frame for auxiliary layers such as subtitles or menus (On Screen Data: OSD) to be included separately in the 3D video signal. In FIG. 4, a data byte 11 can indicate the presence of a caption or OSD overlay. FIG. 13 shows a plurality of video format parameter values for indicating the auxiliary layer. The remaining bytes 20-27 of FIG. 10 can be used to provide specific parameters indicating information for scaled depth and occlusion information related to 3D display.

  It should be noted that the present invention can be implemented as hardware and / or software using programmable components. The method for implementing the present invention has processing steps corresponding to transferring 3D image data described with reference to FIG. Although the present invention has been described primarily by an embodiment using an optical record carrier or the Internet, the present invention further includes a 3D personal computer [PC] display interface or a 3D media center PC coupled to a wireless 3D display device. It is suitable for any image interface environment.

  The present invention can be summarized as follows. A system for transferring three-dimensional (3D) image data is described. The 3D source device provides a 3D display signal for display via a high speed digital interface such as HDMI. The 3D display signal has a sequence of frames constituting 3D image data according to the 3D video transfer format. A sequence of frames has units, each unit corresponding to a frame having video information that is intended to be synthesized and displayed as a three-dimensional image. Each frame has a data structure for representing a sequence of digital image pixel data and represents a partial 3D data structure. The 3D source device includes 3D transfer information having at least information regarding the number of video frames in the unit configured into one 3D image in the 3D display signal. The display detects 3D transfer information and generates a display control signal based on the 3D transfer information. The 3D transfer information preferably further comprises information about a multiplexing scheme for multiplexing the frame into the 3D display signal, most preferably information about the pixel size and the frequency rate of the frame.

  In this document, terms such as “having” and “including” do not exclude the presence of elements or steps other than those listed, and an element expressed in the singular means that there are a plurality of such elements. Without being excluded, any reference signs do not limit the scope of the claims, the invention can be implemented by both hardware and software, and some "means" or "units" may be hardware or software It can be represented by the same item, and the processor can perform the functions of one or more units in cooperation with the hardware elements. Further, the present invention is not limited to the embodiments, but is present in any novel feature or combination of features described above.

Claims (15)

A method of transferring three-dimensional [3D] image data in a 3D source device ,
The 3D image data in accordance with 3D video transmission format having video data period audio and data island period and the active video pixel is a period in which the auxiliary data is transmitted is a period that is transmitted using a packet of a series Processing source image data to generate a 3D display signal having a sequence of frames comprising
Outputting the 3D display signal;
In the 3D display device,
Receiving the 3D display signal;
Processing a 3D display signal to generate a display control signal for rendering the 3D image data on a 3D display;
The sequence of frames has units, wherein the unit is a period from vertical blanking to the next vertical blanking , each unit corresponding to a plurality of frames arranged according to a multiplexing scheme, The frame has video information that is intended to be synthesized and displayed as a 3D image,
Each frame in the unit has a data structure for representing a sequence of digital image pixel data, each frame type represents a partial 3D data structure,
In the 3D source device, the method includes:
During supplementary information frame packets to be transmitted to the data island period, at least information about the multiplexing scheme containing the number of video frames in the to be synthesized unit as one 3D image in the 3D display signal Yes to 3D transfer information, and the information indicating the period of the vertical blanking in accordance with the video format for the transmission of the 3D image including,
The multiplexing scheme is selected from a group of multiplexing schemes including at least frame alternating multiplexing, wherein frame alternating indicates that the plurality of frames are arranged sequentially within the video data period;
The generation of the display control signal is performed depending on the 3D transfer information. In the information regarding the multiplexing scheme, the group of multiplexing schemes is
Field alternating multiplexing,
Line alternate multiplexing,
Side-by-side frame multiplexing, indicating that the plurality of frames are arranged side-by-side within the video data period;
2D and depth frame multiplexing,
2D, depth, graphics and graphics depth frame multiplexing,
The method of claim 1, further comprising at least one of: The 3D transfer information includes information about the transmission frequency with the pixel size and frame The method of claim 1.   The method of claim 3, wherein the video transfer format is HDMI®.   The method of claim 4, wherein the 3D transfer information is included in an AVI information frame.   5. The method of claim 4, wherein the 3D transfer information is included in a Vendor Specific information frame. A 3D source device for transferring 3D image data to a three-dimensional [3D] display device ,
The 3D image data in accordance with 3D video transmission format having video data period audio and data island period and the active video pixel is a period in which the auxiliary data is transmitted is a period that is transmitted using a packet of a series Generating means for processing source image data to generate a 3D display signal having a sequence of frames that constitutes, and an output interface means for outputting the 3D display signal,
Each frame has a data structure for representing a sequence of digital image pixel data, each frame type represents a partial 3D data structure,
The sequence of frames has units, wherein the unit is a period from vertical blanking to the next vertical blanking , each unit corresponding to a plurality of frames arranged according to a multiplexing scheme, The frame has video information that is intended to be synthesized and displayed as a 3D image,
In the auxiliary information frame packet transmitted in the data island period, the output interface means generates one display control signal depending on the 3D transfer information in the display device. the 3D transfer information to at least have the information about the multiplexing scheme containing the number of video frames in the unit to be synthesized as an image, and the vertical blanking in accordance with the video format for the transmission of the 3D image Transmitting information indicating a period of ranking, wherein the multiplexing scheme is selected from a group of multiplexing schemes including at least frame alternating multiplexing, wherein frame alternating means that the plurality of frames are sequentially within the video data period. 3D source device showing that it is placed in The output interface means comprises:
Field alternating multiplexing,
Line alternate multiplexing,
Side-by-side frame multiplexing, indicating that the plurality of frames are arranged side-by-side within the video data period;
2D and depth frame multiplexing,
2D, depth, graphics and graphics depth frame multiplexing,
The 3D source device of claim 7, further providing information on the multiplexing scheme by the group of multiplexing schemes further comprising at least one of: A 3D display device,
3D display for displaying 3D image data ,
The 3D image data in accordance with 3D video transmission format having video data period audio and data island period and the active video pixel is a period in which the auxiliary data is transmitted is a period that is transmitted using a packet of a series Input interface means for receiving a 3D display signal having a frame comprising: a processing means for generating a display control signal for rendering the 3D image data on the 3D display;
Each frame has a data structure for representing a sequence of digital image pixel data, each frame type represents a partial 3D data structure,
The sequence of frames has units, wherein the unit is a period from vertical blanking to the next vertical blanking , each unit corresponding to a plurality of frames arranged according to a multiplexing scheme, The frame has video information that is intended to be synthesized and displayed as a 3D image,
The multiplexing scheme wherein 3D transfer information in the auxiliary information frame packet transmitted during the data island period includes the number of video frames in a unit to be combined as one 3D image in the 3D display signal. The auxiliary information frame packet further includes information indicating a period of the vertical blanking according to a video format for transmission of the 3D image, and the multiplexing scheme includes at least Selected from a group of multiplexing schemes including frame alternating multiplexing, wherein frame alternating indicates that the plurality of frames are sequentially arranged within the video data period;
The 3D display device, wherein the processing means generates the display control signal depending on the 3D transfer information. The processing means is
Field alternating multiplexing,
Line alternate multiplexing,
Side-by-side frame multiplexing, indicating that the plurality of frames are arranged side-by-side within the video data period;
2D and depth frame multiplexing,
2D, depth, graphics and graphics depth frame multiplexing,
Generating the display control signal in dependence on further information regarding the multiplexing scheme by the group of multiplexing schemes further comprising at least one of:
The 3D display device according to claim 9.   The 3D display device according to claim 10, wherein the video transfer format is HDMI (registered trademark).   The 3D display device according to claim 11, wherein the 3D transfer information is included in an AVI information frame.   The 3D display device according to claim 11, wherein the 3D transfer information is included in a vendor-specific information frame. A 3D display signal for transferring 3D image data into a three dimensional [3D] display device, the 3D display signal, data islands audio and auxiliary data are duration that is transmitted using a packet of a series A sequence of frames constituting the 3D image data according to a 3D video transfer format having a period as well as a video data period during which active video pixels are transmitted;
The sequence of frames has units, wherein the unit is a period from vertical blanking to the next vertical blanking , each unit corresponding to a plurality of frames arranged according to a multiplexing scheme, The frame has video information that is intended to be synthesized and displayed as a 3D image,
Each frame has a data structure for representing a sequence of digital image pixel data, each frame type represents a partial 3D data structure,
The 3D display signal is included in the 3D display signal in the auxiliary information frame packet transmitted in the data island period to generate a display control signal depending on 3D transfer information in the display device. one of the 3D transfer information to at least have the information about the multiplexing scheme containing the number of video frames in the unit to be synthesized as a 3D image, and, depending on the video format for the transmission of the 3D image Information indicating the vertical blanking period;
The multiplexing scheme is selected from a group of multiplexing schemes including at least frame alternating multiplexing, wherein frame alternating indicates that the plurality of frames are sequentially arranged within the video data period signal. In the information regarding the multiplexing scheme, the group of multiplexing schemes is
Field alternating multiplexing,
Line alternate multiplexing,
Side-by-side frame multiplexing, indicating that the plurality of frames are arranged side-by-side within the video data period;
2D and depth frame multiplexing,
2D, depth, graphics and graphics depth frame multiplexing,
15. The 3D display signal of claim 14, further comprising at least one of:

Images