Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/15—Processing image signals for colour aspects of image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
  • H04N13/305—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays using lenticular lenses, e.g. arrangements of cylindrical lenses
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2213/00—Details of stereoscopic systems
  • H04N2213/003—Aspects relating to the "2D+depth" image format

Definitions

• This invention pertains in general to the field of image signal processing. More particularly the invention relates to processing of image signals for display on 3D lenticular or barrier displays, and more particularly to preserving the perceived image quality of a signal when rendering image signals for display on 3D lenticular or barrier displays.
• 3D imagery is a function of binocular parallax, which provides relative depth perception to the viewer.
• the resulting retinal disparity provides stimulus from which the sense of stereopsis is created by the viewer's visual system.
• Within the visual system separate neurological sub- systems specializing in different aspects of stereopsis such as fine or coarse stereopsis, or motion- in- depth, static or lateral motion stereopsis performing in combination or separately based upon the stimulus, create a 3D image for the viewer.
• Various means whereby 2D images may be presented to the viewer's visual system as 3D images are currently in existence.
• WO/99/05559 a method for controlling pixel addressing of a display device to drive the display device as an multi-view auto-stereoscopic display when a lenticular screen is overlaid and image data for multiple views to be interlaced is provided. Based on data defining at least the lenticular screen lenticule pitch, and the global lenticular screen position relative to the display device, for each display colour pixel, a derivation is made as to which of the N views it is to carry. The corresponding pixel data for the assigned view is then selected as the display pixel data.
• the image quality of the multi-view display device controlled on basis of the method as described in WO 99/05559 is relatively good, the amount of signal processing needed to produce the displayed image is quite large.
• Fig. 1 illustrates a known signal processing system for rendering 2.5D video signals on 3D displays, wherein the display is constructed such that a view is mapped on individual sub-pixels.
• the rendering system 100 receives a YUVD signal which is converted into a RGBD signal in a known manner by a converter 102.
• the RGBD is then scaled by a scaler 104 into the individual R 5 G 5 B 5 D components.
• a view renderer(s) 106 then renders the RD 5 GD 5 and BD signals to produce new R 5 G 5 B signals.
• the view Tenderer 106 is instantiated 9 times. Each instantiation operates on a single colour component.
• the R 5 G 5 B signals are then merged back together in a merge unit 108 to produce a final RGB signal which can then be displayed on a display screen 110.
• Fig. 2 illustrates the view numbers on the respective sub-pixels for a 9-view display according to the signal processing system illustrated in Fig. 1. It performs two main functions: the depth transformation and resampling to generate the proper sub-pixel grid.
• the computational complexity of a view Tenderer is significant in view of current technology as it operates on sub-pixel positions of high display resolutions. Further, the computational load for each of the colour components is equal. Thus, there is a tremendous amount of computing that must be performed to render the 2.5D video signal on the 3D display. This computational load requires a significant amount of processing power and energy, since it is performed in real-time.
• an improved signal processing system would be advantageous and in particular a rendering system which significantly reduces the amount of computations needed to render the image data on a 3D display while balancing spatial errors and view errors to produce a signal with acceptable image quality.
• the present invention preferably seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination and solves at least the above mentioned problems, at least partly, by providing a method, an apparatus, and a computer-readable medium that provides an efficient rendering of image data on a 3D display, according to the appended patent claims.
• the invention aims at significant cost savings while preserving the perceived image quality when rendering image data on a 3D lenticular or barrier display. This is mainly performed by processing in the YUV domain and reduction of the U/V resolution. The view and spatial errors are balanced by novel selection of sub-pixel values in the YUV7RGB matrix. The perceived image quality is only marginally reduced. Furthermore, the processing in the YUV domain enables adaptive processing of depth dependent brightness/contrast to fit seamlessly in the processing chain. This improves the perceived depth impression. This information reduces the computational load by at least 50%. According to aspects of the invention, a method, an apparatus, and a computer-readable medium for rendering image data on a 3D display are disclosed.
• a method for rendering image data on a 3D display comprising the steps of: - receiving a first image signal; rendering at least one colour component of the first image signal in reduced spatial resolution to produce a second image signal; and spatial filtering said second image signal, wherein spatial errors and view errors are balanced when reconstructing a full resolution signal for the display.
• a signal processing system for rendering image data on a 3D display comprising: means for receiving a first image signal; means for rendering at least one colour component of the first image signal in reduced spatial resolution to produce a second image signal; and - means for spatial filtering said second image signal, wherein spatial errors and view errors are balanced when reconstructing a full resolution signal for the display.
• a computer-readable medium having embodied thereon a computer program for rendering image data for 3D display for processing by a computer, wherein the computer program comprises: - a code segment for rendering at least one colour component of the first image signal in reduced spatial resolution to produce a second image signal; and a code segment for spatial filtering said second image signal, wherein spatial errors and view errors are balanced when reconstructing a full resolution signal for the display.
• the present invention has the advantage over the prior art that it reduces the computational load on a rendering system while maintaining the perceived image quality of the image displayed on a 3D display.
• FIG. 1 illustrates a known signal processing system
• Fig. 2 illustrates view numbers on the respective sub-pixels for a 9-view display according to the signal processing system illustrated in Fig. 1 ;
• Fig. 3 illustrates a schematic perspective view of a multi-view display device which may be used with the various embodiments of the invention
• Fig. 4 illustrates a signal processing system according to one embodiment of the invention
• Fig. 5 illustrates a signal processing system according to another embodiment of the invention
• Fig. 6 illustrates view numbers on the respective sub-pixels for a 9-view display according to the signal processing system illustrated in Fig. 5;
• Fig. 7 illustrates a computer readable medium according to one embodiment of the invention.
• the following description focuses on an embodiment of the present invention applicable to a video display systems and in particular to a 3D video display system.
• the invention is not limited to this application but may be applied to many other video display systems.
• the invention applies to rendering of 2.5D signals (regular video augmented with depth), stereo signals (a left-eye and right-eye regular video signal) or even rendering of multi-view (e.g. 9 images for 9-view display).
• the invention applies to any type of image data such as, for example, video signals, still images, etc., although the calculation load savings is more important for video since it requires real-time processing.
• the display device 10 includes a conventional LC matrix display panel 11 used as a spatial light modulator and comprising a planar array of individually addressable and similarly sized light-generating elements 12 arranged in aligned rows and columns perpendicularly to one another. While only a few light-generating elements are shown, there may, in practice, be around 800 columns (or 2400 columns in colour, with RGB triplets used to provide a full colour display) and 600 rows of display elements. Such panels are well known and will not be described here in more detail.
• the light-generating elements 12 are substantially rectangular in shape and are regularly spaced from one another with the light-generating elements in two adjacent columns being separated by a gap extending in column (vertical) direction and with the display elements in two adjacent rows being separated by a gap extending in the row (horizontal) direction.
• the panel 11 is of the active matrix type in which each light- generating element is associated with a switching element, comprising for example, a TFT or thin film diode, TDF, situated adjacent the light-generating element.
• the display panel 11 is illuminated by a light source 14, which, in this example, comprises a planar backlight extending over the area of the display element array.
• a light source 14 which, in this example, comprises a planar backlight extending over the area of the display element array.
• Light from the source 14 is directed through the panel with the individual light-generating elements being driven, by appropriate application of drive voltages, to modulate this light in conventional manner to produce a display output.
• the array of light-generating elements constituting the display produced thus corresponds with the structure of light-generating elements, each light-generating elements, each light-generating element providing a respective display pixel.
• a computing means 18 computes luminance values for the respective light-generating elements on basis of an input signal.
• a lenticular sheet 15 comprising an array of elongate, parallel, lenticules, or lens elements, acting as optical director means to provide separate images to a viewer's eyes, producing a stereoscopic display to a viewer facing the side of the sheet 15 remote from the panel 11.
• the lenticules of the sheet 15, which is of conventional form, comprise optically (semi) cylindrically converging lenticules, for example, formed as convex cylindrical lenses or graded reflective index cylindrical lenses.
• the rendering process comprises several operations. First, an image is calculated for every view (e.g. from video + depth, or from stereo). The image is then properly scaled to the view resolution. The image is then properly shifted to the subpixel positions of the view. It will be understood by those skilled in the art that some or all of these operations may be combined. For example, as illustrated in Fig. 2, the vertical scaling is done separately and then the view renderer performs all of the horizontal processing of the three operations. In the Human Visual System (HVS), sharpness impression is mainly determined by luminance components, significantly less by chrominance. It is suggested that this also holds for depth perception.
• HVS Human Visual System
• the B and R components are not calculated for every line in the frame.
• the B and R components are only calculate on every even line in the frame and a vertical average between the even lines is used to calculate the B/R signals on the odd lines.
• the B and/or R components have a 50% reduced vertical resolution.
• Fig. 4 illustrates a video signal processing system according to the first embodiment of the invention.
• the video processing system may be part of a display apparatus 200 e.g. a television, computer monitor, handheld device, etc.
• the rendering system 201 receives a YUVD signal, which is converted into a RGBD signal in a known manner by a converter 202.
• the RGBD is then scaled by a sealer 204 into the individual R 5 G 5 B 5 D components.
• the RD 5 GD 5 and BD components for each even line in the frame are sent to at least one view renderer 206 to produce new R 5 G 5 B signals for the even lines.
• the R 5 G 5 B 5 signals for the even lines are then merged together in a merged unit 210.
• the GD component for each odd line in the frame is sent to a view renderer 208 (which behaves similar to 206) to produce a new G signal for each odd line in the frame.
• the view Tenderers 206, 208 spatially filter their respective output signals in such a manner so as to minimize visible artefacts caused by spatial and view errors produced during the rendering process.
• an average value of the R and B signals for the even lines on each side of the odd lines are then merged in the merge unit 212 with the calculated G signal to create an RGB signal of the odd lines of the frame.
• the RGB signals for the even and odd lines are then combined in a unit 214 to create a final RGB signal, which is sent to a display 216.
• the rendering process produces spatial errors and view errors.
• the spatial error refers to the spatial distance. The closer the spatial distance, the more correlated the sample values, so close spatial position provides minimal error.
• the view error refers to the view number. Large differences in view numbers relate to large disparities, hence a minimum view difference provides minimal error.
• a view error of 0 only allows the use of sample values from the same view, resulting in very large spatial distances and thus leads to a significant overall error.
• a minimal spatial error results in some cases in a very large view error resulting in very large disparities and thus leads to a significant overall error.
• the two errors are balanced using spatial filtering resulting in good image quality.
• the spatial filter design takes both the spatial properties and the depth properties of the display into account.
• a spatial filter is selected which tries to balance the correction of the spatial error with the correction of the view error so that neither error produces many visible artefacts. This solution proved to introduce hardly any visible artefacts. Since the computational load of the average operation can be neglected compared to view rendering, this reduces the computations by 1/3.
• the invention may also be used to calculate the R 5 G 5 B values for the odd lines and use the R and B values of the odd lines to estimate the R and B values of the even lines. Furthermore, it will also be understood that the traditional calculation of R and B values for odd lines can be skipped for every other odd line, every 3 rd line, every 4 th line, etc.
• the rendering is performed in the YUV domain.
• Fig. 5 illustrates a video signal processing system according to this embodiment of the invention. It will be understood by those skilled in the art that the video processing system may be part of a display apparatus 300 e.g. a television, computer monitor, handheld device, etc.
• the rendering system 301 receives a YUVD signal, which is applied to a sealer 302.
• the YUVT) signal is scaled by the sealer 302 into individual Y 5 U 5 V 5 D components.
• the YD 5 UD 5 VD components are sent to a view Tenderer 304 to produce new Y 5 U 5 V signals.
• the Y 5 U 5 V signals are then merged back together in a merge unit.
• the merged YUV signal is then converted into an RGB signal by a converter 308.
• the conversion of the YUV signal to the RGB signal takes both the spatial properties and depth properties of the display into account by using a specifically chosen spatial filter as described above.
• the RGB signal is then sent to a display 310. At first glance, this does not provide any cost saving while introducing an error. First, the error should be reduced as mush as possible. Later it will be shown how a reduction of the resolution of the U/V signals leads to significant cost savings.
• the view renderer is designed to operate on the R, G and B sub-pixel locations of the screen. For optimal mapping of YUV on these RGB locations we take the colour space conversion matrix into account; as an example, the ITU-R BT.601-5 colour matrix given by
• V 0.500 * R' - 0.419 * G' - 0.081 * B'
• Y is mapped on G (i.e., it is processed as if it were a G signal); U is mapped on B, V is mapped on R. This mapping of the YUV on RGB sub-pixel locations as is shown in Fig. 6.
• the complexity of U/V processing may be reduced at least by 50% compared to Y processing.
• the above mentioned reduced resolution of U/V signals is exploited.
• the input signal is usually 4:2:2 formatted, only half of the pixels in the horizontal direction should be processed during rendering.
• some filtering may be applied, either using linear or statistical order filters. For example:
• An additional option in the invention is to apply depth depended signal processing. It is known from perception research the depth impression is related to brightness/contrast properties: far-away parts of a scene appear more "misty" than close-by parts. This knowledge can easily be applied in the invention at the rendering stage, since now luminance and depth information are both available at the rendering stage and depth dependent brightness/contrast adaptation can easily be obtained, e.g. by means of a variable gain (depth controlled) or a lookup-table. This results in an improved depth impression.
• depth dependent signal processing relates to sharpness.
• objects in the background are out of focus. This observation can be applied in the signal processing: blurring the background improves the depth impression. Therefore depth dependent sharpness reduction may enhance the depth impression.
• sharpness impression is mainly determined by the luminance component of a video signal, it is advantageous to apply this depth dependent sharpness filter in the YUV domain.
• the current invention provides a particularly advantageous system since this depth dependent filtering can be seamlessly integrated in the rendering unit that is processing the YD signals at relatively low extra cost.
• the main function of the rendering is to provide a disparity depth cue to the observer. By means of dependent signal processing, additional depth cues are provided.
• the various embodiments are designed for easy combination to obtain maximum savings: without even taking the simplified filters of the second embodiment into account, the first and third result both in a reduction of 50% in U/V processing, so 300% for regular RGB becomes 100% for Y and 25% for U and V respectively. This results in a total reduction of 50%.
• the invention can be used in a switchable 2D/3D display where the display can be put in a mode where it operates as a regular 2D display or it can be switched to a 3D mode.
• the pixel selection for the YUV to RGB conversion depends on the selected 2D or 3D display mode.
• a computer readable medium 700 has embodied thereon a computer program 710 for rendering video data on a 3D display, for processing by a computer 713.
• the computer program comprises a code segment 714 for rendering at least one colour component of the first video signal in reduced spatial resolution to produce a second signal; and a code segment 715 for spatial filtering said second video signal wherein spatial errors and view errors are balanced when reconstructing a full resolution signal for the display.
• the invention can be implemented in any suitable form including hardware, software, firmware or any combination of these.
• the elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit, or may be physically and functionally distributed between different units and processors.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Controls And Circuits For Display Device (AREA)
• Image Processing (AREA)
• Processing Or Creating Images (AREA)
• Facsimile Image Signal Circuits (AREA)
• Color Image Communication Systems (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=37064655

Family Applications (1)

Country Status (5)

Cited By (2)

Families Citing this family (15)

Citations (2)

Family Cites Families (8)

• 2006
  • 2006-05-02 JP JP2008510689A patent/JP5213701B2/ja not_active Expired - Fee Related
  • 2006-05-02 EP EP06728110.5A patent/EP1882368B1/en not_active Not-in-force
  • 2006-05-02 US US11/913,877 patent/US7961196B2/en not_active Expired - Fee Related
  • 2006-05-02 CN CN2006800165807A patent/CN101176354B/zh not_active Expired - Fee Related
  • 2006-05-02 WO PCT/IB2006/051374 patent/WO2006120607A2/en not_active Ceased

Patent Citations (2)

Also Published As

Similar Documents

Legal Events