WO2012118231A1 - Method and device for displaying a pair of stereoscopic images - Google Patents

Method and device for displaying a pair of stereoscopic images Download PDF

Info

Publication number
WO2012118231A1
WO2012118231A1 PCT/JP2012/055889 JP2012055889W WO2012118231A1 WO 2012118231 A1 WO2012118231 A1 WO 2012118231A1 JP 2012055889 W JP2012055889 W JP 2012055889W WO 2012118231 A1 WO2012118231 A1 WO 2012118231A1
Authority
WO
WIPO (PCT)
Prior art keywords
disparity
pair
images
graphical element
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2012/055889
Other languages
French (fr)
Inventor
Yeping Su
Hao Pan
Chang YUAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sharp Corp
Original Assignee
Sharp Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sharp Corp filed Critical Sharp Corp
Publication of WO2012118231A1 publication Critical patent/WO2012118231A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G3/00—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes
    • G09G3/001—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background
    • G09G3/003—Control arrangements or circuits, of interest only in connection with visual indicators other than cathode-ray tubes using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background to produce spatial visual effects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
    • H04N13/183—On-screen display [OSD] information, e.g. subtitles or menus
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/398—Synchronisation thereof; Control thereof

Definitions

  • the present invention relates generally to displaying stereoscopic images on a display.
  • planar stereoscopic- display sometimes triggers unpleasant feelings of discomfort or fatigue in the viewer.
  • the discomfort and fatigue may be, at least in part, caused by limitations of existing planar stereoscopic displays .
  • a planar stereoscopic display no matter whether LCD based or projection based, shows two images with disparity between them on the same planar surface .
  • the display results in the left eye seeing one of the stereoscopic images and the right eye seeing the other one of the stereoscopic images. It is the disparity of the two images that results in viewers feeling that they are viewing three dimensional scenes with depth information .
  • This viewing mechanism is different from how eyes normally perceive natural three dimensional scenes, and may cause a vergence-accommodation conflict.
  • the vergence- accommodation conflict strains the eye muscle and sends confusing signals to the brain, and eventually causes discomfort/ fatigue.
  • the preferred solution is to construct a volumetric three dimensional display to replace existing planar stereoscopic displays. Unfortunately, it is difficult to construct such a volumetric display, and is likewise difficult to control such a display.
  • Another solution is based upon signal processing.
  • the signal processing manipulates the stereoscopic image pair sent to the planar stereoscopic display in some manner.
  • the vergence-accommodation conflict can be significantly reduced and thereby reduce the likelihood of discomfort and / or fatigue .
  • What is desired is a display system that reduces the discomfort and/ or fatigue for stereoscopic images.
  • the present invention provides a method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
  • the present invention also includes a method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
  • the present invention also includes a device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
  • a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element.
  • the present invention also includes a device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
  • a receiving section for receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image together with a disparity between said left image and said right image;
  • a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said disparity of said pair of images and said estimated disparity of said graphical element.
  • FIG. 1 illustrates a stereoscopic viewing system for reducing discomfort and / or fatigue.
  • FIG. 2 illustrates disparity estimation with a graphical element.
  • FIG. 3 illustrates a local disparity analysis.
  • FIG. 4 illustrates another local disparity analysis.
  • the system provides a signal processing based technique to reduce the discomfort/ fatigue associated with a three dimensional viewing experience . More specifically, given a planar stereoscopic display, the technique takes in a stereoscopic image pair that may cause viewing discomfort/ fatigue, and outputs a modified stereoscopic pair that causes less or no viewing discomfort/ fatigue .
  • FIG. 1 One simplified example (merely for purposes of illustration) of a stereoscopic processing system for reducing viewer discomfort is illustrated in FIG. 1 .
  • This technique receives a stereoscopic pair of images 100 , 1 10 , in which one image 100 is for the left eye to view (L image) and the other image is for the right eye to view (R image) 1 10 , and outputs a modified stereoscopic pair of images 120 (L image) , 130 (R image) where one or both of the input images may be modified in some manner. If the input stereoscopic image pairs have very large disparities in some areas between two images, the large disparities may cause severe vergence-accommodation conflict that leads to discomfort or even fatigue for some viewers .
  • the technique may include some principal components, namely, a disparity map estimation 200 (receiving section, first estimating section, second estimating section, estimating section) , a disparity map adjustment 300 (modifying section) , and a new view image synthesis 400.
  • a disparity map estimation 200 receiving section, first estimating section, second estimating section, estimating section
  • a disparity map adjustment 300 modifying section
  • a new view image synthesis 400 the system may presume that the input stereoscopic pair has been rectified so the disparity between two images is only horizontal. In other cases, the system may modify accordingly where the input stereoscopic pair is rectified in any other direction or otherwise not rectified.
  • the disparity map estimation 200 outputs two disparity maps, L2R map 202 and R2 L map 204.
  • the L2 R map 202 gives the disparity of each pixel in the L image
  • the R2L map 204 gives the disparity of each pixel in the R image .
  • the data also tends to indicate occlusion regions which may be accounted for in the new view image synthesis 400.
  • the disparity map estimation 200 also provides matching errors of the two disparity maps, which provides a measure of confidence in the map data.
  • the adjustment based on the L2R map 202 and the R2L map 204 in the disparity map adjustment 300 may be controlled by a disparity selection 302.
  • the disparity selection 302 may predict the discomfort based upon the estimated disparity in the image pairs 202 , 204 , viewing conditions, display characteristics, viewer preferences, and/ or any other suitable data. Based upon this estimation, the amount of disparity may be modified. The modification may result in global modification, obj ect based modification, region based modification, or otherwise .
  • a modified set of disparity maps 3 10 , 320 are created by the disparity map adjustment 300.
  • the modified disparity maps may include the newview2 R disparity map 3 10 and the newview2L disparity map 320 which are provided to the new view image synthesis 400.
  • the new view image synthesis 400 synthesizes the new views 120 and/ or 130 based upon data from the disparity map adjustment 300, the disparity map estimation 200 , and the input image pair 100 , 1 10. It is to be understood that any suitable technique may be used to compute disparities or otherwise the disparities may be provided together with the video content, and one or more new images may be computed as a result.
  • disparity information is suitable for image content from a content provider from which disparity information is traditionally provided or computed
  • graphical elements that are rendered on the display that do not contain disparity information from the content provider.
  • Some of the types of graphical elements includes captions, menus, and logos. Often, these graphical elements are inserted into the video stream or otherwise indicated to be displayed on the display, after the video content is obtained .
  • graphical overlays may be provided with scoring or other information that should also be displayed on the display.
  • the horizontal and vertical positions for the graphical element are provided.
  • the disparity information for the graphical element should be provided. That is to say that the graphical elements should be rendered on both the left view and the right view with the appropriate amount of horizontal displacement.
  • One approach for rendering the graphical elements is to use a fixed disparity value , such as for example zero disparity or a pre-determined negative disparity.
  • a fixed disparity tends to result in a visual conflict at the boundaries where the graphical element and the three dimensional video content being display have substantially different disparities. This conflict of different disparities tends to make it hard to fuse both views together and ultimately tends to result in discomfort for the viewers.
  • a modified technique for the suitable presentation of a graphical element on a stereoscopic display should be based upon the local disparity in the region where the graphical element is to be displayed.
  • the local content of the region where the graphical element is to be displayed is used to determine the disparity to be used for the graphical element, and therefore the disparity of the local region and the graphical element will substantially match one another.
  • the result of such a content-dependent technique for determining the disparity information is that fewer artifacts are created and a more comfortable viewing experience is achieved for the viewer.
  • the left-eye image 600 and the right eye image 6 10 are received by the system.
  • the system computes a disparity estimation 620 , which may for example be computed as described in FIG. 1 or any other suitable technique .
  • the result of the disparity estimation 620 may be any suitable data that provides information regarding the disparity between the images at different positions within the images .
  • the disparity information may be provided together with the images.
  • a local disparity analysis 630 may compute a disparity histogram of a local window where the geographical element is to be located. This aggregates disparity data that may be used for determining a suitable disparity for the graphical element.
  • the local disparity estimation 630 may estimate a dense disparity map between the entire left image and the entire right image .
  • the dense disparity map contains disparities for each pixel in either the left image and/ or the right image, with the former being referred to as L2R map and the later being referred to as R2L map.
  • L2R map With the intended spatial location and/ or the size of the graphical overlay being known, a rectangular bounding box/ window (or any other defined region or space) may be defined that generally covers the area where the graphical element is to be located.
  • a horizontal disparity histogram may be collected from the disparity values associated with the pixels in the window.
  • a histogram cleaning step may be applied, that removes outliers with extreme disparities . For example, disparities within the range of 5% and 95% percentiles in the histogram may be retained.
  • the local disparity analysis results in local disparity statistics which are then used to determine a suitable disparity for the graphical element.
  • the local disparity estimation 630 may alternatively substantially compute the disparities within the intended window for the graphical element.
  • the disparity estimation produces a local disparity histogram which reduces the computational complexity of the system.
  • the preferred disparity for the video content of the current frames together with one or more graphical elements 640 may be computed . In this manner, the disparities for the video images are determined and those regions of the image that correspond to graphical elements are likewise determined, both in a manner suitable to reduce viewer discomfort.
  • the disparity for the local window may be computed in any suitable manner, such as by using a histogram.
  • the disparity associated with the graphical element may be a mean disparity, a most frequent disparity, or a minimal disparity.
  • the mean disparity may be determined as the mean disparity of substantially all the samples in the histogram.
  • the most frequent disparity may be determined as the disparity value associated with the highest count in the histogram.
  • the minimal disparity may be determined as the minimal disparity in the histogram .
  • the minimal disparity in the histogram tends to ensure that the graphical element will be placed "on top of" the underlying content in terms of perceived depth, which is usually what a viewer would expect to observe for a graphical element.
  • a temporal smoothing 660 of the disparity calculations 650 is preferably used to smooth out such effects.
  • the system may apply a temporal filter to reduce the disparity from changing to dramatically from frame to frame.
  • the temporal smoothing may be turned off (or substantially reduced) to avoid false disparity information being leaked from one shot to the next shot, as defined by: a ⁇ d(t - ⁇ ) + ( ⁇ - a) ⁇ d(t) no _ shot _ boundary _at _t
  • the disparity estimates for rendering the images are determined 660 and the graphical elements are rendered on both the left image and the right image with the appropriate disparity values.
  • the final disparities estimate d(t) for the on-screen graphical element(s) may be rendered and positioned. Assuming the intended spatial location of the left image is [G x (t), G y (t)], the same overlay may be placed on the right image as [G x (t)+ d(t),
  • the opacity of the graphical overlay may also be modified based on its disparity. For example, if there is a strong conflict between the scene disparities and the graphical disparities, the opacity of the graphical element may be reduced which reduces the visual discomfort.
  • a method for displaying a pair of stereoscopic images together with a graphical element on a display can be provided.
  • the method can comprise following four steps (a) to (d) .
  • said stereoscopic images may include a horizontal disparity.
  • said disparity estimation may provide a LtoR disparity map, a RtoL disparity map, a RtoL disparity matching errors, and LtoR disparity matching errors.
  • said LtoR disparity map and said RtoL disparity map may be adjusted such that respective disparity values of the maps are each increased or decreased.
  • said disparity maps may be further adjusted, where the disparity values are increased or decreased.
  • said LtoR disparity map and said RtoL disparity map may be adjusted with a same offset value for all the pixels .
  • said disparity maps may be adjusted with a same offset value for all the pixels.
  • said LtoR disparity map and said RtoL disparity map may be adjusted with a different offset value for each pixel.
  • said disparity maps may be adjusted with a different offset value for each pixel.
  • said adjusted disparity may be further based upon a viewer preference .
  • said adjusted disparity may be further based upon a model based upon display characteristics of said display.
  • said estimating said disparity of said graphical element may be performed after said estimating said disparity between said left image and said right image .
  • said estimating said disparity of said graphical element may be performed before said estimating said disparity between said left image and said right image .
  • said graphical element may be received without disparity information.
  • a location to position said graphical element within said image may be received.
  • said disparity of said graphical element may not be a fixed disparity value .
  • said disparity of said graphical element may be based upon the disparity proximate said location.
  • said disparity of said graphical element may be substantially the same as the disparity of said pair of images proximate said location .
  • said disparity of said graphical element may be substantially the same as the disparity of said pair of images proximate the boundaries of said graphical element.
  • said disparity of said graphical element may be further based upon a temporal characterization.
  • said disparity of said graphical element may be based upon a histogram.
  • said disparity of said graphical element may be further based upon a minimal disparity.
  • the discomfort and/ or fatigue for stereoscopic images can be reduced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

A method for displaying a pair of stereoscopic images together with a graphical element on a display including receiving a pair of images forming the pair of stereoscopic images, one being a left image and one being a right image. Estimating a disparity between the left image and the right image based upon a matching of a left region of the left image with a right region of the right image. Also, estimating a disparity for a graphical element to be displayed together with the pair of images on the display based upon a disparity of a spatial region of the pair of images. Based upon the estimated disparity of the pair of images and the estimated disparity of the graphical element modifying at least one of the right image and the left image to be displayed upon the display.

Description

DESCRIPTION
TITLE OF INVENTION: METHOD AND DEVICE FOR
DISPLAYING A PAIR OF STEREOSCOPIC IMAGES
TECHNICAL FIELD
The present invention relates generally to displaying stereoscopic images on a display.
BACKGROUND ART
Viewing stereoscopic content on planar stereoscopic- display sometimes triggers unpleasant feelings of discomfort or fatigue in the viewer. The discomfort and fatigue may be, at least in part, caused by limitations of existing planar stereoscopic displays . A planar stereoscopic display, no matter whether LCD based or projection based, shows two images with disparity between them on the same planar surface . By temporal and/ or spatial multiplexing the stereoscopic images, the display results in the left eye seeing one of the stereoscopic images and the right eye seeing the other one of the stereoscopic images. It is the disparity of the two images that results in viewers feeling that they are viewing three dimensional scenes with depth information . This viewing mechanism is different from how eyes normally perceive natural three dimensional scenes, and may cause a vergence-accommodation conflict. The vergence- accommodation conflict strains the eye muscle and sends confusing signals to the brain, and eventually causes discomfort/ fatigue.
The preferred solution is to construct a volumetric three dimensional display to replace existing planar stereoscopic displays. Unfortunately, it is difficult to construct such a volumetric display, and is likewise difficult to control such a display.
Another solution, at least in part, is based upon signal processing. The signal processing manipulates the stereoscopic image pair sent to the planar stereoscopic display in some manner. Although the signal processing cannot fundamentally completely solve the problem, the vergence-accommodation conflict can be significantly reduced and thereby reduce the likelihood of discomfort and / or fatigue .
What is desired is a display system that reduces the discomfort and/ or fatigue for stereoscopic images.
SUMMARY OF INVENTION
The present invention provides a method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image;
(b) estimating a disparity between said left image and said right image based upon a matching of a left region of said left image with a right region of said right image;
(c) estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(d) based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element modifying at least one of said right image and said left image to be displayed upon said display.
The present invention also includes a method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image together with a disparity between said left image and said right image;
(b) estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(c) based upon said disparity of said pair of images and said estimated disparity of said graphical element modifying at least one of said right image and said left image to be displayed upon said display.
The present invention also includes a device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) a receiving section for receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image;
(b) a first estimating section for estimating a disparity between said left image and said right image based upon a matching of a left region of said left image with a right region of said right image;
(c) a second estimating section for estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(d) a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element.
The present invention also includes a device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) a receiving section for receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image together with a disparity between said left image and said right image;
(b) a estimating section for estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(c) a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said disparity of said pair of images and said estimated disparity of said graphical element.
The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings .
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 illustrates a stereoscopic viewing system for reducing discomfort and / or fatigue.
FIG. 2 illustrates disparity estimation with a graphical element. FIG. 3 illustrates a local disparity analysis.
FIG. 4 illustrates another local disparity analysis.
DESCRIPTION OF EMBODIMENTS
The system provides a signal processing based technique to reduce the discomfort/ fatigue associated with a three dimensional viewing experience . More specifically, given a planar stereoscopic display, the technique takes in a stereoscopic image pair that may cause viewing discomfort/ fatigue, and outputs a modified stereoscopic pair that causes less or no viewing discomfort/ fatigue .
One simplified example (merely for purposes of illustration) of a stereoscopic processing system for reducing viewer discomfort is illustrated in FIG. 1 . This technique receives a stereoscopic pair of images 100 , 1 10 , in which one image 100 is for the left eye to view (L image) and the other image is for the right eye to view (R image) 1 10 , and outputs a modified stereoscopic pair of images 120 (L image) , 130 (R image) where one or both of the input images may be modified in some manner. If the input stereoscopic image pairs have very large disparities in some areas between two images, the large disparities may cause severe vergence-accommodation conflict that leads to discomfort or even fatigue for some viewers .
As shown in FIG . 1 , the technique may include some principal components, namely, a disparity map estimation 200 (receiving section, first estimating section, second estimating section, estimating section) , a disparity map adjustment 300 (modifying section) , and a new view image synthesis 400. For simplicity, the system may presume that the input stereoscopic pair has been rectified so the disparity between two images is only horizontal. In other cases, the system may modify accordingly where the input stereoscopic pair is rectified in any other direction or otherwise not rectified.
The disparity map estimation 200 outputs two disparity maps, L2R map 202 and R2 L map 204. The L2 R map 202 gives the disparity of each pixel in the L image, while the R2L map 204 gives the disparity of each pixel in the R image . The data also tends to indicate occlusion regions which may be accounted for in the new view image synthesis 400. The disparity map estimation 200 also provides matching errors of the two disparity maps, which provides a measure of confidence in the map data.
The adjustment based on the L2R map 202 and the R2L map 204 in the disparity map adjustment 300 may be controlled by a disparity selection 302. The disparity selection 302 may predict the discomfort based upon the estimated disparity in the image pairs 202 , 204 , viewing conditions, display characteristics, viewer preferences, and/ or any other suitable data. Based upon this estimation, the amount of disparity may be modified. The modification may result in global modification, obj ect based modification, region based modification, or otherwise . A modified set of disparity maps 3 10 , 320 are created by the disparity map adjustment 300. The modified disparity maps may include the newview2 R disparity map 3 10 and the newview2L disparity map 320 which are provided to the new view image synthesis 400.
The new view image synthesis 400 synthesizes the new views 120 and/ or 130 based upon data from the disparity map adjustment 300, the disparity map estimation 200 , and the input image pair 100 , 1 10. It is to be understood that any suitable technique may be used to compute disparities or otherwise the disparities may be provided together with the video content, and one or more new images may be computed as a result.
While this technique of computing the disparity information is suitable for image content from a content provider from which disparity information is traditionally provided or computed, there are other types of content such as graphical elements that are rendered on the display that do not contain disparity information from the content provider. Some of the types of graphical elements includes captions, menus, and logos. Often, these graphical elements are inserted into the video stream or otherwise indicated to be displayed on the display, after the video content is obtained . For example, while obtaining three dimensional video content of a live football game, graphical overlays may be provided with scoring or other information that should also be displayed on the display. When graphical overlays are rendered on a traditional two-dimensional display, the horizontal and vertical positions for the graphical element are provided. When a graphical element is rendered on a stereoscopic display, the disparity information for the graphical element should be provided. That is to say that the graphical elements should be rendered on both the left view and the right view with the appropriate amount of horizontal displacement.
One approach for rendering the graphical elements is to use a fixed disparity value , such as for example zero disparity or a pre-determined negative disparity. Unfortunately, using a fixed disparity tends to result in a visual conflict at the boundaries where the graphical element and the three dimensional video content being display have substantially different disparities. This conflict of different disparities tends to make it hard to fuse both views together and ultimately tends to result in discomfort for the viewers. A modified technique for the suitable presentation of a graphical element on a stereoscopic display should be based upon the local disparity in the region where the graphical element is to be displayed. In this manner, the local content of the region where the graphical element is to be displayed is used to determine the disparity to be used for the graphical element, and therefore the disparity of the local region and the graphical element will substantially match one another. The result of such a content-dependent technique for determining the disparity information is that fewer artifacts are created and a more comfortable viewing experience is achieved for the viewer.
Referring to FIG. 2 , one implementation of such a system is illustrated . The left-eye image 600 and the right eye image 6 10 are received by the system. The system computes a disparity estimation 620 , which may for example be computed as described in FIG. 1 or any other suitable technique . The result of the disparity estimation 620 may be any suitable data that provides information regarding the disparity between the images at different positions within the images . In addition, the disparity information may be provided together with the images.
A local disparity analysis 630 may compute a disparity histogram of a local window where the geographical element is to be located. This aggregates disparity data that may be used for determining a suitable disparity for the graphical element.
Referring also to FIG. 3 , the local disparity estimation 630 may estimate a dense disparity map between the entire left image and the entire right image . The dense disparity map contains disparities for each pixel in either the left image and/ or the right image, with the former being referred to as L2R map and the later being referred to as R2L map. With the intended spatial location and/ or the size of the graphical overlay being known, a rectangular bounding box/ window (or any other defined region or space) may be defined that generally covers the area where the graphical element is to be located. A horizontal disparity histogram may be collected from the disparity values associated with the pixels in the window. A histogram cleaning step may be applied, that removes outliers with extreme disparities . For example, disparities within the range of 5% and 95% percentiles in the histogram may be retained. The local disparity analysis results in local disparity statistics which are then used to determine a suitable disparity for the graphical element.
Referring also to FIG. 4 , the local disparity estimation 630 may alternatively substantially compute the disparities within the intended window for the graphical element. In this case , the disparity estimation produces a local disparity histogram which reduces the computational complexity of the system.
The preferred disparity for the video content of the current frames together with one or more graphical elements 640 may be computed . In this manner, the disparities for the video images are determined and those regions of the image that correspond to graphical elements are likewise determined, both in a manner suitable to reduce viewer discomfort.
The disparity for the local window may be computed in any suitable manner, such as by using a histogram. For example, the disparity associated with the graphical element may be a mean disparity, a most frequent disparity, or a minimal disparity. The mean disparity may be determined as the mean disparity of substantially all the samples in the histogram. The most frequent disparity may be determined as the disparity value associated with the highest count in the histogram. The minimal disparity may be determined as the minimal disparity in the histogram . The minimal disparity in the histogram tends to ensure that the graphical element will be placed "on top of" the underlying content in terms of perceived depth, which is usually what a viewer would expect to observe for a graphical element.
In some cases, the local disparity will vary significantly between temporal frames, especially in the case of a shot boundary in the video . Accordingly, a temporal smoothing 660 of the disparity calculations 650 is preferably used to smooth out such effects. For example, assuming the disparity of the graphical element is determined to be d(t), the system may apply a temporal filter to reduce the disparity from changing to dramatically from frame to frame. As one particular embodiment, an infinite impulse response filter may be used: d(t) = α· d(t-\) + (\- )»d(t) . If a shot boundary is detected, the temporal smoothing may be turned off (or substantially reduced) to avoid false disparity information being leaked from one shot to the next shot, as defined by: a · d(t - \) + (\- a) · d(t) no _ shot _ boundary _at _t
d(t)
d(t) shot _ boundary _at _t
With the disparities for graphical elements being content-dependent on the particular region(s) of the display, together with temporal smoothing, the disparity estimates for rendering the images are determined 660 and the graphical elements are rendered on both the left image and the right image with the appropriate disparity values. For example, the final disparities estimate d(t) for the on-screen graphical element(s) may be rendered and positioned. Assuming the intended spatial location of the left image is [Gx(t), Gy(t)], the same overlay may be placed on the right image as [Gx(t)+ d(t),
Gy(t)]- The opacity of the graphical overlay may also be modified based on its disparity. For example, if there is a strong conflict between the scene disparities and the graphical disparities, the opacity of the graphical element may be reduced which reduces the visual discomfort.
As described above, according to the present invention, a method for displaying a pair of stereoscopic images together with a graphical element on a display can be provided. The method can comprise following four steps (a) to (d) .
(a) receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image;
(b) estimating a disparity between said left image and said right image based upon a matching of a left region of said left image with a right region of said right image;
(c) estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(d) based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element modifying at least one of said right image and said left image to be displayed upon said display.
Also , according to one aspect of the present invention, said stereoscopic images may include a horizontal disparity.
Also , according to one aspect of the present invention, said disparity estimation may provide a LtoR disparity map, a RtoL disparity map, a RtoL disparity matching errors, and LtoR disparity matching errors. .
Also, according to one aspect of the present invention, said LtoR disparity map and said RtoL disparity map may be adjusted such that respective disparity values of the maps are each increased or decreased. In other words, said disparity maps may be further adjusted, where the disparity values are increased or decreased.
Also, according to one aspect of the present invention, said LtoR disparity map and said RtoL disparity map may be adjusted with a same offset value for all the pixels . In other words, said disparity maps may be adjusted with a same offset value for all the pixels.
Also , according to one aspect of the present invention, said LtoR disparity map and said RtoL disparity map may be adjusted with a different offset value for each pixel. In other words, said disparity maps may be adjusted with a different offset value for each pixel.
Also, according to one aspect of the present invention, said adjusted disparity may be further based upon a viewer preference .
Also , according to one aspect of the present invention, said adjusted disparity may be further based upon a model based upon display characteristics of said display.
Also , according to one aspect of the present invention, said estimating said disparity of said graphical element may be performed after said estimating said disparity between said left image and said right image .
Also, according to one aspect of the present invention, said estimating said disparity of said graphical element may be performed before said estimating said disparity between said left image and said right image .
Also, according to one aspect of the present invention, said graphical element may be received without disparity information.
Also , according to one aspect of the present invention, a location to position said graphical element within said image may be received.
Also, according to one aspect of the present invention, said disparity of said graphical element may not be a fixed disparity value .
Also , according to one aspect of the present invention, said disparity of said graphical element may be based upon the disparity proximate said location.
Also , according to one aspect of the present invention, said disparity of said graphical element may be substantially the same as the disparity of said pair of images proximate said location .
Also, according to one aspect of the present invention, said disparity of said graphical element may be substantially the same as the disparity of said pair of images proximate the boundaries of said graphical element.
Also, according to one aspect of the present invention, said disparity of said graphical element may be further based upon a temporal characterization.
Also , according to one aspect of the present invention, said disparity of said graphical element may be based upon a histogram.
Also, according to one aspect of the present invention, said disparity of said graphical element may be further based upon a minimal disparity.
The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
INDUSTRIAL APPLICABILITY
According to the present invention , the discomfort and/ or fatigue for stereoscopic images can be reduced.

Claims

1 . A method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image;
(b) estimating a disparity between said left image and said right image based upon a matching of a left region of said left image with a right region of said right image;
(c) estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(d) based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element modifying at least one of said right image and said left image to be displayed upon said display.
2. The method of claim 1 wherein said stereoscopic images include a horizontal disparity.
3. The method of claim 1 wherein said disparity estimation provides a LtoR disparity map, a RtoL disparity map, a RtoL disparity matching errors, and LtoR disparity matching errors.
4. The method of claim 3 , wherein said LtoR disparity map and said RtoL disparity map are adjusted such that respective disparity values of the maps are each increased or decreased.
5. The method of claim 3 , wherein said LtoR disparity map and said RtoL disparity map are adjusted with a same offset value for all the pixels.
6. The method of claim 3 , wherein said LtoR disparity map and said RtoL disparity map are adjusted with a different offset value for each pixel.
7. The method as claimed in any one of claims 4 to 6, wherein said adjusted disparity is further based upon a viewer preference .
8. The method of claim 7 , wherein said adjusted disparity is further based upon a model based upon display characteristics of said display.
9. The method of claim 1 , wherein said estimating said disparity of said graphical element is performed after said estimating said disparity between said left image and said right image .
10. The method of claim 1 , wherein said estimating said disparity of said graphical element is performed before said estimating said disparity between said left image and said right image .
1 1 . The method of claim 1 , wherein said graphical element is received without disparity information.
12. The method of claim 1 , wherein a location to position said graphical element within said image is received.
13. The method of claim 12 , wherein said disparity of said graphical element is not a fixed disparity value.
14. The method of claim 13 , wherein said disparity of said graphical element is based upon the disparity proximate said location .
15. The method of claim 14 , wherein said disparity of said graphical element is substantially the same as the disparity of said pair of images proximate said location .
16. The method of claim 15 , wherein said disparity of said graphical element is substantially the same as the disparity of said pair of images proximate the boundaries of said graphical element.
17. The method of claim 14 , wherein said disparity of said graphical element is further based upon a temporal characterization .
18. The method of claim 14 , wherein said disparity of said graphical element is based upon a histogram.
19. The method of claim 18 , wherein said disparity of said graphical element is further based upon a minimal disparity.
20. A method for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image together with a disparity between said left image and said right image;
(b) estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(c) based upon said disparity of said pair of images and said estimated disparity of said graphical element modifying at least one of said right image and said left image to be displayed upon said display.
2 1 . A device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) a receiving section for receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image;
(b) a first estimating section for estimating a disparity between said left image and said right image based upon a matching of a left region of said left image with a right region of said right image;
(c) a second estimating section for estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(d) a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said estimated disparity of said pair of images and said estimated disparity of said graphical element.
22. A device for displaying a pair of stereoscopic images together with a graphical element on a display comprising:
(a) a receiving section for receiving a pair of images forming said pair of stereoscopic images, one being a left image and one being a right image together with a disparity between said left image and said right image;
(b) a estimating section for estimating a disparity for a graphical element to be displayed together with said pair of images on said display based upon a disparity of a spatial region of said pair of images;
(c) a modifying section for modifying at least one of said right image and said left image to be displayed upon said display based upon said disparity of said pair of images and said estimated disparity of said graphical element.
PCT/JP2012/055889 2011-03-02 2012-03-01 Method and device for displaying a pair of stereoscopic images Ceased WO2012118231A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US13/039,148 US20120224037A1 (en) 2011-03-02 2011-03-02 Reducing viewing discomfort for graphical elements
US13/039,148 2011-03-02

Publications (1)

Publication Number Publication Date
WO2012118231A1 true WO2012118231A1 (en) 2012-09-07

Family

ID=46753058

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/055889 Ceased WO2012118231A1 (en) 2011-03-02 2012-03-01 Method and device for displaying a pair of stereoscopic images

Country Status (2)

Country Link
US (1) US20120224037A1 (en)
WO (1) WO2012118231A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9071719B2 (en) * 2011-05-11 2015-06-30 I-Cubed Research Center Inc. Image processing apparatus with a look-up table and a mapping unit, image processing method using a look-up table and a mapping unit, and storage medium in which program using a look-up table and a mapping unit is stored
FR2982448A1 (en) * 2011-11-07 2013-05-10 Thomson Licensing STEREOSCOPIC IMAGE PROCESSING METHOD COMPRISING AN INCRUSTABLE OBJECT AND CORRESPONDING DEVICE
US20150347918A1 (en) * 2014-06-02 2015-12-03 Disney Enterprises, Inc. Future event prediction using augmented conditional random field
US20250173828A1 (en) * 2023-11-27 2025-05-29 Samsung Electronics Co., Ltd. Indirective temporal filtering for adaptive joined parameter smoothing

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011004600A1 (en) * 2009-07-10 2011-01-13 パナソニック株式会社 Recording medium, reproducing device, and integrated circuit
EP2278824A1 (en) * 2009-04-21 2011-01-26 Panasonic Corporation Video processing apparatus and video processing method
WO2011016240A1 (en) * 2009-08-05 2011-02-10 パナソニック株式会社 Video reproducing device
JP2011030200A (en) * 2009-06-29 2011-02-10 Sony Corp Stereoscopic image data transmitter, method for transmitting stereoscopic image data, stereoscopic image data receiver, and method for receiving stereoscopic image data
JP2011029849A (en) * 2009-07-23 2011-02-10 Sony Corp Receiving device, communication system, method of combining caption with stereoscopic image, program, and data structure
JP2011035858A (en) * 2009-08-06 2011-02-17 Panasonic Corp Video processing apparatus
JP2011508557A (en) * 2007-12-26 2011-03-10 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Image processor for overlaying graphics objects

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6215898B1 (en) * 1997-04-15 2001-04-10 Interval Research Corporation Data processing system and method
KR20100135032A (en) * 2009-06-16 2010-12-24 삼성전자주식회사 3D image conversion device and method of 2D image
US8290248B2 (en) * 2009-12-31 2012-10-16 Mitsubishi Electric Research Laboratories, Inc. Determining disparity search range in stereo videos

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011508557A (en) * 2007-12-26 2011-03-10 コーニンクレッカ フィリップス エレクトロニクス エヌ ヴィ Image processor for overlaying graphics objects
EP2278824A1 (en) * 2009-04-21 2011-01-26 Panasonic Corporation Video processing apparatus and video processing method
JP2011030200A (en) * 2009-06-29 2011-02-10 Sony Corp Stereoscopic image data transmitter, method for transmitting stereoscopic image data, stereoscopic image data receiver, and method for receiving stereoscopic image data
WO2011004600A1 (en) * 2009-07-10 2011-01-13 パナソニック株式会社 Recording medium, reproducing device, and integrated circuit
JP2011029849A (en) * 2009-07-23 2011-02-10 Sony Corp Receiving device, communication system, method of combining caption with stereoscopic image, program, and data structure
WO2011016240A1 (en) * 2009-08-05 2011-02-10 パナソニック株式会社 Video reproducing device
JP2011035858A (en) * 2009-08-06 2011-02-17 Panasonic Corp Video processing apparatus

Also Published As

Publication number Publication date
US20120224037A1 (en) 2012-09-06

Similar Documents

Publication Publication Date Title
RU2519433C2 (en) Method and system for processing input three-dimensional video signal
US9451242B2 (en) Apparatus for adjusting displayed picture, display apparatus and display method
US8817073B2 (en) System and method of processing 3D stereoscopic image
US9398289B2 (en) Method and apparatus for converting an overlay area into a 3D image
KR101625830B1 (en) Method and device for generating a depth map
RU2554465C2 (en) Combination of 3d video and auxiliary data
TWI528781B (en) Method and apparatus for customizing three-dimensional effects of stereoscopic content
US20140333739A1 (en) 3d image display device and method
US10110872B2 (en) Method and device for correcting distortion errors due to accommodation effect in stereoscopic display
CN102186023B (en) Binocular three-dimensional subtitle processing method
TWI531212B (en) System and method of rendering stereoscopic images
KR20060088549A (en) Motion Control for Rendering Images
US9686526B2 (en) Objective 3D video quality assessment model
US20100091097A1 (en) Supporting A 3D Presentation
JP2007052304A (en) Video display system
CN107155105A (en) A kind of three-dimensional video-frequency Comfort Evaluation method and device
Kim et al. Visual comfort enhancement for stereoscopic video based on binocular fusion characteristics
US20120229600A1 (en) Image display method and apparatus thereof
US20120224037A1 (en) Reducing viewing discomfort for graphical elements
US20140085434A1 (en) Image signal processing device and image signal processing method
Masia et al. Perceptually-optimized content remapping for automultiscopic displays
US20140085413A1 (en) Image signal processing device and image signal processing method
JP6056459B2 (en) Depth estimation data generation apparatus, pseudo stereoscopic image generation apparatus, depth estimation data generation method, and depth estimation data generation program
CN103152591A (en) Image processor, image processing method, display apparatus and computer program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12751810

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12751810

Country of ref document: EP

Kind code of ref document: A1