CA2982015A1 - Method and apparatus for depth enhanced imaging - Google Patents
Method and apparatus for depth enhanced imaging Download PDFInfo
- Publication number
- CA2982015A1 CA2982015A1 CA2982015A CA2982015A CA2982015A1 CA 2982015 A1 CA2982015 A1 CA 2982015A1 CA 2982015 A CA2982015 A CA 2982015A CA 2982015 A CA2982015 A CA 2982015A CA 2982015 A1 CA2982015 A1 CA 2982015A1
- Authority
- CA
- Canada
- Prior art keywords
- depth
- boundaries
- discontinuities
- imaging devices
- enhanced
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/167—Position within a video image, e.g. region of interest [ROI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
- H04N19/98—Adaptive-dynamic-range coding [ADRC]
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
Abstract
A method and apparatus for enhancing quality of images and videos by emphasizing the boundaries between objects at different depths is introduced. Multiple cameras are placed at strategic locations to detect noticeable differences in depth in a scene while an image or video is being captured. The pixel values at the boundaries with noticeable difference in depth are modified to create a sense of depth visually in images or videos without the use of any stereoscopic visualization devices.
Description
Title: Method and Apparatus for Depth Enhanced Imaging (Application#:
2,982,015) Inventors: Irene Cheng and Anup Basu DESCRIPTION
FIELD OF THE INVENTION
The present invention relates to the field of creating images and videos that arc more appealing to a human viewer.
BACKGROUND OF THE INVENTION
When looking at images and videos, humans try to infer information on the three dimensional (3D) relationship among various objects in a scene. In recent years various methods and apparatus have been developed to capture and display objects and scenes in 3D.
However, these approaches require expensive laser or time-of-flight 3D scanners for image capture, as well as 3D displays for visualization. Current stereoscopic 3D displays are unnatural for the human eyes because of their inability to adjust the display based on the convergence of the axes of the human eyes. Instead, displays are designed assuming that the axes of the left and right eyes are parallel.
This incompatibility between stereoscopic 3D displays and the human eyes causes strain and fatigue to human viewers, and is possibly the reason for the lack of wide spread adoption and success of 3D TV and other types of 3D displays. To address this problem, we propose using a regular 2D display with the pixel values on the display modified to enhance the 3D depth discrepancies in the scene.
BRIEF SUMMARY OF THE INVENTION
The present invention comprises of a method and an apparatus for enhancing the perception of depth in a 3D scene using a 2D display monitor. The present invention provides an apparatus for depth enhanced imaging which includes at least five cameras placed in strategic locations to allow for the extraction of boundaries where depth changes by a significant amount.
A computer processor is provided to receive the data regarding the images or videos captured by the aforementioned five or more cameras. The computer processor automatically processes these images or videos to determine the boundaries of objects in the scene where there is perceptually significant change in depth.
The pixel values around the boundaries with significant change in depth are modified to amplify the perception of depth. This is achieved by changing the luminance or brightness of a pixel depending on the depth of an object in the scene that is imaged by this pixel.
In addition, these boundaries can be sharpened using existing image processing algorithms to amplify sudden change in depth.
There are many ways to adjust the pixel values based on depth. Without loss of generality we outline two methods that can be adopted for this purpose. In the first method, the sharpness of objects at different depths can be modified by adjusting the contrast. This can be achieved by blurring parts of the image, with the extent of blur dependent on the depth;
i.e., objects at greater depth being blurred more. In the second method, the brightness of an object can be adjusted based on distance, with objects at closer objects being made brighter than farther objects.
The depth discrepancies on a 2D display can also be highlighted in an interactive manner depending on where a person is looking. This can be achieved by tracking the eye movements of a person to determine the region where a person is looking and adjusting the display to enhance the effect of depth in this region. Eye trackers are now commercially available with high accuracy and low delay, making interactive enhancement through eye tracking feasible in real time, e.g., at 30 frames per second.
Stereo image capture has been used primarily for 3D reconstruction in the past. However, the information acquired from stereo can also be used to add motion to certain objects in a scene.
This approach can be used to make still images more interesting to look at, and simulate a dynamic aspect into them. In addition to image enhancements based on depth, this aspect can also be incorporated into images using the multi-camera capture set up described in this patent.
The invention described is not limited to enhancement of a single image only, but it can also be used to improve the quality of video by simply applying the improvements suggested to every frame of a video sequence.
CT (camera top) terth CR (camera right) CL (camera left) ,rioA
ft-0 CC mera center) =
Obi CB (camera bottom) Figure 1 CT eN
CL CC CR
=NI NNW
o Min MIN
Figure 2 =
ij Figure 3 Figure 4 Multi- Depth Pixel t\ Output camera Discontinuity 7,/>.Value Image or system Estimator Enhancer Video Frame Figure 5 BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is an arrangement of 5 cameras that can detect depth discontinuities along all four sides (top, bottom, left, and right) of an object.
Figure 2 is an arrangement of 5 cameras as in Figure 1, but in a more compact form within a single unit.
Figure 3 is an arrangement of 9 lenses within a single camera box. This arrangement can achieve the effect of the 5 cameras in Figures 1 and 2; in addition, it can achieve better accuracy for depth discontinuities along diagonal directions.
Figure 4 is an arrangement of an arbitrary number of lenses within a single camera box. This arrangement can achieve the benefits of the configurations in Figures 1, 2 and
FIELD OF THE INVENTION
The present invention relates to the field of creating images and videos that arc more appealing to a human viewer.
BACKGROUND OF THE INVENTION
When looking at images and videos, humans try to infer information on the three dimensional (3D) relationship among various objects in a scene. In recent years various methods and apparatus have been developed to capture and display objects and scenes in 3D.
However, these approaches require expensive laser or time-of-flight 3D scanners for image capture, as well as 3D displays for visualization. Current stereoscopic 3D displays are unnatural for the human eyes because of their inability to adjust the display based on the convergence of the axes of the human eyes. Instead, displays are designed assuming that the axes of the left and right eyes are parallel.
This incompatibility between stereoscopic 3D displays and the human eyes causes strain and fatigue to human viewers, and is possibly the reason for the lack of wide spread adoption and success of 3D TV and other types of 3D displays. To address this problem, we propose using a regular 2D display with the pixel values on the display modified to enhance the 3D depth discrepancies in the scene.
BRIEF SUMMARY OF THE INVENTION
The present invention comprises of a method and an apparatus for enhancing the perception of depth in a 3D scene using a 2D display monitor. The present invention provides an apparatus for depth enhanced imaging which includes at least five cameras placed in strategic locations to allow for the extraction of boundaries where depth changes by a significant amount.
A computer processor is provided to receive the data regarding the images or videos captured by the aforementioned five or more cameras. The computer processor automatically processes these images or videos to determine the boundaries of objects in the scene where there is perceptually significant change in depth.
The pixel values around the boundaries with significant change in depth are modified to amplify the perception of depth. This is achieved by changing the luminance or brightness of a pixel depending on the depth of an object in the scene that is imaged by this pixel.
In addition, these boundaries can be sharpened using existing image processing algorithms to amplify sudden change in depth.
There are many ways to adjust the pixel values based on depth. Without loss of generality we outline two methods that can be adopted for this purpose. In the first method, the sharpness of objects at different depths can be modified by adjusting the contrast. This can be achieved by blurring parts of the image, with the extent of blur dependent on the depth;
i.e., objects at greater depth being blurred more. In the second method, the brightness of an object can be adjusted based on distance, with objects at closer objects being made brighter than farther objects.
The depth discrepancies on a 2D display can also be highlighted in an interactive manner depending on where a person is looking. This can be achieved by tracking the eye movements of a person to determine the region where a person is looking and adjusting the display to enhance the effect of depth in this region. Eye trackers are now commercially available with high accuracy and low delay, making interactive enhancement through eye tracking feasible in real time, e.g., at 30 frames per second.
Stereo image capture has been used primarily for 3D reconstruction in the past. However, the information acquired from stereo can also be used to add motion to certain objects in a scene.
This approach can be used to make still images more interesting to look at, and simulate a dynamic aspect into them. In addition to image enhancements based on depth, this aspect can also be incorporated into images using the multi-camera capture set up described in this patent.
The invention described is not limited to enhancement of a single image only, but it can also be used to improve the quality of video by simply applying the improvements suggested to every frame of a video sequence.
CT (camera top) terth CR (camera right) CL (camera left) ,rioA
ft-0 CC mera center) =
Obi CB (camera bottom) Figure 1 CT eN
CL CC CR
=NI NNW
o Min MIN
Figure 2 =
ij Figure 3 Figure 4 Multi- Depth Pixel t\ Output camera Discontinuity 7,/>.Value Image or system Estimator Enhancer Video Frame Figure 5 BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is an arrangement of 5 cameras that can detect depth discontinuities along all four sides (top, bottom, left, and right) of an object.
Figure 2 is an arrangement of 5 cameras as in Figure 1, but in a more compact form within a single unit.
Figure 3 is an arrangement of 9 lenses within a single camera box. This arrangement can achieve the effect of the 5 cameras in Figures 1 and 2; in addition, it can achieve better accuracy for depth discontinuities along diagonal directions.
Figure 4 is an arrangement of an arbitrary number of lenses within a single camera box. This arrangement can achieve the benefits of the configurations in Figures 1, 2 and
3; in addition, it can achieve better accuracy for depth discontinuities along more directions.
Figure 5 is a flow chart of the steps involved in adjusting the value of a pixel based on information on depth discontinuities.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The preferred embodiment, an apparatus for depth enhanced image or video capture will now be described with reference to Figures 1 through 5.
Referring now to Fig. 1 the center camera (CC) captures an image or video, while the left camera (CL) and right camera (CR) are used to determine vertical discontinuities in image or video captured by CC. Similarly, the top camera (CT) and bottom camera (CB) are used to determine the horizontal discontinuities in CC. Combinations of CL, CR, CT, and CB can be used to determine discontinuities in other orientations.
Referring now to Figure 2, an arrangement of the cameras in Figure 1 placed inside an enclosing box is shown. This collection of cameras along with the enclosing box can function as an integrated unit.
Referring now to Figure 3, alternate arrangement of cameras with additional cameras is shown.
Referring now to Figure 4, another alternate arrangement of cameras is shown.
Though only a few alternate arrangements of cameras is shown here, there are many other arrangements that are possible that can facilitate the primary objective of reliably and precisely estimating the depth discontinuities in an image or a frame of a video sequence.
Referring now to Figure 5, a flow chart of the steps involved in creating a depth enhanced image or video frame is shown. The first key step in the method includes detecting locations in an image discontinuities exist in depth. Following this step, the values of pixels on one or both sides of the boundaries with depth discontinuities are altered to enhance the perception of depth by a human viewer. The region on either side of depth discontinuity boundaries where values of pixels are altered can be varied depending on viewer preference and comfort.
The purpose of altering the pixel values around depth discontinuities is to make viewers clearly perceive that different objects and areas of the scene are at different depths without using aids 3D displays or stereo glasses that help in depth perception from multiple images.
Note that without loss of generality we use the term "frame of video" to mean "a single still frame from a video sequence."
It will be apparent to one skilled in the art that modifications may be made to the illustrated embodiment without departing from the spirit and scope of the invention as hereinafter defined in the claims.
Figure 5 is a flow chart of the steps involved in adjusting the value of a pixel based on information on depth discontinuities.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The preferred embodiment, an apparatus for depth enhanced image or video capture will now be described with reference to Figures 1 through 5.
Referring now to Fig. 1 the center camera (CC) captures an image or video, while the left camera (CL) and right camera (CR) are used to determine vertical discontinuities in image or video captured by CC. Similarly, the top camera (CT) and bottom camera (CB) are used to determine the horizontal discontinuities in CC. Combinations of CL, CR, CT, and CB can be used to determine discontinuities in other orientations.
Referring now to Figure 2, an arrangement of the cameras in Figure 1 placed inside an enclosing box is shown. This collection of cameras along with the enclosing box can function as an integrated unit.
Referring now to Figure 3, alternate arrangement of cameras with additional cameras is shown.
Referring now to Figure 4, another alternate arrangement of cameras is shown.
Though only a few alternate arrangements of cameras is shown here, there are many other arrangements that are possible that can facilitate the primary objective of reliably and precisely estimating the depth discontinuities in an image or a frame of a video sequence.
Referring now to Figure 5, a flow chart of the steps involved in creating a depth enhanced image or video frame is shown. The first key step in the method includes detecting locations in an image discontinuities exist in depth. Following this step, the values of pixels on one or both sides of the boundaries with depth discontinuities are altered to enhance the perception of depth by a human viewer. The region on either side of depth discontinuity boundaries where values of pixels are altered can be varied depending on viewer preference and comfort.
The purpose of altering the pixel values around depth discontinuities is to make viewers clearly perceive that different objects and areas of the scene are at different depths without using aids 3D displays or stereo glasses that help in depth perception from multiple images.
Note that without loss of generality we use the term "frame of video" to mean "a single still frame from a video sequence."
It will be apparent to one skilled in the art that modifications may be made to the illustrated embodiment without departing from the spirit and scope of the invention as hereinafter defined in the claims.
Claims (7)
1. A method and apparatus for depth enhanced imaging, comprising the steps of:
providing at least five imaging devices;
providing a method for computing boundaries with depth discontinuities between any two images acquired by two different imaging devices;
modifying the values of pixels on the boundaries with depth discontinuity to enhance depth perception on a two dimensional video display;
transmitting the depth enhanced video for display or storing it for later viewing.
providing at least five imaging devices;
providing a method for computing boundaries with depth discontinuities between any two images acquired by two different imaging devices;
modifying the values of pixels on the boundaries with depth discontinuity to enhance depth perception on a two dimensional video display;
transmitting the depth enhanced video for display or storing it for later viewing.
2. The apparatus according to Claim 1, wherein the imaging devices are stored in a single enclosure and acts as an integrated device.
3. The method and apparatus according to Claim 1, wherein additional imaging devices are included for higher accuracy in computing boundaries with depth discontinuities.
4. The method and apparatus according to Claim 1, wherein input from imaging devices are used to create a 3D animation of an object of interest.
5. The method and apparatus according to Claim 1, wherein the image characteristics, such as the focus and contrast, are adjusted based on the input from an eye tracker to determine the gaze of a human observer.
6. The method and apparatus according to Claim 1, wherein the depth discontinuities are enhanced by modifying the contrast around boundaries with depth discontinuities.
7. The method and apparatus according to Claim 1, wherein the depth discontinuities are enhanced by modifying the focus around boundaries with depth discontinuities.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA2982015A CA2982015A1 (en) | 2017-10-11 | 2017-10-11 | Method and apparatus for depth enhanced imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA2982015A CA2982015A1 (en) | 2017-10-11 | 2017-10-11 | Method and apparatus for depth enhanced imaging |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2982015A1 true CA2982015A1 (en) | 2019-04-11 |
Family
ID=66096695
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA2982015A Abandoned CA2982015A1 (en) | 2017-10-11 | 2017-10-11 | Method and apparatus for depth enhanced imaging |
Country Status (1)
| Country | Link |
|---|---|
| CA (1) | CA2982015A1 (en) |
-
2017
- 2017-10-11 CA CA2982015A patent/CA2982015A1/en not_active Abandoned
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3035681B1 (en) | Image processing method and apparatus | |
| US8736667B2 (en) | Method and apparatus for processing video images | |
| US8766973B2 (en) | Method and system for processing video images | |
| JP4762994B2 (en) | Parallax map | |
| US20110228051A1 (en) | Stereoscopic Viewing Comfort Through Gaze Estimation | |
| CN105894567B (en) | Scaling pixel depth values of user-controlled virtual objects in a three-dimensional scene | |
| US20100002073A1 (en) | Blur enhancement of stereoscopic images | |
| US8982187B2 (en) | System and method of rendering stereoscopic images | |
| CN108141578A (en) | Camera is presented | |
| EP2517472B1 (en) | Method and apparatus for optimal motion reproduction in stereoscopic digital cinema | |
| Hanhart et al. | Subjective evaluation of two stereoscopic imaging systems exploiting visual attention to improve 3D quality of experience | |
| US10554954B2 (en) | Stereoscopic focus point adjustment | |
| Benzeroual et al. | 3D display size matters: Compensating for the perceptual effects of S3D display scaling | |
| EP2852149A1 (en) | Method and apparatus for generation, processing and delivery of 3D video | |
| KR101172507B1 (en) | Apparatus and Method for Providing 3D Image Adjusted by Viewpoint | |
| US9591290B2 (en) | Stereoscopic video generation | |
| US9693042B2 (en) | Foreground and background detection in a video | |
| CA2982015A1 (en) | Method and apparatus for depth enhanced imaging | |
| KR101939243B1 (en) | Stereoscopic depth adjustment and focus point adjustment | |
| KR101907127B1 (en) | Stereoscopic video zooming and foreground and background detection in a video | |
| US9674500B2 (en) | Stereoscopic depth adjustment | |
| US9609313B2 (en) | Enhanced 3D display method and system | |
| US9661309B2 (en) | Stereoscopic video zooming | |
| Doyen et al. | Towards a free viewpoint and 3D intensity adjustment on multi-view display |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| FZDE | Dead |
Effective date: 20201013 |