Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/128—Adjusting depth or disparity
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T15/00—Three-dimensional [3D] image rendering
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/70—Denoising; Smoothing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

• Implementations are described that relate to coding systems. Various particular implementations relate to joint depth estimation.
• Three-dimensional video (3DV) is a framework directed to generating high-quality 3D rendering at the receiver side. This enables 3D visual experiences with auto-stereoscopic displays, free-view point applications, and stereoscopic displays.
• 3DV applications a reduced number of video views and corresponding depth maps, typically referred as multi- view plus depth (MVD), are transmitted or stored due to limitations in transmission bandwidth and/or storage constraints.
• MMD multi- view plus depth
• At the receiver side additional views are rendered using available views and depth maps.
• a first-view depth indicator for a location in a first view is estimated, and a second-view depth indicator for a corresponding location in a second view is estimated.
• the estimating of one or more of the first-view depth indicator and the second-view depth indicator is based on a constraint.
• the constraint provides a relationship between the first-view depth indicator and the second-view depth indicator for corresponding locations.
• implementations may be configured or embodied in various manners.
• an implementation may be performed as a method, or embodied as apparatus, such as, for example, an apparatus configured to perform a set of operations or an apparatus storing instructions for performing a set of operations, or embodied in a signal.
• apparatus such as, for example, an apparatus configured to perform a set of operations or an apparatus storing instructions for performing a set of operations, or embodied in a signal.
• Figure 1 is an example of a left depth map from a left reference view.
• Figure 2 is an example of a right depth map from a right reference view.
• Figure 3 is a diagram of an implementation of a depth estimator.
• Figure 4 is a diagram of an implementation of a video transmission system.
• Figure 5 is a diagram of an implementation of a video receiving system.
• Figure 6 is a diagram of an implementation of a video processing device.
• Figure 7 is a diagram of an implementation of a first depth estimation process.
• Figure 8 is a diagram of an implementation of a second depth estimation process.
• Rendering may be performed using, for example, the technique of Depth Image Based Rendering (DIBR), which takes the transmitted/stored views (reference views) and the associated per-pixel depth maps as input.
• DIBR Depth Image Based Rendering
• Such input may be, for example, provided by an MVD format.
• Depth may be captured using any of a variety of techniques. However, often only video is provided and depth is estimated. To obtain depth maps, depth estimation techniques are often used to find correspondence among different views.
• noisy depth maps may be inaccurate because they do not have the correct depth value.
• a sequence of noisy depth maps can have inaccurate values that are not consistently inaccurate. For example, an object at constant actual depth may be inaccurately estimated in a first picture with a low depth and inaccurately estimated in a second picture with a high depth.
• the joint depth estimation performs depth estimation for multiple reference views in a joint process, rather than independently.
• the joint depth estimation obtains depth maps of different views that are more consistent, leading to better quality in the DIBR rendered views.
• two depth maps are generated based on an intermediate view.
• the implementation develops a constraint that provides a relationship between corresponding locations in the two depth maps. This constraint can be used in various ways to provide consistency between the two depth maps.
• the joint depth estimation involves a summing operation performed on respective disparity estimation costs (distortions) for disparities between corresponding pixel locations in two or more views.
• the joint depth estimation involves the use of a summing operation based on camera distances.
• the summing operation is weighted based on the camera distances.
• Figure 1 shows an exemplary left depth map generated for a left reference view corresponding to an MPEG test sequence known as "Leaving Laptop”, to which the present principles may be applied in accordance with an embodiment of the present principles.
• Figure 2 shows an exemplary right depth map generated for a right reference view corresponding to the MPEG test sequence known as "Leaving Laptop”, to which the present principles may be applied in accordance with an embodiment of the present principles.
• the depth levels in the depth map pair can be very different.
• the difference can be larger than 40 in the examples of Figures 1 and 2. In such instances, the ideal difference is zero, but the observed difference is, for example, larger than 40. This is an example of views that do not have consistent depth maps.
• depth estimation algorithms may typically be divided into the following three categories: single view; stereopsis; and multiple views. All three categories assume that no depth map is known, and use the video from one or more views to generate a depth map.
• the camera focus is considered as a mean for depth estimation.
• a method referred to as "depth-from-focus" may be used to determine the depth based on the amount of defocus or blurring. This method may not be very reliable because, for example, focus estimation often does not provide good results.
• a stereopsis method may use a pair of views as input to estimate the depth maps for one of the views.
• area-based stereo matching methods such methods typically match neighboring pixel values within a window between the two images. It is typically critical to select an appropriate window size.
• the window size and shape can be iteratively changed based on the local variation of the intensity and current depth estimates.
• Some global constraints may be applied to produce a dense depth map, that is, a depth map having unique values and being continuous almost everywhere.
• One possible approach is to use three views as inputs (a left view, a center view, and a right view) in order to estimate the depth for the center view.
• the aim is to generate a single dense depth map sequence.
• the depth estimation is performed for each target view independently, regardless of the category of the depth estimation algorithm. Hence, inconsistency across the views is likely to be present.
• FIG. 3 shows an exemplary depth estimator 300 to which the present principles may be applied, in accordance with an embodiment of the present principles.
• the depth estimator 300 receives a target view 1, a target view 2, and one or more reference views as inputs.
• the depth estimator 300 provides the estimated depth of target view 1 and the depth of target view 2 as outputs. The operation of the depth estimator 300 is described in further detail herein below.
• the depth estimator 300 provides an estimated depth indicator for target view 1 and an estimated depth indicator for target view 2.
• a depth indicator may be a depth value, or an entire depth map. But a depth indicator may alternatively be, for example, a disparity value, or an entire disparity map. References to depth in the implementations and descriptions that follow are intended to include other depth indicators, such as, for example, disparity.
• the depth indicator may provide a depth indication for, for example, an entire target view or a location in a target view.
• the location may be, for example, a particular pixel, a partition, a sub-macroblock, a macroblock, a slice, or a field.
• FIG. 4 shows an exemplary video transmission system 400, to which the present principles may be applied, in accordance with an implementation of the present principles.
• the video transmission system 400 may be, for example, a head-end or transmission system for transmitting a signal using any of a variety of media, such as, for example, satellite, cable, telephone-line, or terrestrial broadcast.
• the transmission may be provided over the Internet or some other network.
• the video transmission system 400 is capable of generating and delivering compressed video with depth. This is achieved by generating an encoded signal(s) including depth information or information capable of being used to synthesize the depth information at a receiver end that may, for example, have a decoder.
• the video transmission system 400 includes an encoder 410 and a transmitter 420 capable of transmitting the encoded signal.
• the encoder 410 receives video information and generates an encoded signal(s) with depth.
• the encoder 410 may include sub-modules, including for example an assembly unit for receiving and assembling various pieces of information into a structured format for storage or transmission.
• the various pieces of information may include, for example, coded or uncoded video, coded or uncoded depth information, and coded or uncoded elements such as, for example, motion vectors, coding mode indicators, and syntax elements.
• the transmitter 420 may be, for example, adapted to transmit a program signal having one or more bitstreams representing encoded pictures and/or information related thereto. Typical transmitters perform functions such as, for example, one or more of providing error- correction coding, interleaving the data in the signal, randomizing the energy in the signal, and modulating the signal onto one or more carriers.
• the transmitter may include, or interface with, an antenna (not shown). Accordingly, implementations of the transmitter 420 may include, or be limited to, a modulator.
• the video transmission system 400 may also be included, in whole or part, in a variety of user devices.
• Such devices include, for example, a cell phone, a laptop or other computer, and a camcorder.
• FIG. 5 shows an exemplary video receiving system 500 to which the present principles may be applied, in accordance with an embodiment of the present principles.
• the video receiving system 500 may be configured to receive signals over a variety of media, such as, for example, satellite, cable, telephone-line, or terrestrial broadcast. The signals may be received over the Internet or some other network.
• the video receiving system 500 may be, for example, a cell-phone, a computer, a set- top box, a television, or other device that receives encoded video and provides, for example, decoded video for display to a user or for storage.
• the video receiving system 500 may provide its output to, for example, a screen of a television, a computer monitor, a computer (for storage, processing, or display), or some other storage, processing, or display device.
• the video receiving system 500 is capable of receiving and processing video content including video information.
• the video receiving system 500 includes a receiver 510 capable of receiving an encoded signal, such as for example the signals described in the implementations of this application, and a decoder 520 capable of decoding the received signal.
• the receiver 510 may be, for example, adapted to receive a program signal having a plurality of bitstreams representing encoded pictures. Typical receivers perform functions such as, for example, one or more of receiving a modulated and encoded data signal, demodulating the data signal from one or more carriers, de-randomizing the energy in the signal, de-interleaving the data in the signal, and error-correction decoding the signal.
• the receiver 510 may include, or interface with, an antenna (not shown). Implementations of the receiver 510 may include, or be limited to, a demodulator.
• the decoder 520 outputs video signals including, for example, video information and depth information.
• FIG. 6 shows an exemplary video processing device 600 to which the present principles may be applied, in accordance with an embodiment of the present principles.
• the video processing device 600 may be, for example, a set top box or other device that receives encoded video and provides, for example, decoded video for display to a user or for storage.
• the video processing device 600 may provide its output to a television, computer monitor, or a computer or other processing device.
• the video processing device 600 includes a front-end (FE) device 605 and a decoder 610.
• the front-end device 605 may be, for example, a receiver adapted to receive a program signal having a plurality of bitstreams representing encoded pictures, and to select one or more bitstreams for decoding from the plurality of bitstreams.
• Typical receivers perform functions such as, for example, one or more of receiving a modulated and encoded data signal, demodulating the data signal, decoding one or more encodings (for example, channel coding and/or source coding) of the data signal, and/or error-correcting the data signal.
• the front-end device 605 may receive the program signal from, for example, an antenna (not shown).
• the front-end device 605 provides a received data signal to the decoder 610.
• the decoder 610 receives a data signal 620.
• the data signal 620 may include, for example, one or more Advanced Video Coding (AVC), Scalable Video Coding (SVC), or Multi-view Video Coding (MVC) compatible streams.
• AVC Advanced Video Coding
• SVC Scalable Video Coding
• MVC Multi-view Video Coding
• AVC refers more specifically to the existing International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 Recommendation (hereinafter the "H.264/MPEG-4 AVC Standard” or variations thereof, such as the “AVC standard” or simply "AVC”).
• ISO/IEC International Organization for Standardization/International Electrotechnical Commission
• MPEG-4 Moving Picture Experts Group-4
• AVC Advanced Video Coding
• ITU-T International Telecommunication Union, Telecommunication Sector
• H.264/MPEG-4 AVC Standard H.264 Recommendation
• MVC refers more specifically to a multi-view video coding ("MVC") extension (Annex H) of the AVC standard, referred to as H.264/MPEG-4 AVC, MVC extension (the "MVC extension” or simply "MVC").
• MVC multi-view video coding
• SVC refers more specifically to a scalable video coding ("SVC") extension (Annex G) of the AVC standard, referred to as H.264/MPEG-4 AVC, SVC extension (the “SVC extension” or simply "SVC”).
• SVC scalable video coding
• the decoder 610 decodes all or part of the received signal 620 and provides as output a decoded video signal 630.
• the decoded video 630 is provided to a selector 650.
• the device 600 also includes a user interface 660 that receives a user input 670.
• the user interface 660 provides a picture selection signal 680, based on the user input 670, to the selector 650.
• the picture selection signal 680 and the user input 670 indicate which of multiple pictures, sequences, scalable versions, views, or other selections of the available decoded data a user desires to have displayed.
• the selector 650 provides the selected picture(s) as an output 690.
• the selector 650 uses the picture selection information 680 to select which of the pictures in the decoded video 630 to provide as the output 690.
• the selector 650 includes the user interface 660, and in other implementations no user interface 660 is needed because the selector 650 receives the user input 670 directly without a separate interface function being performed.
• the selector 650 may be implemented in software or as an integrated circuit, for example.
• the selector 650 is incorporated with the decoder 610, and in another implementation, the decoder 610, the selector 650, and the user interface 660 are all integrated.
• front-end 605 receives a broadcast of various television shows and selects one for processing. The selection of one show is based on user input of a desired channel to watch. Although the user input to front-end device 605 is not shown in Figure 6, front-end device 605 receives the user input 670.
• the front-end 605 receives the broadcast and processes the desired show by demodulating the relevant part of the broadcast spectrum, and decoding any outer encoding of the demodulated show.
• the front-end 605 provides the decoded show to the decoder 610.
• the decoder 610 is an integrated unit that includes devices 660 and 650. The decoder 610 thus receives the user input, which is a user-supplied indication of a desired view to watch in the show.
• the decoder 610 decodes the selected view, as well as any required reference pictures from other views, and provides the decoded view 690 for display on a television (not shown).
• the user may desire to switch the view that is displayed and may then provide a new input to the decoder 610.
• the decoder 610 decodes both the old view and the new view, as well as any views that are in between the old view and the new view. That is, the decoder 610 decodes any views that are taken from cameras that are physically located in between the camera taking the old view and the camera taking the new view.
• the front-end device 605 also receives the information identifying the old view, the new view, and the views in between. Such information may be provided, for example, by a controller (not shown in Figure 6) having information about the locations of the views, or the decoder 610.
• Other implementations may use a front-end device that has a controller integrated with the front- end device.
• the decoder 610 provides all of these decoded views as output 690.
• a post-processor (not shown in Figure 6) interpolates between the views to provide a smooth transition from the old view to the new view, and displays this transition to the user. After transitioning to the new view, the post-processor informs (through one or more communication links not shown) the decoder 610 and the front-end device 605 that only the new view is needed. Thereafter, the decoder 610 only provides as output 690 the new view.
• the system 600 may be used to receive multiple views of a sequence of images, and to present a single view for display, and to switch between the various views in a smooth manner.
• the smooth manner may involve interpolating between views to move to another view.
• the system 600 may allow a user to rotate an object or scene, or otherwise to see a three-dimensional representation of an object or a scene.
• the rotation of the object for example, may correspond to moving from view to view, and interpolating between the views to obtain a smooth transition between the views or simply to obtain a three-dimensional representation. That is, the user may "select" an interpolated view as the "view" that is to be displayed.
• disparity (d) between two views and object depth (z) are convertible using the follow equation:
• Equation (l ) /is the focal length of the camera lens, / is the baseline spacing (also known as camera distance), and du is the difference in the principal point offset.
• du indicates the difference between the center of the image and the center of the optical system, for a given camera /. That is, u, is the offset of the center of the image from the center of the optical center, for camera /. Then du, or more specifically du tJ , is the difference between u, and U j , where / andy are two cameras/views.
• Disparity refers to the shift or translation between a location in one view and a corresponding location in another view.
• Depth refers to the distance from the camera plane to the object in the video. Both disparity and depth can vary from pixel to pixel. Depth is a value that applies to a single view. Additionally, focal length applies to a single view. However, disparity and du are relative values that describe a relationship between corresponding pixels within two views. Additionally, / describes the distance between two cameras, or more generally / descries a relationship (distance) between two views (cameras are at the view locations).
• disparity (d) is first estimated and then converted to depth (z), because disparity is the correspondence between two views and can be identified by searching for the most similar pixel in the other view(s) with a cost function (also referred to as a distortion) such as mean-squared error (MSE) or sum of absolute difference (SAD), and so forth.
• MSE mean-squared error
• SAD sum of absolute difference
• Equation (2) is as follows:
• Equation (2) can be replaced with a more complicated form, as explained below.
• the relationship between du and dj 4 can still be determined (that is, there are constraints that dj 2 and d ⁇ 4 should satisfy).
• dn and d ⁇ 4 should satisfy certain constraints can be used for a joint depth estimation for a depth map of view 2 and a depth map of view 4.
• Equation (3) can be rearranged as:
• Equation (5) can be rearranged as:
• Equation (7) can be rearranged as:
• Equation (8) simplifies to:
• Equation (9) simplifies to equation (2) if the du values are zero, and the cameras are equally spaced. Equation (9) does illustrate, however, a slightly more general constraint between du and dn- Of course, other constraints are also possible and envisioned.
• Both of equations (2) and (9) describe a constraint that provides a relationship between a depth indicator (disparity) for one view (for example, d ⁇ i) and a depth indicator for another view (for example, ⁇ &,/), for corresponding locations in the two views.
• the locations typically refer to actual pixels.
• the depth indicator in equations (2) and (9) may, equivalently, be represented as depth.
• the constraint is based on one or more camera parameters, such as, for example, du34, du3 2 , 1 34 , and I3 2 .
• du3 2 and du 34 can be derived from camera parameters by taking the difference in principal point offsets of two views. Principal point offset is one of the intrinsic camera parameters. Additionally,/(focal length of the camera lens) is also an intrinsic camera parameter, and may be part of a constraint in other implementations.
• I 34 and I 32 baseline camera spacings can calculated from extrinsic camera parameters.
• Step 1 For a current pixel in view 3, ⁇ 2 and J ⁇ are estimated and stored respectively. If the two disparities cfo and rf? 4 satisfy (or almost satisfy within a given threshold) the relationship in Equation (2), indicating that both are reliable, then the two disparities will be updated, for example using Equation (10) as set forth below. Then the process moves on to the next pixel and performs from Step 1. Equation ( 10) is as follows:
• Estimating a disparity can be performed in various ways.
• a block-based comparison is performed between a block centered on the current pixel in view 3 and a similarly sized block in view 2 centered on a selected pixel.
• the selected pixel reflects a particular disparity.
• the block-based comparison may be a computation of MSE of SAD, for example, and a distortion (or cost) is determined for the particular disparity.
• the selected pixel (and, therefore, the particular disparity) can then be changed, and a new block-based comparison can be performed and a new distortion can be determined.
• a series of possible disparities can be evaluated, and the disparity resulting in the lowest distortion can be selected as the estimate for d 32 .
• Step 2 relates to the case when d 32 and d 34 do not satisfy Equation (2) within a given threshold.
• the given threshold may be, for example, an offset or a scale factor.
• the estimated value of d 32 must be within four pixels (a threshold offset) of the constrained value of d 32 (as predicted, for example, by Equation (2) or Equation (9)).
• the estimated value of d 32 must be within five percent (a scale factor threshold) of the constrained value of d 32 (as predicted, for example, by Equation (2) or Equation (9)).
• E 32 and E 34 represent the estimation cost (distortion) for d 32 and d 34 , respectively.
• the MSE between pixels is a typical estimation cost for disparity estimation. That is, because the estimates are not perfect, there is a cost (penalty) in rendering new views based on the estimated disparity.
• other cost functions can also be utilized, while maintaining the spirit of the present principles. Since we already know that at least one of the estimations of d 32 and d 34 is not reliable as it/they does/do not satisfy (2), if one of E 32 and E 34 is significantly larger than the other, it is likely that the disparity estimation with the larger estimation cost has failed to find a good matching pixel.
• the determination of whether one of E) 2 and E 34 is significantly larger than the other is made by determining whether the difference between E 32 and Eu is larger than a given threshold.
• Equation (2) will be checked again as well as E 32 and E 34 to determine if we can proceed to Step 3.
• the number of iterations should be monitored to avoid an infinite estimation loop. If a given number of iterations has been performed but the process still cannot find reliable disparity values, then a hole pixel will be marked for each disparity map (view 2 and view 4) and the method proceeds to Step 3.
• thresholds to determine whether the disparities adequately satisfy the constraint.
• other implementations assign a confidence to the pair of disparity values in one or more of these scenarios. Based on the measure of confidence, any of a number of actions may be taken. Some implementations provide the measure of confidence to a user or a rendering device.
• Step 3 If there are remaining pixels to be processed, then move on to the next pixel and go to Step 1. Otherwise, go to Step 4.
• Step 4 Hole-filling: If there are hole-pixels in the disparity maps of view 2 and/or view 4, the hole-pixels are being filled in this step.
• the holes can be filled using some interpolation algorithm based on the neighboring disparity pixels.
• the hole can just be filled using the neighboring depth level that is further away (background preferred) from the cameras (that is, simply select the smaller disparity / larger depth).
• the above process of building a disparity determines disparity for locations in view 2 that correspond to locations in view 3.
• the loop is taken over locations in view 3. Accordingly, there may be locations in view 2 for which no location in view 3 has a corresponding location. Such locations in view 2 are simply retained as holes in the view 2 disparity map. Conversely, multiple locations in view 3 may produce a disparity that maps to the same location in view 2. In such cases, the larger disparity (smaller depth) is preferred because it indicates a foreground object.
• FIG. 7 shows an exemplary method 700 for estimating depth in accordance with an embodiment of the present principles.
• Method 700 may be performed, for example, by depth estimator 300 of FIG. 3.
• the view numbers in Figure 7 and the following text are simply for illustration/explanation purposes.
• the general concept can be applied to jointly estimate depth map of two views.
• pixel / is set to the next pixel in view 3.
• disparity estimation is performed on target view 2 to get disparity d 32 .
• disparity estimation is performed on target view 4 to get disparity d 34 .
• step 725 dn and d ⁇ 4 are updated as per Equation (10). Note that this step 725 provides consistency for the corresponding disparity values for depth maps for views 2 and 4.
• step 730 it is determined whether or not the estimation costs E 32 and E 34 are close to each other (for example, within a threshold amount). If so, then the method proceeds to step 735. Otherwise, the method proceeds to step 760.
• step 735 the disparity search parameters are updated.
• step 745 hole pixels in the two disparity maps view 2 and view 4 are marked.
• step 760 a hole pixel in the disparity map with larger disparity error is marked.
• step 750 it is determined whether or not there are more pixels to be processed in view 3. If so, then the method returns to step 705. Otherwise, the method proceeds to step 755. At step 755, hole pixels, if any, in the disparity maps are filled.
• Equation (2) is used to design a joint disparity search process. For every candidate disparity d 32 , we have a corresponding disparity d 34 based on Equation (2) and thus a joint estimation cost function from d 32 and d 34 can be calculated.
• Advanced metrics such as, for example, a weighted sum based on camera distances can be used as alternative metrics.
• Different disparity pairs are evaluated and the one resulting in the lowest estimation cost E Jo ⁇ n , is selected.
• the different disparity pairs are generated in a manner similar to Embodiment 1 , which loops over a predetermined range of possible disparity values for dn- For each pixel, the selected disparity in this embodiment will produce disparity vectors that satisfy Equation (2), leading to consistency in the two depth maps.
• FIG. 8 shows another exemplary method 800 for estimating depth in accordance with an embodiment of the present principles.
• Method 800 may be performed, for example, by depth estimator 300 of FIG. 3.
• the view numbers in Figure 8 and the following text are simply for illustration/explanation purposes.
• the general concept can be applied to jointly estimate depth map of two views.
• pixel / is set to the next pixel in view 3.
• E mm is set to INT MAX, which is the largest available integer.
• du is calculated based on Equation (2).
• the joint estimation error E is calculated based on d 32 and d 34 .
• step 825 it is determined whether or not E is smaller than E mm . If so, then the method proceeds to step 830. Otherwise, the method proceeds to step 835. At step 830, d ⁇ best is set to d 32 , is set to ds4, and E mm is set to E. The method proceeds to step 835. At step 835, it is determined whether or not there exists any more candidate disparity to be evaluated. If so, then the method returns to step 815. Otherwise, the method proceeds to step 840. At step 840, it is determined whether or not there are any more pixels in view 3. If so, then the method returns to step 805. Otherwise, the method is terminated.
• Embodiments 1 and 2 are used together in some implementations. For example, although embodiments 1 and 2 may work equally well in many or all conditions, it is possible that embodiment 2 may work better than embodiment 1 if there are few holes. Accordingly, an implementation uses embodiment 1 in regions where holes are expected, and uses embodiment 2 elsewhere.
• implementations can be performed at an encoder or a decoder.
• depth estimation is performed at an encoder (or, for example, a preprocessor) and then the estimated depth is transmitted with, or without, the video.
• video is transmitted without any depth, and a receiver performs the depth estimation.
• various implementations transmit information about camera parameters.
• one or more constraints are transmitted so that the receiver knows the constraints to use in estimating multiple depth maps jointly.
• a standardized format is used in transmission to encode and transmit the number of cameras, the spacing between the cameras, the du values, and the focal lengths.
• the system encodes and transmits only the spacing between cameras for the determination of the constraints.
• joint depth estimation for multiple depth maps may be performed in which the multiple maps are from different views, or from the same view.
• the multiple maps may be from the same view at different times.
• implementations may jointly estimate depth maps in various relative positions with respect to a common video picture from a given view. For example, several implementations jointly estimate depth maps for views 1 and 2, using video from view 3.
• view 3 is positioned between views 1 and 2.
• view 3 is positioned as the left-most view of views 1-3.
• view 3 is positioned as the right-most view of views 1 -3.
• implementations jointly estimate three or more depth maps.
• one or more constraints are determined that provide a relationship among the three or more depth maps.
• view 3 in the examples above may be used to determine the depth maps not only for views 2 and 4, but also for view 5. It is straightforward to determine the relationship between di 2 , d 34 , and djs using the above- derived equations.
• Implementations may signal information using a variety of techniques including, but not limited to, SEI messages, slice headers, other high level syntax, non-high-level syntax, out-of-band information, datastream data, and implicit signaling. Accordingly, although implementations described herein may be described in a particular context, such descriptions should in no way be taken as limiting the features and concepts to such implementations or contexts.
• such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C).
• This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.
• a picture and “image” are used interchangeably and refer to a still image or a picture from a video sequence.
• a picture may be a frame or a field.
• the implementations described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program).
• An apparatus may be implemented in, for example, appropriate hardware, software, and firmware.
• the methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs”), and other devices that facilitate communication of information between end-users.
• PDAs portable/personal digital assistants
• Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding and decoding.
• equipment include an encoder, a decoder, a post-processor processing output from a decoder, a pre-processor providing input to an encoder, a video coder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cell phone, a PDA, and other communication devices.
• the equipment may be mobile and even installed in a mobile vehicle.
• the methods may be implemented by instructions being performed by a processor, and such instructions (and/or data values produced by an implementation) may be stored on a processor-readable medium such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette, a random access memory ("RAM"), or a read-only memory (“ROM").
• the instructions may form an application program tangibly embodied on a processor-readable medium. Instructions may be, for example, in hardware, firmware, software, or a combination. Instructions may be found in, for example, an operating system, a separate application, or a combination of the two.
• a processor may be characterized, therefore, as, for example, both a device configured to carry out a process and a device that includes a processor-readable medium (such as a storage device) having instructions for carrying out a process. Further, a processor-readable medium may store, in addition to or in lieu of instructions, data values produced by an implementation.
• implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted.
• the information may include, for example, instructions for performing a method, or data produced by one of the described implementations.
• a signal may be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax-values written by a described embodiment.
• Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal.
• the formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream.
• the information that the signal carries may be, for example, analog or digital information.
• the signal may be transmitted over a variety of different wired or wireless links, as is known.
• the signal may be stored on a processor-readable medium.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Signal Processing (AREA)
• Multimedia (AREA)
• Computer Graphics (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Measurement Of Velocity Or Position Using Acoustic Or Ultrasonic Waves (AREA)
• Processing Or Creating Images (AREA)
• Image Processing (AREA)
• Radar Systems Or Details Thereof (AREA)

Priority Applications (5)

Applications Claiming Priority (2)

Publications (2)

Family

ID=43309122

Family Applications (1)

Country Status (7)

Cited By (6)

Families Citing this family (18)

Citations (3)

Family Cites Families (57)

• 2009
  • 2009-12-16 BR BRPI0924045A patent/BRPI0924045A2/pt not_active IP Right Cessation
  • 2009-12-16 KR KR1020117015663A patent/KR20110126103A/ko not_active Ceased
  • 2009-12-16 WO PCT/US2009/006568 patent/WO2010144074A1/en not_active Ceased
  • 2009-12-16 CN CN200980153983.XA patent/CN102272778B/zh not_active Expired - Fee Related
  • 2009-12-16 JP JP2012500763A patent/JP5607716B2/ja not_active Expired - Fee Related
  • 2009-12-16 EP EP09845922.5A patent/EP2377074A4/en not_active Withdrawn
  • 2009-12-16 US US13/143,296 patent/US8913105B2/en not_active Expired - Fee Related

Patent Citations (3)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events