WO2008152607A1 - Procede, appareil, systeme et programme informatique de propagation d'informations relatives a la profondeur - Google Patents

Procede, appareil, systeme et programme informatique de propagation d'informations relatives a la profondeur Download PDF

Info

Publication number
WO2008152607A1
WO2008152607A1 PCT/IB2008/052340 IB2008052340W WO2008152607A1 WO 2008152607 A1 WO2008152607 A1 WO 2008152607A1 IB 2008052340 W IB2008052340 W IB 2008052340W WO 2008152607 A1 WO2008152607 A1 WO 2008152607A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
related information
depth
segmentation
information
Prior art date
Application number
PCT/IB2008/052340
Other languages
English (en)
Inventor
Vasanth Philomin
Fang Liu
Chunfeng Shen
Original Assignee
Koninklijke Philips Electronics N.V.
Philips Intellectual Property & Standards Gmbh
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips Electronics N.V., Philips Intellectual Property & Standards Gmbh filed Critical Koninklijke Philips Electronics N.V.
Publication of WO2008152607A1 publication Critical patent/WO2008152607A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/579—Depth or shape recovery from multiple images from motion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/215—Motion-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Definitions

  • the present invention relates to a method, apparatus, system and computer program product for depth-related information generation.
  • Autostereoscopic displays generally generate an impression of a three-dimensional image by rendering different views of an object for different viewing angles. In this manner a first image can be generated for the left eye of a viewer and a second image for the right eye of the viewer.
  • a first image can be generated for the left eye of a viewer and a second image for the right eye of the viewer.
  • the source material for use with autostereoscopic displays can be generated in a variety of manners. For example multiple image sequences may be recorded using multiple suitably positioned cameras in order to record an image sequence corresponding to each and every view. Alternatively, individual image sequences can be generated using a three dimensional computer model by the autostereoscopic display.
  • depth maps provide depth information indicative of the absolute or relative distance of objects depicted in the image to a/the (virtual) camera.
  • Depth maps can provide depth- information on a per-pixel basis but as will be clear to the skilled person may also provide depth information at a coarser granularity. In certain applications it may be desirable to use a lower resolution depth-map wherein each depth-map value provides depth-information for multiple pixels in the image.
  • Disparity maps can be used as an alternative to the above mentioned depth maps. Disparity refers to the apparent shift of objects in a scene when it is observed from two distinct viewpoints, such as a left-eye and a right-eye viewpoint. This shift is larger for objects near by.
  • Disparity information and depth information are related and can be mapped onto one another using a model, such as a pinhole camera model. More information with regard to the mapping of disparity information to depth-map information can be found in "Depth Estimation from Stereoscopic Image Pairs Assuming Piecewise Continuous Surfaces", hereby incorporated by reference, by L. Falkenhagen and published in Proc. of European Workshop on combined Real and Synthetic Image Processing for Broadcast and Video Production, Hamburg, November 1994.
  • depth-related information is used throughout the description and is understood to comprise information such as depth information as well as disparity information.
  • the autostereoscopic display can render multiple views of the content for one or more viewers.
  • newly created content might be provided with accurately recorded depth-related information
  • conventional 2-D image sequences generally do not comprise the required depth-related information.
  • a known approach for converting 2D to 3D information is presented in International Patent Application WO200213141.
  • a neural network is trained using a manually annotated key image in order to learn the relationship between image characteristics and depth characteristics.
  • the trained neural network is subsequently used to generate depth information for key- frames.
  • depth maps of one or more key- frames are used to generate depth maps for non key- frames, using information such as the relative location, image characteristics and distance to the respective key-frame(s).
  • a problem with the above approach is that it handles scenes wherein objects and/or regions with similar color and/or image characteristics are located at different depths.
  • the present invention proposes to evolve depth-related information and segmentation-related information in an image sequence using a probabilistic network.
  • depth-related information such as depth-information or disparity information
  • segmentation-related information is annotated to at least one image of the image sequence. This information is used as initialization for the probabilistic network.
  • the present invention proposes to pose the depth-related information and segmentation-related information propagation problem as a Bayesian inference problem, wherein the solution is defined as being the maximum a posteriori (MAP) probability estimate of the true labeling.
  • MAP maximum a posteriori
  • a Markov Random Field (MRF) network is used to evolve a labeling comprising the depth-related information as well as a mapping indicative of the relationship between respective (groups of) pixels in the current image and those of the previously processed image.
  • the MRF network evolves the labeling and mapping based on observations in the form of image characteristics such as e.g. color, intensity, texture and/or curvature of edges in the current image.
  • the MRF network can incorporate a variety of prior contextual information, such as the labeling, the mapping in a quantative manner.
  • the MRF network can achieve an optimal solution within the limitations of the probabilistic network by minimizing the posterior energy which comprises contributions from both the labeling and the mapping. In this manner the present invention can generate depth-related information for images while simultaneously taking both depth-related information, segmentation-related information and motion of objects in the image into account.
  • a further advantage of the use of an MRF network is that it is well suited for further extensions. Yet another advantage of using an MRF network it is that is substantially parallel and as a result is well suited for parallel processing.
  • the Bayesian inference problem is extended to also comprise segmentation of the image.
  • a segment here is understood to be a region with some common properties, such as chrominance, luminance, and/or texture, that typically moves as a whole and has depth- related information associated with it.
  • the labeling comprises both segmentation information such as e.g. a segment index, as well as the depth-related information.
  • the MRF network may be further enhanced to capitalize on the fact that in general segment shapes do not exhibit major variations between consecutive images.
  • the MRF network may be enhanced to encode a curvature smoothness model which allows enforcement of constraints on the shape of the respective segments, thereby warranting temporal segment shape continuity.
  • the MRF network is arranged to take account of a time-invariant segment specific color models.
  • a color model can be generated based on image characteristics of the segment from the image for which the depth-related information was annotated.
  • the nodes within the MRF are organized in a pair- wise manner. Consequently the MRF is capable of encoding a spatial smoothness constraint and enables the use of a fast inference algorithm such as e.g. Graph Cut or Belief Propagation, to estimate the MAP solution.
  • a fast inference algorithm such as e.g. Graph Cut or Belief Propagation
  • the neighborhood of a node is defined by its 8- neighborhood. This allows enforcement of more complicated constraints such as curvature smoothness.
  • the image sequence corresponds to a shot, and one image within the shot is annotated, preferably the image with the largest number of visible objects and/or segments.
  • Depth-related information is provided for the image as well as segment information provided that is required by the embodiment in question.
  • the labeling can be evolved both forward and backward in time towards the shot boundaries.
  • the image with the largest number of visible objects is annotated in order to provide an efficient initialization.
  • at least two images within an image sequence are annotated and the MRF network is arranged to apply bi-directional propagation of the depth-related information. To this end the MRF network needs to estimate the labeling and mapping for both the forward and backward propagation at the same time.
  • further constraints such as a constraint with regard to the similarity between forward and backward mapping could be encoded into the MRF network in order to improve temporal consistency of the mapping.
  • a further embodiment of the present invention comprises an apparatus according to claim 14 that is arranged to propagate depth-related information and segmentation-related information in a manner that takes movement of objects in a scene into account in a more direct manner.
  • a further embodiment of the present invention comprises a system according to claim 18 that is arranged to propagate depth-related information and segmentation-related information in a manner that takes movement of objects in a scene into account in a more direct manner.
  • a further embodiment of the present invention comprises a computer program product according to claim 23.
  • Fig. 1 shows a schematic overview of the composition of an image as an overlay of segments
  • Fig. 2 shows a graphical model of a single node in an MRF according to the present invention
  • Fig. 3 shows a graphical model of a MRF according to the present invention
  • Fig. 4 shows several a flow-chart of a method according to the present invention
  • FIG. 5 shows several depth maps propagated using the present invention
  • Fig. 6 shows an apparatus according to the present invention
  • FIG. 7 shows a system according to the present invention.
  • the Figures are not drawn to scale. Generally, identical components are denoted by the same reference numerals in the Figures.
  • the present invention proposes to evolve depth-related information and segmentation-related information in an image sequence using a probabilistic network.
  • the present invention can be applied to images without the need for segmentation.
  • the present invention may advantageously incorporate segmentation of images.
  • the probabilistic network is arranged to evolve both the depth-related information as well as the segments.
  • a segment within the context of this specification is understood to be a region with some common properties, such as chrominance, luminance, and/or texture, that typically moves as a whole and has depth-related information associated with it. It will be clear to the skilled person that such image characteristics can be defined regardless of the color space representation, such as e.g. RGB, or YUV.
  • the depth-related information discussed here may be information such as, but not limited to, depth-information, disparity information and/or (de-)occlusion information
  • the method according to the invention utilizes annotated depth-related information and segmentation information of at least one image of the image sequence. This information may comprise e.g. a segment index and depth-values associated with the respective segments.
  • the method according to this embodiment comprises a step for annotating the image sequence, this is not always required. Instead an annotated image sequence, which was previously annotated, can be used to equal effect.
  • the annotation step in turn might be a manual, semi-automatic, or fully automatic annotation process.
  • the annotated information provides an initial labeling for at least one image of the image sequence. Although it is not mandatory to provide an initial segment labeling for an image, it will be clear to the skilled person that by providing a reliable initial labeling the evolution process can be made much more efficient.
  • the segments are evolved across subsequent images of the image sequence together with their depth-related information using a probabilistic network.
  • the problem of evolving the segments can be posed as a Bayesian inference problem wherein the solution is defined as the Maximum A Posteriori (MAP) labeling.
  • the MAP labeling is obtained by minimizing the a posteriori energy.
  • the MAP labeling is an estimate of the true optimum, but is the best possible based on random observations within the probabilistic network.
  • Sites may represent e.g. pixels, or multiple pixels, but may equally well represent more complex objects such as lines.
  • sites may represent either pixels or multiple pixels that are substantially spatially homogeneous. The actual relationship between sites can be determined by a so-called neighborhood system, which will be discussed later.
  • D be a set of labels. Labeling is to assign a label from the set of labels D to each of the sites in d.
  • F [F 1 ,...,F N ) , i.e. F comprises a random variable for each site.
  • segmentation of an image can be posed as a labeling problem.
  • the set of sites d comprises sites corresponding with the pixels of the image being segmented.
  • Segmentation now corresponds to labeling each of the sites/pixels with a segment index such that all pixels belonging to a segment have one and the same segment index.
  • Bayesian statistics are subsequently used to define an optimization problem that aims to find an optimal configuration/labeling based on quantative criteria that incorporate observations in the form of image characteristic.
  • observations may relate e.g. to the color, intensity, texture distribution within respective segments.
  • Bayesian statistics can be used to incorporate such information in the optimization process.
  • the a priori probability P(f) also referred to as the prior, comprises probability distributions that express uncertainty before evidence is taken into account.
  • the prior depends on how various prior constraints are expressed.
  • the likelihood function p(r I /) in turn relates to how data is observed and is problem domain dependent.
  • MRF Markov Random Field
  • Belief Propagation techniques can estimate the MAP solution of MRF networks through independent local (message-passing) operations. Consequently, they are particularly well suited parallel processing. More background on the use of MRF modeling can be found in "Markov Random Field modeling in image analysis.” by S.Z. Li, Springer- Verlag, 2001, hereby incorporated by reference.
  • the present invention can be applied in the generation of depth-related information from 2D content.
  • the present invention will typically be used to propagate manually or automatically generated depth related-information available for images in the sequence, to further images in the image sequence.
  • the propagation of depth information throughout the image sequence can be posed as a labeling problem.
  • the present invention capitalizes on the fact that there is a strong correlation between the labeling and the mapping in the images. Moreover there is a strong correlation between the mapping and segmentation. By evolving all of these together the present invention effectively improves the depth map propagation compared to the prior art, in that it takes into account both the labeling and the mapping simultaneously, whereas the prior art does not.
  • the use of segments moreover provides a further advantage in that the segments provide a compact abstraction of the image contents.
  • the detection and tracking of segments enables segment based operations. This can be useful e.g. when an object is moving from the background to the foreground, or is appearing from/disappearing behind another segment.
  • segments the present invention effectively simplifies handling of occlusion and de-occlusion in an elegant manner.
  • Segments here are understood to be regions with some common properties, such as chrominance, luminance, texture and/or curvature of the edge. Typically segments move as a whole and have depth-related information associated with it.
  • Fig. 1 shows an image sequence comprising images 100, 101, 102, 103, and 104. Each image of the image sequence can be interpreted as a combination of several segments.
  • Fig. 1 illustrates that image 100 which shows a triangular object 121 and a circular object 131 over a grey background. The image 100 can be interpreted as three overlaid segments; a first segment 110 comprising the grey background, a second segment 120 comprising the triangular object and a third segment 130 comprising the circular object.
  • An MRF network is a model of a joint probability distribution of a set of random variables.
  • An MRF network comprises multiple nodes.
  • the nodes are the basic units of the network.
  • Fig. 2 presents a graphical model of a single node in an MRF network according to the present invention.
  • a node may represent a single pixel, or could alternatively represent multiple pixels, such as a region of an image.
  • An example of a scenario wherein a node represents multiple pixel would be a scenario wherein depth-related information are calculated at a resolution lower than that of the 2D image. In this manner calculating and propagating the depth-related information will be more computationally efficient. For the sake of simplicity here we consider the scenario wherein a node corresponds to a single pixel.
  • Fig. 2 presents a graphical model of a single node in an MRF network according to the present invention.
  • the circles represent hidden state of the node, whereas the boxes represent the observations.
  • this model :
  • / represents the depth-related information and segment labeling of the node of the current image
  • m represents the mapping of the current image with respect to the previous image
  • c represents the set of observed image characteristics such as color characteristics derived from the current image
  • ⁇ represents a set of color HiOcIeIsG 1 , wherein i is a segment index and ⁇ ; represents the color model conditioned on segment i.
  • the hidden state information / will typically comprise the depth-related information and segment label of that pixel.
  • the state information m represents the mapping of the pixel in the current image with respect to the previous image.
  • the state information m can comprise e.g. a 2D vector (x ⁇ y') which indicates the corresponding pixel in the previous image.
  • the 2D vector may encode an offset in the previous image (dx, dy) .
  • the actual format of the information is not relevant as long as it provides information with regard to the mapping.
  • the observed set of image characteristics C represents color characteristics it will be clear to the skilled person that these can be replaced by and/or augmented with intensity characteristics, texture characteristics, or other image characteristics known to those skilled in the art.
  • the set of segment specific color models are typically time-invariant. Segment specific color models are typically constructed using the color information from the respective segment in an annotated image in the image sequence. Based on the initial segment labels color models can be generated for each segment.
  • the color model can be parametric, such as when using Gaussian mixtures, or can be non-parametric, such as when using histograms. Alternatively they may be discriminative; i.e. differentiating between foreground and background.
  • segment specific color models are time-invariant there are situations where time- variant color models may be beneficial.
  • time-variant color models may be beneficial.
  • the image sequence comprises multiple annotated images.
  • the two color models for one and the same segment may be mixed.
  • the contribution of the color models is weighted based on the distance to the annotated image.
  • the joint probability corresponding to the graphical model presented in Fig. 2 can be factorized as shown in eq. 3.
  • p ⁇ l, m,c ⁇ ⁇ ) p ⁇ l)p ⁇ m
  • p(c I m,l, ⁇ ) as such is typically intractable due to the large number of combination of labeling and mapping in practical applications.
  • m,l, ⁇ ) is preferably approximated using p(c ⁇ m)p(c
  • alternative approximations can be used such as approximations based on the use of the principle of structured variational approximation.
  • the variational technique approximates the intractable probability distribution with another tractable probability distribution through minimizing the Kullback-Leibler divergence between them.
  • the variational technique refers to the paper "On structured variational approximations.”, by Ghahramani, Z. (1997), hereby incorporated by reference, Technical Report CRG-TR-97-1, Department of Computer Science, University of Toronto.
  • the MRF network described above comprises multiple nodes which connect to their respective neighbors.
  • the connectivity of the nodes or neighborhood is shown in Fig. 3.
  • the connectivity of the MRF network is pair- wise. It will be clear to the skilled person that the present invention is not limited to two-node cliques MRF networks, but the example is restricted thereto for the sake of simplicity.
  • Fig. 3 the nodes i andy represent neighboring nodes in the MRF.
  • / represent the labeling of the respective nodes.
  • MRF network forms a segment labeling field L.
  • Wi 1 and m ⁇ represent the mapping of the nodes i and j, the set of all mappings in turn defines a mapping field M.
  • C 1 and c ⁇ represent the observations for the respective nodes. These could correspond to e.g. the color value(s) of the corresponding (plurality of) pixel(s) or other image characteristics.
  • the set of all observations defines an observation field C.
  • ⁇ corresponds to the set of color models as defined before.
  • Z is a normalization constant ⁇ ; is defined as the evidence of the hidden state or the compatibility function of the hidden state and the observations
  • ⁇ is defined as the compatibility function between the neighboring nodes.
  • N is the neighborhood system defined on the network, here the set of all possible node pairs.
  • the solution of the segment evolution problem is defined as the MAP probability estimate of the labeling and the mapping.
  • the computation is preferably performed on negative log probability as shown below, which corresponds to the energy function definitions.
  • the definition of the energy functions is given for this particular case by way of example.
  • E(L,M) ⁇ E ⁇ (c 1 ,l 1 ,m 1 ) + ⁇ ⁇ E 2 (l ,,m, , / ⁇ ,m , ) (eq. 5)
  • the energy function E(L, M) comprises the contextual information of a first node i, as well as the contextual information encoded in the links between neighboring nodes i andy.
  • the energy of the contextual information of the first node i can be written as:
  • E 1 (C 1 , l 1 ,m 1 ) E ⁇ (l i ,c 1 ) + E: (m 1 , C 1 ) (eq. 6)
  • ETM (Tn 1 , C 1 ) can be expressed as being:
  • the second energy component E 2 (I 1 is the interaction term between the two neighboring nodes i andy.
  • the second energy component can be expressed as:
  • T 0 is a predefined cost and E 2 (M 1 ,m ⁇ ) is defined as:
  • Fig. 4 shows a flowchart of a method according to the present invention.
  • step 410 image sequence 405 is processed and depth-related information 415 is generated for at least one image of the image sequence 405.
  • depth-related information 415 is generated for an image sequence, such as those disclosed in International Patent Applications WO2005/013623 and WO2005/083630 by the same applicant, hereby incorporated by reference.
  • step 420 the generated depth-related information is combined with manually entered depth-related information 425 and segmentation-related information resulting in annotation information 435 for the at least one image.
  • annotation information 435 may comprise other information that can be annotated to the at least one image.
  • the annotation information 435 is subsequently annotated to the image sequence 405 in step 430.
  • the annotated image sequence 445 is subsequently used to determine the MAP solution for both the labeling and mapping for at least one further image in the image sequence 405, in step 440, in the process taking into account the evidence in the form image sequence 405.
  • the MAP solution in turn comprises the propagated depth-related information.
  • the process of determining a MAP solution can be subsequently repeated for further images in the sequence 405 based on the MAP solution that was just established until all images with the image sequence are annotated.
  • the present invention can be used to propagate depth-related information and segmentation-related information for consecutive images in an image sequence.
  • depth-related information and segmentation-related information propagation would be in a forward direction; i.e. from the current image to another image forward in time
  • the present invention may also be applied on images in an image sequence in reverse order, thereby effectively propagating annotated depth-related information and segmentation-related information backwards in time.
  • Fig. 5 shows an example of three input images for which a depth map and segmentation were generated according to the present invention.
  • the images 501, 502 and 503 represent the original images in sequence.
  • the images 504, 505 and 506 represent the corresponding propagated depth-related information.
  • FIG. 6 shows an apparatus 600 according to the present invention arranged to propagate depth-related information and segmentation-related information in an image sequence.
  • the apparatus comprises two input connectors 605 and 615.
  • the input connectors 615 and 605 are used to receive annotation information 435 for at least one image of the image sequence 405 and the image sequence 405 respectively.
  • the annotating means 610 is arranged to annotate at least one image of the image sequence 405 using annotation information 435.
  • the annotated image sequence 445 is subsequently processed in accordance with the present invention by processing means 620 in order to establish the MAP solution for both the labeling and mapping using a probabilistic network for at least one further image in the image sequence.
  • the processing means 620 may also provide the image sequence 405 on optional output connector 625 and segmentation information on optional output connector 645.
  • the apparatus 600 is well-suited for embedding within more complex devices, such as set-top boxes and/or auto stereoscopic displays.
  • a further apparatus 650 in accordance with the present invention.
  • This device further comprises generation means 630 which is arranged to generate annotation information 435 based on an input sequence 405.
  • the apparatus 650 comprises an input connector 605 for receiving an image sequence 405.
  • the image sequence 405 is subsequently presented to the generation means 630 for generating annotation information 435.
  • the apparatus 650 autonomously generates the annotation information 435 for use in the processing means 620.
  • the present invention may be implemented on a variety of processing platforms. These may range from dedicated hardware platforms that comprise a plurality of massively parallel processor arrays, to general purpose processing on single processor platforms. Moreover the generation means 630, the annotation means 610 and the processing means 620 may be implemented on one and the same processing platform in a substantially sequential or parallel manner, i.e. as far as algorithmic constraints allow parallelism. Finally the implementation of the present invention may be implemented primarily in software e.g. on a programmable computing platform, or alternatively can be mapped primarily on hardware e.g. on a dedicated Application Specific Integrated Circuit (ASIC).
  • ASIC Application Specific Integrated Circuit
  • Fig. 7 shows a system 700 according to the present invention.
  • the system 700 comprises several devices according to the present invention.
  • the system comprises a storage server 755, which might be local or remote and/or a network server 760.
  • Each of these servers 755,760 can provide both image sequence data 405 over network 750.
  • they may be further arranged to also provide annotation information 435 over network 750.
  • This information may be provided e.g. to an apparatus 600 according to the present invention for further processing.
  • the image sequence 405 can be provided to a Set Top Box (STB) 707 comprising an apparatus 650 according to the present invention that is connected to an autostereoscopic display 705.
  • STB Set Top Box
  • the image sequence data 405 can be provided to an apparatus 710 that comprises the functionality of the above mentioned STB 707 and the autostereoscopic display 705.
  • the image sequence 405 may also be provided to a compute server 720 that is arranged to execute instructions stored on a data carrier 730, which instructions when executed by the compute server 720 perform the steps of a method in accordance with the present invention.
  • a compute server 720 that is arranged to execute instructions stored on a data carrier 730, which instructions when executed by the compute server 720 perform the steps of a method in accordance with the present invention.
  • the MRF network can be furthermore enhanced to constrain the effects that such imported depth-related information estimations may have on the labeling and mapping. In this manner temporal stability can be substantially preserved and erratic behavior resulting from external depth-related information can be prevented. It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un appareil, un système, un procédé et un programme informatique de propagation d'informations relatives à la profondeur et d'informations relatives à la segmentation associées à au moins une image issue d'une séquences d'image, à une image consécutive de la séquence d'images au moyen d'un réseau probabiliste. Le procédé selon l'invention consiste à utiliser le réseau probabiliste pour résoudre un problème d'étiquetage bayésien, l'étiquetage contenant les informations relatives à la profondeur et les informations relatives à la segmentation, les liaisons de nœuds du réseau probabiliste étant configurées pour représenter simultanément les contraintes imposées par les informations relatives à la profondeur et les informations relatives à la segmentation de l'image consécutive et des informations de mappage pour le nœud respectif de ladite image au moins à l'image consécutive. Les nœuds prenant en compte des caractéristiques de l'image consécutive, telles que les informations relatives à la profondeur et les informations relatives à la segmentation propagées, sont établis pour l'image consécutive par la création d'une solution a posteriori A maximale à la fois pour l'étiquetage et pour le mappage. Le problème bayésien comprend la segmentation de l'image consécutive et la propagation des informations de profondeur, en fonction des informations de segmentation et de mappage stockées dans le réseau probabiliste construit à partir de ladite image au moins, à l'image consécutive.
PCT/IB2008/052340 2007-06-15 2008-06-13 Procede, appareil, systeme et programme informatique de propagation d'informations relatives a la profondeur WO2008152607A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP07110360 2007-06-15
EP07110360.0 2007-06-15
EP08151409.3 2008-02-14
EP08151409 2008-02-14

Publications (1)

Publication Number Publication Date
WO2008152607A1 true WO2008152607A1 (fr) 2008-12-18

Family

ID=39811683

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2008/052340 WO2008152607A1 (fr) 2007-06-15 2008-06-13 Procede, appareil, systeme et programme informatique de propagation d'informations relatives a la profondeur

Country Status (2)

Country Link
TW (1) TW200907859A (fr)
WO (1) WO2008152607A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103679717A (zh) * 2013-12-05 2014-03-26 河海大学 基于马尔科夫随机场的图像分割方法
US9338424B2 (en) 2011-02-23 2016-05-10 Koninklijlke Philips N.V. Processing depth data of a three-dimensional scene
EP2656315B1 (fr) * 2010-12-22 2016-10-05 Legend3D, Inc. Système et procédé pour flux de travail à itération minimale pour améliorer la profondeur de séquence d'images
WO2016170330A1 (fr) * 2015-04-24 2016-10-27 Oxford University Innovation Limited Traitement d'une série d'images pour identifier au moins une partie d'un objet
CN106570880A (zh) * 2016-10-28 2017-04-19 中国人民解放军第三军医大学 结合模糊聚类和马尔科夫随机场的脑组织mri图像分割方法
CN111951282A (zh) * 2020-08-12 2020-11-17 辽宁石油化工大学 一种基于马尔科夫随机场与区域合并的图像分割改进算法

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
CIPOLLA R ET AL: "Layered motion segmentation and depth ordering by tracking edges", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, IEEE SERVICE CENTER, LOS ALAMITOS, CA, US, vol. 26, no. 4, 1 April 2004 (2004-04-01), pages 479 - 494, XP011107234, ISSN: 0162-8828 *
JIE-YU ZHAO: "Dynamic background discrimination with belief propagation", MACHINE LEARNING AND CYBERNETICS, 2004. PROCEEDINGS OF 2004 INTERNATIO NAL CONFERENCE ON SHANGHAI, CHINA AUG. 26-29, 204, PISCATAWAY, NJ, USA,IEEE, vol. 7, 26 August 2004 (2004-08-26), pages 4342 - 4346, XP010763211, ISBN: 978-0-7803-8403-3 *
SEBASTIAN KNORR ET AL: "A Modular Scheme for 2D/3D Conversion of TV Broadcast", 3D DATA PROCESSING, VISUALIZATION, AND TRANSMISSION, THIRD INTERNATION AL SYMPOSIUM ON, IEEE, PI, 1 June 2006 (2006-06-01), pages 703 - 710, XP031079006, ISBN: 978-0-7695-2825-0 *
YAN LI ET AL: "Object Detection Using 2D Spatial Ordering Constraints", COMPUTER VISION AND PATTERN RECOGNITION, 2005. CVPR 2005. IEEE COMPUTE R SOCIETY CONFERENCE ON SAN DIEGO, CA, USA 20-26 JUNE 2005, PISCATAWAY, NJ, USA,IEEE, vol. 2, 20 June 2005 (2005-06-20), pages 711 - 718, XP010817983, ISBN: 978-0-7695-2372-9 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2656315B1 (fr) * 2010-12-22 2016-10-05 Legend3D, Inc. Système et procédé pour flux de travail à itération minimale pour améliorer la profondeur de séquence d'images
US9338424B2 (en) 2011-02-23 2016-05-10 Koninklijlke Philips N.V. Processing depth data of a three-dimensional scene
CN103679717A (zh) * 2013-12-05 2014-03-26 河海大学 基于马尔科夫随机场的图像分割方法
WO2016170330A1 (fr) * 2015-04-24 2016-10-27 Oxford University Innovation Limited Traitement d'une série d'images pour identifier au moins une partie d'un objet
CN106570880A (zh) * 2016-10-28 2017-04-19 中国人民解放军第三军医大学 结合模糊聚类和马尔科夫随机场的脑组织mri图像分割方法
CN106570880B (zh) * 2016-10-28 2019-10-11 中国人民解放军第三军医大学 结合模糊聚类和马尔科夫随机场的脑组织mri图像分割方法
CN111951282A (zh) * 2020-08-12 2020-11-17 辽宁石油化工大学 一种基于马尔科夫随机场与区域合并的图像分割改进算法

Also Published As

Publication number Publication date
TW200907859A (en) 2009-02-16

Similar Documents

Publication Publication Date Title
CA2668941C (fr) Systeme et procede d'adaptation de modeles et d'enregistrement d'objets pour une conversion 2d->3d
Feng et al. Object-based 2D-to-3D video conversion for effective stereoscopic content generation in 3D-TV applications
Jang et al. Efficient disparity map estimation using occlusion handling for various 3D multimedia applications
US9661307B1 (en) Depth map generation using motion cues for conversion of monoscopic visual content to stereoscopic 3D
KR20110090958A (ko) 이미지 속성들에 대한 오클루젼 데이터의 생성
KR100560464B1 (ko) 관찰자의 시점에 적응적인 다시점 영상 디스플레이 시스템을 구성하는 방법
JP4892113B2 (ja) 画像処理方法及び装置
EP2462536A1 (fr) Systèmes et procédés de génération de vidéo tridimensionnelle
WO2008152607A1 (fr) Procede, appareil, systeme et programme informatique de propagation d'informations relatives a la profondeur
US10218956B2 (en) Method and apparatus for generating a depth cue
Zhang et al. Interactive stereoscopic video conversion
Yang et al. Depth map generation using local depth hypothesis for 2D-to-3D conversion
Lin et al. A 2D to 3D conversion scheme based on depth cues analysis for MPEG videos
Lee et al. Segment-based multi-view depth map estimation using belief propagation from dense multi-view video
Orozco et al. HDR multiview image sequence generation: Toward 3D HDR video
Knorr et al. From 2D-to stereo-to multi-view video
Caviedes et al. Real time 2D to 3D conversion: Technical and visual quality requirements
EP2462539A1 (fr) Systèmes et procédés de génération de vidéo tridimensionnelle
Priya et al. 3d Image Generation from Single 2d Image using Monocular Depth Cues
Cai et al. Image-guided depth propagation for 2-D-to-3-D video conversion using superpixel matching and adaptive autoregressive model
Tsubaki et al. 2D to 3D conversion based on tracking both vanishing point and objects
Raviya et al. Depth and Disparity Extraction Structure for Multi View Images-Video Frame-A Review
Xu et al. Comprehensive depth estimation algorithm for efficient stereoscopic content creation in three-dimensional video systems
Guo et al. Single Image based Fog Information Estimation for Virtual Objects in A Foggy Scene
KR20220071935A (ko) 광학 흐름을 이용한 고해상도 깊이 영상 추정 방법 및 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 08763327

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 08763327

Country of ref document: EP

Kind code of ref document: A1