EP3298578B1 - Method and apparatus for determining a depth map for an image - Google Patents

Method and apparatus for determining a depth map for an image Download PDF

Info

Publication number
EP3298578B1
EP3298578B1 EP16720854.5A EP16720854A EP3298578B1 EP 3298578 B1 EP3298578 B1 EP 3298578B1 EP 16720854 A EP16720854 A EP 16720854A EP 3298578 B1 EP3298578 B1 EP 3298578B1
Authority
EP
European Patent Office
Prior art keywords
depth
pixel
probability
value
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP16720854.5A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3298578C0 (en
EP3298578A1 (en
Inventor
Christiaan Varekamp
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Koninklijke Philips NV
Original Assignee
Koninklijke Philips NV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Koninklijke Philips NV filed Critical Koninklijke Philips NV
Publication of EP3298578A1 publication Critical patent/EP3298578A1/en
Application granted granted Critical
Publication of EP3298578B1 publication Critical patent/EP3298578B1/en
Publication of EP3298578C0 publication Critical patent/EP3298578C0/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G06T5/30—Erosion or dilatation, e.g. thinning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/143—Segmentation; Edge detection involving probabilistic approaches, e.g. Markov random field [MRF] modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06V10/7747—Organisation of the process, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/635—Overlay text, e.g. embedded captions in a TV program
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
    • H04N13/183—On-screen display [OSD] information, e.g. subtitles or menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20182—Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • the invention relates to a method and apparatus for determining a depth map for an image, and in particular for determining a depth map based on estimated disparity values.
  • Three dimensional (3D) displays add a third dimension to the viewing experience by providing a viewer's two eyes with different views of the scene being watched. This can be achieved by having the user wear glasses to separate two views that are displayed.
  • 3D displays that use means at the display (such as lenticular lenses, or barriers) to separate views, and to send them in different directions where they individually may reach the user's eyes.
  • two views are required whereas autostereoscopic displays typically require more views (such as e.g. nine views).
  • lenticular based auto-stereoscopic 3D displays tend to suffer from out-of-screen blur. This effect is similar to what is known as depth-of-field blur in camera systems.
  • the quality of the presented three dimensional image depends on the quality of the received image data, and specifically the three dimensional perception depends on the quality of the received depth information.
  • Three dimensional image information is often provided by a plurality of images corresponding to different view directions for the scene.
  • video content such as films or television programs, are increasingly generated to include some 3D information.
  • Such information can be captured using dedicated 3D cameras that capture two simultaneous images from slightly offset camera positions.
  • the provided images may not directly correspond to the desired directions, or more images may be required.
  • more images are required and indeed often 9-26 view images are used.
  • view point shifting processing may be employed. This is typically performed by a view shifting algorithm which uses an image for a single view direction together with associated depth information.
  • the provided depth information must be sufficiently accurate.
  • depth information may not be as accurate as desired. Indeed, in many scenarios depth information is generated by estimating and extracting depth values by comparing view images for different view directions.
  • three dimensional scenes are captured as stereo images using two cameras at slightly different positions.
  • Specific depth values may then be generated by estimating disparities between corresponding image objects in the two images.
  • depth extraction and estimation is problematic and tends to result in non-ideal depth values. This may again result in artefacts and a degraded three dimensional image quality.
  • Three dimensional image degradation and artefacts tend to be particularly significant for text image objects, such as e.g. subtitle blocks.
  • text image objects tend to be isolated objects that are not perceived as being integrated or embedded in the scene.
  • depth variations for text image objects tend to be more perceptible to the viewer.
  • text (such as especially subtitles) is expected to be sharp and in focus with well-defined edges. Accordingly, it is of high importance to in particular present text image objects, such as subtitle blocks, with a high image quality.
  • US2103/0100260A1 discloses an approach wherein a correction coefficient is applied to scale a depth of a pixel block having a highest probability of being a telop into a given range.
  • US2014/0333720A1 discloses an approach for detecting subtitle areas in left and right images and determining a common disparity for the subtitle areas.
  • an improved approach for determining suitable depth information for text image objects would be advantageous and in particular an approach allowing increased flexibility, facilitated implementation, reduced complexity, an improved 3D experience and/or improved perceived image quality would be advantageous.
  • the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.
  • an apparatus for determining a depth map for an image comprising: an image unit for providing a first image with an associated depth map comprising first depth values for at least some pixels of the first image; a probability unit for determining a probability map for the first image comprising probability values for at least some pixels of the image; the probability value for a pixel being indicative of a probability that the pixel belongs to a text image object; a depth unit for generating a modified depth map for the first image, the depth unit being arranged to determine a modified depth value for at least a first pixel as a weighted combination of a first depth value of the associated depth map for the first pixel and a text image object depth value being the same for a plurality of pixels, the weighting being dependent on the probability value for the first pixel; the weighted combination for the first pixel providing the modified depth value as a function of the first depth value, the text image object depth value, and the probability value for the first pixel with the contribution to the modified depth value
  • the approach may in many embodiments allow an improved depth map to be generated which when used for image processing may provide improved quality.
  • an improved presentation of text image objects such as subtitles
  • the improvement may in particular be significant when the modified depth map is used for image view shifting and/or when presenting three dimensional images on an autostereoscopic display.
  • the approach may further allow a shift of presentation depth for text image objects while mitigating or reducing degradations, inconsistencies and/or artefacts.
  • the maps may be full or partial maps.
  • the probability map may comprise probability values for only a subset of pixels/ pixel groups.
  • the probability value for a given pixel may represent an estimate of the probability or likelihood that the pixel belongs to (is part of) a text image object.
  • the probability value may be an indication of how closely a determined value of a property matches a value expected for a text image object.
  • processing may be applied to the first image to generate a value (or set of values).
  • the probability that the pixel belongs to a text image object may be represented by how closely the value matches an expected value.
  • the determined property may be expected to be b whereas for a pixel not belonging to a text image object, the value may be expected to be a.
  • the exact processing and determination of the value may be different in different embodiments depending on the preferences and requirements of the individual embodiment.
  • a text image object may be a region! area/ subset or segment of the image which comprises text.
  • a text image object may specifically be a subtitle image object.
  • a text image object may correspond to one or more characters or may e.g. include bounding region for the text.
  • the weighted combination may specifically be a weighted summation of the first depth value and the text image object depth value with the weights being a function of the probability value.
  • the weighted combination may specifically be a weighted summation of a monotonic function of the first depth value and the text image object depth value, with the weights being a function of the probability value.
  • the text image object depth value may be a predetermined value.
  • the text image object depth value may specifically be a fixed and/or constant value in the time and/or spatial domain.
  • the text image object depth value may be a desired depth for text image objects.
  • the text image object depth value may indicate a preferred depth for text image objects, and may be a fixed and/or predetermined value.
  • the text image object depth value may indicate a preferred depth for text image objects and may be independent of depth properties of a scene represented by the first image.
  • the text image objects may typically be overlay graphics which are not part of the scene represented by the first image, and the text image object depth value may represent a preferred depth for overlay graphics/ text image objects that are not part of the scene.
  • the text image object depth value is associated with text image objects and may be the same for a plurality of pixels or pixel groups. In many embodiments, the text image object depth value is independent of properties of the first image and/or the associated depth map.
  • the first image may be an image which is part of a plurality of images corresponding to different viewing angles or may e.g. be a single isolated and independent image (having an associated depth map).
  • the first image may be an image of a temporal sequence of images, such as e.g. a frame from a video sequence.
  • the probability unit is arranged to determine pixel groups for the first image, each pixel group comprising a plurality of pixels; and wherein the probability unit is arranged to determine probability values for pixel groups.
  • the determined probability value for a pixel group may be assigned to all pixels belonging to the pixel group.
  • the depth unit is arranged to determine weights for the weighted combination in response to a first probability value for a pixel group to which the first pixel belongs and a second probability value for a neighboring pixel group; a weighting of the first probability value and the second probability value being dependent on a position of the first pixel in the pixel group to which the first pixel belongs.
  • the approach may allow a determination of probability values at a lower resolution than the image pixel resolution while allowing the effective resolution to be increased for the combination of depth levels.
  • the weights for the weighted combination may be determined in response to an interpolated probability value, the interpolated probability value being determined in response to a spatial interpolation between a first probability value for a pixel group to which the first pixel belongs and a second probability value for a neighboring pixel group; the interpolation being dependent on a position of the first pixel in the pixel group to which the first pixel belongs.
  • the probability map may comprises a plurality of probability values, and specifically may comprise a plurality of probability values for each of a plurality of pixels or pixel groups. Similarly, the combination may be performed for each pixel with at least some pixels having different probability values and thus having independent combinations. Thus, the probability values and combination may vary across the image (it may be different for different pixels).
  • the probability unit is arranged to determine probability values for pixel groups of the image, each pixel group comprising at least one pixel; the probability unit being arranged to first determine a classification map comprising values for the pixel groups indicating whether the pixel groups are designated as belonging to a text image object or not belonging to a text image object; and to generate the probability map in response to filtering of the classification map.
  • the classification map comprises binary values for the pixel groups, each binary value indicating either that a pixel group is designated as belonging to a text image object or that the pixel group is designated as not belonging to a text image object.
  • the filtering may convert the binary classification to non-binary probability values which also reflect the characteristics of the temporal and/or spatial neighborhood of the pixel group.
  • the filtering comprises a binary dilation filtering applied to the classification map.
  • This may improve performance and may in particular improve the consistency of detected areas corresponding to text image objects. In many scenarios, it may reduce hole formation in such areas.
  • the filtering comprises a temporal filtering.
  • This may for example allow improved stability and consistency and provide an improved user experience, e.g. when viewing images generated by view shifting based on the modified depth map.
  • the temporal filtering is asymmetric.
  • the filtering comprises spatial filtering.
  • the spatial filtering comprises a softmax filter being a filter having a maximum output value limit.
  • the softmax filter may allow the generation of a consistent depth region corresponding to a text image object while reducing the amount or size of holes and/or while providing a soft transition at edges of the text image object.
  • a softmax filter may be a cascade of a filter and a limiter which limits the output to a maximum value.
  • a softmax low-pass filter may perform an operation corresponding to a cascade of a low-pass filter and a limiter limiting the output value of the low-pass filter to a maximum value.
  • a softmax filter may correspond to a filter having a maximum output value limit.
  • the spatial filtering comprises at least two sequential spatial softmax filters.
  • the softmax filters may allow the generation of a consistent depth region corresponding to a text image object.
  • the filters may reduce the amount or size of holes while also providing a soft transition at edges of the text image object.
  • the two sequential spatial softmax filters may be arranged to have different design/operating parameters, and specifically a kernel dimension, scale factor and/or maximum value may be different for the two filters.
  • the parameters for a first filter may be optimized for reducing hole formation and providing increased consistency while the parameters of the second filter may be optimized to provide a desired depth transition profile at edges of the text image object.
  • the first depth values correspond to disparities for image objects in a plurality of images corresponding to different view directions for a scene of the first image.
  • the invention may in many embodiments improve a depth map generated by disparity estimation, and may specifically mitigate degradations and artefacts associated with text image objects.
  • the image unit is arranged to estimate the disparities for the image objects in the plurality of images.
  • a method of determining a depth map for an image comprising: providing a first image with an associated depth map comprising first depth values for at least some pixels of the first image; determining a probability map for the first image comprising probability values for at least some pixels of the image; the probability value for a pixel being indicative of a probability that the pixel belongs to a text image object; and generating a modified depth map for the first image, the generating comprising determining a modified depth value for at least a first pixel as a weighted combination of a first depth value of the associated depth map for the first pixel and a text image object depth value being the same for a plurality of pixels, the weighting being dependent on the probability value for the first pixel; the weighted combination for the first pixel providing the modified depth value as a function of the first depth value, the text image object depth value, and the probability value for the first pixel with the contribution to the modified depth value from respectively the first depth value and the text image object depth value
  • FIG. 1 illustrates an example of a system in accordance with some embodiments of the window.
  • images corresponding to different views of an autostereoscopic display 101 is generated from an input three dimensional image.
  • the input three dimensional image may for example be represented by a single image with an associated depth map, or may e.g. be represented by stereo images from which an associated depth map is extracted.
  • the image may be an image from a temporal sequence of images, such as a frame from a video sequence/ signal.
  • autostereoscopic displays produce "cones" of views where each cone contains multiple views that correspond to different viewing angles of a scene.
  • the viewing angle difference between adjacent (or in some cases further displaced) views are generated to correspond to the viewing angle difference between a user's right and left eye. Accordingly, a viewer whose left and right eye see two appropriate views will perceive a three dimensional effect.
  • An example of an autostereoscopic display generating nine different views in each viewing cone is illustrated in FIG. 2 .
  • Autostereoscopic displays tend to use means, such as lenticular lenses or parallax barriers/ barrier masks, to separate views and to send them in different directions such that they individually reach the user's eyes.
  • means such as lenticular lenses or parallax barriers/ barrier masks
  • two views are required but most autostereoscopic displays typically utilize more views. Indeed, in some displays a gradual transition of view directions is performed over an image such that different parts of an image may be projected in different viewing directions. Thus, in some more recent autostereoscopic displays a more gradual and continuous distribution of image regions over view directions may be applied rather than the autostereoscopic display rendering a fixed number of complete views.
  • Such an autostereoscopic display is often referred to as providing fractional views rather than full views. More information on fractional views may e.g. be found in WO 2006/117707 .
  • a subtitle box may be identified and the depth level for this box may be set to the screen depth.
  • FIG. 3 illustrates an example of how this may be done
  • the subtitle box is used to place subtitles at screen depth.
  • FIG. 3 illustrates how this can be done.
  • a subtitle box is detected in a luminance image with subtitle text ( FIG. 3a ).
  • detection may be performed per 8x8 block of pixels using features extracted from within each 8x8 block.
  • Each black block may first be classified as being part of a subtitle (shown by white in FIG. 3b ) or not (shown by black in FIG. 3b ).
  • a bounding subtitle box is determined by integrating the detection results first horizontally and finding a start and stop position along the y-coordinate, and then by integrating detection results horizontally and finding a start and stop position for the x-coordinate.
  • the boundaries of the subtitle box may be set to the smallest rectangle that includes all blocks classified as belonging to the subtitle box.
  • the depth for all pixels within the subtitle box may then be set to a specific depth value, and specifically may be set to the screen or display depth. This may reduce artefacts and may reduce blur introduced by the autostereoscopic display to objects that are not at screen level.
  • the approach may be illustrated by FIG. 4 which shows an image with a section 401 that lies behind the screen depth and a section 403 that lies in front of the screen depth. The described approach may identify a subtitle block 405 and position this at the screen depth.
  • the apparatus of FIG. 1 may mitigate or reduce some of the disadvantages known from known approaches.
  • the apparatus comprises a display driver 103 driving the autostereoscopic display 101.
  • the display driver 103 comprises functionality for generating a plurality of view images for the autostereoscopic display 101 and to feed these thereto.
  • the view images are generated for different view directions using a view shifting algorithm which as an input has an input image and a depth map.
  • the system is arranged to process an input depth map (received from an external source or an internal source and often being generated by the display driver 103 from images corresponding to different view directions, such as specifically a stereo image) to generate a modified depth map in which a more subtle rearrangement of subtitle depth is employed.
  • an input depth map received from an external source or an internal source and often being generated by the display driver 103 from images corresponding to different view directions, such as specifically a stereo image
  • This will tend to provide a substantially improved visual experience including a reduction or mitigation of visible artifacts around subtitles or other text image objects.
  • the approach of the display driver 103 involves first generating a probability map, also referred to as an ⁇ -map, comprising probability values that are indicative of whether the corresponding pixels belong to a text image object or not.
  • a probability map also referred to as an ⁇ -map
  • the probability map provides non-binary, and often substantially continuous, values which are indicative of an estimated probability that the pixel, or pixel group, belongs to a text image object or not.
  • the probability values of the probability map also referred to as ⁇ -values, are typically represented by discrete values, and in most embodiments each value may be represented by values with at least 4, but typically at least 8, 16, 32, 64, 128, 256 or even more discrete levels. In many embodiments, each probability value may be represented by a binary value of at least 3, 4, 6, 8, 10 or 16 bits.
  • the depth level for at least some pixels is not generated simply by either selecting an original depth value or setting it to a predetermined depth level (such as screen level). Rather, at least some depth values of the modified depth map are generated by performing a weighted combination of on original depth value and a text image object depth value which is specifically a preferred depth level for text image objects (such as the screen depth). The weighting of the depth levels for a given pixel is determined based on the probability value for that pixel.
  • D text denotes the text image object depth
  • D [ i,j ] is the original depth level.
  • the modified depth map is generated to provide depth levels which more continuously spans the difference between the original depth level and the depth level desired for text image objects.
  • the text image object may effectively be spatially (in the depth direction) "blended" between the original depth presentation and the desired depth.
  • Another advantage of the approach is that by using the text image object detection results to generate a probability map, this can be subjected to spatial and temporal filters to improve the resulting rendering images. Furthermore, such filtering can be performed without affecting the original depth map.
  • the display driver 103 comprises an image unit 105 which is arranged to provide an image with an associated depth map that comprises depth values for at least some pixels of the image.
  • the image will be referred to as an input image and the depth map as the input depth map (and the pixel and depth values as input pixel and depth values).
  • the image unit 105 may in some embodiments be arranged to simply receive the input image and the input depth map from any suitable internal or external source.
  • a video signal comprising three dimensional images represented by an image and an associated depth map may be received from a network (such as the Internet), a broadcast signal, a media carrier etc.
  • the image unit 105 may be arranged to generate the input image and/or the input depth map. Specifically, in many embodiments, the image unit 105 may be arranged to receive a three dimensional image formed by a plurality of images corresponding to different view directions, such as specifically a stereo image, and it may be arranged to perform disparity estimation to generate the input depth map.
  • the depth values comprised in the depth maps may be any suitable representation of depth, such as specifically a depth coordinate (z) value or a disparity value representing shifts between images of different view directions.
  • the input depth values are observed or estimated depth values. Specifically, they may be values which are generated by disparity estimation performed either at an external source or by the image unit 105 itself. Accordingly, the depth values will often be relatively inaccurate and may contain a number of mistakes and errors.
  • the text image object such as subtitles
  • the text image object may not be positioned at the preferred depth when rendered on the autostereoscopic display. This may for example be due to the technically optimally depth (e.g. with respect to sharpness) varying for different displays or due to subjective preferences varying between applications.
  • subtitles are typically positioned in front of the screen, i.e. at a depth level which makes the subtitles appear to be in front of the screen, and often as the most forward object.
  • substantially increased blur is introduced for such depth levels, and therefore it may be preferred to position the subtitles at the screen depth.
  • the display driver 103 may accordingly perform a gradual readjustment of the depth levels for estimated text image objects.
  • the image unit 105 is coupled to a probability unit 107 which is arranged to generate a probability map comprising probability values that are indicative of a probability or likelihood that the corresponding pixel belongs to a text image object.
  • the probability value for a given pixel may represent an estimate of the probability or likelihood that it belongs to a text image object.
  • the probability may for example be an indication of how closely a pixel group comprising the pixel matches a set of expected characteristics for text image objects. The closer the match, the higher the probability that the pixel belongs to a text image object.
  • the probability of a pixel belonging to a text image object may be based on a consideration that a text image object includes only pixels that are part of a text character.
  • a text image object may typically be considered to include a bounding box.
  • a text image object may be considered to include a surrounding subtitle box. It will be appreciated that the exact characteristics and properties considered to represent a text image object will depend on the specific preferences and requirements of the individual embodiment, and that in particular the determination of the probability values can be adapted to reflect the preferred characteristics and properties of what is considered to be a text box.
  • subtitles may be provided e.g. as white characters in a grey or black box.
  • the probability may be determined to indicate the probability of the pixels belonging to a subtitle box resulting in a gradual positioning of the subtitle box at a desired depth.
  • subtitles may be provided simply as white characters on top of the underlying image. In such an example, there is no bounding box and the probability may reflect the probability of a pixel belonging to a text character.
  • the text image object may specifically be a graphics overlay image object, and specifically may be a subtitle or title image object.
  • the probability unit 107 is coupled to a depth unit 109 which is arranged to generate the modified depth map.
  • the depth unit 109 is further coupled to image unit 105 and a memory/ store 111 wherein the text image object depth value is stored.
  • the depth unit 109 accordingly receives the input depth map, the text image object depth value, and the probability map and it proceeds to perform the weighted combination of the input depth values and the text image object depth values where the weighting depends on the probability value.
  • the combination may be a linear combination such as specifically a weighted summation of the input depth value and the text image object depth value with the weights being dependent on the probability value.
  • weighted combinations can be used where the relative contribution from the input depth level and the text image object depth level is dependent on the probability value.
  • non-linear combinations may be used.
  • the combination (for a given pixel) provides an output depth value as a function of the input depth value, the text image object depth value, and the probability value (for the pixel) with the contribution to the output depth value from respectively the input depth value and the text image object depth value depending on the probability value
  • the weighted combination is such that the weighting of the text image object depth is increased and a weighting of the first depth value is reduced for an increasing value of the probability value.
  • the more probable it is estimated to be that a given pixel belongs to a text image object the closer the output depth value will be to the text image object depth value, and the less probable it is estimated to be that a given pixel belongs to a text image object, the closer the output depth value will be to the input depth value.
  • the exact relationship will depend on the preferences and requirements of the individual embodiment.
  • the output depth value may for a given pixel be a function of the probability value where the function is a function that for a probability value of zero outputs the input depth value for the pixel and for a probability of one outputs the text image object depth value.
  • the function of the probability value may specifically be a monotonic function.
  • the function of the probability value may for a given pixel map the probability range from 0 to 1 to the depth value range from the input depth value for the pixel to the text image object depth value.
  • the text image object depth value may be a predetermined value, and specifically may be a constant value for all images and/or for the entire image. However, in some embodiments, the text image object depth value may vary between different regions of the image, e.g. a preferred text image object depth value for a lower part of the image may be different from the preferred text image object depth value for an upper part of the image.
  • the depth unit 109 is in the example coupled to an image generator 113 which is arranged to generate view images for the autostereoscopic display 101.
  • the image generator 113 receives the modified depth map from the depth unit 109 and is further coupled to the image unit 105 from which it receives the input image.
  • the image generator 113 is arranged to generate the view images for the autostereoscopic display 101 by performing view shifting to generate view images for the specific view directions associated with the different views produced by the autostereoscopic display 101.
  • the image generator 113 is arranged to generate these images by a view shifting algorithm based on the input image and the modified depth map.
  • the view images will be generated to present the pixels at depths that gradually reflect whether they are considered likely to belong to text image objects or not.
  • the projected three dimensional image perceived by a viewer will tend to have more consistent rendering of a text image object with a more gradual transition between text image objects and other image regions. This will typically substantially reduce the perceived imperfections of the 3D presentation.
  • the input image and the modified depth map are used by the image generator 113 to generate an image of the scene of the input image but with a different view direction than the view direction of the input image.
  • the image unit 105 may directly receive a depth map together with an input image.
  • the depth maps may be generated at the same time and place as text image objects are included in or added to e.g. a captured image.
  • the depth map may therefore be generated to have depth levels for text image objects such as subtitles which is at a specific preferred depth.
  • the depth map may for a text image object have a full complete and consistent region corresponding to the area of the text image object and the depth values in this region may all be identical. This may allow the text image object be positioned with little error and few artefacts.
  • the depth level of the text image object may not be the preferred depth level for the specific use scenario. For example, a depth level suitable for viewing using a glasses based approach may not be ideal for presentation using an autostereoscopic display. Further, sharp depth transitions may still result in artefacts when performing view shifting.
  • the depth map may not be generated at the same time and placed as the inclusion of a text image object (such as subtitles).
  • a three dimensional image may be represented by stereo images with subtitles included in both images and with the depth of the subtitles being controlled by the disparity between the two stereo images.
  • Such a three dimensional stereo representation does not include any depth map and therefore such a map may if needed be generated as a post-processing operation.
  • the depth map may be generated in the end user device.
  • the image unit 105 may be arranged to receive a plurality of images which correspond to different view directions for the same scene.
  • a stereo image may be received comprising a left eye image and a right eye image.
  • the stereo image may include a text image object, e.g. a subtitle, with the depth of this being reflected by the disparity between the two images.
  • the image unit 105 may then be arranged to generate a depth map in response to a disparity detection between the view direction images. Thus, the image unit 105 may proceed to find corresponding image objects in the images, determine the relative shift! disparity between these, and assign the corresponding depth level to the image objects. It will be appreciated that any suitable algorithm for determining depth based on disparity estimation may be used.
  • Such a disparity estimation may lead to relatively accurate depth maps.
  • the depth maps will still typically comprise a relatively large number of errors and will typically not be fully consistent.
  • artefacts and inconsistencies may be prevalent around large and sharp depth transitions which may often in particular occur for text image objects.
  • the generation of a modified depth map based on probability detections and a merging of the disparity determined depth value and a desired depth value for text image object tends to result in a substantially improved image and specifically in a substantially improved perceived quality around text image objects with increased consistency and reduced artefacts.
  • the presence of errors, inconsistencies or artefacts that may typically occur around text image objects when performing disparity estimation may be substantially mitigated by the described approach.
  • the determination of the probability values may be performed for pixel groups that may comprise a plurality of pixels.
  • pixel groups may comprise a plurality of pixels.
  • the principles described in the following may be applied to individual pixels, they are in the specific example performed on a pixel group basis, and specifically on a rectangular block basis.
  • each pixel group is a block of 8x8 pixels.
  • the probability unit 107 comprises functionality for determining pixel groups for the input image where each pixel group comprises a plurality of pixels. The determination of probability values is then based on these pixel groups.
  • the probability unit 107 is arranged to first determine a classification map comprising values for the pixel groups where each value indicates whether the corresponding pixel group is designated as belonging to a text image object or not belonging to a text image object.
  • the classification values are binary values and each value accordingly indicates that the corresponding pixel group is designated as belonging to a text image object or that the pixel group is designated as not belonging to a text image object.
  • This approach may in many embodiments facilitate the classification process and may generate robust decisions. Furthermore, it may facilitate the processing and has been found to result in probability maps that are highly suited for the following blending or combination of different depth levels. Indeed, it has been found to result in the generation of three dimensional images being perceived to be of high quality.
  • the values of the classification map may be non-binary values.
  • the object classification process may generate soft decision values for indicating whether the pixel groups belong to a text image object or not.
  • the values may be a set of discrete values in the interval of [0;1] where 1 represents a designation as belonging to a text image object and 0 represents a designation of not belonging to a text image object and with values in between reflecting how closely the pixel group is considered to match the requirements for being designated as belonging to a text image object.
  • the values may in some embodiments be considered as initial probability values and the classification map may be considered as an initial probability map.
  • the probability unit 107 may apply a text image object classification process to the input image which provides a binary decision indicating whether each pixel group belongs to a text image object or not.
  • the color distribution within a pixel group may be evaluated and if that predominantly comprises pixel colors corresponding to those used for subtitle boxes (e.g. white and black), the pixel group may be designated as being a text image object and otherwise it may be designated as not being a text image object.
  • a classification may be based on multiple features that are calculated for each block.
  • Features may for instance be the average horizontal pixel gradient and the number of pixels within a certain bin of a pixel intensity histogram.
  • a machine learning approach such as AdaBoost (http://cseweb.ucsd.edu/ ⁇ yfreund/papers/IntroToBoosting.pdf; date: 20-05-2015) may then be used to (automatically) train a so called ⁇ strong classifier' by linearly combining for instance 50 'weak' classification rules. Note that typically a much larger set of candidate features (e.g.
  • the basic features such as average horizontal pixel gradient, are each used as input to multiple spatial convolution (filter) operations with varying kernel size and thus can each generate e.g. 10 new feature candidates.
  • Training of the classifier is at 8x8 block level using a given set of training images. The resulting algorithm may then be used to classify each block as belonging to a text image object or not.
  • the generated classification values are in the described example binary values that indicate either that a pixel group is designated as belonging to a text image object or that it is designated as not belonging to a text image object.
  • the classification process may generate soft-decision values and these may be used instead of binary values.
  • an AdaBoost classifier may internally use and generate soft decision indications which are compared to a threshold in order to designate the pixel group.
  • these soft-decision values may instead be used for the classification map without any comparison to a threshold to generate binary values.
  • the second phase for generating the probability map is to proceed to apply a filtering to the generated classification map.
  • the filtering may impose temporal and spatial restrictions and smoothing to the classification map.
  • these initial binary values may be transformed into non-binary values that are indicative of the probability of individual pixel groups (or pixels) belonging to a text image object or not.
  • the initial binary classification! designation may be converted into gradual non-binary probability values by filtering being applied to the classification map. This conversion not only allows efficient processing with relatively low complexity and resource requirements but has also been found to provide very good results with the probability values reflecting not only the classification of the group itself but also how this relates to other classifications in the temporal and/or spatial neighborhood of the pixel group. Indeed, the approach tends to result in generation of a probability map which has a high degree of consistency and which is highly suitable for the manipulation of depths of text image objects.
  • the filtering may comprise a plurality of filtering operations.
  • the filtering may comprise a plurality of sequential filtering operations with typically a subsequent filtering operation being performed on the result of the previous filtering operation.
  • the filter operations are performed at a pixel group resolution where each pixel group comprises a plurality of pixels.
  • all filters operate at 8x8 block resolution.
  • indices [ m,n ] will accordingly refer to block indices.
  • the resulting probability map will also be at block resolution.
  • the probability values may be interpolated to provide a higher resolution and specifically to provide pixel level resolution.
  • ⁇ [ m,n ] may be (e.g. linearly) interpolated to calculate ⁇ [ i,j ] which is at pixel level.
  • the filtering may include a binary dilation filtering applied to the classification map.
  • ⁇ init [ m,n ] may denote the raw detection result at block indices [ m,n ] i.e. the binary values of the classification map.
  • the raw detection result either takes the value of 1 (belonging to a text image object) or 0 (not belonging to a text image object).
  • the density of the result i.e.
  • the probability unit 107 may also apply a low pass temporal filtering. If a dilation filtering is performed, the temporal filtering may be applied to the result of this filtering. Otherwise, it may e.g. be applied directly to the classification map.
  • the temporal filtering may be asymmetric, i.e. the low pass effect may be more significant in one direction than the other. Specifically, the time constant for changing to increase the likelihood of pixel groups belonging to text image objects is lower than the time constant for changing to decrease the likelihood of pixel groups belonging to text image objects.
  • a temporal, asymmetric recursive filter may be applied to e.g. the output of the dilation filter (or e.g. directly to the classification map).
  • the filtering comprises spatial filtering. This filtering may be performed directly on the classification map (e.g. after dilation filtering if such is included) or may e.g. be performed after temporal filtering.
  • the spatial filtering may be a linear low pass filter. However, in many embodiments, the spatial filtering may specifically include at least one softmax filter.
  • a softmax filter may be a cascade of filter and a limiter which limits the output to a maximum value.
  • a softmax low-pass filter may perform an operation corresponding to a cascade of a low-pass filter and a limiter limiting the output value of the low-pass filter to a maximum value.
  • a softmax filter may correspond to a filter having a maximum output value limit.
  • the maximum value may be a value of 1 but it will be appreciated that this may differ between different embodiments.
  • a softmax filter may be a filter which performs a standard filter operation such as a low-pass filter, but multiplies the filter output with a value greater than 1, such that the filter output is biased to higher values, after which the minimum is taken of the thus obtained output and a pre-defined maximum value.
  • a softmax filter defined in this way thus consists of three components: a (standard) filter operation, a multiplication factor, and a pre-set maximum.
  • a softmax filtering may comprise applying a spatial (low pass) filtering with a limiting of the filter output to a maximum value.
  • the low pass filtering may be a two stage process of first filtering using a (typically normalized) low pass filter followed by a scaling of the filter output values by a scale factor which is typically predetermined.
  • the resulting scaled output is then limited to the maximum value.
  • the scaling may be considered as part of the overall filtering and could for example be implemented by e.g. scaling (kernel) coefficients of the spatial filter.
  • the maximum value is typically predetermined.
  • a softmax filter may provide improved performance. In particular, it may tend to generate regions in which the inner pixels are indicated to (almost certainly) be text image objects while providing a soft and gradual transition towards the borders of the region. Such a characteristic is particularly suitable for identifying e.g. subtitle boxes using probability values.
  • the spatial filtering may comprise at least two sequential spatial soft-max filters.
  • the different softmax filters may have different characteristics.
  • the spatial filtering may provide two functions. One is to fill the regions corresponding to text image objects, i.e. to make the detected areas overlap such that there are preferably no gaps within the detected regions. The other is to provide a spatial smoothing such that the transitions around the detected areas are smooth and gradual. Both functions result in an improved visual experience.
  • a first softmax filter may be applied to "fill in” detected regions corresponding to text image objects.
  • a second softmax filter may then be applied to the result of the first softmax filter to generate smooth transitions.
  • some text objects may result in a very sparsely filled map of indications of areas belonging to text image objects. Therefore, in order to fill holes inside e.g. a subtitle box, a 'soft-max' filter may be applied.
  • a 'soft-max' filter may be applied.
  • ⁇ may be represented by 8-bit numbers such that a value of 1 is represented by 255.
  • Such a softmax filter may in many embodiments fill holes between characters and words.
  • This first softmax filter may be followed by a second softmax filter which seeks to remove the hard spatial border between a region of the depth map corresponding to a text image object and the surrounding areas of the depth map, and thus to provide a soft transition between text and the rest of the three dimensional image when this is presented.
  • the second softmax filter may correspond to the first one but with different parameters, and specifically with a larger denominator such that the filter acts more like a low-pass filter and less like a max-filter.
  • a kernel size of height 11 and width 25 pixels may be used.
  • FIG. 5 illustrates examples of the results of the described exemplary approach for determining the probability map.
  • FIG. 5 shows the input image 501 and the classification map 503 after binary dilation filtering and temporal filtering.
  • the probability map at this stage can be very sparsely populated (white indicates areas with high probability of belonging to a text image object and black indicates areas with low probability of belonging to a text image object).
  • the output 505 of the first softmax filter is shown. As can be seen, this filtering results in a significantly increased area being considered to be likely to belong to a text image object. However, the transition is quite abrupt.
  • the output 507 of the second softmax filter is also shown. As can be seen, this corresponds fairly closely to the output of the first softmax filter but has substantially smoothed transitions resulting in a much more gradual depth transition in the image rendered by the autostereoscopic display 101.
  • the classification map and the filtering! processing thereof was performed using pixel groups, and specifically pixel blocks, comprising a plurality of pixels. Indeed, in the example, blocks of 8x8 pixels were used. This reduces the resolution by a factor of 64 and accordingly can allow a substantially more efficient and less demanding operation. Furthermore, the Inventor has realized that this resolution reduction can be applied while still achieving desirable performance and perceived image quality.
  • the weights for the weighted combination may be generated at a higher resolution than at the block level resolution, and may specifically be generated at the pixel resolution.
  • interpolation between the block based depth values may be achieved by using interpolation between the block based depth values. For example, interpolation between probability values for neighboring blocks may be used to generate depth values for individual pixels of the blocks. The weights for the interpolation may be determined based on the position of the pixel in the pixel group.
  • interpolation between a first and second depth value may be performed.
  • the depth value may be determined by substantially equally weighting the first and second depth values. E.g. a simple averaging may be performed. However, for a pixel in the center of one pixel group, the depth value may be determined simply as the depth value determined for that pixel group.
  • the invention can be implemented in any suitable form including hardware, software, firmware or any combination of these.
  • the invention may optionally be implemented at least partly as computer software running on one or more data processors and/or digital signal processors.
  • the elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Signal Processing (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Human Computer Interaction (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
EP16720854.5A 2015-05-21 2016-05-06 Method and apparatus for determining a depth map for an image Active EP3298578B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP15168577 2015-05-21
PCT/EP2016/060221 WO2016184700A1 (en) 2015-05-21 2016-05-06 Method and apparatus for determining a depth map for an image

Publications (3)

Publication Number Publication Date
EP3298578A1 EP3298578A1 (en) 2018-03-28
EP3298578B1 true EP3298578B1 (en) 2024-04-10
EP3298578C0 EP3298578C0 (en) 2024-04-10

Family

ID=53432966

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16720854.5A Active EP3298578B1 (en) 2015-05-21 2016-05-06 Method and apparatus for determining a depth map for an image

Country Status (10)

Country Link
US (1) US10580154B2 (es)
EP (1) EP3298578B1 (es)
JP (1) JP6715864B2 (es)
KR (1) KR20180011215A (es)
CN (1) CN107636728B (es)
BR (1) BR112017024765A2 (es)
CA (1) CA2986182A1 (es)
RU (1) RU2718423C2 (es)
TW (1) TWI712990B (es)
WO (1) WO2016184700A1 (es)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10491879B2 (en) 2016-01-15 2019-11-26 Blue River Technology Inc. Plant feature detection using captured images
EP3358844A1 (en) * 2017-02-07 2018-08-08 Koninklijke Philips N.V. Method and apparatus for processing an image property map
EP3462408A1 (en) * 2017-09-29 2019-04-03 Thomson Licensing A method for filtering spurious pixels in a depth-map
US10529085B2 (en) * 2018-03-30 2020-01-07 Samsung Electronics Co., Ltd. Hardware disparity evaluation for stereo matching
CN109285164B (zh) * 2018-09-17 2022-04-05 代黎明 医学图像目标区域定位方法及系统
RU2716311C1 (ru) * 2019-11-18 2020-03-12 федеральное государственное бюджетное образовательное учреждение высшего образования "Донской государственный технический университет" (ДГТУ) Устройство для восстановления карты глубины с поиском похожих блоков на основе нейронной сети
RU2730215C1 (ru) * 2019-11-18 2020-08-20 федеральное государственное бюджетное образовательное учреждение высшего образования "Донской государственный технический университет" (ДГТУ) Устройство для восстановления изображений с поиском похожих блоков на основе нейронной сети

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130100260A1 (en) * 2011-10-21 2013-04-25 Kabushiki Kaisha Toshiba Video display apparatus, video processing device and video processing method
US20140333720A1 (en) * 2013-05-08 2014-11-13 Sony Corporation Subtitle detection for stereoscopic video contents

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
PL3522529T3 (pl) 2005-04-29 2021-09-20 Koninklijke Philips N.V. Stereoskopowe urządzenie wyświetlające
US7668394B2 (en) * 2005-12-21 2010-02-23 Lexmark International, Inc. Background intensity correction of a scan of a document
CN101542529B (zh) 2006-11-21 2012-10-03 皇家飞利浦电子股份有限公司 图像的深度图的生成方法和图像处理单元
JP5487298B2 (ja) * 2009-06-11 2014-05-07 株式会社東芝 ３次元画像生成
US20130124148A1 (en) * 2009-08-21 2013-05-16 Hailin Jin System and Method for Generating Editable Constraints for Image-based Models
US9699434B2 (en) * 2009-10-07 2017-07-04 Samsung Electronics Co., Ltd. Apparatus and method for adjusting depth
US8565554B2 (en) * 2010-01-09 2013-10-22 Microsoft Corporation Resizing of digital images
KR101758058B1 (ko) * 2011-01-20 2017-07-17 삼성전자주식회사 깊이 정보를 이용한 카메라 모션 추정 방법 및 장치, 증강 현실 시스템
KR101975247B1 (ko) * 2011-09-14 2019-08-23 삼성전자주식회사 영상 처리 장치 및 그 영상 처리 방법
US8824797B2 (en) * 2011-10-03 2014-09-02 Xerox Corporation Graph-based segmentation integrating visible and NIR information
US8897542B2 (en) * 2011-12-15 2014-11-25 Sony Corporation Depth map generation based on soft classification
RU2012145349A (ru) * 2012-10-24 2014-05-10 ЭлЭсАй Корпорейшн Способ и устройство обработки изображений для устранения артефактов глубины
TW201432622A (zh) * 2012-11-07 2014-08-16 Koninkl Philips Nv 產生一關於一影像之深度圖
US9191643B2 (en) * 2013-04-15 2015-11-17 Microsoft Technology Licensing, Llc Mixing infrared and color component data point clouds
US9363499B2 (en) * 2013-11-15 2016-06-07 Htc Corporation Method, electronic device and medium for adjusting depth values
CN107851174B (zh) * 2015-07-08 2021-06-01 北京市商汤科技开发有限公司 图像语义标注的设备和方法及其模型的生成方法和系统
US10083162B2 (en) * 2016-11-28 2018-09-25 Microsoft Technology Licensing, Llc Constructing a narrative based on a collection of images

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130100260A1 (en) * 2011-10-21 2013-04-25 Kabushiki Kaisha Toshiba Video display apparatus, video processing device and video processing method
US20140333720A1 (en) * 2013-05-08 2014-11-13 Sony Corporation Subtitle detection for stereoscopic video contents

Also Published As

Publication number Publication date
US10580154B2 (en) 2020-03-03
US20180150964A1 (en) 2018-05-31
CN107636728B (zh) 2022-03-01
TWI712990B (zh) 2020-12-11
TW201710998A (zh) 2017-03-16
RU2017144798A (ru) 2019-06-24
CN107636728A (zh) 2018-01-26
RU2718423C2 (ru) 2020-04-02
EP3298578C0 (en) 2024-04-10
BR112017024765A2 (pt) 2018-07-31
JP6715864B2 (ja) 2020-07-01
CA2986182A1 (en) 2016-11-24
KR20180011215A (ko) 2018-01-31
RU2017144798A3 (es) 2019-09-26
WO2016184700A1 (en) 2016-11-24
EP3298578A1 (en) 2018-03-28
JP2018520531A (ja) 2018-07-26

Similar Documents

Publication Publication Date Title
EP3298578B1 (en) Method and apparatus for determining a depth map for an image
EP3395064B1 (en) Processing a depth map for an image
EP3311361B1 (en) Method and apparatus for determining a depth map for an image
US20030053692A1 (en) Method of and apparatus for segmenting a pixellated image
CN110268712B (zh) 用于处理图像属性图的方法和装置
US8817020B2 (en) Image processing apparatus and image processing method thereof
EP2252071A2 (en) Improved image conversion and encoding techniques
EP1815441B1 (en) Rendering images based on image segmentation
US20120113093A1 (en) Modification of perceived depth by stereo image synthesis
KR102161785B1 (ko) ３차원 이미지의 시차의 프로세싱
KR101629414B1 (ko) 휴먼 팩터에 기초한 스테레오스코픽 이미지 추출 방법 및 장치
EP2677496B1 (en) Method and device for determining a depth image

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20171221

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190711

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: KONINKLIJKE PHILIPS N.V.

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 13/00 20180101ALI20231025BHEP

Ipc: G06V 20/64 20220101ALI20231025BHEP

Ipc: G06V 20/62 20220101ALI20231025BHEP

Ipc: G06V 10/80 20220101ALI20231025BHEP

Ipc: G06V 10/774 20220101ALI20231025BHEP

Ipc: G06F 18/25 20230101ALI20231025BHEP

Ipc: H04N 13/128 20180101ALI20231025BHEP

Ipc: G06T 7/143 20170101ALI20231025BHEP

Ipc: G06T 7/11 20170101ALI20231025BHEP

Ipc: G06T 5/30 20060101ALI20231025BHEP

Ipc: G06T 5/00 20060101AFI20231025BHEP

INTG Intention to grant announced

Effective date: 20231116

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602016086812

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

U01 Request for unitary effect filed

Effective date: 20240410

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT SE SI

Effective date: 20240416

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240521

Year of fee payment: 9

U20 Renewal fee paid [unitary effect]

Year of fee payment: 9

Effective date: 20240716

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240810

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240410

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240711