EP3164992A1 - Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées - Google Patents

Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées

Info

Publication number
EP3164992A1
EP3164992A1 EP15814578.9A EP15814578A EP3164992A1 EP 3164992 A1 EP3164992 A1 EP 3164992A1 EP 15814578 A EP15814578 A EP 15814578A EP 3164992 A1 EP3164992 A1 EP 3164992A1
Authority
EP
European Patent Office
Prior art keywords
spectrum
image
coded aperture
disparity
basis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP15814578.9A
Other languages
German (de)
English (en)
Other versions
EP3164992A4 (fr
Inventor
Vladimir Petrovich PARAMONOV
Ivan Andreevich PANCHENKO
Victor Valentinovich Bucha
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Priority claimed from PCT/KR2015/006966 external-priority patent/WO2016003253A1/fr
Publication of EP3164992A1 publication Critical patent/EP3164992A1/fr
Publication of EP3164992A4 publication Critical patent/EP3164992A4/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/214—Image signal generators using stereoscopic image cameras using a single 2D image sensor using spectral multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/257—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • Apparatuses and methods consistent with exemplary embodiments relate to computational photography, and more particularly, to light field capturing and processing.
  • One of the main applications of light field photography is in extraction of image depth information.
  • Examples of apparatuses for light field capturing or image depth information extraction may include a stereo camera, a plenoptic camera, a camera with a binary coded aperture, and a camera with a color coded aperture.
  • these apparatuses may require additional space, increase costs of cameras, or cause a reduction in optical efficiency.
  • a system for image capturing and depth extraction including: a lens system; a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.
  • FIG. 1 is a diagram of a depth extraction/image restoration apparatus according to an exemplary embodiment
  • FIGS. 2a to 2f are diagrams of spectrum coded apertures according to exemplary embodiments
  • FIGS. 3a to 3i are diagrams for describing a channel shift
  • FIG. 4 is a high-level outline diagram of a depth information extraction/image restoration method according to an exemplary embodiment
  • FIG. 5 is a diagram for describing a parabola fitting according to an exemplary embodiment.
  • FIGS. 6a to 6d are diagrams for describing a depth extraction/image restoration apparatus according to an exemplary embodiment.
  • a system for image capturing and depth extraction including: a lens system; a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.
  • the different spectrum channels may form a basis of the spectrum coded aperture.
  • the processing basis may be different from the sensor basis and the basis of the spectrum coded aperture.
  • the spectrum coded aperture may have three regions, and the three regions may include a transparent region in a central portion, and two regions having spectrum bandwidths respectively corresponding to yellow and cyan.
  • the processing basis may three vectors, and the three vectors may include a vector corresponding to yellow, a vector corresponding to cyan, and a vector perpendicular to the two vector.
  • the spectrum coded aperture may include two regions having spectrum bandwidths respectively corresponding to yellow and cyan.
  • the processing basis may include three vectors, and the three vectors may include a vector respectively corresponding to yellow, a vector corresponding to cyan, and a vector perpendicular to the two vector.
  • the spectrum coded aperture may include three congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta.
  • the processing basis may include vectors corresponding to yellow, cyan, and magenta.
  • the spectrum coded aperture may include three non-congruent regions having spectrum bandwidths respectively corresponding to yellow, cyan, and magenta.
  • the processing basis may include vectors respectively corresponding to yellow, cyan, and magenta.
  • the spectrum coded aperture may have a smooth bandwidth change over an aperture region.
  • the spectrum coded aperture may be fixed to the lens system.
  • the spectrum coded aperture may be attachable to and detachable from the lens system.
  • the spectrum coded aperture may be moved from an optical train that does not participate in the image formation.
  • the captured image may be an image selected from a video sequence.
  • the spectrum coded aperture may insert the image selected from the video sequence into the lens system.
  • the spectrum coded aperture may be inserted into an aperture stop of the lens system.
  • the lens system may include a single lens and the spectrum coded aperture may be located in the lens.
  • the spectrum coded aperture may correct a previous video image of the video sequence acquired by the sensor.
  • the spectrum coded aperture may have a combination of an opaque region and a congruent region, and the congruent region may be transparent or transmit ultraviolet light, infrared light, or visible light.
  • the spectrum coded aperture may have a combination of an opaque region and a non-congruent region, and the non-congruent region may be transparent or transmits ultraviolet light, infrared light, or visible light.
  • the spectrum coded aperture may be a spatial light modulator (SLM).
  • SLM spatial light modulator
  • the data processor may include a preprocessing unit configured to perform the converting the captured image, a disparity estimation unit configured to perform the extracting the disparity, and a conversion unit configured to perform the converting the disparity to the depth information.
  • the data processor may further include an image restoration unit configured to restore the captured image based on the extracted disparity.
  • a method of image capturing and depth extraction including: recording at least two shifted spectrum channels of a light field to form an image captured from a video; converting the captured image into an image of a processing basis; estimating a disparity based on a correlation between pixels of the spectrum channels in the processing basis to extract a disparity map; restoring the captured image based on the extracted disparity map; and converting the disparity map into a depth map.
  • the estimating of the disparity may include: generating candidate images having respective shifts in the spectrum channels; computing matching cost involved in the candidate images in the spectrum channels; propagating a matching cost involved in a low textured region of the candidate images; and estimating a matching cost having a sub-pixel accuracy based on the propagated matching cost.
  • the correlation between the pixels of the spectrum channel for requesting the disparity estimation may include a correlation metric computed in a sparse moving window.
  • the correlation between the pixels of the spectrum channel for requesting the disparity estimation may be computed by using at least one stereo matching algorithm.
  • the computing of the correlation by using the stereo matching algorithm may include sum of absolute differences (SAD), normalized cross correlation (NCC), or Laplacian image contrast (LIC).
  • SAD sum of absolute differences
  • NCC normalized cross correlation
  • LIC Laplacian image contrast
  • the correlation metric may include a fast Fourier transform (FFT).
  • FFT fast Fourier transform
  • the correlation metric may include a recursive exponential filter (REF).
  • REF recursive exponential filter
  • the restoring of the captured image may include performing image blurring.
  • the restoring of the captured image may include performing a spectrum channel alignment in the processing basis.
  • a mobile device for image capturing and depth extraction in ultraviolet light, infrared light, or visible light including: a lens system; at least one spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis; and a coded aperture fixture configured to move at least one spectrum coded aperture relatively with respect to the lens system; and a data processor configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.
  • the coded aperture fixture may be configured to replace at least two spectrum coded apertures in an optical train.
  • the coded aperture fixture may be configured to shift all the spectrum coded apertures from the optical train.
  • the coded aperture fixture may be inserted into an aperture stop.
  • the spectrum coded aperture may have a combination of an opaque region and a congruent region, and the congruent region may be transparent or transmit ultraviolet light, infrared light, or visible light.
  • the spectrum coded aperture may have a combination of an opaque region and a non-congruent region, and the non-congruent region may be transparent or transmits ultraviolet light, infrared light, or visible light.
  • an apparatus for image capturing including: a lens system; at least two spectrum coded apertures including a first aperture and a second aperture which have different characteristics of optical efficiency and depth discrimination from each other; a coded aperture fixture adapted to dispose the first aperture in front of the lens system; and a data processor configured to obtain depth information of an image captured through the first spectrum coded aperture, and control the coded aperture fixture to determine whether to switch the first aperture to the second aperture based on the depth information.
  • the first aperture may include a transparent region placed in the center of the first aperture and two regions separated by the transparent region.
  • the two regions pass different color spectrums, respectively.
  • the two regions may pass a yellow spectrum and a cyan spectrum, respectively.
  • the second aperture may include equally divided two regions which may pass yellow and cyan spectrums, respectively.
  • FIG. 1 is a diagram of a depth extraction/image restoration apparatus 101 according to an exemplary embodiment.
  • the depth extraction/image restoration apparatus 101 may include a camera 102 and a data processor 103.
  • the camera 102 may include optical lens (objective lens) 104, a spectrum coded aperture 105, and a sensor 106.
  • the spectrum coded aperture 105 may be inserted into an optical system which is constituted by the combination of the lens 104, the sensor 106, and other optical parts.
  • the spectrum coded aperture 105 may be placed in an optical path that a ray of light follows through the optical system.
  • the spectrum coded aperture 105 may be a diaphragm plane.
  • the sensor 106 may be configured to discriminate different spectrum bandwidths from one another.
  • the sensor 106 may be a sensor covered with a mosaic color/spectrum filter array, or a color stacked photodiode sensor.
  • the data processor 103 may include a preprocessing unit 108, a disparity estimation unit 109, an image restoration unit 110, and a disparity-to-depth conversion unit 111.
  • the data processor 103 may receive a raw image 107 captured by the camera 102.
  • the preprocessing unit 108 may convert the captured image 107 from a sensor basis to a processing basis in which a spectrum coded aperture filter may not be present.
  • the disparity estimation unit 109 may perform disparity estimation.
  • image restoration unit 110 may perform image restoration.
  • the disparity-to-depth conversion unit 111 may perform disparity-to-depth conversion on optical system parameters.
  • the spectrum coded aperture 105 may be divided into sub-regions that respectively have spectrum passbands.
  • the number, geometric structures, and spectrum passbands of the sub-regions may be changed according to applications of optical efficiency, a depth map, and color image restoration image quality. Some of them are illustrated in FIGS. 2a to 2f.
  • FIGS. 2a to 2f are diagrams illustrating patterns of various spectrum coded apertures having a tradeoff relationship among the optical efficiency, the depth map, and the color image restoration image quality.
  • spectrum filters For light field coding, spectrum filters , , and may be used. Examples of the spectrum filters , , and may include a visibly recognizable color filter, an infrared/ultraviolet filter, and a multi-path filter having two or more passbands
  • Main characteristics of a spectrum coded aperture are optical efficiency, depth discrimination ability, and color image restoration image quality.
  • the highest depth discrimination index may be obtained from a geometric structure of a spectrum coded aperture having the longest distance between the centers of aperture sub-regions corresponding to respective optical spectrum bands.
  • FIG. 2a shows an aperture pattern that has a relatively long distance between the centers of sub-regions , , and and a relatively small filter size in the sub-regions. Consequently, an opaque region of the coded aperture may be increased so that the optical system has a reduced optical efficiency. If the aperture design is deformed to enhance optical efficiency as shown in FIG. 2b, the typically extracted disparity accuracy may be deteriorated.
  • FIG. 2c shows a geometric structure of an aperture having a cyan filter (i.e. ) and a yellow filter (i.e., ) on halves
  • FIG. 2d shows a geometric structure of an aperture having a transparent sub-region a cyan filter (i.e., ), a yellow filter (i.e., ), and a green filter (i.e., ).
  • the yellow filter may have a passband including green and red light spectrums.
  • the cyan filter may have a passband including green and blue light spectrums.
  • the transparent region may not filter incoming light.
  • the green channel may not be distorted by these filters and may be used as a reference in an image restoration process.
  • the aperture structure in FIG. 2c may have a better depth map.
  • the aperture structure of FIG. 2d may have an superior optical efficiency to the aperture structure of FIG. 2c.
  • FIG. 2a shows an aperture having a circular filter and an opaque region, which may be used to obtain a high-quality depth map image when light is excessive.
  • the aperture structure of FIG. 2a may compensate for excessive light directed to the camera 102.
  • An aperture structure having infrared light and ultraviolet light on halves as shown in FIG. 2c may be a fully opened aperture and may have the same optical efficiency and have excellent potential with respect to depth extraction.
  • FIG. 2e shows a spectrum coded aperture having three or more spectrum sub-regions with a hive arrangement
  • FIG. 2f illustrates a spectrum coded aperture having a smooth bandwidth change over an aperture region.
  • the light field which is corrected by the spectrum coded aperture 105, may be input to the image sensor 106 that generates the captured raw image 107.
  • the light field having passed through the spectrum coded aperture 105 may be coded. That is, the light field may be divided into different spectrum parts by passing through corresponding aperture sub-regions. Therefore, different views may be extracted from a single captured image with respect to the same scene by dividing the single captured image into spectrum channels correspondingly with respect to the spectrum coded aperture.
  • FIG. 3a illustrates the captured image 107 obtained by a sensor 106 that is capable of discriminating the corresponding spectrum bandwidth with respect to the spectrum coded aperture described above with reference to FIG. 2b.
  • a position of a defocused object 302 in FIG. 3a which is obtained by the presence of the spectrum coded aperture, may be changed with respect to relatively corresponding spectrum filter positions as shown in FIGS. 3d, 3e, and 3f as compared to a focused object 301 in FIG. 3a).
  • Such a view may be used for extracting a disparity map and restoring the captured image 107.
  • the results of image deblurring with respect to the spectrum channels are illustrated in FIGS. 3g, 3h, and 3i.
  • a deblurred color image is illustrated in FIG. 3b.
  • a deblurred image (restored image) aligned in the spectrum channel is illustrated in FIG. 3c.
  • FIG. 4 is a high-level outline diagram of the data processor 103.
  • a system input may be the raw image 107 captured by the camera 102.
  • the captured image 107 may be preprocessed by denoising and demosaic technologies and be translated from a sensor spectrum basis to a processing basis.
  • the processing basis may not be a spectrum filter. is an image color channel acquired by an optical system sensor.
  • a conversion matrix ⁇ needs to be preferentially estimated.
  • the camera 102 uses the aperture structure having a cyan filter and a yellow as described above with reference to FIG. 2C, and a red, green, blue (RGB) mosaic color filter array.
  • a third basis vector is defined as a vector product .
  • Vectors and are respectively a red basis, a green basis, and a blue basis for the camera sensor 106.
  • the sensor spectrum basis In the sensor spectrum basis,
  • any observed color w may be decomposed by an aperture filter response.
  • the matrix ⁇ may be inversely converted. represents an image channel acquired in the processing basis.
  • an inverse conversion matrix (a left inverse matrix and a right inverse matrix) may be used.
  • a disparity may be estimated with respect to all pixels of the image. is a matching cost for disparity estimation and may use a conventional cross-correlation method of a shifted spectrum channel .
  • a generalized mutual correlation metric may be used in the disparity estimation unit 109 so as to process an arbitrary number of spectrum channels. represents a set of nth acquired views in the nth acquired spectrum channel with respect to the same scene from slightly different viewpoints. represents an frame.
  • a conventional correlation matrix may be expressed by the set and a disparity value d .
  • a determinant of the matrix is a good measure of the mutual correlation .
  • the matrix is a singular matrix and the determinant thereof is 0.
  • the determinant of the matrix is 1.
  • the disparity value d corresponding to the least value of the determinant needs to be found from each pixel of the image.
  • operators for cost computation matching may be used.
  • operators may include conventional stereo matching metrics, Laplacian contrast metrics, and feature based metrics.
  • an exponential moving window may be used because this complies with a naturally sparse gradient prior and propagates a matching cost with respect to a low textured region.
  • an exponential kernel filtering may be efficiently computed by using a recursive convolution in a spectrum domain.
  • This equation may also be used for computing an effective approximate value of a joint bilateral filter for propagating disparity information on a small texture region.
  • Sub-pixel estimation may be performed by using a parabola fitting algorithm as shown in FIG. 5.
  • parabola fitting three given points, , may be taken into consideration. may be represented as (i.e., ), and and may be set as a previous argument and a next argument, respectively.
  • a variable of a maximum value of a unique parabola satisfying , ⁇ and may be analytically computed in the following formula.
  • the image restoration unit 110 may perform preliminary image restoration based on the disparity estimation.
  • the captured image of FIG. 3a may be deblurred as shown in FIG. 3b.
  • a color alignment of the deblurred image may be performed as shown in FIG. 3c.
  • FIG. 3a illustrates an example of the image captured by the system.
  • FIG. 2b illustrates a geometric structure of a spectrum coded aperture.
  • the system may be focused on one object 301 and another object 302 may be defocused.
  • the defocused object 302 captured by the camera 102 may cause a spectrum channel misalignment in a photo array to the extent that the blurred images 305, 306, and 307 as shown in FIG. 3d, FIG. 3e, and FIG. 3f are blurred with respect to a conventional imaging system.
  • the image deblurring may be performed based on a deconvolution technology and be applied to images corresponding to different disparity values. For example, while the focused object 301 does not require the deblurring, the images 305, 306, and 307 of the defocused object 302 in the respective spectrum channels are deblurred with respect to the disparity levels thereof.
  • the deblurred image of FIG. 3b is still misaligned with respect to the spectrum channels , , and , as shown in FIGS. 3g, 3h, and 3i.
  • Misalignment vectors , , and respectively corresponding to the spectrum channels , and may be estimated at the respective positions of the captured image 302.
  • a restored image 304 may be acquired by the aligned spectrum channel, based on the misalignment vectors , , and .
  • i is the number of spectrum channels, and and are projections in an x-axis direction and a y-axis direction of a vector , respectively.
  • the image may be converted from a spectrum filter basis to a device play unit basis .
  • the imaging system has a vignetting effect that results in a reduction of an image’s brightness at the periphery of the image, as compared to the center of image.
  • the vignetting effect may be mathematically alleviated by the following equation.
  • the unvignetting coefficient needs to be independently computed with respect to each spectrum channel. This process may be performed by the image restoration unit 110.
  • a final image refinement process may be used to reduce artifact caused by inaccurate disparity estimation.
  • Technologies based on a human’s visual perception for example, bilateral filtering, median filtering, or the like
  • natural image priors for example, sparse gradient prior, color lines prior, or the like
  • the placement-to-depth conversion unit 111 may convert the disparity into a depth map 114 with respect to a single lens optical system by using generalized optical system parameters 112 generalized in a thin lens formula.
  • This formula for a complex object may depend on the design of the optical system.
  • the above-described image capturing apparatus may be extended for performing a temporal coding and a spectral coding.
  • the temporal coding may be performed while moving the spectrum coded aperture with respect to the image capturing apparatus. This extension may remove a motion blur as well as a known defocus blur caused by a movement of the spectrum coded aperture.
  • the above-described image capturing apparatus may extract depth information from a photograph as well as a video stream that is appropriately encrypted by the coded aperture and is appropriately registered by a detector array.
  • the spectrum coded aperture may be modified so as to mix a photograph and depth information on the image captured according to the presence or absence of the spectrum coded aperture.
  • the depth map extraction process may be performed by just using a key frame (for example, every Nth frames) of a video sequence, and other frames may be restored by using image information and a depth map of the key frame. This process may increase time efficiency and image quality of the system.
  • the type of the spectrum coded aperture and the geometric structure may be changed according to the image automatically captured by the detector array.
  • the aperture including the circular filter and the opaque region, as illustrated in FIG. 2a may be used instead of reducing the exposure time or increasing the f-number of the optical system.
  • the depth extraction/image restoration apparatus may be included in mobile phone camera or web camera equipment, but is not limited thereto.
  • the depth extraction/image restoration apparatus according to the exemplary embodiment may be used in a compact optical camera.
  • FIG. 6a is a diagram of a permanently fixed color coded aperture in an optical system of a camera, according to an exemplary embodiment. Since light passes through a fixed color filter aperture, the image quality of a color image may degrade. Each color band may be projected at different positions of a photograph array causing a ghost image effect. A depth estimation and a color image restoration may be performed by the above-described depth estimation method.
  • FIG. 6b is a diagram of a color coded aperture in which an optical system is movable by a mechanical or electromagnetic unit, according to an exemplary embodiment.
  • the color coded aperture may be present in an optical system to acquire depth information on a scene and a computatively restored color image.
  • the color coded aperture may not be present in an optical system that captures an original 2D image without distortion.
  • the slider also referred to as an aperture fixture
  • the slider may switch between the spectrum coded apertures, for example, according to a control signal from the data processor 103.
  • the present embodiment is not limited thereto, and the spectrum coded apertures may be switched manually or under the control of a central processing unit (CPU) in the smartphone.
  • the data processor 103 may extract depth information from the captured image and determine whether to change the aperture to another one based on the depth information.
  • the data processor 103 may send a control signal to the slider so that the previously used aperture is changed to another one which is known to have a better depth discrimination ability.
  • FIG. 6c is a diagram of a spectrum coded aperture with a spatial light modulator (SLM) capable of changing a spectrum passband of a coded color aperture, based on time, according to an exemplary embodiment.
  • SLM spatial light modulator
  • the apparatus of FIG. 6c may operate in a 2D or 3D mode as described above with reference to the exemplary embodiment of FIG. 6b.
  • the apparatuses of FIGS. 6b and 6c may also acquire alternating video frames.
  • one frame may be obtained in the 2D mode and another frame may be obtained in the 3D mode. Consequently, the system may acquire two video streams.
  • One video frame may include an original color frame acquired in the 2D mode, and another video stream may include a frame suitable for the depth extraction.
  • FIG. 6d is a diagram of a spectrum coded aperture that is attachable to a smartphone lens, according to an exemplary embodiment. Due to a larger size of an optical system, the apparatus of FIG. 6d may obtain more excellent depth map image quality as well as more excellent optical efficiency and video image quality than apparatuses with the attached spectrum coded aperture.
  • the apparatus includes a spectrum filtered aperture, and at least one of a RGB color filter, a red, green, blue, and white (RGBW) color filter, a cyan, magenta, yellow (CMY) filter, a cyan, magenta, yellow, green (CMYG) color filter, and an infrared (IR) filter, but is not limited thereto.
  • RGBW red, green, blue, and white
  • CMY cyan, magenta, yellow
  • CYG cyan, magenta, yellow, green
  • IR infrared
  • the exemplary embodiment may be applied to any digital cameras, including a mobile phone camera, so as to perform mirror hardware modification and generate the disparity/depth maps having low cost algorithms.
  • the acquired disparity map may be used in image splitting, custom blur type (bokeh), computational viewpoint disparity, image filtering, and digital post-refocusing having other special effects.
  • unit may mean a hardware component, such as a processor or a circuit, and/or a software component that is executed by a hardware component such as a processor.
  • an exemplary embodiment can be embodied as computer-readable code on a computer-readable recording medium.
  • the computer-readable recording medium is any data storage device that can store data that can be thereafter read by a computer system. Examples of the computer-readable recording medium include read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.
  • the computer-readable recording medium can also be distributed over network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.
  • an exemplary embodiment may be written as a computer program transmitted over a computer-readable transmission medium, such as a carrier wave, and received and implemented in general-use or special-purpose digital computers that execute the programs.
  • one or more units of the above-described apparatuses and devices can include circuitry, a processor, a microprocessor, etc., and may execute a computer program stored in a computer-readable medium.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Color Television Image Signal Generators (AREA)
  • Measurement Of Optical Distance (AREA)

Abstract

L'invention concerne un système pour une capture d'image et une extraction de profondeur, lequel système comprend une caméra et un processeur de données. La caméra comprend : une ouverture à spectre codé comprenant au moins deux régions qui font passer des canaux de spectre d'un champ de lumière incidente qui sont différents les uns des autres ; et un capteur configuré pour enregistrer les au moins deux canaux de spectre pour former une image capturée dans une base de capteur. Le processeur de données est configuré pour convertir l'image capturée dans la base de capteur en une image d'une base de traitement, extraire une disparité à partir de l'image de la base de traitement, et convertir la disparité en informations de profondeur.
EP15814578.9A 2014-07-04 2015-07-06 Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées Withdrawn EP3164992A4 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
RU2014127469/08A RU2595759C2 (ru) 2014-07-04 2014-07-04 Способ и устройство для захвата изображения и одновременного извлечения глубины
KR1020150083666A KR20160004912A (ko) 2014-07-04 2015-06-12 영상 획득 및 동시의 깊이 추출을 위한 장치 및 방법
PCT/KR2015/006966 WO2016003253A1 (fr) 2014-07-04 2015-07-06 Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées

Publications (2)

Publication Number Publication Date
EP3164992A1 true EP3164992A1 (fr) 2017-05-10
EP3164992A4 EP3164992A4 (fr) 2018-02-21

Family

ID=55172768

Family Applications (1)

Application Number Title Priority Date Filing Date
EP15814578.9A Withdrawn EP3164992A4 (fr) 2014-07-04 2015-07-06 Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées

Country Status (4)

Country Link
EP (1) EP3164992A4 (fr)
KR (1) KR20160004912A (fr)
CN (1) CN106471804B (fr)
RU (1) RU2595759C2 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11893668B2 (en) 2021-03-31 2024-02-06 Leica Camera Ag Imaging system and method for generating a final digital image via applying a profile to image information

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI669538B (zh) * 2018-04-27 2019-08-21 點晶科技股份有限公司 立體影像擷取模組及立體影像擷取方法
CN110891131A (zh) 2018-09-10 2020-03-17 北京小米移动软件有限公司 摄像头模组、处理方法及装置、电子设备、存储介质
JP7256368B2 (ja) * 2019-02-06 2023-04-12 ミツミ電機株式会社 測距カメラ
CN112526801B (zh) * 2019-09-03 2022-01-25 宏达国际电子股份有限公司 双镜头成像模块及其提取方法
CN113362224A (zh) * 2021-05-31 2021-09-07 维沃移动通信有限公司 图像处理方法、装置、电子设备及可读存储介质

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7061693B2 (en) * 2004-08-16 2006-06-13 Xceed Imaging Ltd. Optical method and system for extended depth of focus
WO2009153719A1 (fr) * 2008-06-17 2009-12-23 Koninklijke Philips Electronics N.V. Procédé et dispositif d'observation de l'intérieur de milieux troubles
JP4538766B2 (ja) * 2008-08-21 2010-09-08 ソニー株式会社 撮像装置、表示装置および画像処理装置
US8363093B2 (en) * 2009-07-27 2013-01-29 Eastman Kodak Company Stereoscopic imaging using split complementary color filters
CN103210641B (zh) * 2010-02-19 2017-03-15 双光圈国际株式会社 处理多孔径像数据
KR101220413B1 (ko) * 2010-10-15 2013-01-09 중앙대학교 산학협력단 다중 컬러-필터 조리개를 이용하여 촬영된 영상의 화질 개선 장치 및 방법
EP2748797B1 (fr) * 2011-09-28 2015-03-04 Koninklijke Philips N.V. Détermination de la distance à un objet à partir d'une image
CN102595171B (zh) * 2012-02-03 2014-05-14 浙江工商大学 一种多通道空时编码孔径的动态光场成像方法和成像系统
JP2015521411A (ja) * 2012-05-01 2015-07-27 ペリカン イメージング コーポレイション πフィルタ群を用いてパターン化されたカメラモジュール

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11893668B2 (en) 2021-03-31 2024-02-06 Leica Camera Ag Imaging system and method for generating a final digital image via applying a profile to image information

Also Published As

Publication number Publication date
CN106471804A (zh) 2017-03-01
CN106471804B (zh) 2019-01-04
RU2014127469A (ru) 2016-01-27
EP3164992A4 (fr) 2018-02-21
KR20160004912A (ko) 2016-01-13
RU2595759C2 (ru) 2016-08-27

Similar Documents

Publication Publication Date Title
WO2016003253A1 (fr) Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées
EP3164992A1 (fr) Procédé et appareil pour une capture d'image et une extraction de profondeur simultanées
WO2013125768A1 (fr) Appareil et procédé pour détecter automatiquement un objet et des informations de profondeur d'image photographiée par un dispositif de capture d'image ayant une ouverture de filtre à couleurs multiples
US7773115B2 (en) Method and system for deblurring digital camera images using reference image and motion estimation
WO2013103184A1 (fr) Appareil et procédé pour améliorer une image en utilisant des canaux de couleurs
US11570333B2 (en) Image pickup device and electronic system including the same
WO2016137238A1 (fr) Traitement de données d'image à ouvertures multiples
WO2014142417A1 (fr) Système pour améliorer une image de luminance brumeuse à l'aide d'un modèle d'estimation de réduction de la brume
WO2017007096A1 (fr) Appareil de capture d'image et son procédé de fonctionnement
US20120154541A1 (en) Apparatus and method for producing 3d images
WO2020045946A1 (fr) Dispositif de traitement d'image et procédé de traitement d'image
CN106709894B (zh) 一种图像实时拼接方法及系统
WO2014065544A1 (fr) Appareil de traitement d'image utilisant une caméra différentielle
KR101158678B1 (ko) 입체 영상 시스템 및 입체 영상 처리 방법
WO2020055196A1 (fr) Appareil et procédés de génération d'un média à haute gamme dynamique sur la base d'une compensation de mouvement à étages multiples
WO2017209509A1 (fr) Dispositif de traitement d'images, procédé de traitement d'images à cet effet, et support d'enregistrement non temporaire lisible par ordinateur
CN113727042A (zh) 图像处理系统及方法
WO2022103121A1 (fr) Dispositif électronique d'estimation d'illuminant de caméra et procédé associé
WO2021261737A1 (fr) Dispositif électronique comprenant un capteur d'image, et procédé de commande de celui-ci
CN107517367B (zh) 拜尔域图像插值方法、装置、图像处理芯片及存储装置
WO2018216937A1 (fr) Procédé et dispositif de traitement d'image grâce à un capteur d'image double
JP6807538B2 (ja) 画像処理装置、方法、及びプログラム
JP2021005798A (ja) 撮像装置、撮像装置の制御方法およびプログラム
WO2024043428A1 (fr) Appareil pour réaliser un dé-clôturage de trames multiples et procédé associé
GB2563627A (en) Image processing

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20161222

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20180119

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 13/00 20180101AFI20180115BHEP

Ipc: G06T 7/55 20170101ALI20180115BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190515

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20210421