US20160255323A1 - Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling - Google Patents
Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling Download PDFInfo
- Publication number
- US20160255323A1 US20160255323A1 US14/832,062 US201514832062A US2016255323A1 US 20160255323 A1 US20160255323 A1 US 20160255323A1 US 201514832062 A US201514832062 A US 201514832062A US 2016255323 A1 US2016255323 A1 US 2016255323A1
- Authority
- US
- United States
- Prior art keywords
- sampled
- image data
- image
- blur
- bank
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000005070 sampling Methods 0.000 title description 20
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000001914 filtration Methods 0.000 claims abstract description 13
- 238000003384 imaging method Methods 0.000 claims description 65
- 230000003595 spectral effect Effects 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 6
- 230000003247 decreasing effect Effects 0.000 claims description 5
- 238000002329 infrared spectrum Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims 1
- 238000001429 visible spectrum Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 33
- 238000013459 approach Methods 0.000 description 31
- 238000010586 diagram Methods 0.000 description 14
- 230000000694 effects Effects 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 7
- 230000004044 response Effects 0.000 description 6
- 230000035945 sensitivity Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 229910052710 silicon Inorganic materials 0.000 description 4
- 239000010703 silicon Substances 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 230000005670 electromagnetic radiation Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 239000000758 substrate Substances 0.000 description 3
- 230000007704 transition Effects 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000011248 coating agent Substances 0.000 description 2
- 238000000576 coating method Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003331 infrared imaging Methods 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 101100248200 Arabidopsis thaliana RGGB gene Proteins 0.000 description 1
- 102100022717 Atypical chemokine receptor 1 Human genes 0.000 description 1
- 241000579895 Chlorostilbon Species 0.000 description 1
- 240000006829 Ficus sundaica Species 0.000 description 1
- 229910001218 Gallium arsenide Inorganic materials 0.000 description 1
- 101000678879 Homo sapiens Atypical chemokine receptor 1 Proteins 0.000 description 1
- 239000004642 Polyimide Substances 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 229910021417 amorphous silicon Inorganic materials 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 229920001940 conductive polymer Polymers 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 229910052876 emerald Inorganic materials 0.000 description 1
- 239000010976 emerald Substances 0.000 description 1
- 238000009501 film coating Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 229920001721 polyimide Polymers 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000009416 shuttering Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000010409 thin film Substances 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Images
Classifications
-
- H04N13/0018—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G06T7/0051—
-
- G06T7/408—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
- G06T7/571—Depth or shape recovery from multiple images from focus
-
- H04N13/0037—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/15—Processing image signals for colour aspects of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
- H04N13/218—Image signal generators using stereoscopic image cameras using a single 2D image sensor using spatial multiplexing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/257—Colour aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
- H04N23/12—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with one sensor only
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
- H04N5/2226—Determination of depth image, e.g. for foreground/background separation
-
- H04N5/332—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10141—Special mode during image acquisition
- G06T2207/10148—Varying focus
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10141—Special mode during image acquisition
- G06T2207/10152—Varying illumination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20024—Filtering details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
- H04N13/211—Image signal generators using stereoscopic image cameras using a single 2D image sensor using temporal multiplexing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
- H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
Definitions
- This invention relates to a multi-aperture imaging system that uses multiple apertures of different f-numbers to estimate depth of an object.
- a dual-aperture camera has two apertures.
- a narrow aperture typically at one spectral range such as infrared (IR)
- IR infrared
- a wider aperture typically at another spectral range such as RGB
- the pairs of images captured using the two different apertures can be processed to generate distance information of an object, for example as described in U.S. patent application Ser. No. 13/579,568, which is incorporated herein by reference.
- conventional processing methods can be computationally expensive.
- Embodiments relate to different methods for reducing computations used to estimate depth information.
- One aspect relates to scaling the size of blur kernels used in the depth processing.
- the distance range is divided into sub-ranges.
- a bank of blur kernels is used for each sub-range to estimate distance.
- the blur kernels and captured images are down-sampled by different factors. In this way, although the original blur kernels may span a large range of sizes, the down-sampled blur kernels will be more limited in size which reduces computation.
- processing of images takes advantage of edges in the images.
- the same edge in different images may first be normalized to phase match and/or equate energies in the edges of the two images.
- the edges may be binarized. Binarized edges can be used to reduce computationally expensive convolutions into simpler summing operations.
- only partial blur kernels are used.
- single-sided blur kernels may be used in order to accommodate edges caused by occlusions, where the two sides of the edge are at different depths.
- frequency filtering is used to reduce energy and noise at frequencies that are not useful to distinguish between different blur kernels.
- FIG. 1 is a block diagram of a multi-aperture, shared sensor imaging system according to one embodiment of the invention.
- FIG. 2A is a graph illustrating the spectral responses of a digital camera.
- FIG. 2B is a graph illustrating the spectral sensitivity of silicon.
- FIGS. 3A-3C depict operation of a multi-aperture imaging system according to one embodiment of the invention.
- FIGS. 3D-3E depict operation of an adjustable multi-aperture imaging system according to one embodiment of the invention.
- FIG. 4 is a plot of the blur spot sizes B vis and B ir of visible and infrared images, as a function of object distance s.
- FIG. 5 is a table of blur spot and blur kernel as a function of object distance s.
- FIG. 6A is a diagram illustrating one approach to estimating object distance s.
- FIG. 6B is a graph of error e as a function of kernel number k for the architecture of FIG. 6A .
- FIG. 7A is a diagram illustrating another approach to estimating object distance s.
- FIGS. 7B-7D are graphs of error e as a function of kernel number k for the architecture of FIG. 7A .
- FIG. 8 is a diagram illustrating normalization of edges.
- FIGS. 9A-9E illustrate a simplified approach for convolution of binarized edges.
- FIG. 10 is a diagram illustrating the effect of occlusion.
- FIG. 11 is a diagram illustrating a set of single-sided blur kernels with different edge orientations.
- FIG. 12 is a frequency diagram illustrating the effect of frequency filtering.
- FIG. 1 is a block diagram of a multi-aperture, shared sensor imaging system 100 according to one embodiment of the invention.
- the imaging system may be part of a digital camera or integrated in a mobile phone, a webcam, a biometric sensor, image scanner or any other multimedia device requiring image-capturing functionality.
- the system depicted in FIG. 1 includes imaging optics 110 (e.g., a lens and/or mirror system), a multi-aperture system 120 and an image sensor 130 .
- the imaging optics 110 images objects 150 from a scene onto the image sensor. In FIG. 1 , the object 150 is in focus, so that the corresponding image 160 is located at the plane of the sensor 130 . As described below, this will not always be the case. Objects that are located at other depths will be out of focus at the image sensor 130 .
- the multi-aperture system 120 includes at least two apertures, shown in FIG. 1 as apertures 122 and 124 .
- aperture 122 is the aperture that limits the propagation of visible light
- aperture 124 limits the propagation of infrared or other non-visible light.
- the two apertures 122 , 124 are placed together but they could also be separated.
- This type of multi-aperture system 120 may be implemented by wavelength-selective optical components, such as wavelength filters.
- light opticals
- optical optical are not meant to be limited to the visible part of the electromagnetic spectrum but to also include other parts of the electromagnetic spectrum where imaging may occur, including wavelengths that are shorter than visible (e.g., ultraviolet) and wavelengths that are longer than visible (e.g., infrared).
- the sensor 130 detects both the visible image corresponding to aperture 122 and the infrared image corresponding to aperture 124 .
- there are two imaging systems that share a single sensor array 130 a visible imaging system using optics 110 , aperture 122 and sensor 130 ; and an infrared imaging system using optics 110 , aperture 124 and sensor 130 .
- the imaging optics 110 in this example is fully shared by the two imaging systems, but this is not required.
- the two imaging systems do not have to be visible and infrared. They could be other spectral combinations: red and green, or infrared and white (i.e., visible but without color), for example.
- the exposure of the image sensor 130 to electromagnetic radiation is typically controlled by a shutter 170 and the apertures of the multi-aperture system 120 .
- the aperture system controls the amount of light and the degree of collimation of the light exposing the image sensor 130 .
- the shutter 170 may be a mechanical shutter or, alternatively, the shutter may be an electronic shutter integrated in the image sensor.
- the image sensor 130 typically includes rows and columns of photosensitive sites (pixels) forming a two dimensional pixel array.
- the image sensor may be a CMOS (complementary metal oxide semiconductor) active pixel sensor or a CCD (charge coupled device) image sensor.
- the image sensor may relate to other Si (e.g. a-Si), III-V (e.g. GaAs) or conductive polymer based image sensor structures.
- each pixel When the light is projected by the imaging optics 110 onto the image sensor 130 , each pixel produces an electrical signal, which is indicative of the electromagnetic radiation (energy) incident on that pixel.
- a color filter array 132 is interposed between the imaging optics 110 and the image sensor 130 .
- the color filter array 132 may be integrated with the image sensor 130 such that each pixel of the image sensor has a corresponding pixel filter.
- Each color filter is adapted to pass light of a predetermined color band onto the pixel.
- RGB red, green and blue
- the image sensor may have a stacked design where red, green and blue sensor elements are stacked on top of each other rather than relying on individual pixel filters.
- Each pixel of the exposed image sensor 130 produces an electrical signal proportional to the electromagnetic radiation passed through the color filter 132 associated with the pixel.
- the array of pixels thus generates image data (a frame) representing the spatial distribution of the electromagnetic energy (radiation) passed through the color filter array 132 .
- the signals received from the pixels may be amplified using one or more on-chip amplifiers.
- each color channel of the image sensor may be amplified using a separate amplifier, thereby allowing to separately control the ISO speed for different colors.
- pixel signals may be sampled, quantized and transformed into words of a digital format using one or more analog to digital (A/D) converters 140 , which may be integrated on the chip of the image sensor 130 .
- A/D analog to digital
- the digitized image data are processed by a processor 180 , such as a digital signal processor (DSP) coupled to the image sensor, which is configured to perform well known signal processing functions such as interpolation, filtering, white balance, brightness correction, and/or data compression techniques (e.g. MPEG or JPEG type techniques).
- DSP digital signal processor
- the processor 180 may include signal processing functions 184 for obtaining depth information associated with an image captured by the multi-aperture imaging system. These signal processing functions may provide a multi-aperture imaging system with extended imaging functionality including variable depth of focus, focus control and stereoscopic 3D image viewing capabilities. The details and the advantages associated with these signal processing functions will be discussed hereunder in more detail.
- the processor 180 may also be coupled to additional compute resources, such as additional processors, storage memory for storing captured images and program memory for storing software programs.
- additional compute resources such as additional processors, storage memory for storing captured images and program memory for storing software programs.
- a controller 190 may also be used to control and coordinate operation of the components in imaging system 100 . Functions described as performed by the processor 180 may instead be allocated among the processor 180 , the controller 190 and additional compute resources.
- the imaging optics 110 may be configured to allow both visible light and infrared light or at least part of the infrared spectrum to enter the imaging system. Filters located at the entrance aperture of the imaging optics 110 are configured to allow at least part of the infrared spectrum to enter the imaging system.
- imaging system 100 typically would not use infrared blocking filters, usually referred to as hot-mirror filters, which are used in conventional color imaging cameras for blocking infrared light from entering the camera.
- the light entering the multi-aperture imaging system may include both visible light and infrared light, thereby allowing extension of the photo-response of the image sensor to the infrared spectrum.
- the multi-aperture imaging system is based on spectral combinations other than visible and infrared, corresponding wavelength filters would be used.
- FIGS. 2A and 2B are graphs showing the spectral responses of a digital camera.
- curve 202 represents a typical color response of a digital camera without an infrared blocking filter (hot mirror filter). As can be seen, some infrared light passes through the color pixel filters.
- FIG. 2A shows the photo-responses of a conventional blue pixel filter 204 , green pixel filter 206 and red pixel filter 208 .
- the color pixel filters, in particular the red pixel filter may transmit infrared light so that a part of the pixel signal may be attributed to the infrared.
- FIG. 2B depicts the response 220 of silicon (i.e. the main semiconductor component of an image sensor used in digital cameras).
- the sensitivity of a silicon image sensor to infrared radiation is approximately four times higher than its sensitivity to visible light.
- the image sensor 130 in the imaging system in FIG. 1 may be a conventional image sensor.
- the infrared light is mainly sensed by the red pixels.
- the DSP 180 may process the red pixel signals in order to extract the low-noise infrared information.
- the image sensor may be especially configured for imaging at least part of the infrared spectrum.
- the image sensor may include, for example, one or more infrared (I) pixels in addition to the color pixels, thereby allowing the image sensor to produce a RGB color image and a relatively low-noise infrared image.
- An infrared pixel may be realized by covering a pixel with a filter material, which substantially blocks visible light and substantially transmits infrared light, preferably infrared light within the range of approximately 700 through 1100 nm.
- the infrared transmissive pixel filter may be provided in an infrared/color filter array (ICFA) may be realized using well known filter materials having a high transmittance for wavelengths in the infrared band of the spectrum, for example a black polyimide material sold by Brewer Science under the trademark “DARC 400”.
- an ICFA contain blocks of pixels, e.g. a block of 2 ⁇ 2 pixels, where each block comprises a red, green, blue and infrared pixel.
- ICFA image sensor When exposed, such an ICFA image sensor produces a raw mosaic image that includes both RGB color information and infrared information. After processing the raw mosaic image, a RGB color image and an infrared image may be obtained.
- the sensitivity of such an ICFA image sensor to infrared light may be increased by increasing the number of infrared pixels in a block.
- the image sensor filter array uses blocks of sixteen pixels, with four color pixels (RGGB) and twelve infrared pixels.
- the image sensor 130 may use an architecture where each photo-site includes a number of stacked photodiodes.
- the stack contains four stacked photodiodes responsive to the primary colors RGB and infrared, respectively. These stacked photodiodes may be integrated into the silicon substrate of the image sensor.
- the multi-aperture system e.g. a multi-aperture diaphragm, may be used to improve the depth of field (DOF) or other depth aspects of the camera.
- DOF determines the range of distances from the camera that are in focus when the image is captured. Within this range the object is acceptably sharp.
- DOF is determined by the focal length of the imaging optics N, the f-number associated with the lens opening (the aperture), and/or the object-to-camera distance s. The wider the aperture (the more light received) the more limited the DOF.
- DOF aspects of a multi-aperture imaging system are illustrated in FIG. 3 .
- FIG. 3B shows the imaging of an object 150 onto the image sensor 330 .
- Visible and infrared light may enter the imaging system via the multi-aperture system 320 .
- the multi-aperture system 320 may be a filter-coated transparent substrate.
- One filter coating 324 may have a central circular hole of diameter D 1 .
- the filter coating 324 transmits visible light and reflects and/or absorbs infrared light.
- An opaque cover 322 has a larger circular opening with a diameter D 2 . The cover 322 does not transmit either visible or infrared light.
- the multi-aperture system 320 acts as a circular aperture of diameter D 2 for visible light and as a circular aperture of smaller diameter D 1 for infrared light.
- the visible light system has a larger aperture and faster f-number than the infrared light system. Visible and infrared light passing the aperture system are projected by the imaging optics 310 onto the image sensor 330 .
- the pixels of the image sensor may thus receive a wider-aperture optical image signal 352 B for visible light, overlaying a second narrower-aperture optical image signal 354 B for infrared light.
- the wider-aperture visible image signal 352 B will have a shorter DOF, while the narrower-aperture infrared image signal 354 will have a longer DOF.
- the object 150 B is located at the plane of focus N, so that the corresponding image 160 B is in focus at the image sensor 330 .
- Objects 150 close to the plane of focus N of the lens are projected onto the image sensor plane 330 with relatively small defocus blur.
- Objects away from the plane of focus N are projected onto image planes that are in front of or behind the image sensor 330 .
- the image captured by the image sensor 330 is blurred. Because the visible light 352 B has a faster f-number than the infrared light 354 B, the visible image will blur more quickly than the infrared image as the object 150 moves away from the plane of focus N. This is shown by FIGS. 3A and 3C and by the blur diagrams at the right of each figure.
- FIG. 3B shows the propagation of rays from object 150 B to the image sensor 330 .
- the righthand side of FIG. 3B also includes a blur diagram 335 , which shows the blurs resulting from imaging of visible light and of infrared light from an on-axis point 152 of the object.
- the on-axis point 152 produces a visible blur 332 B that is relatively small and also produces an infrared blur 334 B that is also relatively small. That is because, in FIG. 3B , the object is in focus.
- FIGS. 3A and 3C show the effects of defocus.
- the object 150 A is located to one side of the nominal plane of focus N.
- the corresponding image 160 A is formed at a location in front of the image sensor 330 .
- the light travels the additional distance to the image sensor 330 , thus producing larger blur spots than in FIG. 3B .
- the visible light 352 A is a faster f-number, it diverges more quickly and produces a larger blur spot 332 A.
- the infrared light 354 is a slower f-number, so it produces a blur spot 334 A that is not much larger than in FIG. 3B . If the f-number is slow enough, the infrared blur spot may be assumed to be constant size across the range of depths that are of interest.
- FIG. 3C shows the same effect, but in the opposite direction.
- the object 150 C produces an image 160 C that would fall behind the image sensor 330 .
- the image sensor 330 captures the light before it reaches the actual image plane, resulting in blurring.
- the visible blur spot 332 C is larger due to the faster f-number.
- the infrared blur spot 334 C grows more slowly with defocus, due to the slower f-number.
- the DSP 180 may be configured to process and combine the captured color and infrared images. Improvements in the DOF and the ISO speed provided by a multi-aperture imaging system are described in more detail in U.S. application Ser. No. 13/144,499, “Improving the depth of field in an imaging system”; U.S. application Ser. No. 13/392,101, “Reducing noise in a color image”; U.S. application Ser. No. 13/579,568, “Processing multi-aperture image data”; U.S. application Ser. No. 13/579,569, “Processing multi-aperture image data”; and U.S. application Ser. No. 13/810,227, “Flash system for multi-aperture imaging.” All of the foregoing are incorporated by reference herein in their entirety.
- the multi-aperture imaging system allows a simple mobile phone camera with a typical f-number of 2 (e.g. focal length of 3 mm and a diameter of 1.5 mm) to improve its DOF via a second aperture with a f-number varying e.g. between 6 for a diameter of 0.5 mm up to 15 or more for diameters equal to or less than 0.2 mm.
- the f-number is defined as the ratio of the focal length f and the effective diameter of the aperture.
- Preferable implementations include optical systems with an f-number for the visible aperture of approximately 2 to 4 for increasing the sharpness of near objects, in combination with an f-number for the infrared aperture of approximately 16 to 22 for increasing the sharpness of distance objects.
- the multi-aperture imaging system may also be used for generating depth information for the captured image.
- the DSP 180 of the multi-aperture imaging system may include at least one depth function, which typically depends on the parameters of the optical system and which in one embodiment may be determined in advance by the manufacturer and stored in the memory of the camera for use in digital image processing functions.
- the depth function typically will also include the dependence on the adjustment.
- a fixed lens camera may implement the depth function as a lookup table
- a zoom lens camera may have multiple lookup tables corresponding to different focal lengths, possibly interpolating between the lookup tables for intermediate focal lengths.
- it may store a single lookup table for a specific focal length but use an algorithm to scale the lookup table for different focal lengths.
- a similar approach may be used for other types of adjustments, such as an adjustable aperture.
- a lookup table or a formula when determining the distance or change of distance of an object from the camera, provides an estimate of the distance based on one or more of the following parameters: the blur kernel providing the best match between IR and RGB image data; the f-number or aperture size for the IR imaging; the f-number or aperture size for the RGB imaging; and the focal length.
- the physical aperture is constrained in size, so that as the focal length of the lens changes, the f-number changes. In this case, the diameter of the aperture remains unchanged but the f-number changes.
- the formula or lookup table could also take this effect into account.
- adjusting the relative size of the two apertures may be used to compensate for different lighting conditions. In some cases, it may be desirable to turn off the multi-aperture aspect. As another example, different ratios may be preferable for different object depths, or focal lengths or accuracy requirements. Having the ability to adjust the ratio of IR to RGB provides an additional degree of freedom in these situations.
- FIG. 3D is a diagram illustrating adjustment of the relative sizes of an IR aperture 324 and visible aperture 322 .
- the hashed annulus is a mechanical shutter 370 .
- the mechanical shutter 370 is fully open so that the visible aperture 322 has maximum area.
- the shutter 370 is stopped down, so that the visible aperture 322 has less area but the IR aperture 324 is unchanged so that the ratio between visible and IR can be adjusted by adjusting the mechanical shutter 370 .
- the IR aperture 324 is located near the edge of the visible aperture 322 .
- Stopping down the mechanical shutter 370 reduces the size (and changes the shape) of the IR aperture 324 and the dual-aperture mode can be eliminated by stopping the shutter 370 to the point where the IR aperture 324 is entirely covered. Similar effects can be implemented by other mechanisms, such as adjusting electronic shuttering or exposure time.
- a scene may contain different objects located at different distances from the camera lens so that objects closer to the focal plane of the camera will be sharper than objects further away from the focal plane.
- a depth function may relate sharpness information for different objects located in different areas of the scene to the depth or distance of those objects from the camera.
- a depth function is based on the sharpness of the color image components relative to the sharpness of the infrared image components.
- the sharpness parameter may relate to the circle of confusion, which corresponds to the blur spot diameter measured by the image sensor.
- the blur spot diameter representing the defocus blur is small (approaching zero) for objects that are in focus and grows larger when moving away to the foreground or background in object space.
- the blur disk is smaller than the maximum acceptable circle of confusion, it is considered sufficiently sharp and part of the DOF range. From the known DOF formulas it follows that there is a direct relation between the depth of an object, e.g. its distance s from the camera, and the amount of blur or sharpness of the captured image of that object. Furthermore, this direct relation is different for the color image than it is for the infrared image, due to the difference in apertures and f-numbers.
- the increase or decrease in sharpness of the RGB components of a color image relative to the sharpness of the IR components in the infrared image is a function of the distance to the object. For example, if the lens is focused at 3 meters, the sharpness of both the RGB components and the IR components may be the same. In contrast, due to the small aperture used for the infrared image for objects at a distance of 1 meter, the sharpness of the RGB components may be significantly less than those of the infrared components. This dependence may be used to estimate the distances of objects from the camera.
- the imaging system is set to a large (“infinite”) focus point. That is, the imaging system is designed so that objects at infinity are in focus. This point is referred to as the hyperfocal distance H of the multi-aperture imaging system.
- the system may then determine the points in an image where the color and the infrared components are equally sharp. These points in the image correspond to objects that are in focus, which in this example means that they are located at a relatively large distance (typically the background) from the camera.
- the hyperfocal distance H i.e., closer to the camera
- the relative difference in sharpness between the infrared components and the color components will change as a function of the distance s between the object and the lens.
- the sharpness may be obtained empirically by measuring the sharpness (or, equivalently, the blurriness) for one or more test objects at different distances s from the camera lens. It may also be calculated based on models of the imaging system. In one embodiment, sharpness is measured by the absolute value of the high-frequency infrared components in an image. In another approach, blurriness is measured by the blur size or point spread function (PSF) of the imaging system.
- PSF point spread function
- FIG. 4 is a plot of the blur spot sizes B vis and B ir of the visible and infrared images, as a function of object distance s.
- FIG. 4 shows that around the focal distance N, which in this example is the hyperfocal distance, the blur spots are the smallest. Away from the focal distance N, the color components experience rapid blurring and rapid increase in the blur spot size B vis . In contrast, as a result of the relatively small infrared aperture, the infrared components do not blur as quickly and, if the f-number is slow enough, the blur spot size B ir may be approximated as constant in size over the range of depths considered.
- the infrared image is produced with a blur spot 410 and the visible image is produced with a blur spot 420 .
- this information could be used to estimate the object distance s x .
- the blur spot also referred to as the point spread function, is the image produced by a single point source. If the object were a single point source, then the infrared image will be a blur spot of size 410 and the corresponding visible image will be a blur spot of size 420 .
- FIG. 5 illustrates one approach to estimating the object distance based on the color and infrared blur spots.
- FIG. 5 is a table of blur spot as a function of object distance s. For each object distance s k , there is shown a corresponding IR blur spot (PSF ir ) and color blur spot (PSF vis ).
- the IR image I n is the convolution of an ideal image I ideal with PSF ir
- the color image I vis is the convolution of the ideal image I ideal with PSF vis .
- B is a blur kernel that accounts for deblurring of the IR image followed by blurring of the visible image.
- the blur kernels B can be calculated in advance or empirically measured as a function of object depth s, producing a table as shown in FIG. 5 .
- the blur kernel B is shown as similar in size to the visible blur spot PSF vis .
- the IR blur spot PSF ir may be neglected or otherwise accounted for.
- the IR blur spot is small relative to the visible blur spot PSF vis , then neglecting the effect of the IR blur may be negligible.
- the IR blur spot does not vary significantly with object distance, then it may be neglected for purposes of calculating the blur kernel B, but may be accounted for by a systematic adjustment of the results.
- FIG. 6A is a diagram illustrating a method for producing an estimate s* of the object distance s using a bank 610 of blur kernels B k .
- the infrared image I n is blurred by each of the blur kernels B k in the bank.
- the blurring is accomplished by convolution, although faster approaches will be discussed below. This results in estimated visible images I* vis .
- Each of these estimated images I* vis is compared 620 to the actual visible image I vis .
- the comparison is a sum squared error e k between the two images.
- FIG. 6B is a graph of error e as a function of kernel number k for the architecture of FIG. 6A .
- each kernel number k corresponds to a specific object distance s.
- the error metrics e are processed 630 to yield an estimate s* of the object distance.
- the minimum error e k is identified, and the estimated object distance s* is the object depth s k corresponding to the minimum error e k .
- the functional pairs (s k ,e k ) can be interpolated for the value of s that yields the minimum e.
- the infrared image I ir and visible image I vis in FIG. 6A typically are not the entire captured images. Rather, the approach of FIG. 6A can be applied to different windows within the image in order to estimate the depth of the objects in the window. In this way, a depth map of the entire image can be produced.
- the approach of FIG. 6A includes a convolution for each blur kernel. If the window and blur kernel B k are each large, the convolution can be computationally expensive.
- the blur kernels B k by definition will vary in size. For example, the smallest blur kernel may be 3 ⁇ 3 while the largest may be 25 ⁇ 25 or larger.
- the window In order to accommodate the largest blur kernels, the window should be at least the same size as the largest blur kernel, which means a large window size is required for a bank that includes a large blur kernel. Furthermore, the same window should be used for all blur kernels in order to allow direct comparison of the calculated error metrics. Therefore, if the bank includes a large blur kernel, a large window will be used for all blur kernels, which can lead to computationally expensive convolutions.
- FIG. 7A is a diagram illustrating a variation of FIG. 6A that addresses this issue.
- the approach of FIG. 7A uses multiple banks 710 a -M of blur kernels.
- Each bank contains multiple blur kernels.
- each bank 710 is down-sampled by a different down-sampling factor.
- bank 710 a may use the smallest blur kernels and the original images without down-sampling
- bank 710 b may use the next smallest set of kernels but with down-sampling of 2 ⁇ , and so on.
- bank 710 m uses down-sampling of mx.
- the visible image and the infrared image are also down-sampled by mx, as indicated by the boxes marked “/m”.
- Bank 710 m uses blur kernels J to (J+K), each of which is also down-sampled by mx, as indicated by the “/m” in “*B J /m”.
- Each bank 710 produces a result, for example an estimated object distance s m * and these are combined 730 into an overall depth estimate s*.
- the table below shows a set of 9 blur kernels, ranging in size from 3 ⁇ 3 for blur kernel 1 , to 25 ⁇ 25 for blur kernel 9 .
- blur kernel 9 would be 25 ⁇ 25 with a corresponding number of multiply-accumulates used to implement convolution.
- all blur kernels are down-sampled so that no convolution uses a kernel larger than 5 ⁇ 5.
- FIGS. 7B and 7C are graphs of error as a function of blur kernel number k for the architecture of FIG. 7A . If the down-sampling is performed without normalizing energies, then the error curve may exhibit discontinuities when transitioning from one bank to the next bank.
- FIG. 7B shows an error curve using five banks Each piece of the curve corresponds to one of the banks Each curve is continuous because the same down-sampling factor is used for all blur kernels in that bank. However, the down-sampling factor changes from one bank to the next so the different pieces of the curve may not align correctly. However, the minimum error can still be determined.
- curve 750 c is the only curve that has a minimum within that curve. The other four curves are either monotonically increasing or monotonically decreasing.
- the minimum error occurs within curve 750 c .
- More sophisticated approaches may also be used. For example, differentials across the entire range of curves may be analyzed to predict the point of minimum error. This approach can be used to avoid local minima, which may be caused by noise or other effects.
- FIG. 7B the curves are shown as continuous within each bank. However, there may be a limited number of samples for each bank.
- FIG. 7C is the same as FIG. 7B , except that there are only three samples for each bank.
- the dashed ovals identify each of the banks
- Each of the banks can be classified as monotonically increasing, monotonically decreasing or containing an extremum.
- banks 750 a and 750 b are monotonically decreasing
- bank 750 c contains an extremum
- banks 750 d and 750 e are monotonically increasing. Based on these classifications, the minimum error e occurs somewhere within bank 750 c . Finer resolution sampling within bank 750 c can then be performed to more accurately locate the location of the minimum value.
- banks 750 a and 750 b are monotonically decreasing, and banks 750 c and 750 d are monotonically increasing.
- the minimum lies in the range covered by banks 750 b and 750 c .
- another bank can be constructed that spans the gap between banks 750 b and 750 c . That bank will then have an internal minimum.
- the error function e(k) may be coarsely sampled at first in order to narrow the range of k where the minimum error e exists. Finer and finer sampling may be used as the range is narrowed. Other sampling approaches can be used to find the value of kernel number k (and the corresponding object distance) where the extremum of the error function e(k) occurs.
- Down-sampling can be implemented in other ways.
- the visible images may be down-sampled first.
- the blur kernels are then down-sampled to match the down-sampling of the visible images.
- the down-sampled blur kernels are applied to the full resolution IR images.
- the result is an intermediate form which retains the fill resolution of the IR image but then is down-sampled to match the resolution of the down-sampled visible images.
- This method is not as efficient as fully down-sampling the IR but is more efficient than not using down-sampling at all. This approach may be beneficial to reduce computation while still maintaining a finer resolution.
- FIG. 6A Another aspect is that the approach of FIG. 6A depends on the content of the window.
- a window for which the only object is a single point source object e.g., a window containing a single star surrounded entirely by black night sky
- a window that contains the image of only an edge will also yield a good result because that image is a direct measure of the underlying point spread functions albeit only along one direction.
- a window that is constant and has no features will not yield any estimate because every estimated visible image will also be a constant so there is no way to distinguish the different blur kernels. Other images may be somewhere between these extremes.
- Features will help distinguish the different blur kernels. Featureless areas will not and typically will also add unwanted noise.
- the windows are selected to include edges.
- Edge identification can be accomplished using known algorithms. Once identified, edges preferably are processed to normalize variations between the different captured images.
- FIG. 8 shows one example.
- the green component I gm of the color image is the fast f-number image and the IR image I ir is the slow f-number image.
- the left column of FIG. 8 shows processing of the green image while the right column shows processing of the IR image.
- the top row shows the same edge appearing in both images.
- the object is not in focus so that the green edge is blurred relative to the IR edge.
- the edge has different phase in the two images.
- the green edge transitions from high to low amplitude, while the IR edge transitions from low to high amplitude.
- FIG. 8 shows one approach to normalize these edges to allow comparisons using blur kernels as described above.
- the second row of FIG. 8 shows both edges after differentiation 810 .
- the absolute value 820 of the derivatives is then taken, yielding the third row of FIG. 8 .
- the two edges are then scaled 830 , resulting in the bottom row of FIG. 8 .
- the IR image is binarized to take on only the values 0 or 1, and the green image is scaled in amplitude to have equal energy as the IR image.
- the blur kernels are also scaled in amplitude so that, although a blur kernel might spread the energy in an image over a certain area, it does not increase or decrease the total energy. This then allows a direct comparison between the actual green edge and the estimated green edges calculated by applying the blur kernels to the IR edge.
- the IR edge looks like a line source. This is not uncommon since the IR point spread function is small and fairly constant over a range of depths, compared to the color point spread function. Also recall that in FIG. 6 , the IR image is convolved with many different blur kernels. The convolution can be simplified as follows. First, the IR edge is binarized, so that the IR image is a binary image taking on only the values of 0 or 1. (In step 830 above, the color image is then scaled in amplitude to have equal energy as the binary IR image). Convolution generally requires multiplies and adds. However, when the image only takes values of 0 or 1, the multiplies are simplified.
- Multiplying by 0 yields all 0's so that pixels with 0 value can be ignored.
- Multiplying by 1 yields the blur kernel so that no actual multiplication is required. Rather, any pixel with 1 value causes an accumulation of the blur kernel centered on that pixel.
- FIGS. 9A-9E illustrate this concept.
- FIG. 9A shows a 4 ⁇ 4 window with a binarized edge, where the pixels are either 1 or 0.
- FIG. 9B shows a 3 ⁇ 3 blur kernel to be convolved with the window.
- FIGS. 9C-9E show progression of the convolution using only adds and no multiplies.
- the lefthand side shows the binarized edge of FIG. 9A and the righthand side shows progression of the convolution.
- FIG. 9C pixel 910 has been processed, meaning that the blur kernel centered on pixel 910 has been added to the moving sum on the right.
- FIG. 9D the next pixel along the edge 911 has been processed.
- FIG. 9E shows the final result after all four edge pixels have been processed. This is the estimated green edge, which can then be compared to the actual green edge. If the two match well, then the blur kernel shown in FIG. 9B is the correct blur kernel for this window and can be used to estimate the object distance for this edge.
- Edges in an image may be caused by a sharp transition within an object, for example the border between black and white squares on a checkerboard. In that case, the approach shown in FIG. 9 may be implemented using entire blur kernels. However, edges may also be caused by occlusion, when a closer object partially blocks a more distant object.
- the sign 1010 in the foreground partially blocks the house 1020 in the background. This creates an edge 1030 in the image.
- the left side of the edge is the sign 1010 , which is at a closer object distance
- the right side of the edge is the house 1020 , which is at a farther object distance.
- the two different object distances correspond to different blur kernels. Applying a single blur kernel to the edge will not give good results, because when one side is matched to the blur kernel, the other side will not be.
- a single-sided blur kernel is half a blur kernel instead of an entire blur kernel.
- FIG. 11 shows a set of eight single-sided blur kernels with different edge orientations based on the 3 ⁇ 3 blur kernel of FIG. 9B . The full 3 ⁇ 3 blur kernel is reproduced in the center of FIG. 11 . Note that different single-sided blur kernels can be derived from the same full blur kernel, depending on the orientation of the edge. In FIG. 11 , the solid line 1110 represents the edge. These single-sided blur kernels can be applied to binarized edges, as described above, to yield different depth estimates for each side of the edge.
- FIG. 12 illustrates another aspect of the approach described above.
- a bank of blur kernels of varying sizes is used to estimate the object depth.
- Blur kernels effectively act as low pass filters. Larger blur kernels cause more blurring and therefore have lower cutoff frequencies compared to smaller blur kernels.
- FIG. 12 shows a generalized frequency response for a bank of blur kernels.
- Blur kernel 1210 A is the low pass filter with the lowest cutoff frequency in the bank, which corresponds to the blur kernel with the largest blur size.
- Blur kernel 1210 B is the second largest blur kernel and so on to blur kernel 1210 D, which has the highest cutoff frequency and smallest blur size.
- the IR image is blurred by each of these blur kernels, and the results are compared to determine which blur kernel corresponds to the object depth.
- the blur kernels 1210 A-D differ only within the frequency range 1220 . Outside this frequency range 1220 , all of the blur kernels 1210 A-D in the bank have the same behavior. Therefore, content outside the frequency range 1220 will not distinguish between the different blur kernels 1210 A-D. However, that content will add to background noise. Therefore, in one approach, frequency filtering is added to reduce energy and noise from outside the frequency range 1220 .
- the original images are frequency filtered.
- the blur kernels may be frequency filtered versions.
- the frequency filtering may be low pass filtering to reduce frequency content above frequency 1220 B, high pass filtering to reduce frequency content below frequency 1220 A, or bandpass filtering to reduce both the low frequency and high frequency content.
- the filtering may take different forms and may be performed regardless of whether down-sampling is also used. When it is used, down-sampling is a type of low pass filtering.
- the filtering may also be applied to less than or more than all the blur kernels in a bank.
- a narrower bandpass filter may be used if it is desired to distinguish only blur kernels 1210 A and 1210 B (i.e., to determine the error gradient between blur kernels 1210 A- 1210 B). Most of the difference between those two blur kernels occurs in the frequency band 1230 , so a bandpass filter that primarily passes frequencies within that range and rejects frequencies outside that range will increase the relative signal available for distinguishing the two blur kernels 1210 A and 1210 B.
- Window sizes and locations preferably are selected based on the above considerations, and the window size may be selected independent of the blur kernel size.
- window size may be selected to be large enough to contain features such as edges, small enough to avoid interfering features such as closely spaced parallel edges, and generally only large enough to allow processing of features since larger windows will add more noise.
- the size of the blur kernel may be selected to reduce computation (e.g., by down-sampling) and also possibly in order to provide sufficient resolution for the depth estimation.
- the window size may be different (typically, larger) than the size of the blur kernels.
- the number of windows and window locations may also be selected to contain features such as edges, and to reduce computation.
- a judicious choice of windows can reduce power consumption by having fewer pixels to power up and to read out, which in turn can be used to increase the frame rate.
- a higher frame rate may be advantageous for many reasons, for example in enabling finer control of gesture tracking.
- Embodiments of the invention may be implemented as a program product for use with a computer system.
- the program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media.
- Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored.
- non-writable storage media e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory,
- any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments.
- the invention is not limited to the embodiments described above, which may be varied within the scope of the accompanying claims.
- aspects of this technology have been described with respect to different f-number images captured by a multi-aperture imaging system.
- these approaches are not limited to multi-aperture imaging systems. They can also be used in other systems that estimate depth based on differences in blurring, regardless of whether a multi-aperture imaging system is used to capture the images.
- two images may be captured in time sequence, but at different f-number settings.
- Another method is to capture two or more images of the same scene but with different focus settings, or to rely on differences in aberrations (e.g., chromatic aberrations) or other phenomenon that cause the blurring of the two or more images to vary differently as a function of depth so that these variations can be used to estimate the depth.
- aberrations e.g., chromatic aberrations
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
Abstract
Embodiments relate to different methods for reducing computations used to estimate depth information. One aspect relates to using down-sampled blur kernels. Another aspect relates to processing of edges in the images. Yet another aspect relates to using partial blur kernels, such as single-sided blur kernels. Yet another aspect relates to frequency filtering to reduce energy and noise at frequencies that do not distinguish between different blur kernels.
Description
- This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No. 62/121,203, “Dual-Aperture Depth Map Using Adaptive PSF Sizing,” filed Feb. 26, 2015. The subject matter of all of the foregoing is incorporated herein by reference in its entirety.
- 1. Field of the Invention
- This invention relates to a multi-aperture imaging system that uses multiple apertures of different f-numbers to estimate depth of an object.
- 2. Description of Related Art
- A dual-aperture camera has two apertures. A narrow aperture, typically at one spectral range such as infrared (IR), produces relatively sharp images over a long depth of focus. A wider aperture, typically at another spectral range such as RGB, produces sometimes blurred images for out of focus objects. The pairs of images captured using the two different apertures can be processed to generate distance information of an object, for example as described in U.S. patent application Ser. No. 13/579,568, which is incorporated herein by reference. However, conventional processing methods can be computationally expensive.
- Therefore, there is a need to improve approaches for depth map generation.
- Embodiments relate to different methods for reducing computations used to estimate depth information. One aspect relates to scaling the size of blur kernels used in the depth processing. The distance range is divided into sub-ranges. A bank of blur kernels is used for each sub-range to estimate distance. For different sub-ranges, the blur kernels and captured images are down-sampled by different factors. In this way, although the original blur kernels may span a large range of sizes, the down-sampled blur kernels will be more limited in size which reduces computation.
- In another aspect, processing of images takes advantage of edges in the images. The same edge in different images may first be normalized to phase match and/or equate energies in the edges of the two images. In another aspect, the edges may be binarized. Binarized edges can be used to reduce computationally expensive convolutions into simpler summing operations.
- In another aspect, rather than using full blur kernels, only partial blur kernels are used. For example, single-sided blur kernels may be used in order to accommodate edges caused by occlusions, where the two sides of the edge are at different depths.
- In yet another aspect, frequency filtering is used to reduce energy and noise at frequencies that are not useful to distinguish between different blur kernels.
- Other aspects include components, devices, systems, improvements, methods, processes, applications, computer readable mediums, and other technologies related to any of the above.
- Embodiments of the disclosure have other advantages and features which will be more readily apparent from the following detailed description and the appended claims, when taken in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a block diagram of a multi-aperture, shared sensor imaging system according to one embodiment of the invention. -
FIG. 2A is a graph illustrating the spectral responses of a digital camera. -
FIG. 2B is a graph illustrating the spectral sensitivity of silicon. -
FIGS. 3A-3C depict operation of a multi-aperture imaging system according to one embodiment of the invention. -
FIGS. 3D-3E depict operation of an adjustable multi-aperture imaging system according to one embodiment of the invention. -
FIG. 4 is a plot of the blur spot sizes Bvis and Bir of visible and infrared images, as a function of object distance s. -
FIG. 5 is a table of blur spot and blur kernel as a function of object distance s. -
FIG. 6A is a diagram illustrating one approach to estimating object distance s. -
FIG. 6B is a graph of error e as a function of kernel number k for the architecture ofFIG. 6A . -
FIG. 7A is a diagram illustrating another approach to estimating object distance s. -
FIGS. 7B-7D are graphs of error e as a function of kernel number k for the architecture ofFIG. 7A . -
FIG. 8 is a diagram illustrating normalization of edges. -
FIGS. 9A-9E illustrate a simplified approach for convolution of binarized edges. -
FIG. 10 is a diagram illustrating the effect of occlusion. -
FIG. 11 is a diagram illustrating a set of single-sided blur kernels with different edge orientations. -
FIG. 12 is a frequency diagram illustrating the effect of frequency filtering. - The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.
-
FIG. 1 is a block diagram of a multi-aperture, sharedsensor imaging system 100 according to one embodiment of the invention. The imaging system may be part of a digital camera or integrated in a mobile phone, a webcam, a biometric sensor, image scanner or any other multimedia device requiring image-capturing functionality. The system depicted inFIG. 1 includes imaging optics 110 (e.g., a lens and/or mirror system), amulti-aperture system 120 and animage sensor 130. Theimaging optics 110images objects 150 from a scene onto the image sensor. InFIG. 1 , theobject 150 is in focus, so that thecorresponding image 160 is located at the plane of thesensor 130. As described below, this will not always be the case. Objects that are located at other depths will be out of focus at theimage sensor 130. - The
multi-aperture system 120 includes at least two apertures, shown inFIG. 1 asapertures aperture 122 is the aperture that limits the propagation of visible light, andaperture 124 limits the propagation of infrared or other non-visible light. In this example, the twoapertures multi-aperture system 120 may be implemented by wavelength-selective optical components, such as wavelength filters. As used in this disclosure, terms such as “light” “optics” and “optical” are not meant to be limited to the visible part of the electromagnetic spectrum but to also include other parts of the electromagnetic spectrum where imaging may occur, including wavelengths that are shorter than visible (e.g., ultraviolet) and wavelengths that are longer than visible (e.g., infrared). - The
sensor 130 detects both the visible image corresponding toaperture 122 and the infrared image corresponding toaperture 124. In effect, there are two imaging systems that share a single sensor array 130: a visible imagingsystem using optics 110,aperture 122 andsensor 130; and an infrared imagingsystem using optics 110,aperture 124 andsensor 130. Theimaging optics 110 in this example is fully shared by the two imaging systems, but this is not required. In addition, the two imaging systems do not have to be visible and infrared. They could be other spectral combinations: red and green, or infrared and white (i.e., visible but without color), for example. - The exposure of the
image sensor 130 to electromagnetic radiation is typically controlled by ashutter 170 and the apertures of themulti-aperture system 120. When theshutter 170 is opened, the aperture system controls the amount of light and the degree of collimation of the light exposing theimage sensor 130. Theshutter 170 may be a mechanical shutter or, alternatively, the shutter may be an electronic shutter integrated in the image sensor. Theimage sensor 130 typically includes rows and columns of photosensitive sites (pixels) forming a two dimensional pixel array. The image sensor may be a CMOS (complementary metal oxide semiconductor) active pixel sensor or a CCD (charge coupled device) image sensor. Alternatively, the image sensor may relate to other Si (e.g. a-Si), III-V (e.g. GaAs) or conductive polymer based image sensor structures. - When the light is projected by the
imaging optics 110 onto theimage sensor 130, each pixel produces an electrical signal, which is indicative of the electromagnetic radiation (energy) incident on that pixel. In order to obtain color information and to separate the color components of an image which is projected onto the imaging plane of the image sensor, typically acolor filter array 132 is interposed between theimaging optics 110 and theimage sensor 130. Thecolor filter array 132 may be integrated with theimage sensor 130 such that each pixel of the image sensor has a corresponding pixel filter. Each color filter is adapted to pass light of a predetermined color band onto the pixel. Usually a combination of red, green and blue (RGB) filters is used. However other filter schemes are also possible, e.g. CYGM (cyan, yellow, green, magenta), RGBE (red, green, blue, emerald), etc. Alternately, the image sensor may have a stacked design where red, green and blue sensor elements are stacked on top of each other rather than relying on individual pixel filters. - Each pixel of the exposed
image sensor 130 produces an electrical signal proportional to the electromagnetic radiation passed through thecolor filter 132 associated with the pixel. The array of pixels thus generates image data (a frame) representing the spatial distribution of the electromagnetic energy (radiation) passed through thecolor filter array 132. The signals received from the pixels may be amplified using one or more on-chip amplifiers. In one embodiment, each color channel of the image sensor may be amplified using a separate amplifier, thereby allowing to separately control the ISO speed for different colors. - Further, pixel signals may be sampled, quantized and transformed into words of a digital format using one or more analog to digital (A/D)
converters 140, which may be integrated on the chip of theimage sensor 130. The digitized image data are processed by aprocessor 180, such as a digital signal processor (DSP) coupled to the image sensor, which is configured to perform well known signal processing functions such as interpolation, filtering, white balance, brightness correction, and/or data compression techniques (e.g. MPEG or JPEG type techniques). - The
processor 180 may include signal processing functions 184 for obtaining depth information associated with an image captured by the multi-aperture imaging system. These signal processing functions may provide a multi-aperture imaging system with extended imaging functionality including variable depth of focus, focus control and stereoscopic 3D image viewing capabilities. The details and the advantages associated with these signal processing functions will be discussed hereunder in more detail. - The
processor 180 may also be coupled to additional compute resources, such as additional processors, storage memory for storing captured images and program memory for storing software programs. Acontroller 190 may also be used to control and coordinate operation of the components inimaging system 100. Functions described as performed by theprocessor 180 may instead be allocated among theprocessor 180, thecontroller 190 and additional compute resources. - As described above, the sensitivity of the
imaging system 100 is extended by using infrared imaging functionality. To that end, theimaging optics 110 may be configured to allow both visible light and infrared light or at least part of the infrared spectrum to enter the imaging system. Filters located at the entrance aperture of theimaging optics 110 are configured to allow at least part of the infrared spectrum to enter the imaging system. In particular,imaging system 100 typically would not use infrared blocking filters, usually referred to as hot-mirror filters, which are used in conventional color imaging cameras for blocking infrared light from entering the camera. Hence, the light entering the multi-aperture imaging system may include both visible light and infrared light, thereby allowing extension of the photo-response of the image sensor to the infrared spectrum. In cases where the multi-aperture imaging system is based on spectral combinations other than visible and infrared, corresponding wavelength filters would be used. -
FIGS. 2A and 2B are graphs showing the spectral responses of a digital camera. InFIG. 2A ,curve 202 represents a typical color response of a digital camera without an infrared blocking filter (hot mirror filter). As can be seen, some infrared light passes through the color pixel filters.FIG. 2A shows the photo-responses of a conventionalblue pixel filter 204,green pixel filter 206 andred pixel filter 208. The color pixel filters, in particular the red pixel filter, may transmit infrared light so that a part of the pixel signal may be attributed to the infrared.FIG. 2B depicts theresponse 220 of silicon (i.e. the main semiconductor component of an image sensor used in digital cameras). The sensitivity of a silicon image sensor to infrared radiation is approximately four times higher than its sensitivity to visible light. - In order to take advantage of the spectral sensitivity provided by the image sensor as illustrated by
FIGS. 2A and 2B , theimage sensor 130 in the imaging system inFIG. 1 may be a conventional image sensor. In a conventional RGB sensor, the infrared light is mainly sensed by the red pixels. In that case, theDSP 180 may process the red pixel signals in order to extract the low-noise infrared information. Alternatively, the image sensor may be especially configured for imaging at least part of the infrared spectrum. The image sensor may include, for example, one or more infrared (I) pixels in addition to the color pixels, thereby allowing the image sensor to produce a RGB color image and a relatively low-noise infrared image. - An infrared pixel may be realized by covering a pixel with a filter material, which substantially blocks visible light and substantially transmits infrared light, preferably infrared light within the range of approximately 700 through 1100 nm. The infrared transmissive pixel filter may be provided in an infrared/color filter array (ICFA) may be realized using well known filter materials having a high transmittance for wavelengths in the infrared band of the spectrum, for example a black polyimide material sold by Brewer Science under the trademark “
DARC 400”. - Such filters are described in more detail in US2009/0159799, “Color infrared light sensor, camera and method for capturing images,” which is incorporated herein by reference. In one design, an ICFA contain blocks of pixels, e.g. a block of 2×2 pixels, where each block comprises a red, green, blue and infrared pixel. When exposed, such an ICFA image sensor produces a raw mosaic image that includes both RGB color information and infrared information. After processing the raw mosaic image, a RGB color image and an infrared image may be obtained. The sensitivity of such an ICFA image sensor to infrared light may be increased by increasing the number of infrared pixels in a block. In one configuration (not shown), the image sensor filter array uses blocks of sixteen pixels, with four color pixels (RGGB) and twelve infrared pixels.
- Instead of an ICFA image sensor (where color pixels are implemented by using color filters for individual sensor pixels), in a different approach, the
image sensor 130 may use an architecture where each photo-site includes a number of stacked photodiodes. Preferably, the stack contains four stacked photodiodes responsive to the primary colors RGB and infrared, respectively. These stacked photodiodes may be integrated into the silicon substrate of the image sensor. - The multi-aperture system, e.g. a multi-aperture diaphragm, may be used to improve the depth of field (DOF) or other depth aspects of the camera. The DOF determines the range of distances from the camera that are in focus when the image is captured. Within this range the object is acceptably sharp. For moderate to large distances and a given image format, DOF is determined by the focal length of the imaging optics N, the f-number associated with the lens opening (the aperture), and/or the object-to-camera distance s. The wider the aperture (the more light received) the more limited the DOF. DOF aspects of a multi-aperture imaging system are illustrated in
FIG. 3 . - Consider first
FIG. 3B , which shows the imaging of anobject 150 onto theimage sensor 330. Visible and infrared light may enter the imaging system via themulti-aperture system 320. In one embodiment, themulti-aperture system 320 may be a filter-coated transparent substrate. Onefilter coating 324 may have a central circular hole of diameter D1. Thefilter coating 324 transmits visible light and reflects and/or absorbs infrared light. Anopaque cover 322 has a larger circular opening with a diameter D2. Thecover 322 does not transmit either visible or infrared light. It may be a thin-film coating which reflects both infrared and visible light or, alternatively, the cover may be part of an opaque holder for holding and positioning the substrate in the optical system. This way, themulti-aperture system 320 acts as a circular aperture of diameter D2 for visible light and as a circular aperture of smaller diameter D1 for infrared light. The visible light system has a larger aperture and faster f-number than the infrared light system. Visible and infrared light passing the aperture system are projected by theimaging optics 310 onto theimage sensor 330. - The pixels of the image sensor may thus receive a wider-aperture
optical image signal 352B for visible light, overlaying a second narrower-apertureoptical image signal 354B for infrared light. The wider-aperturevisible image signal 352B will have a shorter DOF, while the narrower-aperture infrared image signal 354 will have a longer DOF. InFIG. 3B , theobject 150B is located at the plane of focus N, so that thecorresponding image 160B is in focus at theimage sensor 330. -
Objects 150 close to the plane of focus N of the lens are projected onto theimage sensor plane 330 with relatively small defocus blur. Objects away from the plane of focus N are projected onto image planes that are in front of or behind theimage sensor 330. Thus, the image captured by theimage sensor 330 is blurred. Because thevisible light 352B has a faster f-number than theinfrared light 354B, the visible image will blur more quickly than the infrared image as theobject 150 moves away from the plane of focus N. This is shown byFIGS. 3A and 3C and by the blur diagrams at the right of each figure. - Most of
FIG. 3B shows the propagation of rays fromobject 150B to theimage sensor 330. The righthand side ofFIG. 3B also includes a blur diagram 335, which shows the blurs resulting from imaging of visible light and of infrared light from an on-axis point 152 of the object. InFIG. 3B , the on-axis point 152 produces avisible blur 332B that is relatively small and also produces aninfrared blur 334B that is also relatively small. That is because, inFIG. 3B , the object is in focus. -
FIGS. 3A and 3C show the effects of defocus. InFIG. 3A , theobject 150A is located to one side of the nominal plane of focus N. As a result, thecorresponding image 160A is formed at a location in front of theimage sensor 330. The light travels the additional distance to theimage sensor 330, thus producing larger blur spots than inFIG. 3B . Because thevisible light 352A is a faster f-number, it diverges more quickly and produces alarger blur spot 332A. The infrared light 354 is a slower f-number, so it produces ablur spot 334A that is not much larger than inFIG. 3B . If the f-number is slow enough, the infrared blur spot may be assumed to be constant size across the range of depths that are of interest. -
FIG. 3C shows the same effect, but in the opposite direction. Here, theobject 150C produces animage 160C that would fall behind theimage sensor 330. Theimage sensor 330 captures the light before it reaches the actual image plane, resulting in blurring. Thevisible blur spot 332C is larger due to the faster f-number. Theinfrared blur spot 334C grows more slowly with defocus, due to the slower f-number. - The
DSP 180 may be configured to process and combine the captured color and infrared images. Improvements in the DOF and the ISO speed provided by a multi-aperture imaging system are described in more detail in U.S. application Ser. No. 13/144,499, “Improving the depth of field in an imaging system”; U.S. application Ser. No. 13/392,101, “Reducing noise in a color image”; U.S. application Ser. No. 13/579,568, “Processing multi-aperture image data”; U.S. application Ser. No. 13/579,569, “Processing multi-aperture image data”; and U.S. application Ser. No. 13/810,227, “Flash system for multi-aperture imaging.” All of the foregoing are incorporated by reference herein in their entirety. - In one example, the multi-aperture imaging system allows a simple mobile phone camera with a typical f-number of 2 (e.g. focal length of 3 mm and a diameter of 1.5 mm) to improve its DOF via a second aperture with a f-number varying e.g. between 6 for a diameter of 0.5 mm up to 15 or more for diameters equal to or less than 0.2 mm. The f-number is defined as the ratio of the focal length f and the effective diameter of the aperture. Preferable implementations include optical systems with an f-number for the visible aperture of approximately 2 to 4 for increasing the sharpness of near objects, in combination with an f-number for the infrared aperture of approximately 16 to 22 for increasing the sharpness of distance objects.
- The multi-aperture imaging system may also be used for generating depth information for the captured image. The
DSP 180 of the multi-aperture imaging system may include at least one depth function, which typically depends on the parameters of the optical system and which in one embodiment may be determined in advance by the manufacturer and stored in the memory of the camera for use in digital image processing functions. - If the multi-aperture imaging system is adjustable (e.g., a zoom lens), then the depth function typically will also include the dependence on the adjustment. For example, a fixed lens camera may implement the depth function as a lookup table, and a zoom lens camera may have multiple lookup tables corresponding to different focal lengths, possibly interpolating between the lookup tables for intermediate focal lengths. Alternately, it may store a single lookup table for a specific focal length but use an algorithm to scale the lookup table for different focal lengths. A similar approach may be used for other types of adjustments, such as an adjustable aperture. In various embodiments, when determining the distance or change of distance of an object from the camera, a lookup table or a formula provides an estimate of the distance based on one or more of the following parameters: the blur kernel providing the best match between IR and RGB image data; the f-number or aperture size for the IR imaging; the f-number or aperture size for the RGB imaging; and the focal length. In some imaging systems, the physical aperture is constrained in size, so that as the focal length of the lens changes, the f-number changes. In this case, the diameter of the aperture remains unchanged but the f-number changes. The formula or lookup table could also take this effect into account.
- In certain situations, it is desirable to control the relative size of the IR aperture and the RGB aperture. This may be desirable for various reasons. For example, adjusting the relative size of the two apertures may be used to compensate for different lighting conditions. In some cases, it may be desirable to turn off the multi-aperture aspect. As another example, different ratios may be preferable for different object depths, or focal lengths or accuracy requirements. Having the ability to adjust the ratio of IR to RGB provides an additional degree of freedom in these situations.
-
FIG. 3D is a diagram illustrating adjustment of the relative sizes of anIR aperture 324 andvisible aperture 322. In this diagram, the hashed annulus is amechanical shutter 370. On the lefthand side, themechanical shutter 370 is fully open so that thevisible aperture 322 has maximum area. On the righthand side, theshutter 370 is stopped down, so that thevisible aperture 322 has less area but theIR aperture 324 is unchanged so that the ratio between visible and IR can be adjusted by adjusting themechanical shutter 370. InFIG. 3E , theIR aperture 324 is located near the edge of thevisible aperture 322. Stopping down themechanical shutter 370 reduces the size (and changes the shape) of theIR aperture 324 and the dual-aperture mode can be eliminated by stopping theshutter 370 to the point where theIR aperture 324 is entirely covered. Similar effects can be implemented by other mechanisms, such as adjusting electronic shuttering or exposure time. - As described above in
FIGS. 3A-3C , a scene may contain different objects located at different distances from the camera lens so that objects closer to the focal plane of the camera will be sharper than objects further away from the focal plane. A depth function may relate sharpness information for different objects located in different areas of the scene to the depth or distance of those objects from the camera. In one embodiment, a depth function is based on the sharpness of the color image components relative to the sharpness of the infrared image components. - Here, the sharpness parameter may relate to the circle of confusion, which corresponds to the blur spot diameter measured by the image sensor. As described above in
FIGS. 3A-3C , the blur spot diameter representing the defocus blur is small (approaching zero) for objects that are in focus and grows larger when moving away to the foreground or background in object space. As long as the blur disk is smaller than the maximum acceptable circle of confusion, it is considered sufficiently sharp and part of the DOF range. From the known DOF formulas it follows that there is a direct relation between the depth of an object, e.g. its distance s from the camera, and the amount of blur or sharpness of the captured image of that object. Furthermore, this direct relation is different for the color image than it is for the infrared image, due to the difference in apertures and f-numbers. - Hence, in a multi-aperture imaging system, the increase or decrease in sharpness of the RGB components of a color image relative to the sharpness of the IR components in the infrared image is a function of the distance to the object. For example, if the lens is focused at 3 meters, the sharpness of both the RGB components and the IR components may be the same. In contrast, due to the small aperture used for the infrared image for objects at a distance of 1 meter, the sharpness of the RGB components may be significantly less than those of the infrared components. This dependence may be used to estimate the distances of objects from the camera.
- In one approach, the imaging system is set to a large (“infinite”) focus point. That is, the imaging system is designed so that objects at infinity are in focus. This point is referred to as the hyperfocal distance H of the multi-aperture imaging system. The system may then determine the points in an image where the color and the infrared components are equally sharp. These points in the image correspond to objects that are in focus, which in this example means that they are located at a relatively large distance (typically the background) from the camera. For objects located away from the hyperfocal distance H (i.e., closer to the camera), the relative difference in sharpness between the infrared components and the color components will change as a function of the distance s between the object and the lens.
- The sharpness may be obtained empirically by measuring the sharpness (or, equivalently, the blurriness) for one or more test objects at different distances s from the camera lens. It may also be calculated based on models of the imaging system. In one embodiment, sharpness is measured by the absolute value of the high-frequency infrared components in an image. In another approach, blurriness is measured by the blur size or point spread function (PSF) of the imaging system.
-
FIG. 4 is a plot of the blur spot sizes Bvis and Bir of the visible and infrared images, as a function of object distance s.FIG. 4 shows that around the focal distance N, which in this example is the hyperfocal distance, the blur spots are the smallest. Away from the focal distance N, the color components experience rapid blurring and rapid increase in the blur spot size Bvis. In contrast, as a result of the relatively small infrared aperture, the infrared components do not blur as quickly and, if the f-number is slow enough, the blur spot size Bir may be approximated as constant in size over the range of depths considered. - Now consider the object distance sx. At this object distance, the infrared image is produced with a
blur spot 410 and the visible image is produced with ablur spot 420. Conversely, if the blur spot sizes were known, or the ratio of the blur spot sizes were know, this information could be used to estimate the object distance sx. Recall that the blur spot, also referred to as the point spread function, is the image produced by a single point source. If the object were a single point source, then the infrared image will be a blur spot ofsize 410 and the corresponding visible image will be a blur spot ofsize 420. -
FIG. 5 illustrates one approach to estimating the object distance based on the color and infrared blur spots.FIG. 5 is a table of blur spot as a function of object distance s. For each object distance sk, there is shown a corresponding IR blur spot (PSFir) and color blur spot (PSFvis). The IR image In is the convolution of an ideal image Iideal with PSFir, and the color image Ivis is the convolution of the ideal image Iideal with PSFvis. -
I ir =I ideal *PSF ir (1) -
I vis =I ideal *PSF vis (2) - where * is the convolution operator. Manipulating these two equations yields
-
I vis =I ir *B (3) - where B is a blur kernel that accounts for deblurring of the IR image followed by blurring of the visible image. The blur kernels B can be calculated in advance or empirically measured as a function of object depth s, producing a table as shown in
FIG. 5 . - In
FIG. 5 , the blur kernel B is shown as similar in size to the visible blur spot PSFvis. Under certain circumstances, the IR blur spot PSFir may be neglected or otherwise accounted for. For example, if the IR blur spot is small relative to the visible blur spot PSFvis, then neglecting the effect of the IR blur may be negligible. As another example, if the IR blur spot does not vary significantly with object distance, then it may be neglected for purposes of calculating the blur kernel B, but may be accounted for by a systematic adjustment of the results. -
FIG. 6A is a diagram illustrating a method for producing an estimate s* of the object distance s using abank 610 of blur kernels Bk. The infrared image In is blurred by each of the blur kernels Bk in the bank. In this example, the blurring is accomplished by convolution, although faster approaches will be discussed below. This results in estimated visible images I*vis. - Each of these estimated images I*vis is compared 620 to the actual visible image Ivis. In this example, the comparison is a sum squared error ek between the two images.
-
FIG. 6B is a graph of error e as a function of kernel number k for the architecture ofFIG. 6A . Recall that each kernel number k corresponds to a specific object distance s. The error metrics e are processed 630 to yield an estimate s* of the object distance. In one approach, the minimum error ek is identified, and the estimated object distance s* is the object depth sk corresponding to the minimum error ek. Other approaches can also be used. For example, the functional pairs (sk,ek) can be interpolated for the value of s that yields the minimum e. - The infrared image Iir and visible image Ivis in
FIG. 6A typically are not the entire captured images. Rather, the approach ofFIG. 6A can be applied to different windows within the image in order to estimate the depth of the objects in the window. In this way, a depth map of the entire image can be produced. - The approach of
FIG. 6A includes a convolution for each blur kernel. If the window and blur kernel Bk are each large, the convolution can be computationally expensive. The blur kernels Bk by definition will vary in size. For example, the smallest blur kernel may be 3×3 while the largest may be 25×25 or larger. In order to accommodate the largest blur kernels, the window should be at least the same size as the largest blur kernel, which means a large window size is required for a bank that includes a large blur kernel. Furthermore, the same window should be used for all blur kernels in order to allow direct comparison of the calculated error metrics. Therefore, if the bank includes a large blur kernel, a large window will be used for all blur kernels, which can lead to computationally expensive convolutions. -
FIG. 7A is a diagram illustrating a variation ofFIG. 6A that addresses this issue. Rather than using a single bank of blur kernels, as inFIG. 6A , the approach ofFIG. 7A uses multiple banks 710 a-M of blur kernels. Each bank contains multiple blur kernels. However, each bank 710 is down-sampled by a different down-sampling factor. For example,bank 710 a may use the smallest blur kernels and the original images without down-sampling,bank 710 b may use the next smallest set of kernels but with down-sampling of 2×, and so on. InFIG. 7A ,bank 710 m uses down-sampling of mx. The visible image and the infrared image are also down-sampled by mx, as indicated by the boxes marked “/m”.Bank 710 m uses blur kernels J to (J+K), each of which is also down-sampled by mx, as indicated by the “/m” in “*BJ/m”. Each bank 710 produces a result, for example an estimated object distance sm* and these are combined 730 into an overall depth estimate s*. - One advantage of this approach is that down-sampled blur kernels are smaller and therefore require less computation for convolution and other operations. The table below shows a set of 9 blur kernels, ranging in size from 3×3 for
blur kernel 1, to 25×25 for blur kernel 9. In the approach ofFIG. 6A , blur kernel 9 would be 25×25 with a corresponding number of multiply-accumulates used to implement convolution. In contrast, in the table below, all blur kernels are down-sampled so that no convolution uses a kernel larger than 5×5. -
TABLE 1 Kernel Size of Down-sampling number (k) blur kernel factor 1 3 × 3 1x 2 5 × 5 2x 3 8 × 8 2x 4 11 × 11 3x 5 14 × 14 3x 6 17 × 17 4x 7 20 × 20 4x 8 23 × 23 5x 9 25 × 25 5x -
FIGS. 7B and 7C are graphs of error as a function of blur kernel number k for the architecture ofFIG. 7A . If the down-sampling is performed without normalizing energies, then the error curve may exhibit discontinuities when transitioning from one bank to the next bank.FIG. 7B shows an error curve using five banks Each piece of the curve corresponds to one of the banks Each curve is continuous because the same down-sampling factor is used for all blur kernels in that bank. However, the down-sampling factor changes from one bank to the next so the different pieces of the curve may not align correctly. However, the minimum error can still be determined. In this example,curve 750 c is the only curve that has a minimum within that curve. The other four curves are either monotonically increasing or monotonically decreasing. Therefore, the minimum error occurs withincurve 750 c. More sophisticated approaches may also be used. For example, differentials across the entire range of curves may be analyzed to predict the point of minimum error. This approach can be used to avoid local minima, which may be caused by noise or other effects. - In
FIG. 7B , the curves are shown as continuous within each bank. However, there may be a limited number of samples for each bank.FIG. 7C is the same asFIG. 7B , except that there are only three samples for each bank. InFIG. 7C , the dashed ovals identify each of the banks Each of the banks can be classified as monotonically increasing, monotonically decreasing or containing an extremum. In this example,banks bank 750 c contains an extremum, andbanks bank 750 c. Finer resolution sampling withinbank 750 c can then be performed to more accurately locate the location of the minimum value. - In
FIG. 7D ,banks banks banks banks - These figures effectively illustrate different sampling approaches to find the extremum of the error function e(k). As another variation, the error function e(k) may be coarsely sampled at first in order to narrow the range of k where the minimum error e exists. Finer and finer sampling may be used as the range is narrowed. Other sampling approaches can be used to find the value of kernel number k (and the corresponding object distance) where the extremum of the error function e(k) occurs.
- Down-sampling can be implemented in other ways. For example, the visible images may be down-sampled first. The blur kernels are then down-sampled to match the down-sampling of the visible images. The down-sampled blur kernels are applied to the full resolution IR images. The result is an intermediate form which retains the fill resolution of the IR image but then is down-sampled to match the resolution of the down-sampled visible images. This method is not as efficient as fully down-sampling the IR but is more efficient than not using down-sampling at all. This approach may be beneficial to reduce computation while still maintaining a finer resolution.
- Another aspect is that the approach of
FIG. 6A depends on the content of the window. For example, a window for which the only object is a single point source object (e.g., a window containing a single star surrounded entirely by black night sky) will yield a good result because that image is a direct measure of the underlying point spread functions. Similarly, a window that contains the image of only an edge will also yield a good result because that image is a direct measure of the underlying point spread functions albeit only along one direction. At the other extreme, a window that is constant and has no features will not yield any estimate because every estimated visible image will also be a constant so there is no way to distinguish the different blur kernels. Other images may be somewhere between these extremes. Features will help distinguish the different blur kernels. Featureless areas will not and typically will also add unwanted noise. - In one approach, the windows are selected to include edges. Edge identification can be accomplished using known algorithms. Once identified, edges preferably are processed to normalize variations between the different captured images.
FIG. 8 shows one example. In this example, the green component Igm of the color image is the fast f-number image and the IR image Iir is the slow f-number image. The left column ofFIG. 8 shows processing of the green image while the right column shows processing of the IR image. The top row shows the same edge appearing in both images. The object is not in focus so that the green edge is blurred relative to the IR edge. Also note that the edge has different phase in the two images. The green edge transitions from high to low amplitude, while the IR edge transitions from low to high amplitude.FIG. 8 shows one approach to normalize these edges to allow comparisons using blur kernels as described above. - The second row of
FIG. 8 shows both edges afterdifferentiation 810. Theabsolute value 820 of the derivatives is then taken, yielding the third row ofFIG. 8 . This effectively removes the phase mismatch between the two edges, yielding two phase matched edges. The two edges are then scaled 830, resulting in the bottom row ofFIG. 8 . In this example, the IR image is binarized to take on only thevalues - Note that the IR edge looks like a line source. This is not uncommon since the IR point spread function is small and fairly constant over a range of depths, compared to the color point spread function. Also recall that in
FIG. 6 , the IR image is convolved with many different blur kernels. The convolution can be simplified as follows. First, the IR edge is binarized, so that the IR image is a binary image taking on only the values of 0 or 1. (Instep 830 above, the color image is then scaled in amplitude to have equal energy as the binary IR image). Convolution generally requires multiplies and adds. However, when the image only takes values of 0 or 1, the multiplies are simplified. Multiplying by 0 yields all 0's so that pixels with 0 value can be ignored. Multiplying by 1 yields the blur kernel so that no actual multiplication is required. Rather, any pixel with 1 value causes an accumulation of the blur kernel centered on that pixel. -
FIGS. 9A-9E illustrate this concept.FIG. 9A shows a 4×4 window with a binarized edge, where the pixels are either 1 or 0.FIG. 9B shows a 3×3 blur kernel to be convolved with the window.FIGS. 9C-9E show progression of the convolution using only adds and no multiplies. In these figures, the lefthand side shows the binarized edge ofFIG. 9A and the righthand side shows progression of the convolution. InFIG. 9C ,pixel 910 has been processed, meaning that the blur kernel centered onpixel 910 has been added to the moving sum on the right. InFIG. 9D , the next pixel along theedge 911 has been processed. The blur kernel centered onpixel 911 is added to the moving sum, which already contains the effect ofpixel 910. The result is shown on the right. This continues for all pixels with value of 1.FIG. 9E shows the final result after all four edge pixels have been processed. This is the estimated green edge, which can then be compared to the actual green edge. If the two match well, then the blur kernel shown inFIG. 9B is the correct blur kernel for this window and can be used to estimate the object distance for this edge. - Edges in an image may be caused by a sharp transition within an object, for example the border between black and white squares on a checkerboard. In that case, the approach shown in
FIG. 9 may be implemented using entire blur kernels. However, edges may also be caused by occlusion, when a closer object partially blocks a more distant object. InFIG. 10 , thesign 1010 in the foreground partially blocks thehouse 1020 in the background. This creates anedge 1030 in the image. However, the left side of the edge is thesign 1010, which is at a closer object distance, and the right side of the edge is thehouse 1020, which is at a farther object distance. The two different object distances correspond to different blur kernels. Applying a single blur kernel to the edge will not give good results, because when one side is matched to the blur kernel, the other side will not be. - Single-sided blur kernels can be used instead. A single-sided blur kernel is half a blur kernel instead of an entire blur kernel.
FIG. 11 shows a set of eight single-sided blur kernels with different edge orientations based on the 3×3 blur kernel ofFIG. 9B . The full 3×3 blur kernel is reproduced in the center ofFIG. 11 . Note that different single-sided blur kernels can be derived from the same full blur kernel, depending on the orientation of the edge. InFIG. 11 , thesolid line 1110 represents the edge. These single-sided blur kernels can be applied to binarized edges, as described above, to yield different depth estimates for each side of the edge. -
FIG. 12 illustrates another aspect of the approach described above. As described above, a bank of blur kernels of varying sizes is used to estimate the object depth. Blur kernels effectively act as low pass filters. Larger blur kernels cause more blurring and therefore have lower cutoff frequencies compared to smaller blur kernels.FIG. 12 shows a generalized frequency response for a bank of blur kernels.Blur kernel 1210A is the low pass filter with the lowest cutoff frequency in the bank, which corresponds to the blur kernel with the largest blur size.Blur kernel 1210B is the second largest blur kernel and so on to blurkernel 1210D, which has the highest cutoff frequency and smallest blur size. The IR image is blurred by each of these blur kernels, and the results are compared to determine which blur kernel corresponds to the object depth. - However, note that the
blur kernels 1210A-D differ only within thefrequency range 1220. Outside thisfrequency range 1220, all of theblur kernels 1210A-D in the bank have the same behavior. Therefore, content outside thefrequency range 1220 will not distinguish between thedifferent blur kernels 1210A-D. However, that content will add to background noise. Therefore, in one approach, frequency filtering is added to reduce energy and noise from outside thefrequency range 1220. In one approach, the original images are frequency filtered. In another approach, the blur kernels may be frequency filtered versions. The frequency filtering may be low pass filtering to reduce frequency content abovefrequency 1220B, high pass filtering to reduce frequency content belowfrequency 1220A, or bandpass filtering to reduce both the low frequency and high frequency content. The filtering may take different forms and may be performed regardless of whether down-sampling is also used. When it is used, down-sampling is a type of low pass filtering. - The filtering may also be applied to less than or more than all the blur kernels in a bank. For example, a narrower bandpass filter may be used if it is desired to distinguish only blur
kernels blur kernels 1210A-1210B). Most of the difference between those two blur kernels occurs in thefrequency band 1230, so a bandpass filter that primarily passes frequencies within that range and rejects frequencies outside that range will increase the relative signal available for distinguishing the twoblur kernels - Window sizes and locations preferably are selected based on the above considerations, and the window size may be selected independent of the blur kernel size. For example, window size may be selected to be large enough to contain features such as edges, small enough to avoid interfering features such as closely spaced parallel edges, and generally only large enough to allow processing of features since larger windows will add more noise. The size of the blur kernel may be selected to reduce computation (e.g., by down-sampling) and also possibly in order to provide sufficient resolution for the depth estimation. As a result, the window size may be different (typically, larger) than the size of the blur kernels.
- The number of windows and window locations may also be selected to contain features such as edges, and to reduce computation. A judicious choice of windows can reduce power consumption by having fewer pixels to power up and to read out, which in turn can be used to increase the frame rate. A higher frame rate may be advantageous for many reasons, for example in enabling finer control of gesture tracking.
- Embodiments of the invention may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored.
- It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Moreover, the invention is not limited to the embodiments described above, which may be varied within the scope of the accompanying claims. For example, aspects of this technology have been described with respect to different f-number images captured by a multi-aperture imaging system. However, these approaches are not limited to multi-aperture imaging systems. They can also be used in other systems that estimate depth based on differences in blurring, regardless of whether a multi-aperture imaging system is used to capture the images. For example, two images may be captured in time sequence, but at different f-number settings. Another method is to capture two or more images of the same scene but with different focus settings, or to rely on differences in aberrations (e.g., chromatic aberrations) or other phenomenon that cause the blurring of the two or more images to vary differently as a function of depth so that these variations can be used to estimate the depth.
Claims (23)
1. A method for processing blurred image data, comprising:
downsampling first image data associated with a first image of an object, the first image captured using a first imaging system characterized by a first point spread function;
downsampling second image data associated with a second image of the object, the second image captured using a second imaging system characterized by a second point spread function that varies as a function of depth differently than the first point spread function;
for each blur kernel from a bank of down-sampled blur kernels, wherein each blur kernel corresponds to the first point spread function relative to the second point spread function at a different object depth, and the bank of blur kernels spans a range of object depths:
blurring the down-sampled second image data with the down-sampled blur kernel; and
comparing the blurred down-sampled second image data and the down-sampled first image data; and
generating depth information for the object based on said comparisons.
2. The method of claim 1 , wherein:
for each blur kernel, comparing the blurred down-sampled second image data and the down-sampled first image data comprises calculating an error between the blurred down-sampled second image data and the down-sampled first image data; and
generating depth information for the object based on said comparisons comprises generating depth information based on a depth that corresponds to the blur kernel with a lowest calculated error.
3. The method of claim 2 , wherein blurring the down-sampled second image data with the down-sampled blur kernel comprises:
first deblurring the down-sampled second image data; and
then blurring the deblurred, down-sampled second image data with the down-sampled blur kernel.
4. The method of claim 2 , wherein blurring the down-sampled second image data with the down-sampled blur kernel comprises convolving the down-sampled second image data with the down-sampled blur kernel.
5. The method of claim 1 , wherein:
the first image data and the second image data each contain a same edge;
for each blur kernel:
blurring the down-sampled second image data with the down-sampled blur kernel comprises blurring the edge in the down-sampled second image data with the down-sampled blur kernel; and
comparing the blurred down-sampled second image data and the down-sampled first image data comprises comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data.
6. The method of claim 5 , wherein blurring the edge in the down-sampled second image data comprises:
binarizing the edge in the down-sampled second image data; and
blurring the binarized edge with the down-sampled blur kernel.
7. The method of claim 5 , wherein comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data comprises phase matching the edges in the first and second image data.
8. The method of claim 5 , wherein comparing the blurred edge in the down-sampled second image data and the same edge in the down-sampled first image data comprises equating energy in the edges in the first and second image data.
9. The method of claim 1 , wherein said blurring and comparing for each blur kernel and said generating depth information is performed for each of a plurality of banks of down-sampled blur kernels, each bank down-sampled by a different downsampling factor.
10. The method of claim 9 , wherein the plurality of banks span a contiguous range of object depths.
11. The method of claim 9 , further comprising:
classifying each bank as containing or not containing an extremum with respect to said comparison; and
generating depth information for the object based on said classifications for the bank that contains the extremum.
12. The method of claim 9 , further comprising:
classifying each bank as monotonically increasing, monotonically decreasing or containing an extremum with respect to said comparison; and
generating depth information for the object based on said classifications for the banks.
13. The method of claim 12 , further comprising:
if the classifications indicate an extremum occurs between two banks, then creating an additional bank that spans between the two banks.
14. The method of claim 9 , wherein each bank is down-sampled by a different integer downsampling factor.
15. The method of claim 9 , wherein a largest down-sampled blur kernel for each bank is a same size for all the banks.
16. The method of claim 9 , wherein all of the blur kernels are sufficiently down-sampled so that no down-sampled blur kernel is larger than 5×5.
17. The method of claim 1 , wherein the first imaging system has a first f-number and the second imaging system has a second f-number that is slower than the first f-number, wherein the f-number is defined as a ratio of a focal length and an effective diameter of an aperture and whereby a size of the second point spread function varies as a function of depth more slowly than a size of the first point spread function.
18. The method of claim 17 , further comprising:
exposing an image sensor in a multi-aperture shared sensor imaging system to light from the object, using a first aperture with the first f-number to expose the first image and a second aperture with the second f-number to expose the second image.
19. The method of claim 18 , wherein the first aperture exposes the first image using light from a first spectral band, and the second aperture exposes the second image using light from a different second spectral band.
20. The method of claim 18 , wherein the first aperture exposes the first image using light from a visible spectrum, and the second aperture exposes the second image using light from an infrared spectrum.
21. The method of claim 1 , wherein the bank of down-sampled blur kernels comprises a bank of down-sampled single-sided blur kernels.
22. The method of claim 1 , further comprising:
frequency filtering the second image data.
23. A non-transitory computer-readable storage medium storing executable computer program instructions for processing blurred image data, the instructions executable by a processor and causing the processor to perform a method comprising:
downsampling first image data associated with a first image of an object, the first image captured using a first imaging system characterized by a first point spread function;
downsampling second image data associated with a second image of the object, the second image captured using a second imaging system characterized by a second point spread function that varies as a function of depth differently than the first point spread function;
for each blur kernel from a bank of down-sampled blur kernels, wherein each blur kernel corresponds to the first point spread function of the first imaging system relative to the second point spread function at a different object depth, and the bank of blur kernels spans a range of object depths:
blurring the down-sampled second image data with the down-sampled blur kernel; and
comparing the blurred down-sampled second image data and the down-sampled first image data; and
generating depth information for the object based on said comparisons.
Priority Applications (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/832,062 US20160255323A1 (en) | 2015-02-26 | 2015-08-21 | Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling |
PCT/KR2016/001838 WO2016137241A1 (en) | 2015-02-26 | 2016-02-25 | Multi-aperture depth map using blur kernels and down-sampling |
US15/162,154 US9721344B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using partial blurring |
US15/162,147 US9721357B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using blur kernels and edges |
US15/163,435 US20160269600A1 (en) | 2015-02-26 | 2016-05-24 | Multi-Aperture Depth Map Using Frequency Filtering |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562121203P | 2015-02-26 | 2015-02-26 | |
US14/832,062 US20160255323A1 (en) | 2015-02-26 | 2015-08-21 | Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling |
Related Child Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/162,147 Continuation US9721357B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using blur kernels and edges |
US15/162,154 Continuation US9721344B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using partial blurring |
US15/163,435 Continuation US20160269600A1 (en) | 2015-02-26 | 2016-05-24 | Multi-Aperture Depth Map Using Frequency Filtering |
Publications (1)
Publication Number | Publication Date |
---|---|
US20160255323A1 true US20160255323A1 (en) | 2016-09-01 |
Family
ID=56788898
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/832,062 Abandoned US20160255323A1 (en) | 2015-02-26 | 2015-08-21 | Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling |
US15/162,147 Expired - Fee Related US9721357B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using blur kernels and edges |
US15/162,154 Expired - Fee Related US9721344B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using partial blurring |
US15/163,435 Abandoned US20160269600A1 (en) | 2015-02-26 | 2016-05-24 | Multi-Aperture Depth Map Using Frequency Filtering |
Family Applications After (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/162,147 Expired - Fee Related US9721357B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using blur kernels and edges |
US15/162,154 Expired - Fee Related US9721344B2 (en) | 2015-02-26 | 2016-05-23 | Multi-aperture depth map using partial blurring |
US15/163,435 Abandoned US20160269600A1 (en) | 2015-02-26 | 2016-05-24 | Multi-Aperture Depth Map Using Frequency Filtering |
Country Status (2)
Country | Link |
---|---|
US (4) | US20160255323A1 (en) |
WO (1) | WO2016137241A1 (en) |
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160267667A1 (en) * | 2015-02-26 | 2016-09-15 | Dual Aperture International Co. Ltd. | Multi-aperture Depth Map Using Blur Kernels and Edges |
US20170054910A1 (en) * | 2015-08-20 | 2017-02-23 | Kabushiki Kaisha Toshhiba | Image processing apparatus and image capturing apparatus |
US9584717B2 (en) * | 2015-06-04 | 2017-02-28 | Lite-On Electronics (Guangzhou) Limited | Focusing method, and image capturing device for implementing the same |
US20170150019A1 (en) * | 2015-11-23 | 2017-05-25 | Center For Integrated Smart Sensors Foundation | Multi-aperture camera system using disparity |
US20180286066A1 (en) * | 2015-09-18 | 2018-10-04 | The Regents Of The University Of California | Cameras and depth estimation of images acquired in a distorting medium |
US20190205614A1 (en) * | 2018-01-03 | 2019-07-04 | Samsung Electronics Co., Ltd. | Method and apparatus for recognizing object |
US10412283B2 (en) * | 2015-09-14 | 2019-09-10 | Trinamix Gmbh | Dual aperture 3D camera and method using differing aperture areas |
US10775505B2 (en) | 2015-01-30 | 2020-09-15 | Trinamix Gmbh | Detector for an optical detection of at least one object |
US10785412B2 (en) | 2015-08-20 | 2020-09-22 | Kabushiki Kaisha Toshiba | Image processing apparatus and image capturing apparatus |
US10823818B2 (en) | 2013-06-13 | 2020-11-03 | Basf Se | Detector for optically detecting at least one object |
US10890491B2 (en) | 2016-10-25 | 2021-01-12 | Trinamix Gmbh | Optical detector for an optical detection |
US10948567B2 (en) | 2016-11-17 | 2021-03-16 | Trinamix Gmbh | Detector for optically detecting at least one object |
US10955936B2 (en) | 2015-07-17 | 2021-03-23 | Trinamix Gmbh | Detector for optically detecting at least one object |
US11041718B2 (en) | 2014-07-08 | 2021-06-22 | Basf Se | Detector for determining a position of at least one object |
US11125880B2 (en) | 2014-12-09 | 2021-09-21 | Basf Se | Optical detector |
US11211513B2 (en) | 2016-07-29 | 2021-12-28 | Trinamix Gmbh | Optical sensor and detector for an optical detection |
US11428787B2 (en) | 2016-10-25 | 2022-08-30 | Trinamix Gmbh | Detector for an optical detection of at least one object |
WO2023240452A1 (en) * | 2022-06-14 | 2023-12-21 | 北京小米移动软件有限公司 | Image processing method and apparatus, electronic device, and storage medium |
US11860292B2 (en) | 2016-11-17 | 2024-01-02 | Trinamix Gmbh | Detector and methods for authenticating at least one object |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TWI577971B (en) * | 2015-10-22 | 2017-04-11 | 原相科技股份有限公司 | Dual-aperture ranging system |
JP2019015575A (en) * | 2017-07-05 | 2019-01-31 | 株式会社東芝 | Image processor, distance measuring device, and processing system |
CN109493376B (en) * | 2017-09-13 | 2022-02-22 | 腾讯科技(深圳)有限公司 | Image processing method and apparatus, storage medium, and electronic apparatus |
US10419664B2 (en) | 2017-12-28 | 2019-09-17 | Semiconductor Components Industries, Llc | Image sensors with phase detection pixels and a variable aperture |
US11831858B2 (en) | 2020-05-08 | 2023-11-28 | Shenzhen GOODIX Technology Co., Ltd. | Passive three-dimensional image sensing based on referential image blurring |
CN112465712B (en) * | 2020-11-09 | 2022-08-09 | 华中光电技术研究所(中国船舶重工集团公司第七一七研究所) | Motion blur star map restoration method and system |
Family Cites Families (134)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US2921509A (en) | 1955-10-24 | 1960-01-19 | Freund Karl | Multiple exposure camera |
US3971065A (en) | 1975-03-05 | 1976-07-20 | Eastman Kodak Company | Color imaging array |
US4238760A (en) | 1978-10-06 | 1980-12-09 | Recognition Equipment Incorporated | Multi-spectrum photodiode devices |
JPS60130274A (en) | 1983-12-19 | 1985-07-11 | Toshiba Corp | Solid-state image pickup device |
US4878113A (en) | 1987-08-11 | 1989-10-31 | Olympus Optical Co., Ltd. | Endoscope apparatus |
US4965840A (en) | 1987-11-27 | 1990-10-23 | State University Of New York | Method and apparatus for determining the distances between surface-patches of a three-dimensional spatial scene and a camera system |
US4987295A (en) | 1989-03-31 | 1991-01-22 | General Electric Company | Multichip imager with improved optical performance near the butt region |
JP2800364B2 (en) | 1990-04-27 | 1998-09-21 | 松下電器産業株式会社 | Optical low-pass filter |
US5148209A (en) | 1990-07-12 | 1992-09-15 | The Research Foundation Of State University Of New York | Passive ranging and rapid autofocusing |
US5231443A (en) | 1991-12-16 | 1993-07-27 | The Research Foundation Of State University Of New York | Automatic ranging and automatic focusing |
JP3614898B2 (en) | 1994-11-08 | 2005-01-26 | 富士写真フイルム株式会社 | PHOTOGRAPHIC APPARATUS, IMAGE PROCESSING APPARATUS, AND STEREOGRAPHIC CREATION METHOD |
US6292212B1 (en) | 1994-12-23 | 2001-09-18 | Eastman Kodak Company | Electronic color infrared camera |
US5631703A (en) | 1996-05-29 | 1997-05-20 | Eastman Kodak Company | Particular pattern of pixels for a color filter array which is used to derive luminance and chrominance values |
GB2317771A (en) | 1996-09-27 | 1998-04-01 | Sharp Kk | Observer tracking directional display |
EP0858208A1 (en) | 1997-02-07 | 1998-08-12 | Eastman Kodak Company | Method of producing digital images with improved performance characteristic |
US5998090A (en) | 1997-12-01 | 1999-12-07 | Brewer Science, Inc. | High optical density ultra thin organic black matrix system |
US6034372A (en) | 1997-12-03 | 2000-03-07 | The United States Of America As Represented By The Secretary Of The Air Force | Pupil stop for multi-band focal plane arrays |
EP1053644A1 (en) | 1998-02-04 | 2000-11-22 | Applied Science Fiction, Inc. | Multilinear array sensor with an infrared line |
US7006132B2 (en) | 1998-02-25 | 2006-02-28 | California Institute Of Technology | Aperture coded camera for three dimensional imaging |
EP1062636A1 (en) | 1998-03-13 | 2000-12-27 | Applied Science Fiction, Inc. | Image defect correction method |
US6771314B1 (en) | 1998-03-31 | 2004-08-03 | Intel Corporation | Orange-green-blue (OGB) color system for digital image sensor applications |
US6657663B2 (en) | 1998-05-06 | 2003-12-02 | Intel Corporation | Pre-subtracting architecture for enabling multiple spectrum image sensing |
US6459450B2 (en) | 1998-06-24 | 2002-10-01 | Intel Corporation | Infrared filterless pixel structure |
US7683926B2 (en) | 1999-02-25 | 2010-03-23 | Visionsense Ltd. | Optical device |
US8248457B2 (en) | 1999-02-25 | 2012-08-21 | Visionsense, Ltd. | Optical device |
US6727521B2 (en) | 2000-09-25 | 2004-04-27 | Foveon, Inc. | Vertical color filter detector group and array |
US20040252867A1 (en) | 2000-01-05 | 2004-12-16 | Je-Hsiung Lan | Biometric sensor |
US7053928B1 (en) | 2000-03-20 | 2006-05-30 | Litton Systems, Inc. | Method and system for combining multi-spectral images of a scene |
US7274383B1 (en) | 2000-07-28 | 2007-09-25 | Clairvoyante, Inc | Arrangement of color pixels for full color imaging devices with simplified addressing |
US6768565B1 (en) | 2000-09-07 | 2004-07-27 | Xerox Corporation | Infrared correction in color scanners |
US6316284B1 (en) | 2000-09-07 | 2001-11-13 | Xerox Corporation | Infrared correction in color scanners |
CN100446264C (en) | 2000-10-19 | 2008-12-24 | 量子半导体有限公司 | Method of fabricating heterojunction photodiodes integrated with CMOS |
SE518050C2 (en) | 2000-12-22 | 2002-08-20 | Afsenius Sven Aake | Camera that combines sharply focused parts from various exposures to a final image |
US7072508B2 (en) | 2001-01-10 | 2006-07-04 | Xerox Corporation | Document optimized reconstruction of color filter array images |
JP3626101B2 (en) | 2001-01-12 | 2005-03-02 | コニカミノルタフォトイメージング株式会社 | Digital camera |
US7176962B2 (en) | 2001-03-01 | 2007-02-13 | Nikon Corporation | Digital camera and digital processing system for correcting motion blur using spatial frequency |
US7053908B2 (en) | 2001-04-12 | 2006-05-30 | Polaroid Corporation | Method and apparatus for sensing and interpolating color image data |
JP2002344999A (en) | 2001-05-21 | 2002-11-29 | Asahi Optical Co Ltd | Stereoscopic image pickup device |
JP4931288B2 (en) | 2001-06-08 | 2012-05-16 | ペンタックスリコーイメージング株式会社 | Image detection device and diaphragm device |
US6930336B1 (en) | 2001-06-18 | 2005-08-16 | Foveon, Inc. | Vertical-color-filter detector group with trench isolation |
JP2003084344A (en) | 2001-09-14 | 2003-03-19 | Casio Comput Co Ltd | Flash device, camera device equipped with the same, and color temperature control method for flash device |
US6870684B2 (en) | 2001-09-24 | 2005-03-22 | Kulicke & Soffa Investments, Inc. | Multi-wavelength aperture and vision system and method using same |
US7248297B2 (en) | 2001-11-30 | 2007-07-24 | The Board Of Trustees Of The Leland Stanford Junior University | Integrated color pixel (ICP) |
US7057654B2 (en) | 2002-02-26 | 2006-06-06 | Eastman Kodak Company | Four color image sensing apparatus |
US6998660B2 (en) | 2002-03-20 | 2006-02-14 | Foveon, Inc. | Vertical color filter sensor group array that emulates a pattern of single-layer sensors with efficient use of each sensor group's sensors |
US6783900B2 (en) | 2002-05-13 | 2004-08-31 | Micron Technology, Inc. | Color filter imaging array and method of formation |
US7164444B1 (en) | 2002-05-17 | 2007-01-16 | Foveon, Inc. | Vertical color filter detector group with highlight detector |
JP2005533463A (en) | 2002-06-26 | 2005-11-04 | ヴイケイビー・インコーポレーテッド | Multi-function integrated image sensor and application to virtual interface technology |
WO2004047421A2 (en) | 2002-11-14 | 2004-06-03 | Donnelly Corporation | Imaging system for vehicle |
US7405860B2 (en) | 2002-11-26 | 2008-07-29 | Texas Instruments Incorporated | Spatial light modulators with light blocking/absorbing areas |
US20040174446A1 (en) | 2003-02-28 | 2004-09-09 | Tinku Acharya | Four-color mosaic pattern for depth and image capture |
US7274393B2 (en) | 2003-02-28 | 2007-09-25 | Intel Corporation | Four-color mosaic pattern for depth and image capture |
US7269295B2 (en) | 2003-07-31 | 2007-09-11 | Hewlett-Packard Development Company, L.P. | Digital image processing methods, digital image devices, and articles of manufacture |
KR101081000B1 (en) | 2003-10-23 | 2011-11-09 | 소니 가부시키가이샤 | Image processing apparatus and image processing method, and recording medium |
JP4578797B2 (en) | 2003-11-10 | 2010-11-10 | パナソニック株式会社 | Imaging device |
US7123298B2 (en) | 2003-12-18 | 2006-10-17 | Avago Technologies Sensor Ip Pte. Ltd. | Color image sensor with imaging elements imaging on respective regions of sensor elements |
US20050146634A1 (en) | 2003-12-31 | 2005-07-07 | Silverstein D. A. | Cameras, optical systems, imaging methods, and optical filter configuration methods |
US20060054782A1 (en) | 2004-08-25 | 2006-03-16 | Olsen Richard I | Apparatus for multiple camera devices and method of operating same |
JP2006109120A (en) | 2004-10-06 | 2006-04-20 | Funai Electric Co Ltd | Infrared imaging device |
JP3926363B2 (en) | 2004-11-04 | 2007-06-06 | 三菱電機株式会社 | Pixel signal processing apparatus and pixel signal processing method |
JP4534756B2 (en) | 2004-12-22 | 2010-09-01 | ソニー株式会社 | Image processing apparatus, image processing method, imaging apparatus, program, and recording medium |
US7224540B2 (en) | 2005-01-31 | 2007-05-29 | Datalogic Scanning, Inc. | Extended depth of field imaging system using chromatic aberration |
US20060182364A1 (en) | 2005-02-16 | 2006-08-17 | George John | System and method for sharpening vector-valued digital images |
JP4434991B2 (en) | 2005-03-01 | 2010-03-17 | キヤノン株式会社 | Image sensor |
JP4622629B2 (en) | 2005-03-31 | 2011-02-02 | 株式会社ニコン | Imaging device |
US7435962B2 (en) | 2005-05-18 | 2008-10-14 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Imaging device and method for producing an infrared filtered digital image |
DE102005026912A1 (en) | 2005-06-10 | 2006-12-14 | Arnold & Richter Cine Technik Gmbh & Co. Betriebs Kg | Exposed film`s e.g. motion image film, image information scanner, has optical screen with filter area surrounding transparent central area, where filter area has different transmission properties in visible and infrared spectral areas |
US7577309B2 (en) | 2005-06-18 | 2009-08-18 | Muralidhara Subbarao | Direct vision sensor for 3D computer vision, digital imaging, and digital video |
US20070102622A1 (en) | 2005-07-01 | 2007-05-10 | Olsen Richard I | Apparatus for multiple camera devices and method of operating same |
JP4984634B2 (en) | 2005-07-21 | 2012-07-25 | ソニー株式会社 | Physical information acquisition method and physical information acquisition device |
CA2553473A1 (en) | 2005-07-26 | 2007-01-26 | Wa James Tam | Generating a depth map from a tw0-dimensional source image for stereoscopic and multiview imaging |
US8274715B2 (en) | 2005-07-28 | 2012-09-25 | Omnivision Technologies, Inc. | Processing color and panchromatic pixels |
US7400458B2 (en) | 2005-08-12 | 2008-07-15 | Philips Lumileds Lighting Company, Llc | Imaging optics with wavelength dependent aperture stop |
US7940994B2 (en) | 2005-11-15 | 2011-05-10 | Teledyne Licensing, Llc | Multi-scale image fusion |
JP2007139893A (en) | 2005-11-15 | 2007-06-07 | Olympus Corp | Focusing detection device |
US7609291B2 (en) | 2005-12-07 | 2009-10-27 | Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. | Device and method for producing an enhanced color image using a flash of infrared light |
US20070133983A1 (en) | 2005-12-14 | 2007-06-14 | Matilda Traff | Light-controlling element for a camera |
JP4501855B2 (en) | 2005-12-22 | 2010-07-14 | ソニー株式会社 | Image signal processing apparatus, imaging apparatus, image signal processing method, and computer program |
JP4730082B2 (en) | 2005-12-22 | 2011-07-20 | ソニー株式会社 | Image signal processing apparatus, imaging apparatus, image signal processing method, and computer program |
US20070145273A1 (en) | 2005-12-22 | 2007-06-28 | Chang Edward T | High-sensitivity infrared color camera |
JP4147273B2 (en) | 2006-01-20 | 2008-09-10 | 松下電器産業株式会社 | Compound eye camera module and manufacturing method thereof |
US7819591B2 (en) | 2006-02-13 | 2010-10-26 | 3M Innovative Properties Company | Monocular three-dimensional imaging |
US20070189750A1 (en) | 2006-02-16 | 2007-08-16 | Sony Corporation | Method of and apparatus for simultaneously capturing and generating multiple blurred images |
CN101390131B (en) | 2006-02-27 | 2013-03-13 | 皇家飞利浦电子股份有限公司 | Rendering an output image |
US7585122B2 (en) | 2006-03-15 | 2009-09-08 | Nokia Corporation | Aperture construction for a mobile camera |
JP4695550B2 (en) | 2006-06-22 | 2011-06-08 | 富士フイルム株式会社 | Solid-state imaging device and driving method thereof |
US7612805B2 (en) | 2006-07-11 | 2009-11-03 | Neal Solomon | Digital imaging system and methods for selective image filtration |
US8124870B2 (en) | 2006-09-19 | 2012-02-28 | Itn Energy System, Inc. | Systems and processes for bifacial collection and tandem junctions using a thin-film photovoltaic device |
JP5315574B2 (en) | 2007-03-22 | 2013-10-16 | 富士フイルム株式会社 | Imaging device |
JP4757221B2 (en) | 2007-03-30 | 2011-08-24 | 富士フイルム株式会社 | Imaging apparatus and method |
JP4386096B2 (en) | 2007-05-18 | 2009-12-16 | ソニー株式会社 | Image input processing apparatus and method |
JP2009004605A (en) | 2007-06-22 | 2009-01-08 | Fujifilm Corp | Image sensor and imaging device |
JP4359855B2 (en) | 2007-07-09 | 2009-11-11 | Smc株式会社 | Solenoid valve drive circuit and solenoid valve |
US7956924B2 (en) | 2007-10-18 | 2011-06-07 | Adobe Systems Incorporated | Fast computational camera based on two arrays of lenses |
US20090159799A1 (en) | 2007-12-19 | 2009-06-25 | Spectral Instruments, Inc. | Color infrared light sensor, camera, and method for capturing images |
US20090175535A1 (en) | 2008-01-09 | 2009-07-09 | Lockheed Martin Corporation | Improved processing of multi-color images for detection and classification |
WO2009097552A1 (en) | 2008-02-01 | 2009-08-06 | Omnivision Cdm Optics, Inc. | Image data fusion systems and methods |
JP2009232348A (en) | 2008-03-25 | 2009-10-08 | Nikon Corp | Imaging apparatus, distance information acquiring method, image processing method, and drive control method of optical system |
US8958539B2 (en) | 2008-04-23 | 2015-02-17 | Centurylink Intellectual Property Llc | System and method for network based call transfers |
KR20090120159A (en) | 2008-05-19 | 2009-11-24 | 삼성전자주식회사 | Apparatus and method for combining images |
US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
EP3876510A1 (en) | 2008-05-20 | 2021-09-08 | FotoNation Limited | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US7773317B2 (en) | 2008-07-01 | 2010-08-10 | Aptina Imaging Corp. | Lens system with symmetrical optics |
US8184196B2 (en) * | 2008-08-05 | 2012-05-22 | Qualcomm Incorporated | System and method to generate depth data using edge detection |
GB2463480A (en) | 2008-09-12 | 2010-03-17 | Sharp Kk | Camera Having Large Depth of Field |
US8588541B2 (en) * | 2008-09-24 | 2013-11-19 | Nikon Corporation | Method and device for image deblurring using joint bilateral filtering |
US8406564B2 (en) * | 2008-09-24 | 2013-03-26 | Microsoft Corporation | Removing blur from an image |
US9282926B2 (en) | 2008-12-18 | 2016-03-15 | Sirona Dental Systems Gmbh | Camera for recording surface structures, such as for dental purposes |
JP5486017B2 (en) | 2009-01-16 | 2014-05-07 | アイピーリンク・リミテッド | Improving the depth of field of an imaging system |
JP5424679B2 (en) | 2009-03-18 | 2014-02-26 | キヤノン株式会社 | Imaging apparatus and signal processing apparatus |
US8125546B2 (en) | 2009-06-05 | 2012-02-28 | Omnivision Technologies, Inc. | Color filter array pattern having four-channels |
US8228417B1 (en) | 2009-07-15 | 2012-07-24 | Adobe Systems Incorporated | Focused plenoptic camera employing different apertures or filtering at different microlenses |
JP5552277B2 (en) | 2009-07-30 | 2014-07-16 | 森永製菓株式会社 | Royal jelly extract and human osteoblast growth inhibitor |
US20120154596A1 (en) | 2009-08-25 | 2012-06-21 | Andrew Augustine Wajs | Reducing noise in a color image |
EP2454876B1 (en) * | 2009-10-21 | 2013-12-04 | Ron Banner | Real-time video deblurring |
CN103210641B (en) | 2010-02-19 | 2017-03-15 | 双光圈国际株式会社 | Process multi-perture image data |
JP5670481B2 (en) * | 2010-02-19 | 2015-02-18 | デュアル・アパーチャー・インコーポレーテッド | Multi-aperture image data processing |
EP2594062B1 (en) | 2010-07-16 | 2016-09-14 | Dual Aperture International Co. Ltd. | Flash system for multi-aperture imaging |
KR101739880B1 (en) | 2010-12-01 | 2017-05-26 | 삼성전자주식회사 | Color filter array, image sensor having the same, and image processing system having the same |
US8478123B2 (en) | 2011-01-25 | 2013-07-02 | Aptina Imaging Corporation | Imaging devices having arrays of image sensors and lenses with multiple aperture sizes |
JP6023087B2 (en) | 2011-02-04 | 2016-11-09 | コーニンクレッカ フィリップス エヌ ヴェKoninklijke Philips N.V. | Method for recording image, method for obtaining 3D information from image, camera system |
US8593565B2 (en) | 2011-03-25 | 2013-11-26 | Gary S. Shuster | Simulated large aperture lens |
US8624986B2 (en) | 2011-03-31 | 2014-01-07 | Sony Corporation | Motion robust depth estimation using convolution and wavelet transforms |
US8749636B2 (en) | 2011-07-12 | 2014-06-10 | Lockheed Martin Corporation | Passive multi-band aperture filters and cameras therefrom |
AU2011224051B2 (en) * | 2011-09-14 | 2014-05-01 | Canon Kabushiki Kaisha | Determining a depth map from images of a scene |
JP5830348B2 (en) | 2011-10-26 | 2015-12-09 | オリンパス株式会社 | Imaging device |
US9117281B2 (en) | 2011-11-02 | 2015-08-25 | Microsoft Corporation | Surface segmentation from RGB and depth images |
US9264676B2 (en) | 2012-01-06 | 2016-02-16 | Microsoft Technology Licensing, Llc | Broadband imager |
JP6129309B2 (en) * | 2012-07-12 | 2017-05-17 | デュアル・アパーチャー・インターナショナル・カンパニー・リミテッド | Gesture based user interface |
US8687913B2 (en) * | 2012-07-17 | 2014-04-01 | Adobe Systems Incorporated | Methods and apparatus for image deblurring and sharpening using local patch self-similarity |
US8792710B2 (en) * | 2012-07-24 | 2014-07-29 | Intel Corporation | Stereoscopic depth reconstruction with probabilistic pixel correspondence search |
AU2012258467A1 (en) | 2012-12-03 | 2014-06-19 | Canon Kabushiki Kaisha | Bokeh amplification |
AU2013206601A1 (en) * | 2013-06-28 | 2015-01-22 | Canon Kabushiki Kaisha | Variable blend width compositing |
US20160255323A1 (en) * | 2015-02-26 | 2016-09-01 | Dual Aperture International Co. Ltd. | Multi-Aperture Depth Map Using Blur Kernels and Down-Sampling |
-
2015
- 2015-08-21 US US14/832,062 patent/US20160255323A1/en not_active Abandoned
-
2016
- 2016-02-25 WO PCT/KR2016/001838 patent/WO2016137241A1/en active Application Filing
- 2016-05-23 US US15/162,147 patent/US9721357B2/en not_active Expired - Fee Related
- 2016-05-23 US US15/162,154 patent/US9721344B2/en not_active Expired - Fee Related
- 2016-05-24 US US15/163,435 patent/US20160269600A1/en not_active Abandoned
Cited By (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10845459B2 (en) | 2013-06-13 | 2020-11-24 | Basf Se | Detector for optically detecting at least one object |
US10823818B2 (en) | 2013-06-13 | 2020-11-03 | Basf Se | Detector for optically detecting at least one object |
US11041718B2 (en) | 2014-07-08 | 2021-06-22 | Basf Se | Detector for determining a position of at least one object |
US11125880B2 (en) | 2014-12-09 | 2021-09-21 | Basf Se | Optical detector |
US10775505B2 (en) | 2015-01-30 | 2020-09-15 | Trinamix Gmbh | Detector for an optical detection of at least one object |
US20160269600A1 (en) * | 2015-02-26 | 2016-09-15 | Dual Aperture International Co. Ltd. | Multi-Aperture Depth Map Using Frequency Filtering |
US20160269710A1 (en) * | 2015-02-26 | 2016-09-15 | Dual Aperture International Co. Ltd. | Multi-aperture Depth Map Using Partial Blurring |
US9721344B2 (en) * | 2015-02-26 | 2017-08-01 | Dual Aperture International Co., Ltd. | Multi-aperture depth map using partial blurring |
US9721357B2 (en) * | 2015-02-26 | 2017-08-01 | Dual Aperture International Co. Ltd. | Multi-aperture depth map using blur kernels and edges |
US20160267667A1 (en) * | 2015-02-26 | 2016-09-15 | Dual Aperture International Co. Ltd. | Multi-aperture Depth Map Using Blur Kernels and Edges |
US9584717B2 (en) * | 2015-06-04 | 2017-02-28 | Lite-On Electronics (Guangzhou) Limited | Focusing method, and image capturing device for implementing the same |
US10955936B2 (en) | 2015-07-17 | 2021-03-23 | Trinamix Gmbh | Detector for optically detecting at least one object |
US10785412B2 (en) | 2015-08-20 | 2020-09-22 | Kabushiki Kaisha Toshiba | Image processing apparatus and image capturing apparatus |
US10382684B2 (en) * | 2015-08-20 | 2019-08-13 | Kabushiki Kaisha Toshiba | Image processing apparatus and image capturing apparatus |
US20170054910A1 (en) * | 2015-08-20 | 2017-02-23 | Kabushiki Kaisha Toshhiba | Image processing apparatus and image capturing apparatus |
US10412283B2 (en) * | 2015-09-14 | 2019-09-10 | Trinamix Gmbh | Dual aperture 3D camera and method using differing aperture areas |
US11024047B2 (en) * | 2015-09-18 | 2021-06-01 | The Regents Of The University Of California | Cameras and depth estimation of images acquired in a distorting medium |
US20180286066A1 (en) * | 2015-09-18 | 2018-10-04 | The Regents Of The University Of California | Cameras and depth estimation of images acquired in a distorting medium |
US20170150019A1 (en) * | 2015-11-23 | 2017-05-25 | Center For Integrated Smart Sensors Foundation | Multi-aperture camera system using disparity |
US10021282B2 (en) * | 2015-11-23 | 2018-07-10 | Center For Integrated Smart Sensors Foundation | Multi-aperture camera system using disparity |
US11211513B2 (en) | 2016-07-29 | 2021-12-28 | Trinamix Gmbh | Optical sensor and detector for an optical detection |
US10890491B2 (en) | 2016-10-25 | 2021-01-12 | Trinamix Gmbh | Optical detector for an optical detection |
US11428787B2 (en) | 2016-10-25 | 2022-08-30 | Trinamix Gmbh | Detector for an optical detection of at least one object |
US10948567B2 (en) | 2016-11-17 | 2021-03-16 | Trinamix Gmbh | Detector for optically detecting at least one object |
US11415661B2 (en) | 2016-11-17 | 2022-08-16 | Trinamix Gmbh | Detector for optically detecting at least one object |
US11635486B2 (en) | 2016-11-17 | 2023-04-25 | Trinamix Gmbh | Detector for optically detecting at least one object |
US11698435B2 (en) | 2016-11-17 | 2023-07-11 | Trinamix Gmbh | Detector for optically detecting at least one object |
US11860292B2 (en) | 2016-11-17 | 2024-01-02 | Trinamix Gmbh | Detector and methods for authenticating at least one object |
US10936851B2 (en) * | 2018-01-03 | 2021-03-02 | Samsung Electronics Co., Ltd. | Method and apparatus for recognizing object |
US20190205614A1 (en) * | 2018-01-03 | 2019-07-04 | Samsung Electronics Co., Ltd. | Method and apparatus for recognizing object |
WO2023240452A1 (en) * | 2022-06-14 | 2023-12-21 | 北京小米移动软件有限公司 | Image processing method and apparatus, electronic device, and storage medium |
Also Published As
Publication number | Publication date |
---|---|
WO2016137241A1 (en) | 2016-09-01 |
US9721344B2 (en) | 2017-08-01 |
US9721357B2 (en) | 2017-08-01 |
US20160269600A1 (en) | 2016-09-15 |
US20160267667A1 (en) | 2016-09-15 |
US20160269710A1 (en) | 2016-09-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9721344B2 (en) | Multi-aperture depth map using partial blurring | |
US11856291B2 (en) | Thin multi-aperture imaging system with auto-focus and methods for using same | |
US9495751B2 (en) | Processing multi-aperture image data | |
US9635275B2 (en) | Flash system for multi-aperture imaging | |
US9871980B2 (en) | Multi-zone imaging sensor and lens array | |
US20160286199A1 (en) | Processing Multi-Aperture Image Data for a Compound Imaging System | |
US10397465B2 (en) | Extended or full-density phase-detection autofocus control | |
US20130033579A1 (en) | Processing multi-aperture image data | |
US9774880B2 (en) | Depth-based video compression | |
US20160042522A1 (en) | Processing Multi-Aperture Image Data | |
US20170230638A1 (en) | Depth Measurement Techniques for a Multi-Aperture Imaging System | |
US20170034456A1 (en) | Sensor assembly with selective infrared filter array | |
US20160255334A1 (en) | Generating an improved depth map using a multi-aperture imaging system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: DUAL APERTURE INTERNATIONAL CO. LTD., KOREA, REPUB Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WAJS, ANDREW;REEL/FRAME:036404/0240 Effective date: 20150820 |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |