US10572974B2 - Image demosaicer and method - Google Patents
Image demosaicer and method Download PDFInfo
- Publication number
- US10572974B2 US10572974B2 US15/950,845 US201815950845A US10572974B2 US 10572974 B2 US10572974 B2 US 10572974B2 US 201815950845 A US201815950845 A US 201815950845A US 10572974 B2 US10572974 B2 US 10572974B2
- Authority
- US
- United States
- Prior art keywords
- pixel
- values
- primary
- green channel
- red
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4015—Demosaicing, e.g. colour filter array [CFA], Bayer pattern
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/64—Circuits for processing colour signals
- H04N9/646—Circuits for processing colour signals for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20216—Image averaging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20224—Image subtraction
Definitions
- Many consumer electronics products include at least one camera. These products include tablet computers, mobile phones, and smart watches. In such products, and in digital still cameras themselves, images and video captured in low-light environments suffer from low signal-to-noise ratio, which limits image and video quality. High-dynamic range images also face this problem, for example, in low-luminosity regions of the image.
- Embodiments disclosed herein ameliorate said image quality problems resulting from relatively low illumination.
- a method for demosaicing a raw image captured by an image sensor includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CF r , green color filters CF g , and blue color filters CF b each covering a respective one of a plurality of pixel sub-arrays of the pixel array.
- CFA Bayer-pattern color-filter array
- the pixel array including a plurality of pixels p(x,y) ⁇ located at a respective one of a plurality of coordinates (x,y) ⁇ , and beneath a color filter CF ⁇ , and each generating a respective one of a plurality of primary pixel-values B ⁇ (x,y) ⁇ , where ⁇ is one of r, g, and b.
- the method includes the following steps denoted 1-8:
- an image demosaicer for demosaicing a raw image includes a memory and a microprocessor.
- the raw image is captured by the image sensor of the aforementioned method.
- the memory stores non-transitory computer-readable instructions and is adapted to store the raw image.
- the microprocessor is adapted to execute the instructions to execute steps 1-8 of the aforementioned method.
- FIG. 1 depicts a scene imaged by a camera that includes an image sensor, which includes a pixel array.
- FIG. 2 is a schematic plan view of a region of an image sensor that includes the pixel array of FIG. 1 , and a four-cell Bayer color filter array (CFA) thereon, in an embodiment.
- CFA Bayer color filter array
- FIG. 3 is a schematic plan view of a region of a raw image corresponding to the region of the pixel array illustrated in FIG. 2 .
- FIGS. 4, 5, and 6 illustrate, respectively, a schematic view of a red channel, a green channel, and a blue channel generated from the raw image of FIG. 3 , in an embodiment.
- FIG. 7 is a schematic functional block diagram of an image demosaicer configured to generate a demosaiced image from raw the image of FIG. 3 .
- FIGS. 8 and 9 illustrate a horizontally-interpolated green channel and a vertically-interpolated green channel, respectively, generated by the image demosaicer of FIG. 7 , in an embodiment.
- FIGS. 10 and 11 illustrate refined versions of the interpolated green channels of FIG. 8 and FIG. 9 , respectively, generated by the image demosaicer of FIG. 7 , in an embodiment.
- FIGS. 12-15 graphically illustrate pixel-values involved in computing, horizontal-supplements and vertical-supplements used to produce the refined green channels of FIGS. 10 and 11 , in an embodiment.
- FIGS. 16 and 17 illustrate a horizontal-gradient image and a vertical-gradient image, respectively, each of which are respective examples of the gradient images generated by the image demosaicer of FIG. 7 , in an embodiment.
- FIG. 18 illustrates a full-resolution green channel that is an average of refined interpolated green channels of FIGS. 10 and 11 weighted by the gradient images of FIGS. 16 and 17 , in an embodiment.
- FIG. 19 illustrates a region of a binned raw image, which is generated from the raw image of FIG. 3 , in an embodiment.
- FIG. 20 illustrates a region of a binned green channel, which is generated from the full-resolution green channel of FIG. 18 , in an embodiment.
- FIG. 21 is the schematic illustration of a region of a demosaiced image, which is an example of the demosaiced image generated by the image demosaicer of FIG. 7 , in an embodiment.
- FIG. 22 is a flowchart illustrating a method for demosaicing a raw image, in an embodiment.
- FIG. 1 depicts a camera 100 imaging a scene 120 having a high dynamic range of luminance.
- Scene 120 includes a person 121 in front of a window 122 , through which a sunny scene 123 is visible.
- Camera 100 includes an imaging lens (not shown), an image sensor 130 , a memory 104 , and a microprocessor 102 communicatively coupled to image sensor 130 .
- Image sensor 130 includes a pixel array 132 A, which may have a color filter array (CFA) 140 thereon.
- Pixel array 132 A includes a plurality of pixels 132 , not shown in FIG. 1 for clarity of illustration.
- Each color filter of CFA 140 may be aligned with a respective pixel 132 of pixel array 132 A or with a center of a respective pixel sub-array of pixel array 132 A.
- a pixel sub-array is, for example, a m ⁇ n array of pixels 132 , where m and n are integers, at least one of which is greater than one.
- the imaging lens images scene 120 onto image sensor 130 .
- Image sensor 130 also includes circuitry 138 that includes at least one analog-to-digital converter.
- Each pixel 132 has a respective pixel charge corresponding to a respective intensity of light from a scene 120 imaged onto pixel array 132 A.
- Circuitry 138 converts each pixel charge to a respective one of a first plurality of pixel-values of a raw image 190 , which is stored in memory 104 .
- Camera 100 may include a display 139 configured to display a demosaiced version of raw image 190 as a demosaiced image 198 .
- FIG. 2 is a schematic plan view of a region of an image sensor 200 that includes pixel array 132 A and a four-cell Bayer CFA 240 thereon, hereinafter CFA 240 .
- Image sensor 200 and CFA 240 are examples of image sensor 130 and CFA 140 respectively.
- the two-by-two grouping of red, green, and blue color filter regions results in image sensor 200 having a low-light response resembling that of an image sensor with larger pixels, and hence requires a shorter exposure time for sufficient exposure of person 121 .
- a shortcoming of such a color filter array is lost image resolution.
- Embodiments disclosed herein restore this lost image resolution.
- Image sensor 200 and CFA 240 are positioned with respect to an x-y coordinate system 209 .
- all coordinates are expressed as (x,y) coordinates, that is, with the x value being listed first.
- the notation ⁇ (i,j) refers to a color filter at coordinates (i,j), where ⁇ denotes a transmission spectrum of a color filter and i and j are integers.
- ⁇ is one of r, g, b, c, m, y, and k which represent, respectively, red, green, blue, cyan, magenta, yellow, and clear (panchromatic) color filters.
- a subscript on a coordinate pair indicates the type color filter, of CFA 240 , covering a pixel 132 located at the coordinate.
- pixel p(x,y) ⁇ denotes a pixel 132 , located at coordinate (x,y), that is covered by a color filter with a transmission spectrum denoted by ⁇ .
- the pixel p(x,y) ⁇ generates a primary pixel-value B ⁇ (x,y).
- CFA 240 includes a plurality of red (R), green (G), and blue (B) color filters arranged in a four-cell Bayer pattern.
- R red
- G green
- B blue
- each color filter is aligned with a respective pixel of the pixel array.
- a conventional Bayer pattern may be viewed as a “single-cell Bayer pattern.”
- a four-pixel sub-array of the pixel array may be beneath one color filter of the same color.
- each color filter may be aligned with a two-by-two pixel sub-array of pixel array 132 A.
- pixel array 132 A includes a pixel sub-array 236 formed of four pixels 132 located at a respective one of four coordinates (5,5), (6,5), (5, 6), and (6,6) hereinafter coordinates C 1 .
- CFA 240 includes a blue color filter aligned to pixels 132 that constitute pixel sub-array 236 . This blue color filter may be a single continuous optical element, or include a plurality of optical elements, such as four distinct optical elements arranged in a two-by-two array.
- CFA 240 includes red and green color filters each aligned with a respective two-by-two pixel sub-array of pixel array 132 A.
- FIG. 3 is a schematic plan view of a region of a raw image 390 corresponding to the region of pixel array 132 A illustrated in FIG. 2 .
- Raw image 390 is an example of raw image 190 , and includes a plurality of primary pixel-values B r , a plurality of primary pixel-values B g , and a plurality of primary pixel-values B b .
- Each primary pixel-value B r is generated by a respective pixel 132 beneath a red color filter of CFA 140 .
- Each primary pixel-value B g is generated by a respective pixel 132 beneath a green color filter of CFA 140 .
- Each primary pixel-value B b is generated by a respective pixel 132 beneath a blue color filter of CFA 140 .
- primary pixel-values B r , B g , and B b at respective coordinates (3,4), (3,6), and (5,6) are generated by respective pixels of image sensor 200 at the same respective coordinates (3,4), (3,6), and (5,6).
- primary pixel-values B r (3,4), B b (3,6), and B g (5,6) are generated by respective pixels p(3,4) r , p(3,6) g , and p(5,6) b of image sensor 200 .
- FIGS. 4-6 illustrate, respectively, a schematic view of a red channel 400 , a green channel 500 , and a blue channel 600 generated from raw image 390 .
- Channels 400 , 500 , and 600 are each illustrated with respect to x-y coordinate system 209 .
- Red channel 400 includes primary pixel-values B r of raw image 390 at locations corresponding to red color filters of CFA 240 .
- Green channel 500 includes primary pixel-values B g of raw image 390 at locations corresponding to green color filters of CFA 240 .
- Blue channel 600 includes primary pixel-values B b of raw image 390 at locations corresponding to blue color filters of CFA 240 .
- FIG. 7 is a schematic functional block diagram of an image demosaicer 700 configured to generate, from raw image 790 , a demosaiced image 798 .
- Image demosaicer 700 may be implemented within camera 100 .
- Raw image 390 FIG. 3
- FIG. 3 is an example of raw image 790 .
- Image demosaicer 700 includes a microprocessor 702 and a memory 704 that stores software 720 that includes computer-readable instructions.
- Microprocessor 702 may be a digital signal processor such as an image processor.
- Memory 704 may be transitory and/or non-transitory and may include one or both of volatile memory (e.g., SRAM, DRAM, VRAM, or any combination thereof) and nonvolatile memory (e.g., FLASH, ROM, magnetic media, optical media, or any combination thereof).
- Memory 704 and microprocessor 702 may function as memory 104 and microprocessor 102 , respectively, of camera 100 , FIG. 1 .
- Microprocessor 702 is adapted to execute the instructions to perform functions of image demosaicer 700 as described herein.
- Memory 704 may store at least one of raw image 790 , software 720 , and intermediate outputs 740 .
- Raw image 790 includes color channels 791 , 792 , and 793 .
- Examples of color channels 791 , 792 , and 793 are, respectively, red channel 400 , green channel 500 , and blue channel 600 .
- Software 720 includes one or more of the following software modules that when executed by microprocessor 702 produce respective data outputs: a green-channel interpolator 722 , a resolution restorer 723 , a gradient filter 724 , a combiner 727 , a red-channel interpolator 728 , a blue-channel interpolator 729 , and a combiner 732 .
- Green-channel interpolator 722 is configured to interpolate color channel 792 to yield a horizontally-interpolated green channel 742 H and vertically-interpolated green channel 742 V.
- Resolution restorer 723 produces a refined green channel 743 H and a refined green channel 743 V from, respectively, horizontally-interpolated green channel 742 H and vertically-interpolated green channel 742 V.
- Gradient filter 724 produces a gradient image 744 H and a gradient image 744 V from, respectively, horizontally-interpolated green channel 742 H and vertically-interpolated green channel 742 V.
- Gradient images 744 H and 744 V may be obtained by applying a gradient filter to channels 743 H and 743 V, respectively.
- gradient images 744 H and 744 V may be obtained by applying a gradient filter to channels 742 H and 742 V, respectively.
- Combiner 727 produces a full-resolution green channel 747 from refined green channels 743 H and 743 V.
- Red-channel interpolator 728 generates full-resolution red channel from full-resolution green channel 747 and color channel 791 .
- Blue-channel interpolator 729 generates full-resolution blue channel from full-resolution green channel 747 and color channel 793 .
- Combiner 732 combines full-resolution channels 747 , 748 , and 79 to form demosaiced image 798 .
- FIGS. 8 and 9 are schematic plan views of a region of a horizontally-interpolated green channel 800 and a vertically-interpolated green channel 900 , respectively.
- Interpolated channels 800 and 900 are examples of interpolated channels 742 H and 742 V, respectively.
- Interpolated channel 800 includes a plurality of primary pixel-values B g (x,y) g , and a plurality of green pixel-values 801 ( x,y ) r , and a plurality of green pixel-values 803 ( x,y ) b .
- Coordinates (x,y) r and (x,y) b correspond to, respectively, locations of red color filters and blue color filters of CFA 240 .
- Green pixel-values 801 ( x,y ) r and 803 ( x,y ) b are interpolated from horizontally-adjacent primary pixel-values B g (x,y) g of green channel 500 .
- non-primary pixel-value 803 (6,5) b is interpolated from at least two primary pixel-values B g (6,y ⁇ 5) g .
- Green pixel-values 801 ( x,y ) r and 803 ( x,y ) b are each examples of non-primary pixel-values because they are part of interpolated green channel 800 at coordinates (x,y) beneath non-green color filters.
- Interpolated channel 900 includes a plurality of primary pixel-values B g (x,y) g and a plurality of green pixel-values 901 (x,y) r , and a plurality of green pixel-values 903 ( x,y ) b .
- Green pixel-values 901 ( x,y ) r and 903 ( x,y ) b are interpolated from vertically-adjacent primary pixel-values B g (x,y) g .
- green pixel-value 903 (6,5) b is interpolated from at least two primary pixel-values B g (x ⁇ 6,5) g .
- Green pixel-values 901 ( x,y ) r and 903 ( x,y ) b are each examples of non-primary pixel-values because they are part of interpolated green channel 900 at coordinates (x,y) beneath non-green color filters.
- Both interpolated channels 800 and 900 may be logarithmically interpolated from green channel 500 .
- the logarithms of interpolated pixel-values 801 , 803 , 901 , and 903 may be interpolated from logarithms of primary pixel-values B g (x,y) g of green channel 500 .
- Logarithmic interpolation enables accurate interpolation without requiring information about white balance and illumination associated with raw image 390 .
- Interpolating logarithms of pixel-values also more heavily weights smaller pixel-values than does interpolating pixel-values themselves, which contributes to accurate gradient detection unperturbed by high-pixel-value noise.
- FIGS. 10 and 11 are schematic plan views of a region of a refined horizontally-interpolated green channel 1000 and a refined vertically-interpolated green channel 1100 , respectively.
- Refined green channels 1000 and 1100 are examples of refined green channels 743 H and 743 V, respectively.
- Refined green channel 1000 includes the following green pixel-values: primary pixel-values B g (x,y) g of raw image 390 , a plurality of green pixel-values 1001 ( x,y ) r , and a plurality of green pixel-values 1003 ( x,y ) b .
- Each green pixel-value 1001 ( x,y ) r may be equal to a sum of green pixel-value 801 ( x,y ) r and a red horizontal-supplement H r (x,y), as shown by equation 1.
- Each green pixel-value 1003 ( x,y ) b may be equal to a sum of green pixel-value 803 ( x,y ) b and a blue horizontal-supplement H b (x,y), as shown by equation 2.
- 1001( x,y ) r 801( x,y ) r +H r ( x,y ) (1)
- 1003( x,y ) b 803( x,y ) b +H b ( x,y ) (2)
- Refined green channel 1100 includes the following green pixel-values: plurality of primary pixel-values B g (x,y) g of raw image 390 , a plurality of green pixel-values 1101 ( x,y ) r , and a plurality of green pixel-values 1103 ( x,y ) b .
- Each green pixel-value 1101 ( x,y ) r may be equal to a sum of green pixel-value 901 ( x,y ) r and a red vertical-supplement V r (x,y), as shown by equation 3.
- Each green pixel-value 1103 ( x,y ) b may be equal to a sum of green pixel-value 903 ( x,y ) b and a blue vertical-supplement V b (x,y), as shown by equation 4.
- 1101( x,y ) r 901( x,y ) r +V r ( x,y )
- 1103( x,y ) b 903( x,y ) b +V b ( x,y ) (4)
- Coordinate (x,y) of horizontal-supplements H(x,y) and vertical-supplements V (x,y) corresponds to an m ⁇ n pixel sub-array of pixel array 132 A, discussed above regarding image sensor 200 and CFA 240 .
- red horizontal-supplement H r (x,y) is a linear combination of
- Horizontal coordinates y 1 and y 2 may satisfy
- Green pixel-value 1001 ( x 1 ,y 1 ) r may equal the sum of green pixel-value 801 (x 1 ,y 1 ) r and horizontal-supplement H r (x 1 ,y 1 ) r .
- red vertical-supplement V r (x,y) is a linear combination of:
- Vertical coordinates x 1 and x 2 may satisfy
- Supplements H r (x,y) and V r (x,y) may include non-zero terms for pixel-values B r (x,y) r further away from coordinate (x 1 ,y 1 ) than those indicated above.
- blue horizontal-supplement H b (x,y) is a linear combination of:
- Green pixel-value 1001 ( x 2 ,y 2 ) b may equal the sum of green pixel-value 803 (x 2 ,y 2 ) b and horizontal-supplement H b (x 2 ,y 2 ) b .
- blue vertical-supplement V b (x,y) is a linear combination of:
- Green pixel-value 1101 ( x 2 ,y 2 ) b may equal the sum of green pixel-value 901 ( x 2 ,y 2 ) b and vertical-supplement V b (x 2 ,y 2 ) b .
- FIGS. 16 and 17 illustrate respective a gradient images 1600 and 1700 , which are respective examples of horizontal-gradient image 744 H and vertical-gradient image 744 V.
- Horizontal-gradient image 1600 is a difference between refined horizontally-interpolated green channel 1000 and raw image 390 .
- Vertical-gradient image 1700 is a difference between refined vertically-interpolated green channel 1100 and raw image 390 . Accordingly, values of gradient images 1600 and 1700 beneath green color filters of CFA 240 equal zero, as both channels 1000 and 1100 include primary pixel-values B g as such locations. At least one of a color gradient and a luminance gradient may be applied to gradient images 1600 and 1700 .
- Gradient images 1600 and 1700 may be spatially-averaged, e.g., via a convolution filter, over a group of pixels such that its resolution is less than resolutions of channel 1000 , 1100 , and raw image 390 .
- gradient images 1600 and 1700 may be averaged over a weighted or unweighted q x ⁇ q y kernel, where each of q x and q y exceed both m and n, where m ⁇ n is the size of the pixel sub-array beneath a same color filter in image sensor 200 .
- an average vertical-gradient ⁇ v of gradient image 1700 at coordinate (x,y) may be expressed by equation (6).
- Equations 3 and 4 are example expressions of pixel-values 1101 and 1103 .
- Gradient filter 724 may also suppress noise in gradient images 744 H and 744 V.
- gradient filter may reduce pixel values of gradient images 1600 and 1700 at coordinates (x,y) where gradient images 1600 and 1700 differ by less than a predetermined threshold. Such small differences denote noise, and hence do not contribute to accurate gradient values.
- FIG. 18 illustrates a full-resolution green channel 1800 , which is an example of full-resolution green channel 747 .
- full-resolution green channel 1800 may be an average of refined interpolated green channels of 1000 and 1100 , weighted by the gradient images 1700 and 1600 respectively, and normalized by a sum of gradient images 1600 and 1700 .
- Equations 7 and 8 express, respectively, this relationship in general and as applied to channels 1000 , 1100 and gradient images 1600 and 1700 .
- ⁇ h and ⁇ v represent gradient images 744 H and 744 V, respectively.
- pixel-value 1802 ( x,y ) denotes a pixel-value of full-resolution green channel 1800 at coordinate (x,y).
- full-resolution green channel 1800 has pixel-values equal to primary pixel-values B g . For example, at coordinates (x,y) g corresponding to green color filters of CFA 240 , full-resolution green
- FIG. 19 illustrates a region of a binned raw image 1900 , which is generated from raw image 390 .
- FIG. 20 illustrates a region of a binned green channel 2000 , which is generated from full-resolution green channel 1800 . Binning raw image 390 and full-resolution green channel 1800 enable a chrominance interpolation process with reduced color aliasing and increased speed.
- Binned raw image 1900 includes a plurality of binned pixel-values: binned red pixel-values 1901 , binned green pixel-values 1902 , and binned blue pixel-values 1903 .
- Each binned pixel-value may be an average of pixel-values of raw image 390 beneath a pixel sub-array of pixel array 132 A.
- binned red pixel-value 1901 (1) is an average of red primary pixel-values B r at coordinates (3,3), (3,4), (4,3), and (4,4);
- binned green pixel-value 1902 (1) is an average of green primary pixel-values B g at coordinates (3,5), (3,6), (4,5), and (4,6);
- binned blue pixel-value 1903 (1) is an average of blue primary pixel-values B b at coordinates (5,5), (5,6), (6,5), and (6,6).
- Binned green channel 2000 includes a plurality of binned green pixel-values 2001 , 2002 , and 2003 generated by pixel-subarray, of pixel array 132 A, located beneath red, green, and blue color filters of CFA 240 , respectively.
- Each binned pixel-value 2001 - 2003 may be an average of pixel-values of full-resolution green channel 1800 beneath a pixel sub-array of pixel array 132 A.
- binned pixel-value 2001 (1) is an average of pixel-values 1802 (3,3), 1802 (4,4) and primary pixel-values B g (3,4) and B g (4,3);
- binned green pixel-value 2002 (1) is an average of green primary pixel-values B g (3,5), B g (3,6), and pixel-values 1802 (4,5) and 1802 (4,6);
- binned blue pixel-value 2003 (1) is an average of blue primary pixel-values B b at coordinates pixel-values 1802 (5,5), 1802 (5,6) and primary pixel-values B g (6,5), and B g (6,6).
- FIG. 21 is the schematic illustration of a region of a demosaiced image 2100 that shows one primary pixel-value B a and two non-primary pixel-values V ⁇ corresponding to a location of each pixel 132 of pixel array 132 A.
- the fill-pattern at each coordinate (x,y) indicates the type of color filter of CFA 240 aligned with the pixel 132 at the coordinate: vertical lines denote red, diagonal lines denote green, and horizontal lines denote blue.
- Demosaiced image 2100 is an example of demosaiced image 798 produced by image demosaicer 700 , FIG. 7 .
- the following examples illustrate computation of a blue pixel-value at a coordinate beneath a red color filter and red pixel-value at a coordinate beneath a blue color filter.
- Pixel-values at these coordinates are enclosed by a dashed box in FIG. 21 , and pixel-values therein computed below are presented in bold letters. Equations below refer to pixel-values of binned raw image 1900 and binned green channel 2000 , the locations of which are indicated by solid-line boxes in FIG. 21 .
- demosaiced image 2100 includes red pixel-value I r (5,5), green pixel-value I g (5,5), blue primary pixel-value B b (5,5).
- Green pixel-value I g (5,5) equals pixel-value 1802 (5,5) of full-resolution green channel 1800 , FIG. 18 .
- Red pixel-value I r (5,5) may be computed by chrominance interpolation equations (21r-1) through (21r-8).
- R avg is an average of binned red pixel-values 1901 (1-4) of binned raw image 1900 .
- G avg is an average of binned pixel-values 2001 (1-4) of binned green channel 2000 , which correspond to pixels 132 beneath a green color filter of CFA 240 .
- RG avg is an average of an addend-by-addend multiplication of R avg and G avg .
- GG avg is an average of an addend-by-addend multiplication of G avg and G avg . Equations (21r-5) through (21r-7) derive a weight based on a covariance and variance between R avg and G avg .
- R avg 1 4 ⁇ ( 1901 ⁇ ( 1 ) + 1902 ⁇ ( 2 ) + 1901 ⁇ ( 3 ) + 1901 ⁇ ( 4 ) ) ( 21 ⁇ r ⁇ - ⁇ 1 )
- G avg 1 4 ⁇ ( 2001 ⁇ ( 1 ) + 2001 ⁇ ( 2 ) + 2001 ⁇ ( 3 ) + 2001 ⁇ ( 4 ) ) ( 21 ⁇ r ⁇ - ⁇ 2 )
- RG avg 1 4 ⁇ ( 1901 ⁇ ( 1 ) ⁇ 2001 ⁇ ( 1 ) + 1901 ⁇ ( 2 ) ⁇ 2001 ⁇ ( 2 ) + 1901 ⁇ ( 3 ) ⁇ 2001 ⁇ ( 3 ) + 1901 ⁇ ( 4 ) ⁇ 2001 ⁇ ( 4 ) ) ( 21 ⁇ r ⁇ - ⁇ 3 )
- GG avg 1 4 ⁇ ( 2001 ⁇ ( 1 ) 2 + 2001 ⁇ ( 2 ) 2 + 2001 ⁇ ( 3 )
- demosaiced image 2100 includes red primary pixel-value B r (7,7) of raw image 390 , green pixel-value I g (7,7), and blue pixel-value I b (7,7).
- Green pixel-value I g (7,7) equals pixel-value 1802 (7,7) of full-resolution green channel 1800 , FIG. 18 .
- Blue pixel-value I b (7,7) may be computed by chrominance interpolation equations (21b-1) through (21b-8), e.g., by blue-channel interpolator 729 of image demosaicer 700 .
- B avg is an average of binned blue pixel-values 1903 (1-4) of binned raw image 1900 .
- G avg is an average of binned pixel-values 2003 (1-4) of binned green channel 2000 , which correspond to pixels 132 beneath a green color filter of CFA 240 .
- BG avg is an average of an addend-by-addend multiplication of B avg and G avg .
- GG avg is an average of an addend-by-addend multiplication of G avg and G avg . Equations (21b-5) through (21b-8) derive a weight based on a covariance and variance between B avg and G avg .
- the weight of equations (21r-8) and (21b-8) are determined at least in part by applying a guided filter to full-resolution green channel 1800 and raw image 390 , and/or images derived therefrom.
- the guided filter may be applied to full-resolution green channel 1800 and red channel 400 and blue channel 600 of raw image 390 , for example, such that weights are derived from a correlation between channels 1800 , 400 , and 600 .
- Use of a guided filter reduces color aliasing that may arise during chrominance interpolation.
- a guided filter is, for example, one described in K. He, J. Sun and X. Tang, “Guided Image Filtering,” in IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 35, no. 6, pp. 1397-1409, June 2013.
- FIG. 22 is a flowchart illustrating a method 2200 for demosaicing a raw image captured by an image sensor that includes a pixel array and a Bayer-pattern CFA having a plurality of red color filters CF r , green color filters CF g , and blue color filters CF b each covering a respective one of a plurality of m ⁇ n pixel sub-arrays of the pixel array.
- the pixel array includes a plurality of pixels p(x,y) ⁇ located at a respective one of a plurality of coordinates (x,y) ⁇ and beneath a color filter CF ⁇ .
- Method 2200 may be implemented by image demosaicer 700 processing a raw image 390 , e.g., by microprocessor 702 executing computer-readable instructions of software 720 .
- Method 2200 includes at least one of steps 2210 H, 2210 V, 2220 H, 2220 V, 2230 , 2240 R, 2240 B, and 2250 .
- Step 2210 H includes horizontally-interpolating a green channel, of the raw image, formed of primary pixel-values B g (x,y) g of the plurality of primary pixel-values B ⁇ (x,y) ⁇ , to yield a horizontally-interpolated green channel that includes both primary pixel-values B g (x,y) g and non-primary pixel-values I gh (x,y) r,b .
- green-channel interpolator 722 interpolates green channel 500 to yield horizontally-interpolated green channel 800 .
- green-channel interpolator 722 may interpolate a logarithm of green channel 500 to yield a logarithm of channel 800 , and also apply an inverse logarithmic function thereto to yield horizontally-interpolated green channel 800
- Step 2210 V includes vertically-interpolating a green channel, of the raw image, formed of primary pixel-values B g (x,y) g of the plurality of primary pixel-values B ⁇ (x,y) ⁇ , to yield a vertically-interpolated green channel that includes both primary pixel-values B g (x,y) g and non-primary pixel-values I gv (x,y) r,b
- green-channel interpolator 722 interpolates green channel 500 to yield vertically-interpolated green channel 900 .
- green-channel interpolator 722 may interpolate a logarithm of green channel 500 to yield a logarithm of channel 900 , and also apply an inverse logarithmic function thereto to yield vertically-interpolated green channel 900 .
- Step 2220 H includes restoring resolution to the horizontally-interpolated green channel by modifying each non-primary pixel-value I gh (x,y) r,b by a plurality of horizontally-neighboring non-primary pixel-values I gh (x,y) r,b , to yield a refined horizontally-interpolated green channel.
- resolution restorer 723 generates refined horizontally-interpolated green channel 1000 from horizontally-interpolated green channel 800 .
- Step 2220 V includes restoring resolution to the vertically-interpolated green channel by modifying each non-primary pixel-value I gh (x,y) r,b by a plurality of vertically-neighboring non-primary pixel-values I gh (x,y) r,b , to yield a refined vertically-interpolated green channel.
- resolution restorer 723 generates refined vertically-interpolated green channel 1100 from vertically-interpolated green channel 900 .
- Step 2230 includes generating a full-resolution green channel from the horizontally-interpolated green channel, vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel.
- combiner 727 generates full-resolution green channel 1800 from refined horizontally-interpolated green channel 1000 and refined vertically-interpolated green channel 1100 .
- Step 2240 R includes generating a full-resolution red channel by determining a plurality of non-primary red pixel-values I r (x t ,y t ) g,b , each corresponding to a respective one of the plurality of coordinates (x t ,y t ) g,b, from a local-red mean value of neighboring red primary pixel-values B r (x,y) r and the full-resolution green channel.
- red-channel interpolator 728 generates red pixel-values I r (x,y) of demosaiced image 2100 , for example, by executing one or more of equations (21r-1)-(21r-8).
- an example of full-resolution red channel 748 includes red pixel-values I r (x,y).
- Step 2240 B includes generating a full-resolution blue channel by determining a plurality of non-primary blue pixel-values I b (x t ,y t ) g,r , each corresponding to a respective one of the plurality of coordinates (x t ,y t ) g,r , from a local-blue mean value of neighboring blue primary pixel-values B b (x,y) b and the full-resolution green channel.
- blue-channel interpolator 729 generates blue pixel-values I b (x,y) of demosaiced image 2100 , for example, by executing one or more of equations (21b-1)-(21b-8).
- an example of full-resolution blue channel 749 includes blue pixel-values I b (x,y).
- Step 2250 includes combining the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.
- combiner 732 combines full-resolution red channel 748 , full-resolution green channel 1800 , and full-resolution blue channel 749 to yield demosaiced image 2100 .
- (A1) denotes a method for demosaicing a raw image captured by an image sensor.
- the image sensor includes a pixel array and a Bayer-pattern color-filter array (CFA) having a plurality of red color filters CF r , green color filters CF g , and blue color filters CF b each covering a respective one of a plurality of pixel sub-arrays of the pixel array.
- CFA Bayer-pattern color-filter array
- the pixel array including a plurality of pixels p(x,y) ⁇ located at a respective one of a plurality of coordinates (x,y) ⁇ , and beneath a color filter CF ⁇ , and each generating a respective one of a plurality of primary pixel-values B ⁇ (x,y) ⁇ , where ⁇ is one of r, g, and b.
- Method (A1) includes the following steps denoted 1-8:
- both the horizontally-interpolating the green channel and the vertically-interpolating the green channel may include linearly interpolating a logarithm of primary pixel-values B g (x,y) g .
- the step of modifying each non-primary pixel-value I gh (x,y) r,b may include, for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between (i) a pair of horizontally-neighboring source pixel sub-arrays each beneath respective red color filters, of the CFA and horizontally collinear with the target pixel sub-array: determining a red horizontal-supplement H r (x,y) as a linear combination of B r (x,y) r , primary pixel-values B r (x,y 1 ) r of horizontally-adjacent pixels p(x,y 1 ) beneath the target pixel sub-array, and primary pixel-values B r
- the step of modifying each non-primary pixel-value I gv (x,y) r,b may include, for each pixel p(x,y) beneath a red color filter of a target pixel sub-array, of the plurality of pixel sub-arrays, located between a pair of vertically-neighboring source pixel sub-arrays each beneath a respective red color filter of the CFA and vertically collinear with the target pixel sub-array: determining a red vertical-supplement V r (x,y) as a linear combination of B r (x,y) r , primary pixel-values B r (x 1 ,y) r of vertically-adjacent pixels p(x 1 ,y) beneath the target pixel sub-array, and primary pixel-values B r (x
- Any method denoted by one of (A1) through (A4) may further include determining the horizontal-gradient and the vertical-gradient by: (a) subtracting the raw image from the refined horizontally-interpolated green channel, and (b) subtracting the raw image from the refined vertical-interpolated green channel.
- At least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel may include binning the full-resolution green channel, and binning the raw image.
- the at least one of (i) generating the full-resolution red channel and (ii) generating the full-resolution blue channel may include: computing a variance of the binned full-resolution green channel and a covariance of (a) the binned full-resolution green channel and (b) the binned raw image.
- (A8) In any method denoted by one of (A6) and (A7): (a) binning of the full-resolution green channel may yield a binned green channel that includes a plurality of binned green pixel-values; and (b) binning the raw image may yield a binned raw image that includes a plurality of binned red pixel-values based on the plurality of primary pixel-values B r and a plurality of binned blue pixel-values based on the plurality of primary pixel-values B b , the local-red mean value being an average of binned red pixel-values, the local-blue mean value being an average of binned blue pixel-values.
- each of the plurality of non-primary red pixel-values I r (x t ,y t ) g,b may have a magnitude that increases in response to an increasing covariance of the average of binned red pixel-values and an average of binned green pixel-values; and each of the plurality of non-primary blue pixel-values I b (x t ,y t ) g,r may have a magnitude that increases in response to an increasing covariance of the average of binned blue pixel-values and the average of binned green pixel-values.
- each of the plurality of non-primary red pixel-values I r (x t ,y t ) g,b may have a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values; and each of the plurality of non-primary blue pixel-values I b (x t ,y t ) g,r may have a magnitude that decreases in response to an increasing variance of the average of binned green pixel-values.
- An image demosaicer (B1) for demosaicing a raw image includes a memory and a microprocessor.
- the raw image is captured by the image sensor of the aforementioned method.
- the memory stores non-transitory computer-readable instructions and is adapted to store the raw image.
- the microprocessor is adapted to execute the instructions to execute any of the methods denoted by (A1) through (A10).
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Color Television Image Signal Generators (AREA)
Abstract
Description
-
- (1) horizontally-interpolating a green channel, of the raw image, formed of primary pixel-values Bg(x,y)g of the plurality of primary pixel-values Bα(x,y)α, to yield a horizontally-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igh(x,y)r,b;
- (2) modifying each non-primary pixel-value Igh(x,y)r,b by a plurality of horizontally-neighboring non-primary pixel-values Igh(x,y)r,b, to yield a refined horizontally-interpolated green channel;
- (3) vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igv(x,y)r,b;
- (4) modifying each non-primary pixel-value Igv(x,y)r,b by a plurality of vertically-neighboring non-primary pixel-values Igv(x,y)r,b, to yield a refined vertically-interpolated green channel;
- (5) generating a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;
- (6) generating a full-resolution red channel by determining a plurality of non-primary red pixel-values Ir(xt,yt)g,b, each corresponding to a respective one of the plurality of coordinates (xt,yt)g,b, from a local-red mean value of neighboring red primary pixel-values Br(x,y)r and the full-resolution green channel;
- (7) generating a full-resolution blue channel by determining a plurality of non-primary blue pixel-values Ib(xt,yt)g,r, each corresponding to a respective one of the plurality of coordinates (xt,yt)g,r, from a local-blue mean value of neighboring blue primary pixel-values Bb(x,y)b and the full-resolution green channel; and
- (8) combining the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.
1001(x,y)r=801(x,y)r +H r(x,y) (1)
1003(x,y)b=803(x,y)b +H b(x,y) (2)
1101(x,y)r=901(x,y)r +V r(x,y) (3)
1103(x,y)b=903(x,y)b +V b(x,y) (4)
-
- (a) Br(x,y),
- (b) primary pixel-values Br(x,y1) of horizontally-adjacent pixels p(x,y1) beneath the target pixel sub-array, and
- (c) primary pixel-values Br(x,y2) of horizontally-collinear pixels p(x,y2) beneath red color filters of the pair of neighboring source pixel sub-arrays.
H r(x 1 ,y 1)r =α 0 B r(x 1 ,y 1)r −α 1 B r(x 1 ,y 1−4)r−α2 B r(x 1 ,y 1−3)r −α 3 B r(x 1 ,y 1+1)r −α 4 B r(x 1 ,y 1+4)r −α 5 B r(x 1 ,y 1+5)r.
Green pixel-value 1001(x 1,y1)r may equal the sum of green pixel-value 801(x1,y1)r and horizontal-supplement Hr(x1,y1)r. Coefficients α0-5 are, for example: α0=4, α1-4=1, α5=0.
V r(x 1 ,y 1)r =c 0 B r(x 1 ,y 1)r −c 1 B r(x 1−5,y 1)r −c 2 B r(x 1−4,y 1)r −c 3 B r(x 1−1,y 1)r −c 4 B r(x 1+3,y 1)r −c 5 B r(x 1+4,y 1)r
Green pixel-value 1101(x 1,y1)r may equal the sum of green pixel-value 901(x 1,y1)r and vertical-supplement Vr(x1,y1)r. Coefficients c0-5 are, for example: c0=4, c1=0, and c2-5=1.
-
- (a) Bb (x,y)
- (b) primary pixel-values Bb (x,y1) of horizontally-adjacent pixels p(x,y1) beneath the target pixel sub-array, and
- (c) primary pixel-values Bb (x,y2) of horizontally-collinear pixels p(x,y2).
Horizontal coordinates y1 and y2 may satisfy |y−y1|<n and |y−y2|≥n.
H b(x 2 ,y 2)b =α 0 B b(x 2 ,y 2)b −α 1 B b(x 2 ,y 2−5)b −α 2 B b(x 2 ,y 2−4)b −α 3 B b(x 2 ,y 2−1)b −α 4 B b(x 2 ,y 2+3)b −α 5 B b(x 2 ,y 2+4)b.
Green pixel-value 1001(x 2,y2)b may equal the sum of green pixel-value 803 (x2,y2)b and horizontal-supplement Hb(x2,y2)b. Coefficients α0-5 are, for example: α0=4, α1-4=1, α5=0.
-
- (a) Bb(x,y),
- (b) primary pixel-values Bb (x1,y) of vertically-adjacent pixels p(x1,y) beneath the target pixel sub-array, and
- (c) primary pixel-values Bb (x2,y) of vertically-collinear pixels p(x2,y) beneath blue color filters of the pair of neighboring source pixel sub-arrays.
Vertical coordinates x1 and x2 satisfy |x−x1|<m and |x−x2|≥m.
V b(x 2 ,y 2)b =c 0 B b(x 2 ,y 2)b −c 1 B b(x 2−5,y 2)b −c 2 B b(x 2 −y 2)b −c 3 B b(x 2+1,y 2)b −c 4 B b(x 2+3,y 2)b −c 5 B b(x 2+4,y 2)b.
Green pixel-value 1101(x 2,y2)b may equal the sum of green pixel-value 901(x 2,y2)b and vertical-supplement Vb (x2,y2)b. Coefficients c0-5 are, for example: c0=4, c1=0, and c2-5=1.
1600(x,y)=|1001−B r L + B g −B r G|+|1003−B b L + B g −B b G|, (5)
where each right-hand-side term is evaluated at coordinate (x,y), · L and · G denote, respectively, a local average within the kernel, e.g.,
1700(x,y)=|1101−B r L + B g −B r G|+|1103−B b L + B g −B b G|. (6)
In equation (6), each right-hand-side term is evaluated at coordinate (x,y).
At pixel coordinates (x,y)g corresponding to green color filters of
Chrominance Interpolation
Red-
Blue-
-
- (1) horizontally-interpolating a green channel, of the raw image, formed of primary pixel-values Bg(x,y)g of the plurality of primary pixel-values Bα(x,y)α, to yield a horizontally-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igh(x,y)r,b;
- (2) modifying each non-primary pixel-value Igh(x,y)r,b by a plurality of horizontally-neighboring non-primary pixel-values Igh(x,y)r,b, to yield a refined horizontally-interpolated green channel;
- (3) vertically-interpolating the green channel to yield a vertically-interpolated green channel that includes both primary pixel-values Bg(x,y)g and non-primary pixel-values Igv(x,y)r,b;
- (4) modifying each non-primary pixel-value Igv (x,y)r,b by a plurality of vertically-neighboring non-primary pixel-values Igv(x,y)r,b, to yield a refined vertically-interpolated green channel;
- (5) generating a full-resolution green channel from the refined horizontally-interpolated green channel, refined vertically-interpolated green channel, a horizontal-gradient of the horizontally-interpolated green channel, and a vertical-gradient of the vertically-interpolated green channel;
- (6) generating a full-resolution red channel by determining a plurality of non-primary red pixel-values Ir(xt,yt)g,b, each corresponding to a respective one of the plurality of coordinates (xt,yt)g,b, from a local-red mean value of neighboring red primary pixel-values Br(x,y)r and the full-resolution green channel;
- (7) generating a full-resolution blue channel by determining a plurality of non-primary blue pixel-values Ib(xt,yt)g,r, each corresponding to a respective one of the plurality of coordinates (xt,yt)g,r, from a local-blue mean value of neighboring blue primary pixel-values Bb(x,y)b and the full-resolution green channel; and
- (8) combining the full-resolution red channel, the full-resolution green channel, and the full-resolution blue channel to yield a demosaiced raw image.
Claims (20)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/950,845 US10572974B2 (en) | 2018-04-11 | 2018-04-11 | Image demosaicer and method |
CN201910227536.XA CN110365961B (en) | 2018-04-11 | 2019-03-25 | Image demosaicing device and method |
TW108111554A TWI737979B (en) | 2018-04-11 | 2019-04-01 | Image demosaicer and method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/950,845 US10572974B2 (en) | 2018-04-11 | 2018-04-11 | Image demosaicer and method |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190318451A1 US20190318451A1 (en) | 2019-10-17 |
US10572974B2 true US10572974B2 (en) | 2020-02-25 |
Family
ID=68162011
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/950,845 Active 2038-07-28 US10572974B2 (en) | 2018-04-11 | 2018-04-11 | Image demosaicer and method |
Country Status (3)
Country | Link |
---|---|
US (1) | US10572974B2 (en) |
CN (1) | CN110365961B (en) |
TW (1) | TWI737979B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11263724B2 (en) * | 2019-10-02 | 2022-03-01 | Hanwha Techwin Co., Ltd. | Device for interpolating colors and method thereof |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10726531B2 (en) * | 2018-09-04 | 2020-07-28 | Richard H. Vollmerhausen | Resolution enhancement of color images |
RU2712436C1 (en) * | 2019-08-09 | 2020-01-28 | Самсунг Электроникс Ко., Лтд. | Rccb image processing method |
CN110852953B (en) * | 2019-11-15 | 2022-09-02 | 展讯通信(上海)有限公司 | Image interpolation method and device, storage medium, image signal processor and terminal |
TWI743628B (en) * | 2019-12-18 | 2021-10-21 | 瑞昱半導體股份有限公司 | Method for over-exposed image correction and circuit system |
CN113781349A (en) * | 2021-09-16 | 2021-12-10 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, electronic device, and storage medium |
CN116896688B (en) * | 2023-09-11 | 2023-12-26 | 归芯科技(深圳)有限公司 | Bayer image processing method, device, chip and storage medium |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6791609B2 (en) * | 1999-12-20 | 2004-09-14 | Texas Instruments Incorporated | Digital still camera system and method |
US7643676B2 (en) * | 2004-03-15 | 2010-01-05 | Microsoft Corp. | System and method for adaptive interpolation of images from patterned sensors |
US7825965B2 (en) * | 2007-09-07 | 2010-11-02 | Seiko Epson Corporation | Method and apparatus for interpolating missing colors in a color filter array |
US20110090242A1 (en) | 2009-10-20 | 2011-04-21 | Apple Inc. | System and method for demosaicing image data using weighted gradients |
US8165389B2 (en) * | 2004-03-15 | 2012-04-24 | Microsoft Corp. | Adaptive interpolation with artifact reduction of images |
US8170331B2 (en) * | 2008-04-02 | 2012-05-01 | Himax Imaging, Inc. | Apparatus and method for pattern interpolation |
US9113118B2 (en) * | 2012-05-31 | 2015-08-18 | Apple Inc. | Green non-uniformity correction |
US20170085818A1 (en) | 2015-09-17 | 2017-03-23 | Sony Semiconductor Solutions Corporation | Power reduction for image sensor with raw image scaler |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1111904A3 (en) * | 1999-12-20 | 2005-03-16 | Texas Instruments Incorporated | Digital camera system and method |
CN101411205A (en) * | 2006-03-27 | 2009-04-15 | Nxp股份有限公司 | Method and arrangement for generating a color video signal |
US8724895B2 (en) * | 2007-07-23 | 2014-05-13 | Nvidia Corporation | Techniques for reducing color artifacts in digital images |
KR101324198B1 (en) * | 2007-10-05 | 2013-11-06 | 삼성전자주식회사 | Improved solid state image sensing device, Method for arranging pixels and processing signals for the same |
CN102938843B (en) * | 2012-11-22 | 2014-12-03 | 华为技术有限公司 | Image processing method, image processing device and imaging device |
CN103595980B (en) * | 2013-10-25 | 2015-08-05 | 西安电子科技大学 | Based on the color filter array image demosaicing method of outline non-local mean value |
CN103595981B (en) * | 2013-10-25 | 2015-09-30 | 西安电子科技大学 | Based on the color filter array image demosaicing method of non-local low rank |
CN105828693B (en) * | 2013-12-20 | 2018-11-06 | 奥林巴斯株式会社 | Endoscope apparatus |
CN103905802B (en) * | 2014-04-21 | 2016-07-13 | 浙江宇视科技有限公司 | A kind of demosaicing methods based on P pattern colour filter array and device |
CN104159091B (en) * | 2014-07-30 | 2016-08-24 | 广东顺德中山大学卡内基梅隆大学国际联合研究院 | A kind of color interpolation method based on rim detection |
-
2018
- 2018-04-11 US US15/950,845 patent/US10572974B2/en active Active
-
2019
- 2019-03-25 CN CN201910227536.XA patent/CN110365961B/en active Active
- 2019-04-01 TW TW108111554A patent/TWI737979B/en active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6791609B2 (en) * | 1999-12-20 | 2004-09-14 | Texas Instruments Incorporated | Digital still camera system and method |
US7643676B2 (en) * | 2004-03-15 | 2010-01-05 | Microsoft Corp. | System and method for adaptive interpolation of images from patterned sensors |
US8165389B2 (en) * | 2004-03-15 | 2012-04-24 | Microsoft Corp. | Adaptive interpolation with artifact reduction of images |
US7825965B2 (en) * | 2007-09-07 | 2010-11-02 | Seiko Epson Corporation | Method and apparatus for interpolating missing colors in a color filter array |
US8170331B2 (en) * | 2008-04-02 | 2012-05-01 | Himax Imaging, Inc. | Apparatus and method for pattern interpolation |
US20110090242A1 (en) | 2009-10-20 | 2011-04-21 | Apple Inc. | System and method for demosaicing image data using weighted gradients |
US8638342B2 (en) * | 2009-10-20 | 2014-01-28 | Apple Inc. | System and method for demosaicing image data using weighted gradients |
US9113118B2 (en) * | 2012-05-31 | 2015-08-18 | Apple Inc. | Green non-uniformity correction |
US20170085818A1 (en) | 2015-09-17 | 2017-03-23 | Sony Semiconductor Solutions Corporation | Power reduction for image sensor with raw image scaler |
Non-Patent Citations (3)
Title |
---|
Chun-Ye Su, et al., "Effective Demosaicing using subband correlation", Apr. 17, 2009 (Year: 2009). * |
He et al. (2013) "Guided Image Filtering," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, No. 6, pp. 1397-1409. |
Taiwan Patent Application No. 108111554, English translation of Office Action dated Dec. 6, 2019, 4 pages. |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11263724B2 (en) * | 2019-10-02 | 2022-03-01 | Hanwha Techwin Co., Ltd. | Device for interpolating colors and method thereof |
Also Published As
Publication number | Publication date |
---|---|
US20190318451A1 (en) | 2019-10-17 |
TWI737979B (en) | 2021-09-01 |
TW201944773A (en) | 2019-11-16 |
CN110365961A (en) | 2019-10-22 |
CN110365961B (en) | 2021-09-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10572974B2 (en) | Image demosaicer and method | |
US20220414829A1 (en) | Systems and Methods for Synthesizing High Resolution Images Using Images Captured by an Array of Independently Controllable Imagers | |
US10547772B2 (en) | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras | |
US10136107B2 (en) | Imaging systems with visible light sensitive pixels and infrared light sensitive pixels | |
EP2642757B1 (en) | Imaging systems with clear filter pixels | |
KR101263888B1 (en) | Image processing apparatus and image processing method as well as computer program | |
US7092020B2 (en) | Resizing images captured by an electronic still camera | |
EP2359604B1 (en) | Modifying color and panchromatic channel cfa image | |
US8823808B2 (en) | Method for improved digital video image quality | |
JP4551486B2 (en) | Image generation device | |
US8248495B2 (en) | Image picking-up processing device, image picking-up device, image processing method and computer program | |
US20080158396A1 (en) | Image Signal Processor For CMOS Image Sensors | |
US20130002936A1 (en) | Image pickup apparatus, image processing apparatus, and storage medium storing image processing program | |
US20110216210A1 (en) | Providing improved high resolution image | |
US20100104209A1 (en) | Defective color and panchromatic cfa image | |
US8018500B2 (en) | Image picking-up processing device, image picking-up device, image processing method and computer program | |
US9916644B1 (en) | Ghost artifact removal system and method | |
US20180330529A1 (en) | Image processing apparatus, image processing method, and computer readable recording medium | |
KR20170074602A (en) | Apparatus for outputting image and method thereof | |
WO2014027511A1 (en) | Image processing device, image processing method, and program | |
US20200177807A1 (en) | Compound-eye imaging device, image processing method, and recording medium | |
US20180047176A1 (en) | Image processing device and method, and recording medium | |
CN113068011B (en) | Image sensor, image processing method and system | |
US20180365801A1 (en) | Method for processing signals from a matrix for taking colour images, and corresponding sensor | |
US20140118580A1 (en) | Image processing device, image processing method, and program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: OMNIVISION TECHNOLOGIES, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SWAMI, SARVESH;DUAN, YUBO;LIU, CHENGMING;SIGNING DATES FROM 20180612 TO 20180620;REEL/FRAME:047134/0550 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |