JP2009536470A - Method and apparatus for green channel gain balancing of an adaptive self-calibrating sensor - Google Patents

Abstract

A method and apparatus for adaptive green channel odd-even mismatch removal that eliminates artifacts caused by odd-even mismatch in demosaiced images. One adaptive approach determines the calibrated GR channel gain for the red row and the calibrated GB channel gain for the blue row, which are functions of only the effective pixels in the individual regions. After the calibration, in the correction process, the green pixels in the red row of the region are multiplied by the calibrated GR channel gain. On the other hand, the green pixel in the blue row is multiplied by the calibrated GB channel gain. Thus, after demosaicing, the corrected image is essentially free of artifacts caused by the even-numbered mismatch of the green channel. Alternatively, the adaptive green channel odd / even mismatch removal method replaces the central green pixel in a region with an odd number of columns and rows with a normalized weighted green pixel sum. The total of the weighted green pixels is the sum of the center green pixel weighted with the first weighting factor, the weighted green pixel value of the first hierarchy based on the second weighting factor, and the third weighting factor. It is the sum of the weighted green pixel values of the second layer based on it.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is related to provisional patent application 60 / 760,769 filed on January 19, 2006 and provisional patent application 60/759, filed January 18, 2006. , 842, both of which are hereby incorporated by reference as if set forth in full below.

  The present invention relates generally to image correction methods, and more particularly to a process for adaptively removing green channel odd-even mismatches.

  As the number of pixels of the sensor increases, the area of each pixel photodiode decreases. The signal readout circuit must handle weak signal level readout and transfer. In the case of an RGB bayer pattern sensor, the odd and even rows of green channels are usually read out through different circuits. In particular, the green channel of the Bayer pattern sensor exhibits an unbalanced response due to the metal wiring design of the photodiode, electron leakage, light incident angle, and signal output circuit. This imbalance includes both global and local variations. Even with the same circuit design, read / amplifier circuits can become mismatched due to an incomplete manufacturing process. In addition, the green channel may exhibit odd-even mismatch due to the lack of color filter array and lens coverage and mounting. Thus, the all-green channel odd-even mismatch is position dependent and non-uniform. Since the green channel odd / even mismatch results in an artifact cross hatched pattern as shown in FIG. 1, the green channel odd / even mismatch makes the image processing task difficult.

  In FIG. 1, a flat field image 10 is generated by a demosaic operation. Since the lens is covered with a diffusing lens, the flat field image is considered flat. The processed image is considered to have no texture at all. However, as can be seen in FIG. 1, the cross hatch pattern spreads over the entire image 10. Further investigation revealed that this artifact was caused by a green channel odd-even mismatch.

  Since 50% of the Bayer pixels are green, the demosaic algorithm typically determines edges based heavily on the green channel signal. A typical Bayer pixel arrangement is shown in FIG. 10B. However, when there is an odd / even mismatch in the green channel, such a mismatch is treated as an edge and the demosaic module tries to keep such an edge in either the vertical or horizontal direction. The final result is the cross hatch pattern shown in FIG. 1 after demosaic processing. This artifact is most noticeable when the image is magnified to about 300%.

The failing solution proposed an overall green channel gain balance. If the channel readout / amplifier circuit is the only source of green odd-even mismatch, this problem can be solved by applying an overall green channel gain balance. However, for the Sony ^™ 3MP sensor, using the overall green channel gain balance did not work. Further analysis revealed that the odd / even mismatch was not uniform across the image.

  Region-based channel balance calibration is performed on flat field images by dividing the 3MP sensor image into regions of 32 × 32 pixels per region. The required Gr and Gb gains for balancing the green channel are shown in FIGS. 2A and 2B. As can be readily seen from FIGS. 2A and 2B, the balance of the green channel is very uneven throughout the image. As a result, applying the overall green channel gain cannot solve the problem or eliminate the artifact cross-hatch pattern shown in FIG.

  Another possible solution uses an adaptive Bayer filter. An adaptive Bayer filter can be applied only to green pixels to smooth out odd-even mismatches. For the Sony sensor under study, the problem is that some areas show an even / even mismatch of the 13% green channel. Attempting to smooth out such large misalignments can also exacerbate the true edges of the image. As a result, the image becomes unclear.

  Furthermore, the computational cost of adaptive Bayer filters is relatively high with respect to software / firmware. In addition, this calculation adds significant delay time to the processing of the snapshot image. FIG. 3 shows an image 20 obtained after applying an adaptive Bayer filter to the flat field image of FIG. The resulting image 20 has gone through the entire processing pipeline. An appropriate amount of smoothing is applied with an adaptive Bayer filter. The resulting image 20 has some smooth crosshatch pattern artifacts, but some still remain.

  If a greater amount of smoothing is applied with the adaptive Bayer filter, the cross-hatch pattern can be completely removed, but the image texture will be blurred.

  When direct smoothing is performed on the raw image of the Bayer domain, the edges and texture are degraded. Forcing each pair of green pixels (Gr and Gb) to be equal makes the high frequency edge worse.

Summary of the Invention

  An object of the present invention is to provide an adaptive green channel odd-even mismatch removal method that eliminates artifacts caused by such mismatch.

  It is also an object of the present invention to provide an adaptive green channel odd-even mismatch removal module that eliminates artifacts caused by such mismatch.

  It is also an object of the present invention to provide program instructions executable by a processor that adaptively removes green channel odd-even mismatches to eliminate artifacts caused by such mismatches.

  It is a further object of the present invention to provide adaptive green channel odd-even mismatch removal that is easily implemented to minimize computational complexity and not reduce image processing speed.

  A further object of the present invention is an adaptive green channel odd-even mismatch that adaptively calibrates to correct the even-odd mismatch for each region to compensate for image content variance and indoor and outdoor image variance. Is to provide removal.

  It is a further object of the present invention to provide adaptive green channel odd / even mismatch removal to adaptively compensate for spatially varying green channel odd / even mismatches.

  A further object of the present invention is to use an adaptive approach to resolve green channel odd-even mismatches while preserving many edges, including high frequency edges and either vertical or horizontal edges. To provide de-matching.

  With respect to the above object, the object of the present invention is to divide the original image from the sensor into a plurality of regions and to eliminate the green channel odd-even mismatch of the original image for each region in order to eliminate artifacts of the demosaiced image. Adaptively removing the adaptive green channel odd-even mismatch removal method.

  An object of the present invention is to provide a method for adaptively removing green channel odd-even mismatches by calibrating the green (GR) channel gain of the red row and the green (GB) channel gain of the blue row for each region of the original image. Executed. After the step of calibrating, the GR channel gain is applied to the green pixels in the red row and the green pixels in the blue row calibrated for each region to remove the green channel odd-even mismatch. GB channel gain is applied for each region.

  The object of the present invention is to generate, for each region of the original image, a weighted central green pixel value based on the first weighting factor of the central green pixel and form a sum of the first hierarchy. Summing the weighted green pixel values based on the second weighting factors of the surrounding green pixels of the first hierarchy for the central green pixel and forming a second hierarchy sum for the central green pixel of the region Summing the weighted green pixel values based on the third weighting factors of the peripheral green pixels of the second hierarchy and forming a total of the weighted green pixels, This is performed by a method that adaptively removes the green channel odd-even mismatch by summing the sum of the second hierarchy and the sum of the second hierarchy. After generating the weighted green pixel total, the weighted green pixel total is normalized. To remove the green channel odd mismatch, replace the normalized weighted green pixel sum with the central green pixel value of the region.

  The object of the invention is carried out by a method for removing the green channel odd-even mismatch from the original Bayer image.

  The object of the invention is carried out by a method of removing the green channel odd-even mismatch before demosaicing by removing edge pixels for each region of the image when calibrating the gain.

  The object of the present invention is performed by an adaptive green channel odd-even mismatch removal method that filters out bad pixels and edge pixels in each region to form a set of effective pixel pairs when calibrating.

  The purpose of the present invention is to calculate the average number of effective green pixels in the red row, calculate the average number of effective green pixels in the red row, and calculate the average number of effective green pixels in the blue row. Performed by the adaptive green channel odd-even mismatch removal method to calculate.

  An object of the present invention is performed by an adaptive green channel odd-even mismatch removal method that reduces the noise variance by filtering the GR channel gain and the GB channel gain with the GR channel gain and GB channel gain of the preceding image when calibrating. Is done. The applied GR channel gain and the applied GB channel gain are the filtered GR channel gain and the filtered GB channel gain, respectively.

  An object of the present invention is to perform demosaic processing by multiplying the green pixel in the red row of each region by the GR channel gain, multiplying the green pixel in the blue row by the GB channel gain, and correcting the odd-even mismatch. Performed by an adaptive green channel odd-even mismatch removal method that includes eliminating the artifacts later.

  The object of the present invention is achieved by program code executed by a processing device comprising instructions that can be implemented to calibrate the GR channel gain and GB channel gain for each region of the image at runtime. The instructions can also be implemented to apply the calibrated GR channel gain and the GB channel gain for each region for each region to adaptively remove the green channel odd-even mismatch from the image.

  The object of the invention is performed by an adaptive green channel odd-even mismatch removal module comprising means for calibrating the GR channel gain and GB channel gain for each region of the image. The module also removes the green channel odd-even mismatch from the green pixel in the red row and the GR channel gain from the green row in the blue row, calibrated for each region. Means for applying to each region.

  The object of the present invention is performed by an adaptive green channel odd-even mismatch removal module comprising means for generating a weighted central green pixel value based on a first weighting factor of the central green pixel. The module further includes means for summing weighted green pixel values based on a second weighting factor of peripheral green pixels of the first hierarchy for the central green pixel of the region to form a first hierarchy sum. And a means for summing weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a sum of the second hierarchy ing. The module also includes means for summing the weighted median green pixel value, the sum of the first tier, and the sum of the second tier to form a sum of weighted green pixels; Means for normalizing the pixel total and means for replacing the central green pixel value with the normalized weighted green pixel total to remove the green channel odd-even mismatch.

  The object of the present invention is performed by program code executed by a processing device comprising instructions executable at run time to generate a weighted central green pixel value based on a first weighting factor of the central green pixel. The program code further sums weighted green pixel values based on a second weighting factor of the surrounding green pixels of the first hierarchy for the central green pixel of the region to form a sum of the first hierarchy. To sum the weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a sum of the second hierarchy. is there. The program code further sums the weighted central green pixel value, the sum of the first hierarchy, and the sum of the second hierarchy to form a sum of weighted green pixels, It can be implemented to normalize the sum and replace the pixel value of the central green pixel with the normalized weighted green pixel sum to remove the green channel odd-even mismatch.

  The foregoing summary, as well as the following detailed description of the preferred embodiments of the present invention, will be better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there are shown in the drawings embodiments which are presently preferred. However, it should be understood that the invention is not limited to the exact configuration as shown.

Detailed description

  While the invention is susceptible to many different forms of embodiments, the present specification and the accompanying drawings disclose only a few forms of use of the invention. The present invention is not intended to be limited to such described embodiments, but the scope of the present invention is set forth in the appended claims.

  A preferred embodiment of the green channel odd-even mismatch removal method according to the present invention will be described below with respect to a specific application for snapshot images. However, it will be recognized by those skilled in the art that the present invention is well adapted to other types of images that require green channel correction. Referring now to the drawings in detail, like reference numerals designate like elements throughout, and according to the present invention, in FIGS. 6A, 6B, and 7, the self-calibration and correction processes are generally referred to as reference numerals 100 and 120.

  However, to enable an understanding of the present invention, odd-even mismatch is a mismatch between a green pixel in a red row having red and green pixels and a green pixel in a blue row having blue and green pixels. It points to something. For the reasons described above, the green pixel response is different even if the scene is a smooth flat field image. Mismatch is usually characterized as a ratio of Gr / Gb. Gr means a green pixel in the red row, and Gb means a green pixel in the blue row. Ideally this ratio should be 1.0.

  As shown in FIGS. 2A and 2B, the green channel odd-even mismatch is very non-uniform across the image. Inconsistent patterns cannot be easily modeled. It is clear that the green channel mismatch varies from sensor to sensor. In addition, misalignment may occur in different modules of the same sensor model. By comparing the green channel mismatch of the indoor image of FIG. 4 captured by the same sensor with the green channel mismatch of the outdoor image of FIG. 5, the green channel mismatch is also dependent on the image content. Is easily understood.

  In the first exemplary embodiment, the green channel odd / even mismatch removal method includes a per-region adaptive green channel gain self-calibration process 100 described with reference to FIGS. 6A and 6B, and FIG. And a correction process 120 as described. In general, the green channel odd-even mismatch removal method compensates for green channel mismatch by a region-by-region adaptive green channel gain self-calibration process 100 and then in a correction process 120 the sensor module 210 (FIG. 11) of the snapshot imaging device 200. Apply the green channel gain for each region to all final snapshot images output from.

  Referring now to FIGS. 6A, 6B, 8, 9, 10A, and 10B, the per-region adaptive green channel gain self-calibration process 100 begins at step S102, where a sensor module 210 (FIG. 11) is divided into X × Y areas (FIG. 10A) made up of M × M pixels (FIG. 10B) (M is a multiple of 2). In the exemplary embodiment, image 150 is divided into 4 × 3 (X = 4, Y = 3) regions, and each region is divided into 8 × 8 pixels. In this example, image 150 has 12 regions. The hatched area labeled R1 is divided into M × M pixels as shown in FIG. 10B. Image 150 is an original Bayer image and has not undergone demosaic processing 230. Accordingly, FIG. 10B shows the original Bayer notation for region R1.

  For illustrative purposes only, the first row in FIG. 10B is a blue row with alternating green and blue pixels. The green pixels in the blue row are marked GB. The second row immediately after the first row is a red row in which green and red pixels are alternated. The green pixel in the red row is denoted as GR. In this exemplary embodiment, the first column of FIG. 10B includes alternating green and red pixels.

  Referring again to the flowchart of FIG. 6A, step S102 is followed by step S103, where the region is set to 1. Step S103 is followed by step S104 where the ratio of the adjacent green pixels GR and GB on the red and blue rows is calculated for each region. In this exemplary embodiment, two GB, GR pixel pairs in each region are selected. Step S104 is followed by step S106, where defective pixels and edge pixels are filtered out.

  A defective pixel can be detected based on adjacent pixels of the same color. For example, if a comparison between a current pixel of the same color and an adjacent pixel exceeds a certain threshold, the current pixel may be determined to be defective. On the other hand, edge pixel detection can use an A × A size window and 2-D convolution. The output of the 2-D convolution is compared to a threshold value. If the output is greater than the threshold, this output is an edge. Otherwise, the output is not an edge. There are many bad pixel detection and edge pixel detection algorithms. Therefore, the above description regarding defective pixel detection and edge pixel detection is for illustrative purposes only.

  Step S106 is followed by step S108, where the average of GB and GR pixel values (denoted Gr_avg and Gb_avg) of non-defective pixels in the region is calculated. Step S108 is followed by step S110 in FIG. 6B. Gr_avg and Gb_avg are calculated based on the equations (1) and (2) described below, respectively.

  Next, a process for calculating pixel values of Gr_avg and Gb_avg will be described with reference to FIG. Exemplary code for calculating Gr_avg and Gb_avg is also provided in the Appendix following this section. The process of step S108 begins with step S140, where the number of valid pixel pairs (#VP) in the region under consideration is counted or measured. An effective pair is a pair of non-defective pixels remaining after the filtering step S106. Step S140 is followed by step S142, where i is set to zero. Step S142 is followed by step S144, which is a determination step, to determine whether i is less than the number of valid pairs in the region (#VP). If the determination is “Yes”, in step S146, the sum of GR pixel values (denoted as Gr_sum) of the green pixels GR that are not defective in the red row is calculated. Step S146 is followed by step S148, where the sum of GB pixel values (denoted as Gb_sum) of green pixels GB that are not defective in the blue row is calculated. Step S148 is followed by step S150, where i is incremented by one.

The process returns from step S150 to step S144. Steps S144, S146, 148, and 150 are loops and are repeated until i is less than the number of valid pairs. Accordingly, in step S146, the sum is incremented by the green pixel value of each corresponding non-defective GR pixel in the region. In step S148, the sum is incremented by the green pixel value of each corresponding non-defective GB pixel. When all non-defective GR and GB pixels are individually summed, step S144 is followed by step S152, where Gr_avg (the average pixel value of the non-defective green pixels in the region's red row) is: Is calculated based on equation (1) defined as:
Gr_avg = Gr_sum / number of effective pairs per region (1)
Step S152 is followed by step S154, where Gb_avg (average pixel value of non-defective green pixels in the region's blue row) is calculated based on equation (2) defined as follows:
Gb_avg = Gb_sum / number of effective pairs per region (2)
Next, referring to FIG. 6B and FIG. 9, in step S110, the average gains Gr_gain and Gb_gain of the green pixel values of the two channels are calculated. This means that the weak green channel is applied with a digital gain greater than 1, whereas the strong green channel is applied with a gain less than 1.0. It should be noted that the application of a gain less than 1.0 does not result in color misregistration because the purpose of this process is to balance the green pixels of the two channels rather than between the channels of different colors. . Therefore, the channel gain of each (GB, GR) pair can be derived from equation (3) in step S160, equation (4) in step S162, and equation (5) in step S164 defined as follows. it can:
avg = (Gr_avg + Gb_avg) / 2 Formula (3)
Gr_gain = avg / GR_avg Formula (4)
Gb_gain = avg / GB_avg Formula (5)
Here, avg is calculated by the valid (non-defective) green pixel GR of the red row calculated by the expression (1) and the expression (2) for the non-defective or effective pixel pair in the region. It is an average value calculated from the average of valid (not defective) green pixels GR in the blue row.

  Step S110 generates channel gains Gr_gain and Gb_gain, which are passed to step S112. In step S112, Gr_gain and Gb_gain of the current image 150 can be low-pass filtered with the channel gains (Gr_gain and Gb_gain) of the previous image to reduce noise variance. The filtered Gr_gain and Gb_gain of the current image are denoted as Gr_gain 'and Gb_gain'.

  The box representing step S112 has two outputs labeled Gr_gain 'and Gb_gain', which are stored for use in the calculation in the correction process. Step S112 is followed by step S114, where the region is incremented.

  The process of FIGS. 6A and 6B is repeated to calculate Gr_gain and Gb_gain or filtered Gr_gain 'and Gb_gain' for each region. Accordingly, a determination step S116 subsequent to step S114 determines whether there is more area. If “yes”, step S116 returns to step S104 in FIG. 6A to self-calibrate the next region. On the other hand, if there are no more regions, the self-calibration process 100 for calibrating the two channel gains ends.

  Next, referring to FIG. 7, a correction process 120 using Gr_gain 'and Gb_gain' for each region will be described next. Process 120 begins at step S122, where the region is set to one. Step S122 is followed by step S124, in which the pixel value of each green pixel GB in the blue row is multiplied by Gb_gain '. Step S124 is followed by step S126, where the pixel value of each green pixel GR in the red row is multiplied by Gr_gain '. Step S126 is followed by step S128, where the region is incremented. Step S128 is followed by step S130, where it is determined whether there are more areas. If no, process 120 ends. On the other hand, if “yes”, step S130 returns to step S124, where correction is applied to the next region.

  Using an area size of 32 × 32 pixels, self-calibration and correction processes 100 and 120 are performed on the test image, and the demosaic output of this test image no longer shows a cross-hatch pattern. The 32x32 region size is small enough so that the corrected image does not show any recognizable region boundary artifacts. However, if the region size is too large, such as 256 × 256, blockiness artifacts may be recognizable. FIG. 12 shows the corrected image 10 ′ after demosaic processing. Compared to FIG. 1, the image 10 'of FIG. 12 is greatly improved.

  Referring now to FIG. 11, the snapshot imaging device 200 includes a lens 202 and a sensor module 210 having an image processing unit 212 and a color filtering unit 214. The color filtering unit 214 is a Bayer color filter array that generates an original Bayer image. The original Bayer image is corrected by the green channel odd / even mismatch removal module 220. The sensor values for all three primary colors red, green, and blue at the position of one pixel are interpolated from adjacent pixels. This interpolation process is performed by demosaic processing unit 230. There are a number of demosaicing methods such as pixel replication, bilinear interpolation, and median interpolation. The output of the adaptive green channel odd / even mismatch removal module 220 provides the corrected original Bayer image to the demosaic processing unit 230.

  The green channel odd / even mismatch removal method performed by the green channel odd / even mismatch removal module 220 can be implemented using firmware, software, and hardware. In the case of a firmware implementation, the digital signal process (DSP) 222 reads one region at a time and the Advanced RISC Machine (ARM) 226 supplies Gr_gain 'and Gb_gain' to the DSP 222. The DSP 222 performs multiplication on the green pixel. This process is in place, ie input and output pixels share the same buffer 228. In other words, image pixels can be directly replaced with new values without having to allocate another buffer for processing. Program instructions 224 may be implemented to perform a per-region adaptive green channel gain self-calibration process 100 and a correction process 120 at run time.

  Although DSP 222 and ARM 226 are shown as part of green channel odd-even mismatch removal module 220, snapshot imaging device 200 performs the functions of image processing unit 212, color filtering unit 214, and demosaic processing unit 230. In some cases, DSP 222 and ARM226 are originally included. Thus, a processing device that executes the program instructions 224 may already exist.

  On the other hand, in the case of software implementation, although not limited, program instructions written in a programming language such as C code, for example, are executed on the ARM 226, and an original image such as an original Bayer image is divided into a plurality of areas. Divide and use the Gr_gain ′ and Gb_gain ′ for that region to multiply the green pixels. ARM 226 is generally existing and can be used to execute program instructions 224. Accordingly, ARM 226 performs both self-calibration and correction processes 100 and 120. In a software implementation, this process is in the same place, so the image pixel can be directly replaced with a new value without having to allocate another buffer to the process.

  For hardware implementations, the self-calibration and correction processes 100 and 120 can be implemented in hardware as long as the size of the lookup table is not an issue.

  The green channel odd-even mismatch creates a whole new problem for image processor video front end (VFE) processing. Due to the nature of the non-uniform mismatch distribution, the overall channel gain has not solved this problem. The area-by-area calibration and correction processes 100 and 120 provide an efficient and quick way to solve the problem with non-uniform mismatch distribution.

  FIG. 13A shows a normal RGB Bayer pattern with green pixels indexed, as described in detail below. An alternative adaptive green channel odd / even mismatch removal method 300 that adaptively balances the green channel to remove the odd / even mismatch is described with reference to the flowcharts of FIGS. 14A-14E and the images of FIGS. 13A and 13B. In this embodiment, program instructions 224 (FIG. 11) are modified to include instructions that can be implemented to perform adaptive green channel odd-even mismatch removal method 300 described herein.

  The adaptive green channel odd / even mismatch removal method 300 begins at step S302, where an original image such as the original Bayer image best seen in FIG. 13A is acquired. To enable an understanding of the method 300, FIG. 13A indexes the green pixels. Step S302 is followed by step S304, where an N × N pixel region is created from the image. In the exemplary embodiment, N is odd and equal to 5. Step S304 is followed by step S306, where the central green pixel (CGP) is selected. In this embodiment, the CGP is labeled G22 in FIG. 13A. Step S306 is followed by step S308, in which a first weighting factor is assigned to CGP G22. In the exemplary embodiment, the first weighting factor is 8. Step S308 is followed by step S310, in which a second weighting factor is assigned to all green pixels in the region N × N at a distance of one pixel from CGP. In the exemplary embodiment, there are four neighboring green pixels in opposing green channels with a distance to the CGP of 1. These adjacent pixels at a distance of one pixel are referred to herein as “GP1” and define the first hierarchy as a whole. GP1 in the first hierarchy includes green pixels indexed with G11, G13, G31, and G33. In the exemplary embodiment, the second weighting factor is 4.

Step S310 is followed by step S312 where a third weighting factor is assigned to the green pixels at a distance of 2 pixels from CGP G22. These neighboring pixels at a distance of two pixels are referred to herein as “GP2” and define the second hierarchy as a whole. In the exemplary embodiment, there are eight GP2s indexed G00, G02, G04, G20, G24, G40, G42, and G44, each taking a weighting factor of one. Thus, since the total weight factor is 32, normalization is facilitated by downshifting by 5 bits or dividing by ²⁵ , and pixel values are represented in 8, 10, or 12 bits using a binary representation. Can be done. The normalization will be described later.

  Step S312 is followed by step S314, where F_max and F_min are set and calculated. F_max is an upper bound threshold of the maximum green mismatch ratio. F_min is a lower bound threshold of the maximum green mismatch ratio. Step S314 is followed by step 316, where an offset is calculated, which is an intensity threshold for performing smoothing.

One important factor for green channel mismatch is due to the cross talk of surrounding red pixels. That is, the Gr / Gb channel mismatch depends on the red channel value. Therefore, the offset is aligned with the surrounding red pixels to properly remove the spatially varying green channel odd-even mismatch. In the exemplary embodiment, the peripheral red pixels are marked with an index such as R10, R12, R14, R30, R32, and R34 (FIG. 13B). In the exemplary embodiment, there are six peripheral red pixels. The offset parameter is defined by equation (6) as follows:
Offset = k ^* Average (R10, R12, R14, R30, R32, R34) Formula (6)
Here, k is a parameter for adjusting the magnitude of crosstalk correction, and R10, R12, R14, R30, R32, and R34 indicate pixel values of red pixels with corresponding indexes.

  Furthermore, an upper limit is set for the offset by a constant described as an upper limit (Offset Cap) of the offset to avoid an offset threshold that is too large. Accordingly, if the offset is greater than the upper limit of the offset, step S316 is followed by step S317, where the offset is set to the upper limit of the offset or other constant. Step S318 is followed by step S319. However, if the offset is not greater than the upper limit of the offset, step S317 is followed by step S319.

In step S319, for CGP G22, variables P_max, P_min, and G_sum are calculated according to equations (7), (8), and (9a) defined as follows:
P_max = max (F_max ^* G22, G22 + offset) Expression (7)
P_min = min (F_min ^* G22, G22-offset) Formula (8)
G_sum = G22 << 3 Formula (9a)
Here, G22 indicates the green pixel value of the central pixel G22, P_max is the maximum value of the green pixel, and P_min is the minimum value of the green pixel. Furthermore, the sign “<<” indicates a 3-bit upshift. In other words, G_sum is equal to the pixel value of the green central pixel G22 multiplied by the weighting factor 8 (2 ³ ). Thus, equation 9 (a) can also be written as equation (9b):
G_sum = pixel value of G22 ^* weight coefficient of G22 Formula (9b)
As can be readily seen, G_sum in equation (9a) or equation (9b) generates a weighted central green pixel value based on the first weighting factor of central green pixel (CGP) G22.

Step S319 is followed by step S320, where the first green pixel at distance 1 from the central green pixel (CGP) G22 is acquired in the first hierarchy. Step S320 is followed by step S322, in which it is determined whether the pixel value of the green pixel GP1 is greater than or equal to P_min and less than or equal to P_max (see step S322). In other words, step S322 determines whether the green pixel under evaluation is within range. If the determination in step S322 is yes, step S322 is followed by step S324, where the value G_sum is first shifted up by 2 bits to generate a weighted pixel value for the first hierarchy. It is incremented by the green pixel value of the green pixel GP1 (eg, the indexed pixel G11). More specifically, G_sum is increased by equation (10a) or equation (10b):
G_sum + = GP1 << 2 Equation (10a), or G_sum = G_sum + (GP1 ^* GP1 weighting factor) Equation (10b)
(Weighting factor of the first layer = 4)
Here, GP1 is the pixel value of the green pixel indexed in the first hierarchy surrounding the central green pixel. In the exemplary embodiment, the first tier GP1 includes G11, G13, G31, and G33. As a result, when the green pixel value GP1 under evaluation is within the range defined by P_min and P_max, G_sum in the expression (10a) or the expression (10b) is the weight coefficient (4) of the first layer, that is, ( 2 ² ) is multiplied by the pixel value of each GP1 (G11, G13, G31, and G33) multiplied.

On the other hand, when the determination in step S322 is “No”, the pixel value of the green pixel GP1 is smaller than P_min and / or larger than P_max. In other words, the green pixel value GP1 under evaluation is out of range. Accordingly, step S322 is followed by step S326, where the value G_sum is increased by the pixel value of the central green pixel value denoted as G22 upshifted by 2 bits. More specifically, G_sum is increased by equation (11a) or equation (11b):
G_sum + = G22 << 2 Equation (11a), or G_sum = G_sum + (G22 ^* GP1 weighting factor) Equation (11b)
(Weighting factor of the first layer = 4)
Here, G22 indicates the pixel value of the central green pixel G22 to which the index is attached. The same calculation and weighting are performed on G13, G31, and G33 of the first hierarchy. When the green pixel value GP1 under evaluation is outside the range defined by P_min and P_max, the expression of G_sum in Expression (11a) or Expression (11b) is used. As can be easily understood from Expression (11a) or Expression (11b), the green pixel outside the range of the first hierarchy is replaced with the pixel value of the central green pixel G22.

  Steps S324 and S326 are followed by step S328 to determine if there is another GP1. If there is, step S328 returns to step S320, and steps S322, S324, and S326 are reevaluated based on the pixel value of the next GP1.

  At the end of the loop defined by steps S320, S322, S324, S326, and S328, G_sum in equations (10a), (10b), (11a), and / or (11b) is the weight of the first hierarchy The green pixel values are added together, which usually forms the sum of the first hierarchy. In the proposed program code provided, the G_sum formula further adds the sum of the first hierarchy to the G_sum calculated before the weighted central green pixel value.

If there is no more GP1 in the first layer, step S328 is followed by step S330, where a first green pixel at a distance of 2 from the central green pixel (CGP) G22 is acquired in the second layer. Step S330 is followed by step S332, in which it is determined whether the pixel value of the green pixel GP2 is greater than or equal to P_min and less than or equal to P_max (see step S332), that is, within the range. If the determination in step S332 is “yes”, step S332 is followed by step S334, where the value G_sum is increased by the green pixel value of the first green pixel GP2 (eg, indexed pixel G00). The More specifically, G_sum is increased by equation (12a) or equation (12b):
G_sum + = GP2 Equation (12a), or G_sum = G_sum + GP2 ^* GP2 weight coefficient Equation (12b)
(Second layer weighting factor = 1)
Here, GP2 is the pixel value of the green pixel indexed in the second hierarchy. In the exemplary embodiment, the second tier GP2 includes G00, G02, G04, G20, G24, G42, and G44. As a result, when the green pixel value GP2 being evaluated is within the range defined by P_min and P_max, the weighting factor is 1, so G_sum is increased by the pixel value of GP2.

On the other hand, if the determination in step S332 is that the pixel value of the green pixel GP2 is less than P_min and / or greater than P_max, ie out of range, step S332 is followed by step S336, where G_sum is , G22 is incremented by the pixel value of the central green pixel value. More specifically, G_sum is increased by equation (13a) or equation (13):
G_sum + = G22 Expression (13a) or G_sum = G_sum + (G22 ^* GP2 weighting coefficient) Expression (13b)
(Weighting factor of the second layer = 1)
Here, G22 indicates the pixel value of the central green pixel G22 to which the index is attached. The same calculation and weighting is performed on GP2 labeled G00, G02, G04, G20, G24, G42, and G44 in the second hierarchy. When the green pixel value GP2 under evaluation is outside the range defined by P_min and P_max, the expression of G_sum of Expression (13a) or Expression (13b) is used. Therefore, the green pixel value of the green pixel outside the range of the second hierarchy is replaced with the pixel value of the central green pixel G22.

  As can be readily seen, equations (12a), (12b), (13a), and / or (13b) sum the weighted pixel values of the second hierarchy.

  Steps S334 and S336 are followed by step S338 to determine if there is another GP2. If there is, step S338 returns to step S330, where steps S332, S334, S336 are evaluated again based on the next GP2 pixel value. At the end of the loop defined in steps S330, S332, S334, S336, and S338, G_sum adds together the weighted green pixel values of the second hierarchy to form the sum of the second hierarchy. In the proposed program code provided, G_sum further adds together the sum of the second tier, the sum of the first tier, and the weighted central green pixel value to form the sum of the weighted green pixels. To do.

Referring now to FIG. 14E, after the green pixels in the N × N (N = 5) region have been processed, G_sum (total weighted green pixels) is normalized in step S340. G_sum (total weighted green pixels) is normalized by downshifting G_sum by 5 bits (2 ⁵ = total weighting factor 32 in binary representation). The sum of the weighting factors is the first weighting factor, the product of the second weighting factor and the number of green pixels in the first layer, and the product of the third weighting factor and the number of green pixels in the second layer. Is equal to the sum of In the exemplary embodiment, the first weighting factor is 8. Multiplying the second weighting factor by the number of green pixels in the first hierarchy is equal to 16. Multiplying the third weighting factor by the number of green pixels in the second hierarchy is equal to 8.

In step S342, the pixel value of the central green pixel G22 is replaced with the normalized G_sum calculated in step S340. More specifically, the new pixel value for the central green pixel (G22) is defined by equation (14):
New G22 = G_sum >> 5 Equation (14)
Here, G22 indicates the pixel value of the central green pixel G22, the sign “>>” indicates a downshift, and G_sum in equation (14) is the total of the weighted green pixels. A 5-bit downshift is the same as a division by 2 ⁵ or 32.

  In the adaptive green channel odd / even mismatch removal method 300, for green pixels close to the central green pixel G22, they are used to perform low pass filtering. If the green pixels are outside the defined proximity range, they are skipped (replaced with the pixel value of the central green pixel). In the exemplary embodiment, the defined proximity is a pixel that is a distance of one pixel or a distance of two pixels. Therefore, the normalization coefficient is a constant. Thus, division can be replaced with a simple downshift.

  Step 342 is followed by step S344 where the method 300 is repeated on the next region of the image until there are no more regions. If the determination is “No” in step S344, then there is no more area and the method 300 ends. On the other hand, if the determination is “yes”, step S344 loops back to step S304 of FIG. 14A to repeat the process for the next region in the entire frame image.

  Instead, step S344 can be moved to a position before normalization step S340, which will normalize all the central pixels simultaneously.

  Note that the values of P_max and P_min utilize both ratio and offset parameters. For small signals, this ratio cannot produce a useful range. With the help of offset, it is possible to provide a sufficient range of [P_min, P_max]. As a side benefit, noise is also reduced. For large signals, this ratio dominates and matches the worst calibrated Gr / Gb ratio mismatch estimated from the sharp gray signal during the calibration process.

  When using the adaptive green channel odd-even mismatch removal method 300, only the worst mismatch of the sensor (such as sensor module 210) should be known as prior information. There are no parameters to change or adjust at runtime. The worst mismatch is known during the sensor calibration procedure.

Experimental Results FIGS. 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19A, 19B, 20A, and 20B are performed with and without demosaic processing and adaptive green channel odd / even mismatch removal method 300. Comparison of images is shown. The adaptive green channel odd / even mismatch removal method 300 (hereinafter referred to as “adaptive green channel balancing method”) recognizes high frequency components (clear lines and curves) very well and removes the cross-hatch pattern. It is easy to understand. New artifacts are not captured in the image by the adaptive green channel balancing method. 15A, 15B, 16A, 16B, 17A, 17B, 18A, 18B, 19A, 19B, 20A, and 20B images processed with offsets limited to F_max = 1.13, F_min = 0.87, and 5 Has been. FIGS. 15A and 15B show flat field images with and without the adaptive green channel balancing method (enlarged to 300% with demosaic), and FIGS. 16A and 16B show adaptive green channel balancing. FIGS. 17A and 17B show resolution chart images (center circle) with and without the method (enlarged to 300% and with demosaic processing), and FIGS. 17A and 17B show the case with the adaptive green channel balancing method A resolution chart image (vertical line) when not performed is shown (enlarged to 300%, with demosaic processing), and FIGS. 18A and 18B are resolution charts with and without the adaptive green channel balancing method. Images (horizontal lines) are shown (enlarged to 300% with demosaic processing), and FIGS. 19A and 19B show adaptive channels Shown are MachBeth chart images with and without the equilibration algorithm (enlarged to 300% with demosaic processing), and FIGS. 20A and 20B show the case with and without adaptive channel balancing. A Macbeth chart image is shown when there is not (with demosaic processing).

  As can easily be seen from FIGS. 15B, 16B, 17B, 18B, 19B, and 20B, the adaptive green channel balancing method 300 keeps the edges of the image very good, while the even-odd mismatch artifact (cross-hatch pattern). Can be removed. The adaptive green channel balancing method 300 does not require run-time coordination or management. Only the worst-case even-number mismatch off-line calibration is required to determine the parameters (F_max and F_min) used in the adaptive green channel balancing method 300. The calibration procedure provides green channel mismatch gain and gain dispersion. Based on this, the parameters (F_max and F_min) can be derived.

  The adaptive green channel balancing method 300 is suitable for implementation in hardware, firmware, or software.

  The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the exact form disclosed, and modifications and variations are possible in light of the above teachings or may be obtained from practice of the invention. The embodiments are selected in order to explain the principles of the invention and its practical application and to enable those skilled in the art to utilize the invention in various embodiments and modifications suitable for the specific use contemplated. Has been explained. It is intended that the scope of the invention be defined by the claims of the invention and their equivalents.

Appendix
The values of Gr_avg and Gb_avg for non-defective green pixels in the region are calculated according to the following procedure:

The channel gain of each (GB, GR) pair can be derived using pseudo code;
avg = (Gr_avg + Gb_avg) / 2;
Gr_gain = avg / GR_avg;
Gb_gain = avg / GB_avg
The following codes may be used for an alternative adaptive green channel odd / even mismatch removal method 300.

The following exemplary code may be used in the adaptive green channel odd / even mismatch removal method 300 instead of summing the parameters G_sum based on green pixels at a distance 1 from the first level green center pixel.

The same calculation and weighting is performed on G13, G31 and G33.

The following exemplary code may be used for the adaptive green channel odd-even mismatch removal method 300 instead of summing the parameters G_sum based on green pixels at a distance 2 from the second level green center pixel.

The same calculation and weighting is performed on G02, G04, G20, G24, G40, G42, and G44.

The figure which shows the flat field image (enlarged to 300%) after demosaic operation | movement. Plot of Gb gain distribution of green channel odd-even mismatch distribution where each point represents one region (32 × 32 pixels). Plot of Gr gain distribution of green channel odd-even mismatch distribution with each point representing one region (32 × 32) pixels. FIG. 5 shows a modified flat field image after applying an adaptive Bayer filter to deal with green channel odd-even mismatches with moderate smoothing. The figure which shows the plot (Gr / Gb) of the green channel mismatch of the indoor image in which each point represents one area | region (32x32 pixel). The figure which shows the plot (Gr / Gb) of the green channel mismatch of the outdoor image in which each point represents one area | region (32x32 pixel). FIG. 6 is a flow diagram of an adaptive green channel gain self-calibration process for each region of the green channel odd-even mismatch removal method. FIG. 6 is a flow diagram of an adaptive green channel gain self-calibration process for each region of the green channel odd-even mismatch removal method. The flowchart of the correction process of the green channel odd / even mismatch removal method. The flowchart which calculates the average GB and GR value of an effective pixel pair. The flowchart which calculates the average gain for every GB and GR pair. The figure which shows the divided | segmented original Bayer image which has 4x3 area | regions, and one of them is a hatched area | region. The figure which shows the oblique line area | region of FIG. 10A divided | segmented into 8x8 pixel. The block diagram of the snapshot imaging device incorporating a green channel odd-even mismatch removal module. The figure which shows the flat field image (enlarged to 300%) after the gain correction | amendment for every area | region and demosaic whose area size is 32x32. The figure which shows the Bayer pattern which attached | subjected the index to the green pixel. The figure which shows the Bayer pattern which attached | subjected the index to the green pixel and attached the index to the red pixel. 6 is a flowchart of another adaptive green channel odd / even mismatch removal method for adaptive channel balancing. 6 is a flowchart of another adaptive green channel odd / even mismatch removal method for adaptive channel balancing. 6 is a flowchart of another adaptive green channel odd / even mismatch removal method for adaptive channel balancing. 6 is a flowchart of another adaptive green channel odd / even mismatch removal method for adaptive channel balancing. 6 is a flow diagram of an alternative adaptive green channel odd / even mismatch removal method for adaptive channel balancing. The figure which shows the flat field image when not performing adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the flat field image at the time of carrying out adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (center circle) when not performing adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (center circle) at the time of carrying out adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (vertical line) when not performing adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (vertical line) at the time of carrying out adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (horizontal line) when not performing adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows the resolution chart image (horizontal line) at the time of carrying out adaptive channel balancing (it expands to 300% and there is a demosaic process). The figure which shows the Macbeth chart image when not performing adaptive channel balancing (enlarged to 300% and with demosaic processing). The figure which shows a Macbeth chart image in case there exists an adaptive channel balancing algorithm (expanded to 300% and with demosaic processing). The figure which shows the Macbeth chart image which performed demosaic processing, without performing adaptive channel balancing. The figure which shows the Macbeth chart image which performed adaptive channel balancing and performed the demosaic process.

Claims (50)

Calibrating the green (GR) channel gain of the red row and the green (GB) channel gain of the blue row for each region of the image;
Apply the calibrated GR channel gain for each region to the green pixels in the red row and the GB channel gain to the green pixels in the blue row for each region to remove green channel odd-even mismatch. And an adaptive green channel odd-even mismatch removal method.   The method of claim 1, wherein the image is an original Bayer image.   The method of claim 1, wherein the calibrating includes filtering out defective and edge pixels of the individual regions to form a set of effective pixel pairs.   The step of calibrating includes counting the number of effective pixel pairs in the region, calculating an average number of the effective green pixels in the red row, and calculating an average number of the effective green pixels in the blue row. The method of claim 3 comprising the step of calculating.   The method of claim 4, wherein the GR channel gain and the GB channel gain are a function of the calculated average number of the effective green pixels in the red row and the blue row.   The calibrating step includes filtering the GR channel gain and the GB channel gain with a GR channel gain and a GB channel gain of a previous image, and the applied GR channel gain and the application of the applying step. The method of claim 1, wherein the filtered GB channel gain is the filtered GR channel gain and the filtered GB channel gain.   The step of applying comprises: multiplying the green pixels in red rows of the individual regions by the GR channel gain; and multiplying the green pixels in blue rows by the GB channel gain. The method described in 1. Program code executed by a processing device,
Instructions executable to calibrate the green (GR) channel gain of the red row and the green (GB) channel gain of the blue row for each region of the image;
To adaptively remove green channel odd-even mismatch from the image, the GR channel gain calibrated for each region is the green pixel in the red row and the GB channel gain is the green pixel in the blue row. Program code comprising instructions that can be implemented to apply to each region.   9. The instructions executable to calibrate include instructions executable to filter out defective pixels and edge pixels of the individual regions to form a set of valid pixel pairs. The code described in.   The instructions executable to calibrate count the number of effective pixel pairs in the region, calculate the average number of effective green pixels in the red row, and calculate the average number of effective green pixels in the blue row. The code of claim 9, comprising instructions executable to calculate.   The code of claim 10, wherein the GR channel gain and the GB channel gain are a function of the calculated average number of the effective green pixels in the red row and the blue row.   The instructions executable to calibrate include instructions executable to filter the GR channel gain and the GB channel gain with a GR channel gain and a GB channel gain of a previous image, and the GR channel gain and The code of claim 8, wherein the instructions executable to apply the GB channel gain apply the filtered GR channel gain and the filtered GB channel gain.   The instructions executable to apply are such that for each individual region, the green pixels in the red row are multiplied by the GR channel gain, and the green pixels in the blue row are multiplied by the GB channel gain. The code of claim 8, comprising executable instructions. Means for calibrating the red row green (GR) channel gain and the blue row green (GB) channel gain for each region of the image;
Means for applying the GR channel gain calibrated for each region to the green pixels in the red row and the GB channel gain to the green pixels in the blue row for each region to eliminate green channel odd-even mismatch And an adaptive green channel odd-even mismatch removal module.   The module of claim 14, wherein the image is an original Bayer image.   15. The module of claim 14, wherein the means for calibrating includes means for filtering out defective pixels and edge pixels of the individual regions to form a set of effective pixel pairs.   The means for calibrating calculates means for counting the number of effective pixel pairs in the region, means for calculating the average number of effective green pixels in the red row, and calculating the average number of effective green pixels in the blue row. 17. A module according to claim 16, comprising means.   The module of claim 17, wherein the GR channel gain and the GB channel gain are a function of the calculated average number of the effective green pixels of the red row and the blue row.   The means for calibrating includes means for filtering the GR channel gain and the GB channel gain with a GR channel gain and a GB channel gain of a preceding image, and the applied GR channel gain and the application of the applying means The module of claim 17, wherein the filtered GB channel gain is the filtered GR channel gain and the filtered GB channel gain.   The means for applying comprises: means for multiplying the green pixels in the red row of the individual regions by the GR channel gain; and means for multiplying the green pixels in the blue row by the GB channel gain. Item 20. The module according to Item 19. Dividing the original image from the sensor into a plurality of regions;
Adaptive green channel odd / even mismatch removal method comprising: adaptively removing the green channel odd / even mismatch of the original image on a region-by-region basis to eliminate artifacts in the demosaiced image.   The method of claim 21, wherein the original image is an original Bayer image. The removing step includes
Calibrating the green (GR) channel gain of the red row and the green (GB) channel gain of the blue row for each region of the original image;
Apply the calibrated GR channel gain for each region to the green pixels in the red row and the GB channel gain to the green pixels in the blue row for each region to remove the green channel odd / even mismatch. The method of claim 21 comprising the step of: The removing step includes
For each individual area of the original image,
Generating a weighted central green pixel value based on a first weighting factor of the central green pixel;
Summing weighted green pixel values based on a second weighting factor of peripheral green pixels of the first hierarchy for the central green pixel of the region to form a first hierarchy sum;
Summing weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a second hierarchy sum;
Summing the weighted central green pixel value, the sum of the first tier, and the sum of the second tier to form a total of weighted green pixels;
Normalizing the total of the weighted green pixels;
23. The method of claim 21, comprising replacing a pixel value of the central green pixel with the normalized weighted green pixel total. The generating step includes
25. The method of claim 24, comprising multiplying the central green pixel value by the first weighting factor to generate the weighted central green pixel value. Before the summing step for the first hierarchy,
Comparing a pixel value of each green pixel in the first hierarchy with a pixel maximum value and a pixel minimum value to determine whether the pixel value of each green pixel in the first hierarchy is within a range; ,
Multiplying the pixel value of each green pixel by the first weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the first hierarchy in the range; ,
Multiplying the central green pixel value by the first weighting factor to form a weighted green pixel value outside the corresponding range for each green pixel in the first hierarchy outside the range. A method,
The summing step for the first hierarchy includes weighted green pixel values within the range of all green pixels of the first hierarchy within the range and all green pixels of the first hierarchy outside the range. 25. The method of claim 24, wherein the weighted green pixel values outside the range are summed. Before the summing step for the second hierarchy,
The pixel value of each green pixel in the second hierarchy is compared with the pixel maximum value and the pixel minimum value to determine whether the pixel value for each green pixel in the second hierarchy is within the range. And steps to
Multiplying the pixel value of each green pixel by the second weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the second hierarchy in the range;
Multiplying the central green pixel value by the second weighting factor to form a corresponding out-of-range weighted green pixel value for each second-tier green pixel outside the range. A method,
The summing step for the second hierarchy includes weighted green pixel values within the range of all green pixels of the second hierarchy within the range and all green pixels of the second hierarchy outside the range. 27. The method of claim 26, wherein the weighted green pixel values outside the range are summed. Setting an upper threshold (F_max) for the maximum green mismatch ratio;
Setting a lower threshold (F_min) for the maximum green mismatch ratio;
Calculating an offset adapted to a red pixel surrounding the central green pixel (CGP) to remove a spatially varying green channel odd-even mismatch;
Calculating a pixel maximum value (P_max) based on an expression defined as P_max = max (F_max ^* CGP, CGP + offset);
27. The method of claim 26, further comprising: calculating a pixel minimum value (P_min) based on an expression defined as P_min = min (F_min ^* CGP, CGP-offset). Calculating the offset comprises:
29. The method of claim 28, comprising the step of multiplying k by an average of the peripheral red pixel values of the peripheral red pixels, where k is a parameter that adjusts the magnitude of crosstalk correction. Means for dividing the original image from the sensor into a plurality of regions;
An adaptive green channel odd / even mismatch removal module comprising: means for adaptively removing green channel odd / even mismatches in each region of the original image to eliminate artifacts in the demosaiced image.   32. The module of claim 30, wherein the original image is an original Bayer image. The removal module is
Means for calibrating the green (GR) channel gain of the red row and the green (GB) channel gain of the blue row for each region of the original image;
In order to remove the green channel odd-even mismatch, the calibrated GR channel gain for each region is applied to the green pixels in the red row and the GB channel gain is applied to the green pixels in the blue row for each region. 32. The module of claim 30, comprising means. The means for removing is
Means for generating a weighted central green pixel value based on the first weighting factor of the central green pixel;
Means for summing weighted green pixel values based on a second weighting factor of peripheral green pixels of the first hierarchy for the central green pixel of the region to form a first hierarchy sum;
Means for summing weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a second hierarchy total;
Means for summing the weighted central green pixel value, the sum of the first hierarchy, and the sum of the second hierarchy to form a total of weighted green pixels;
Means for normalizing the sum of the weighted green pixels;
35. The module of claim 32, comprising: means for replacing a pixel value of the central green pixel with a sum of the normalized weighted green pixels. The means for generating is
34. The module of claim 33, comprising means for multiplying the central green pixel value by the first weighting factor to generate the weighted central green pixel value. Means for comparing the pixel value of each green pixel in the first hierarchy with a pixel maximum value and a pixel minimum value to determine whether the pixel value of each green pixel in the first hierarchy is within a range; ,
Means for multiplying the pixel value of each green pixel by the first weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the first hierarchy in the range; ,
Means for multiplying the central green pixel value by the first weighting factor to form a corresponding out-of-range weighted green pixel value for each green pixel in the first hierarchy outside the range; A module,
The means for summing for the first hierarchy includes weighted green pixel values within the range of all green pixels of the first hierarchy within the range and all green pixels of the first hierarchy outside the range. 34. The module of claim 33, summing weighted green pixel values outside the range. Prior to the summation for the second hierarchy, the pixel value of each green pixel of the second hierarchy is compared with the pixel maximum value and the pixel minimum value, and the green pixel of the second hierarchy Means for determining whether a pixel value is within the range;
Means for multiplying the pixel value of each green pixel by the second weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the second hierarchy in the range;
Means for multiplying the central green pixel value by the second weighting factor to form a corresponding out-of-range weighted green pixel value for each second-tier green pixel outside the range. And
The means for summing for the second hierarchy comprises weighted green pixel values within the range of all green pixels of the second hierarchy within the range and all green pixels of the second hierarchy outside the range. 36. The module of claim 35, summing with weighted green pixel values outside the range. Means for setting an upper threshold (F_max) for the maximum green mismatch ratio;
Means for setting a lower threshold (F_min) of the maximum green mismatch ratio;
Means for calculating an offset adapted to a red pixel surrounding the central green pixel (CGP) to remove a spatially varying green channel odd-even mismatch;
Means for calculating a pixel maximum value (P_max) based on an expression defined as P_max = max (F_max ^* CGP, CGP + offset);
37. The module according to claim 36, further comprising means for calculating a pixel minimum value (P_min) based on an expression defined as P_min = min (F_min ^* CGP, CGP-offset). The means for calculating the offset is:
38. The module of claim 37, comprising means for multiplying k by an average of the peripheral red pixel values of the peripheral red pixels, wherein k is a parameter for adjusting the magnitude of crosstalk correction. For each area of the original image,
Generating a weighted central green pixel value based on a first weighting factor of the central green pixel;
Summing weighted green pixel values based on a second weighting factor of peripheral green pixels of the first hierarchy for the central green pixel of the region to form a first hierarchy sum;
Summing weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a second hierarchy sum;
Summing the weighted central green pixel value, the sum of the first tier, and the sum of the second tier to form a total of weighted green pixels;
Normalizing the total of the weighted green pixels;
An adaptive green channel odd / even mismatch removal method comprising: substituting pixel values of the central green pixel with the normalized weighted green pixel sum to remove the green channel odd / even mismatch. The generating step includes
40. The method of claim 39, comprising multiplying the central green pixel value by the first weighting factor to generate the weighted central green pixel value. Before the summing step for the first hierarchy,
Comparing a pixel value of each green pixel in the first hierarchy with a pixel maximum value and a pixel minimum value to determine whether the pixel value of each green pixel in the first hierarchy is within a range; ,
Multiplying the pixel value of each green pixel by the first weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the first hierarchy in the range; ,
Multiplying the central green pixel value by the first weighting factor to form a weighted green pixel value outside the corresponding range for each green pixel in the first hierarchy outside the range. A method,
The summing step for the first hierarchy includes weighted green pixel values within the range of all green pixels of the first hierarchy within the range and all green pixels of the first hierarchy outside the range. 41. The method of claim 40, wherein the weighted green pixel values outside the range are summed. Before the summing step for the second hierarchy,
The pixel value of each green pixel in the second hierarchy is compared with the pixel maximum value and the pixel minimum value to determine whether the pixel value for each green pixel in the second hierarchy is within the range. And steps to
Multiplying the pixel value of each green pixel by the second weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the second hierarchy in the range; ,
Multiplying the central green pixel value by the second weighting factor to form a corresponding out-of-range weighted green pixel value for each second-tier green pixel outside the range. A method,
The summing step for the second hierarchy includes weighted green pixel values within the range of all green pixels of the second hierarchy within the range and all green pixels of the second hierarchy outside the range. 42. The method of claim 41, wherein the weighted green pixel values outside the range are summed. Setting an upper threshold (F_max) for the maximum green mismatch ratio;
Setting a lower threshold (F_min) for the maximum green mismatch ratio;
Calculating an offset adapted to a red pixel surrounding the central green pixel (CGP) to remove a spatially varying green channel odd-even mismatch;
Calculating a pixel maximum value (P_max) based on an expression defined as P_max = max (F_max ^* CGP, CGP + offset);
42. The method of claim 41, further comprising: calculating a pixel minimum value (P_min) based on an expression defined as P_min = min (F_min ^* CGP, CGP-offset). Calculating the offset comprises:
44. The method of claim 43, comprising the step of multiplying k by an average of the peripheral red pixel values of the peripheral red pixels, where k is a parameter that adjusts the magnitude of crosstalk correction. Means for generating a weighted central green pixel value based on the first weighting factor of the central green pixel;
Means for summing weighted green pixel values based on a second weighting factor of peripheral green pixels of the first hierarchy for the central green pixel of the region to form a first hierarchy sum;
Means for summing weighted green pixel values based on a third weighting factor of peripheral green pixels of the second hierarchy for the central green pixel of the region to form a second hierarchy total;
Means for summing the weighted central green pixel value, the sum of the first hierarchy, and the sum of the second hierarchy to form a total of weighted green pixels;
Means for normalizing the sum of the weighted green pixels;
Means for replacing a pixel value of the central green pixel with a sum of the normalized weighted green pixels, and an adaptive green channel odd-even mismatch removal module. The means for generating is
46. The module of claim 45, comprising means for multiplying the central green pixel value by the first weighting factor to generate the weighted central green pixel value. Means for comparing the pixel value of each green pixel in the first hierarchy with a pixel maximum value and a pixel minimum value to determine whether the pixel value of each green pixel in the first hierarchy is within a range; ,
Means for multiplying the pixel value of each green pixel by the first weighting factor to form a weighted green pixel value in the corresponding range for each green pixel in the first hierarchy in the range; ,
Means for multiplying the central green pixel value by the first weighting factor to form a corresponding out-of-range weighted green pixel value for each green pixel in the first hierarchy outside the range; A module,
The means for summing for the first hierarchy includes weighted green pixel values within the range of all green pixels of the first hierarchy within the range and all green pixels of the first hierarchy outside the range. 46. The module of claim 45, summing weighted green pixel values outside the range. Before summing by the means for summing the second hierarchy, the pixel value of each green pixel of the second hierarchy is compared with the pixel maximum value and the pixel minimum value, and each of the second hierarchy Means for determining whether the pixel value of a green pixel is within the range;
Means for multiplying the pixel value of each green pixel by the second weighting factor to form a weighted green pixel value within the corresponding range for each green pixel of the second hierarchy within the range; ,
Means for multiplying the central green pixel value by the second weighting factor to form a corresponding out-of-range weighted green pixel value for each second-tier green pixel outside the range; A module,
The means for summing for the second hierarchy comprises weighted green pixel values within the range of all green pixels of the second hierarchy within the range and all green pixels of the second hierarchy outside the range. 48. The module of claim 47, summing the weighted green pixel values outside the range. Means for setting an upper threshold (F_max) for the maximum green mismatch ratio;
Means for setting a lower threshold (F_min) of the maximum green mismatch ratio;
Means for calculating an offset adapted to red pixels surrounding the central green pixel (CPG) to remove spatially varying green channel odd-even mismatch;
Means for calculating a pixel maximum value (P_max) based on an expression defined as P_max = max (F_max ^* CGP, CGP + offset);
39. The module according to claim 38, further comprising means for calculating a pixel minimum value (P_min) based on an expression defined as P_min = min (F_min ^* CGP, CGP-offset). The means for calculating the offset is:
50. The module of claim 49, comprising means for multiplying k by an average of the peripheral red pixel values of the peripheral red pixels, wherein k is a parameter for adjusting the magnitude of crosstalk correction.

Images