JP2009206553A - Image processing apparatus and program for image processing - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To achieve false color reduction and/or noise reduction processing to an interpolated image, where image data in a Bayer array are subjected to pixel interpolation, without damaging the quality of original images, in an operation time which is relatively shorter than conventionally. <P>SOLUTION: Sorting processing sections 12, 13 perform sorting processing, in the order of pixel values in 25-pixel regions to color difference signals HR, HB outputted from HR, HB calculation sections 11, and output medium sorting results, namely the ninth to seventeenth in the sorting processing results obtained. Operation sections 16-19 calculate a plurality of candidate values of an R signal and those of a B signal, and calculate the update values of the color difference signals HR, HB, based on the candidate values of the R, B signals. In the R, B signals, the value of the difference, obtained in comparison with the values of the pixels around the pixels to be processed and the pixels of primary colors corresponding to the primary colors of the candidate values, is less than the threshold. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program, and more particularly to an image processing apparatus that performs post processing for the purpose of false color reduction and noise reduction on an interpolated image obtained by subjecting Bayer array image data to pixel interpolation. And an image processing program.

  2. Description of the Related Art A solid-state imaging device in which a Bayer array color filter is provided on a light receiving surface is known. As schematically shown in FIG. 4, the Bayer array color filter includes a red filter portion R for one pixel that transmits red light, a green filter portion G for one pixel that transmits green light, and blue light. The blue filter part B for 1 pixel which permeate | transmits is arrange | positioned. The Bayer array color filter shown in FIG. 4 has a checkered pattern of green filter portions G that transmit green light having a high contribution ratio of the luminance signal in every other pixel pitch among the color filter portions R, G, and B described above. The red filter part R and the blue filter part B are arranged in a checkered pattern every other pixel pitch in the remaining portions.

  In this way, the color filter of the Bayer array is output from a solid-state imaging device in which the color filter of the Bayer array is provided on the light receiving surface because the color filter portion of the same color element exists at the geometrically skipped pixel position. The image processing apparatus that processes the captured image signal performs an interpolation process for interpolating the colors existing at the skipped pixel positions in order to obtain a signal in which the color elements of the three primary colors are aligned for all the pixels.

  Here, when the above-described image processing apparatus performs the above-described interpolation processing by calculating the average of simple neighboring pixels, jaggies and false colors (colors that do not exist in the original image) of the edge portion are widely generated. Are known. For this reason, various image processing apparatuses for reducing these jaggies and false colors have been proposed. Many of the proposals perform processing that controls the interpolation conditions based on the continuity and correlation of colors.

  However, even if various efforts are made in the interpolation technique, there are limits to the reduction effect such as false color and zipper noise (noise on the dotted line that does not exist in the original image; generated as an isolated point in the edge or corner of the original image). Therefore, in the past, attempts have been made to reduce these separately from the interpolation technique (see, for example, Patent Documents 1 to 5 and Non-Patent Document 1).

  Since the image processing apparatus described in Patent Document 1 includes a signal having a frequency (fs / 2) that is half the sampling frequency in the output of the image sensor, erroneous detection is likely to occur in the vertical / horizontal correlation determination. When the component is detected and the color is chromatic, pre-processing is performed in which the three primary color signals from which the frequency component of fs / 2 is removed are used.

  The image processing apparatus described in Patent Document 2 performs post processing that suppresses false colors while avoiding color loss by thinning the color of the line segment component (straight line portion) in the interpolated signal. The image processing apparatus described in Patent Document 3 performs post processing that suppresses false colors by controlling the color density in accordance with the content of high-frequency components in the interpolated signal.

  In addition, the image processing apparatuses described in Patent Document 4 and Patent Document 5 perform post-processing that suppresses false colors by high-frequency removal of interpolated RG and BG signals. Hereinafter, the R signal indicates a red signal, the G signal indicates a green signal, the B signal indicates a blue signal, the RG signal indicates a difference signal (color difference signal) between the red signal R and the green signal G, and a BG signal. Indicates a difference signal (color difference signal) between the blue signal B and the green signal G.

  Furthermore, the image processing apparatus described in Non-Patent Document 1 performs the following processing.

(1) Select a portion including a high frequency for the G signal (2) Deriving RG and BG signals for 25 pixels (5 pixels in the vertical direction and 5 pixels in the horizontal direction) in the vicinity of the interpolation target pixel (3) ) Deriving the median (intermediate value) for both the RG and BG signals (4) The R and B signals calculated from the RG and BG signals of the center interpolation target pixel; The R signal and B signal calculated from the median of the RG signal and BG signal are correlated with the surroundings so that the ratio of the R signal and B signal that give values close to the surroundings is high. Blend the RG and BG signals of the center pixel to be interpolated with the median of the RG and BG signals. (5) Using the blended signal of (4), the R signal, G Update signal and B signal (6) Return to (2) and repeat (total 5 times)
In the image processing apparatus described in Non-Patent Document 1, in (4), in order to prevent a luminance isolated point that causes zipper noise, a balance is provided in a direction in which there is no difference between the sample color value in the vicinity and the converted color value. It is characteristic.

Japanese Patent Laid-Open No. 11-177995 JP 2002-209224 A JP 2003-224859 A JP 2006-245916 A JP 2006-311314 A Kai, Takahashi, Kikuchi, Muramatsu: “Color Demosaicing Considering Color Signals with High Frequency”, Journal of the Institute of Image Information and Television Engineers, Vol.61, No.8, pp.1209-1217 (2007)

  However, since the image processing apparatus described in Patent Document 1 performs preprocessing, it cannot be used to improve a video signal that has already been shot and interpolated. Further, according to the image processing apparatuses described in Patent Document 2 and Patent Document 3, the color density can be controlled for a portion where false colors are likely to be generated or a portion where the false colors are easily noticeable, which is effective. However, as a practical matter, it is difficult to make a setting so as to avoid fading of a necessary portion by thinning only the false color portion.

  In the method of smoothing the RG signal and the BG signal as in the image processing apparatuses described in Patent Document 4 and Patent Document 5, a false color is generated because the correlation between the primary colors is locally disturbed in the interpolation. It is very effective in mitigating the situation. However, in the image processing apparatuses described in Patent Document 4 and Patent Document 5, image information such as edges may be lost when smoothing by simple high-frequency removal is performed.

  The image processing apparatus described in Non-Patent Document 1 takes measures to avoid the above-mentioned problem of losing image information by using a median filter, and to prevent the occurrence of independent luminance points. However, the image processing apparatus described in Non-Patent Document 1 has a problem that it takes a relatively long calculation time because it repeatedly performs calculations.

  The present invention has been made in view of the above points, and performs false color reduction / noise reduction processing for an interpolated image obtained by performing pixel interpolation on Bayer array image data without compromising the image quality of the original image as much as possible, and compared with the prior art. An object of the present invention is to provide an image processing apparatus and an image processing program that can be realized in a relatively short calculation time.

In order to achieve the above object, according to a first aspect of the present invention, for each pixel constituting an imaging signal output from a solid-state imaging device in which a Bayer array color filter is provided on a light receiving surface, red, blue in the Bayer array The three primary colors that have undergone color demosaicing processing, which is the process of interpolating the values of the other two color elements whose values other than the value of one color element of its own pixel are unknown among the three primary color elements of green An image processing apparatus that performs image processing on a signal,
The three primary color elements of the three primary color signals obtained by the color demosaicing process for each pixel in a predetermined pixel area unit consisting of the pixel to be processed in the three primary color signals and surrounding pixels centering on the pixel to be processed A color difference signal calculating means (11, step S1) for calculating a first color difference signal that is a color difference signal between red and green and a second color difference signal that is a color difference signal between blue and green from the value of Each pixel in the pixel area unit is sorted in the order of the magnitude of the values of the first and second color difference signals, and the sorting order is within a middle range which is a predetermined range including the central order. Sorting processing means (12, 13, steps S2, S3) for outputting the plurality of values of the first color difference signal and the plurality of values of the second color difference signal, respectively, and the first output from the sorting processing means A plurality of values of the color difference signal of 1 and the second value Among the plurality of values of the color difference signal, when the pixel to be processed is a central pixel, a plurality of types of pixels depending on which color element value of the three primary color elements of the Bayer array is the central pixel. When the pixel type classification means (14, 15, step S6, S7) for classifying into the type and the pixel type classification means classify that the central pixel is the pixel type having the value of the red color element The update value of the green color element is calculated from the known red value in the center pixel and the value of the first color difference signal having the sorting order output from the sorting processing means, and the color element of the center pixel is calculated. Is classified as the pixel type having the value of the blue color element, the value of the second primary color signal known in the center pixel and the sorting order output from the sorting processing means are in the middle order. Second color of The first update value calculation means (16, 18, steps S8, S18) for calculating the update value of the green color element from the signal value and the pixel type classification means, the color element of the center pixel is a red color element When it is classified as a pixel type having a value other than, a plurality of candidate values for updating the value of the red color element are calculated based on the plurality of values of the first color difference signal. When the pixel type classification means classifies the central pixel as a pixel type having a value other than the blue color element, the blue color element is determined based on the plurality of values of the second color difference signal. Candidate value calculation means (17, 18, steps S10, S12, S15, S20, S22, S25) for calculating a plurality of candidate values for updating the values and the red color element calculated by the candidate value calculation means A plurality of candidate values for updating the value and a plurality of candidate values for updating the value of the blue color element are respectively processed. Compared with the value of the color element of the pixel surrounding the heart pixel and having the same color element value as the color element of the central pixel to be processed, the difference value is calculated. Difference value calculated using difference calculation means (17, 18, steps S10, S12, S15, S20, S22, S25) and a plurality of candidate values for updating the value of the red color element by the difference calculation means The update value of the red color element is calculated from the cumulative average value of the candidate values for updating the value of one or more red color elements whose value is less than a predetermined threshold, and the difference calculation means The blue color by the cumulative average value of candidate values for updating the value of one or more blue color elements whose difference value calculated using a plurality of candidate values for updating the value is less than a predetermined threshold value And second update value calculation means (17, 18, steps S11, S13, S16, S21, S23, S26) for calculating an update value of the element.

  In order to achieve the above object, the second invention is calculated using a plurality of candidate values for updating the value of the red color element by the difference calculation means in addition to the configuration of the first invention. When all the difference values are equal to or greater than the threshold, the sorting color output from the sorting processing means is added with the first color difference value in the middle order and the update value calculated by the first update value calculating means. A plurality of candidate values for updating the value of the blue color element are used by the first update value setting means (17, steps S14, S17) whose value is the update value of the red color element and the difference calculation means. When the difference values calculated in this way are all equal to or greater than the threshold value, the sorting order output from the sorting processing means is the second color difference signal value in the middle order and the update calculated by the second update value calculating means. The value added to the value is used as the update value for the blue color element. Updated value setting means (18, step S24, S27) and characterized by further comprising a.

  In order to achieve the above object, an image processing program according to a third invention causes a computer to function as each constituent unit of the first invention. Furthermore, in order to achieve the above object, an image processing program according to a fourth invention causes a computer to function as each constituent means of the second invention.

  According to the present invention, false color reduction and noise reduction processing is performed on an interpolated image obtained by performing pixel interpolation on Bayer array image data without compromising the image quality of the original image and relatively shorter than before. It can be realized in calculation time.

  Next, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 shows a block diagram of an embodiment of an image processing apparatus according to the present invention. In the figure, an image processing apparatus 10 according to the present embodiment uses a pixel interpolator to perform predetermined processing on an imaging signal output from a solid-state imaging device (not shown) having a Bayer array color filter provided on a light receiving surface. A red signal R and a blue signal obtained by performing a known color demosaicing process (decoding process) for interpolating the pixel value data of the nonexistent color element to a pixel for which no pixel value data exists for the color element of B and three primary color signals of green signal G are received as input. Usually, the three primary color signals often contain false colors.

  The image processing apparatus 10 calculates the color difference signals HR (= RG) and HB (= BG) in units of a predetermined image area centering on the pixel to be processed from the input three primary color signals. , HB calculation unit 11. Further, the image processing apparatus 10 includes sorting processing units 12 and 13, pixel type classification units 14 and 15, and calculation units 16 to 19, following the HR and HB calculation unit 11.

  The sorting processing units 12 and 13 perform a sorting process on the color difference signals HR and HB output from the HR and HB calculation unit 11. Then, a signal with a medium result is output. Details of this signal will be described later.

  The pixel type classification units 14 and 15 receive as input signals that are intermediate in the sorting results output from the sorting processing units 12 and 13. Since the input signal is a signal based on a Bayer array, when the input signal is a known center pixel, the input signal is classified into four types of pixel types based on the relationship between the center pixel and the peripheral pixels. As will be described later, the pixel type includes a pixel having a red signal R (hereinafter also referred to as R pixel), a pixel having a blue signal B (hereinafter also referred to as B pixel), and a pixel having a green signal G at the center. (Hereinafter also referred to as G pixels), and the right and left pixels are R pixel types, the center is a G pixel, and the left and right pixels are B pixel types.

  When the color element of the pixel from which the sorting result output from the pixel type classifying unit 14 generates the intermediate color difference signal HR is an R pixel, the calculation unit 16 is an intermediate between the pixel signal of the R pixel and the color difference signal HR. The G signal is calculated from the value, and the B signal is calculated from the G signal and a signal HBhat described later from the calculation unit 19.

  When the color element of the pixel from which the sorting result output from the pixel type classifying unit 14 generates the intermediate color difference signal HR is other than the R pixel, the calculation unit 17 determines the input pixel and the surrounding R pixel. While comparing, the intermediate color difference signal HR is accumulated and averaged to calculate the signal HRhat, and further, the calculation of determining the R signal from the G signal and HRhat is performed.

  When the color element of the pixel from which the sorting result output from the pixel type classifying unit 14 generates the intermediate color difference signal HB is a B pixel, the calculation unit 18 is an intermediate between the pixel signal of the B pixel and the color difference signal HB. The G signal is calculated from the value, and the R signal is calculated from the G signal and a signal HRhat described later from the calculation unit 17.

  When the color element of the pixel from which the sorting result output from the pixel type classification unit 14 generates the intermediate color difference signal HB is other than B pixels, the arithmetic unit 19 determines the input pixel and the surrounding B pixels. While comparing, the intermediate color difference signal HB is accumulated and averaged to calculate the signal HBhat, and further, the calculation for determining the B signal from the G signal and HBhat is performed.

  Next, the operation of the present embodiment will be described with reference to the flowchart of FIG. 2 and the pixel type explanatory diagram of FIG. The HR and HB calculation unit 11 includes, in the three primary color signals obtained by performing the demosaicing process on the above-described Bayer array image pickup signal, 5 pixels in the vertical direction and 5 pixels in the horizontal direction centering on the center pixel to be processed. Based on signals of a total of 25 pixels, the color difference signal HR (= RG) and the color difference signal HB (= BG) are respectively calculated and generated (step S1).

  The sorting processing unit 12 performs the ascending order of the HR pixel value from the largest to the smallest in the 25 pixel area described above with respect to the color difference signal HR output from the HR and HB calculation unit 11 (or vice versa). (2) A sorting process is performed (step S2). On the other hand, the sorting processing unit 13 increases the color difference signal HB output from the HR / HB calculation unit 11 in ascending order from the largest HB pixel value within the 25 pixel region described above (or vice versa). Sorting process is performed in descending order (step S3). The sorted HR and HB are expressed as hr (num) and hb (num) with the rank being num (1 to 25).

  Subsequently, the sorting processing unit 12 outputs the intermediate sorting results hr (9) to hr (17) from the 9th place to the 17th place among the sorting results hr (num) from the 1st place to the 25th place (step) S4). Similarly, the sorting processing unit 13 outputs the intermediate sorting results hb (9) to hb (17) from the 9th to the 17th among the sorting results hb (num) from the 1st to the 25th (Step). S5).

  Here, the reason why the intermediate sorting results from the 9th to the 17th are output and used in the subsequent stage is that a smoothing function like a median filter is secured and a plurality of candidate values for the color difference signals HR and HB are secured. This is because it is possible to ensure the freedom to select the R signal and B signal calculated from them, which have high correlation with the surroundings and do not easily generate a luminance isolated point.

  In other words, the median filter performs a filtering process in which neighboring pixels of a pixel to be processed are arranged in order of increasing values, for example, to obtain a median value and the value of the pixel to be processed is replaced with the median value in order to remove noise. Therefore, the median filtering smoothing function is ensured by using the middle sorting results from the 9th to the 17th. As for sorting, the present embodiment is for the purpose of evaluating the intermediate sorting result, so either ascending order or descending order may be used.

  The pixel type classification unit 14 receives as input the intermediate sorting results from the 9th to the 17th of the color difference signal HR output from the sorting processing unit 12, and the pixels around the central pixel that is the basis of the input signal The pixel types are classified based on the positional relationship (step S6). Similarly to the above, the pixel type classifying unit 15 also receives the intermediate sorting result from the 9th to the 17th of the color difference signal HB output from the sorting processing unit 13 as an input, and the central pixel from which the input signal is based The pixel type is classified based on the positional relationship with the surrounding pixels (step S7).

  Here, the sorting result input to the pixel type classification units 14 and 15 is a color difference signal, and the central pixel of the color difference signal is the center of the three primary color signals of the Bayer array that is the generation element of the color difference signals HR and HB in step S1. Corresponds to a pixel. The central pixel classified by the pixel type classifying units 14 and 15 is a pixel corresponding to the color element of the color filter unit of the Bayer array that is known when the three primary color signals are generated by interpolation. The Bayer shown in FIG. The pixel is a red signal R pixel (R pixel), a green signal G pixel (G pixel), or a blue signal B pixel (B pixel) based on the arrangement.

  When the central pixel is the central pixel of the pixel region of 3 pixels in the vertical direction and 3 pixels in the horizontal direction, the pixel type classification units 14 and 15 have four types of pixel types as shown in FIGS. Classify. Here, since the known pixel corresponding to the color filter portion of the color filter of the Bayer array has a known pixel coordinate of the solid-state image sensor (in other words, the coordinate of the color filter portion), the R pixel from this coordinate position, It can be seen whether the pixel is a B pixel or a G pixel.

  FIG. 3A shows a case where the central pixel at the position of the color filter section R shown in FIG. 4 is an R pixel, the upper, lower, left, and right pixels are G pixels, and the left and right diagonal pixels are B pixels. is there. FIG. 3B shows the case where the central pixel at the position of the color filter section B shown in FIG. 4 is a B pixel, the upper, lower, left, and right pixels are G pixels, and the left and right diagonal pixels are R pixels. Pixel.

  When the known center pixel at the position of the color filter portion G shown in FIG. 4 is a G pixel, there are two types shown in FIGS. 3C and 3D depending on the upper, lower, left, and right pixels. The pixel arrays shown in FIGS. 3C and 3D are common in that the center pixel is a G pixel and the left and right diagonal pixels are also G pixels, but the pixel array in FIG. 3 is different from the pixel arrangement shown in FIG. 3D in that the upper and lower pixels are R pixels and the left and right pixels are B pixels. Here, the central pixel in the pixel array in FIG. 3C is referred to as a G_a pixel, and the central pixel in the pixel array in FIG. 3D is referred to as a G_b pixel.

  Returning to FIG. 1 and FIG. It is assumed that the decoded R signal, B signal, and G signal have been obtained for all pixels before the processing of this embodiment.

  First, a case will be described in which sorting results of the color difference signals HR and HB, in which the center pixel is determined to be the R pixel, are output from the pixel type classification units 14 and 15, respectively. The sorting result hr [num] (here, num = 9 to 17) of the color difference signal HR at this time is supplied to the arithmetic unit 16, and the sorting result hb [num] (here, num = 9 to 17) of the color difference signal HB. ) Is supplied to the arithmetic unit 19.

  The calculation unit 16 calculates hr [13], which is the median value (intermediate value) of the sorting result hr [num] of the color difference signal HR determined from the pixel type classification unit 14 that the center pixel is an R pixel. By subtracting from the original R signal already obtained, an updated value of the G signal in the R pixel is obtained (step S8). At this time, the arithmetic unit 16 may further generate the color difference signal RG using the obtained G signal and the known R signal, and output this. The calculation unit 16 adds a signal HBhat supplied from the calculation unit 19 described later and the G signal obtained in step S8 to obtain an updated value of the B signal in the R pixel (step S9).

  On the other hand, the calculation unit 19 sorts the color difference signal HB (hb [9] to hb [17]) when the central pixel output from the pixel type classification unit 15 is determined to be an R pixel, The candidate value of the B signal in the nine R pixels obtained by the sum of the G signal already obtained in the R pixel to be compared is compared with the values of the upper, lower, left and right B pixels, and the difference is a predetermined threshold value. Candidate values less than th are detected. Then, the detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S20). The sorting results (hb [9] to hb [17]) indicate the values of the nine color difference signals HB, and the candidate values of the B signal in the R pixel obtained by the sum with the G signal are as follows. The provisional B signal value is shown.

  Subsequently, the calculation unit 19 sets the average value of the accumulated values accumulated in step S20 as the update value HBhat of the color difference signal BG (step S21). In addition, when all the differences are equal to or greater than the threshold th and there is no cumulative value as a result of the comparison in step S20, the calculation unit 19 is in the input signal. The intermediate value hb [13] is used as the update value HBhat of the color difference signal BG (step S21). The calculation unit 19 outputs this HBhat to the calculation unit 16.

  Therefore, when the center pixel is an R pixel, the update value of the G signal and the update value of the B signal at the target R pixel are output from the calculation unit 16. In step S21, the reason why the average value of the accumulated values in step S20 is calculated to be HBhat is to make it as close as possible to the original median value among the values satisfying continuity (other steps S11, The same applies to S13, S16, S23, and S26).

  Next, a case will be described in which the sorting results of the color difference signals HR and HB, in which the center pixel is determined to be the B pixel, are output from the pixel type classification units 14 and 15, respectively. The sorting result hr [num] (here, num = 9 to 17) of the color difference signal HR at this time is supplied to the arithmetic unit 17, and the sorting result hb [num] (here, num = 9 to 17) of the color difference signal HB. ) Is supplied to the calculation unit 18.

  The calculation unit 17 sorts the color difference signal HR (hr [9] to hr {17}) output when the pixel type classification unit 14 determines that the central pixel is the B pixel, and the target B. The candidate value of the R signal in the nine B pixels obtained by the sum of the G signal already obtained in the pixel is compared with the values of the upper, lower, left and right R pixels, and the difference is less than a predetermined threshold th The candidate value of is detected. Then, the detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S10). The sorting results (hr [9] to hr [17]) indicate the values of the nine color difference signals HR, and the candidate value of the R signal in the B pixel obtained by the sum with the G signal is The value of the provisional R signal is shown.

  Subsequently, the calculation unit 17 sets the average value of the accumulated values accumulated in step S10 as the update value HRhat of the color difference signal RG (step S11). In addition, when all the differences are equal to or greater than the threshold th and there is no cumulative value (this is a case where there is no condition that satisfies the continuity condition), the arithmetic unit 17 determines the intermediate value hr [13] in the input signal. Is the update value HRhat of the color difference signal RG (step S11). The calculation unit 17 outputs this HRhat to the calculation unit 18.

  On the other hand, the calculation unit 18 determines hb [, which is the median value (intermediate value) of the sorting result hb [num] of the color difference signal HB when it is determined that the center pixel output from the pixel type classification unit 15 is the B pixel. 13] is subtracted from the already obtained original B signal to obtain an updated value of the G signal in the B pixel (step S18). The calculation unit 18 adds the signal HRhat supplied from the calculation unit 17 and the G signal obtained in step S18 to obtain an updated value of the R signal in the B pixel (step S19).

  Therefore, when the center pixel is a B pixel, the calculation unit 18 outputs an updated value of the G signal and an updated value of the R signal in the target B pixel. At this time, the calculation unit 18 may further generate the color difference signal BG using the obtained G signal and the known B signal, and output this.

  Next, a case will be described in which the sorting results of the color difference signals HR and HB, in which the center pixel is determined to be the G_a pixel, are output from the pixel type classification units 14 and 15, respectively. The sorting result hr [num] (here, num = 9 to 17) of the color difference signal HR at this time is supplied to the arithmetic unit 17, and the sorting result hb [num] (here, num = 9 to 17) of the color difference signal HB. ) Is supplied to the arithmetic unit 19.

  The calculation unit 17 sorts the color difference signal HR (hr [9] to hr [17]) output when the pixel type classification unit 14 determines that the central pixel is the G_a pixel, and the target G_a The candidate value of the R signal in the nine G_a pixels obtained by the sum of each of the G signals already obtained in the pixel is compared with the values of the left and right R pixels, and the difference is less than a predetermined threshold th Detect candidate values. Then, the detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S12). The sorting results (hr [9] to hr [17]) indicate the values of the nine color difference signals HR, and the candidate value of the R signal in the G_a pixel obtained by the sum with the G signal is The value of the provisional R signal is shown.

  Subsequently, the calculation unit 17 sets the average value of the accumulated values accumulated in step S12 as the update value HRhat of the color difference signal RG (step S13). In addition, when all the differences are equal to or greater than the threshold th and there is no accumulated value as a result of the comparison in step S12, the calculation unit 17 is in the input signal. The intermediate value hr [13] is used as the update value HRhat of the color difference signal RG (step S13). Subsequently, the arithmetic unit 17 adds the calculated HRhat and the G signal already obtained in the G_a pixel that is the central pixel to obtain an updated value of the R signal in the target G_a pixel (step S14). ).

  On the other hand, the calculating unit 19 sorts the color difference signal HB (hb [9] to hb [17]) when the central pixel output from the pixel type classifying unit 15 is determined to be a G_a pixel, The candidate values of the B signal in the nine G_a pixels obtained by the respective sums with the G signal already obtained in the G_a pixel to be compared are compared with the values of the upper and lower B pixels, and the difference is a predetermined threshold th Detect candidate values less than Then, the detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S22). The sorting results (hb [9] to hb [17]) indicate the values of the nine color difference signals HB, and the candidate values of the B signal in the G_a pixel obtained by the sum with the G signal are The provisional B signal value is shown.

  Subsequently, the calculation unit 19 sets the average value of the accumulated values accumulated in step S22 as the update value HBhat of the color difference signal BG (step S23). In addition, as a result of the comparison in step S22, the arithmetic unit 19 determines that the input signal is in the input signal when all the differences are equal to or larger than the threshold th and there is no cumulative value (this is a case where there is no continuity condition). The intermediate value hb [13] is used as the update value HBhat of the color difference signal BG (step S23). Subsequently, the computing unit 19 adds the calculated HBhat and the G signal already obtained in the G_a pixel that is the central pixel to obtain an updated value of the B signal in the target G_a pixel (step S24). ).

  Therefore, when the central pixel is the G_a pixel, the calculation unit 17 outputs the updated value of the R signal in the target G_a pixel, and the calculation unit 19 outputs the updated value of the B signal in the target G_a pixel. Is done. At this time, the computing unit 17 may further generate the color difference signal RG using the obtained G signal and the known R signal, and output this. The calculation unit 19 may also generate the color difference signal BG using the obtained G signal and the known B signal, and output this.

  Next, a case will be described in which the sorting results of the color difference signals HR and HB, in which the center pixel is determined to be the G_b pixel, are output from the pixel type classification units 14 and 15, respectively. The sorting result hr [num] (here, num = 9 to 17) of the color difference signal HR at this time is supplied to the arithmetic unit 17, and the sorting result hb [num] (here, num = 9 to 17) of the color difference signal HB. ) Is supplied to the arithmetic unit 19.

  The calculation unit 17 sorts the color difference signal HR (hr [9] to hr [17]) output when the pixel type classification unit 14 determines that the central pixel is a G_b pixel, and the target G_b. The candidate value of the R signal in the 9 G_b pixels obtained by the sum of the G signal already obtained in the pixel is compared with the values of the upper and lower R pixels, and the difference is less than the predetermined threshold th Detect candidate values. Then, the detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S15). The sorting results (hr [9] to hr [17]) indicate the values of the nine color difference signals HR, and the candidate value of the R signal in the G_b pixel obtained by the sum with the G signal is The value of the provisional R signal is shown.

  Subsequently, the calculation unit 17 sets the average value of the accumulated values accumulated in step S15 as the update value HRhat of the color difference signal RG (step S16). In addition, when all the differences are equal to or greater than the threshold th and there is no cumulative value as a result of the comparison in step S15, the arithmetic unit 17 is in the input signal. The intermediate value hr [13] is used as the update value HRhat of the color difference signal RG (step S16). Subsequently, the arithmetic unit 17 adds the calculated HRhat and the G signal already obtained in the G_b pixel that is the central pixel to obtain an updated value of the R signal in the target G_b pixel (step S17). ).

  On the other hand, the calculation unit 19 sorts the color difference signal HB (hb [9] to hb [17]) when it is determined that the center pixel output from the pixel type classification unit 15 is a G_b pixel, The candidate value of the B signal in the nine G_b pixels obtained by the sum of the G signal already obtained in the G_b pixel to be compared is compared with the values of the left and right B pixels, and the difference is a predetermined threshold th Detect candidate values less than The detected candidate values are regarded as being correlated (high continuity), and the candidate values are accumulated (step S25). The sorting results (hb [9] to hb [17]) indicate the values of the nine color difference signals HB, and the candidate values of the B signal in the G_b pixel obtained by the sum with the G signal are as follows. The provisional B signal value is shown.

  Subsequently, the calculation unit 19 sets the average value of the accumulated values accumulated in step S25 as the update value HBhat of the color difference signal BG (step S26). In addition, when all the differences are equal to or greater than the threshold th and there is no cumulative value as a result of the comparison in step S25, the calculation unit 19 is in the input signal. The intermediate value hb [13] is used as the update value HBhat of the color difference signal BG (step S26). Subsequently, the computing unit 19 adds the calculated HBhat and the G signal already obtained in the G_b pixel that is the central pixel to obtain an updated value of the B signal in the target G_b pixel (step S27). ).

  Therefore, when the central pixel is a G_b pixel, the calculation unit 17 outputs the updated value of the R signal in the target G_b pixel, and the calculation unit 19 outputs the updated value of the B signal in the target G_b pixel. Is done. At this time, the computing unit 17 may further generate the color difference signal RG using the obtained G signal and the known R signal, and output this. The calculation unit 19 may also generate the color difference signal BG using the obtained G signal and the known B signal, and output this.

  As described above, according to the present embodiment, pixel interpolation is performed on Bayer array image data, and the image signal in a state in which all data of R pixel, G pixel, and B pixel are gathered, In order to reduce the false color generated because the correlation between the primary colors is locally disturbed at the interpolation stage, the color difference signal RG and the color difference signal BG are smoothed. At this time, the smoothing according to the present embodiment is performed by using a plurality of color difference signals RG and BG located within a middle range in which the order of the sorting results is a predetermined range including the middle order. Since the value is used, it acts as a de facto median filter and the edges are not damaged (the original image is not damaged).

  In the present embodiment, for each of a plurality of values of the color difference signals RG and BG, which are the intermediate sorting results, the surrounding pixels are calculated for the R pixel and B pixel values calculated from them. When the continuity is high, the color difference signals RG and BG are used as updated values of the color difference signals RG and BG, and when there is no continuity, the sorting result is obtained. Is used as an updated value of the color difference signals RG and BG, and necessary primary colors are updated based on these updated values. Therefore, the luminance isolation that causes false color reduction and zipper noise is achieved. The avoidance of points can be realized without performing repeated calculations. Therefore, according to the present embodiment, it is possible to remove the false color in a relatively short calculation time compared to the conventional case.

  The present invention is not limited to an image processing apparatus that performs hardware Bayer image interpolation processing, and includes a computer program that causes a computer to execute the configuration of the block diagram of FIG. The image processing program which is a computer program in this case may be taken into the computer from a recording medium, or may be taken into the computer via a network.

It is a block diagram of one embodiment of an image processing device of the present invention. It is a flowchart for operation | movement description of FIG. It is explanatory drawing of the classification of the pixel type in this invention. It is a figure which shows an example of a Bayer arrangement | sequence.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image processing apparatus 11 HR, HB calculation part 12, 13 Sorting process part 14, 15 Pixel type classification | category part 16-19 Calculation part

Claims (4)

For each pixel constituting an image signal output from a solid-state image sensor having a Bayer array color filter provided on the light receiving surface, among the three primary color elements of red, blue, and green in the Bayer array, its own pixel is An image processing apparatus that performs image processing on three primary color signals subjected to color demosaicing processing, which is processing for interpolating values of other two color elements whose values other than one color element value are unknown. And
The three primary color signals obtained by the color demosaicing process for each pixel in a predetermined pixel region unit composed of a pixel to be processed in the three primary color signals and surrounding pixels centering on the pixel to be processed Color difference signal calculating means for calculating a first color difference signal that is a color difference signal between red and green and a second color difference signal that is a color difference signal between blue and green from the values of the three primary color elements;
Each pixel in the predetermined pixel area unit is sorted in the order of the magnitudes of the values of the first and second color difference signals, and the sorting order is a middle range that includes a middle order. Sorting processing means for outputting a plurality of values of the first color difference signal and a plurality of values of the second color difference signal respectively located within
Of the plurality of values of the first color difference signal and the plurality of values of the second color difference signal output from the sorting processing means, when the pixel to be processed is a central pixel, the central pixel is the Bayer A pixel type classification means for classifying into a plurality of types of pixel types according to which color element value of the three primary color elements of the array is a pixel;
When the pixel type classification unit classifies the center pixel as a pixel type having a red color element value, the known red value in the center pixel is output from the sorting processing unit. A pixel type in which an updated value of a green color element is calculated from the value of the first color difference signal of which the sorting order is the middle order, and the color element of the central pixel has a value of a blue color element Is classified from the value of the second primary color signal which is known in the center pixel and the value of the second color difference signal having the sorting order output from the sorting processing means as the center order. First update value calculation means for calculating an update value of a green color element;
When the pixel type classification unit classifies the color element of the central pixel as a pixel type having a value other than the red color element, based on a plurality of values of the first color difference signal, A plurality of candidate values for updating the value of the red color element are calculated, and the pixel type classification unit classifies the central pixel as a pixel type having a value other than the blue color element. And a candidate value calculation means for calculating a plurality of candidate values for updating the value of the blue color element based on a plurality of values of the second color difference signal.
A plurality of candidate values for updating the value of the red color element calculated by the candidate value calculating means and a plurality of candidate values for updating the value of the blue color element are each set as the center of the processing target. Compared with the value of the color element of the pixel surrounding the pixel and having the same color element value as the color element of the central pixel to be processed, the difference value is calculated. A difference calculating means;
One or more red color element values whose difference values calculated using a plurality of candidate values for updating the red color element values by the difference calculation means are less than a predetermined threshold are updated. The update value of the red color element is calculated by the cumulative average value of the candidate values for performing the calculation, and the difference calculation means calculates the plurality of candidate values for updating the value of the blue color element. Second update value calculation for calculating an update value of the blue color element based on a cumulative average value of candidate values for updating the value of one or more blue color elements whose difference value is less than a predetermined threshold value Means,
An image processing apparatus comprising: When all the difference values calculated by using the plurality of candidate values for updating the value of the red color element by the difference calculation means are equal to or larger than the threshold value, the difference is output from the sorting processing means. A first update value in which a value obtained by adding the first color difference value of the central sorting order and the update value calculated by the first update value calculation means is an update value of the red color element Setting means;
When all the difference values calculated by the difference calculation means using a plurality of candidate values for updating the value of the blue color element are equal to or greater than the threshold value, the difference is output from the sorting processing means. A value obtained by adding a value of the second color difference signal having a sorting order of the middle order and the update value calculated by the second update value calculation means is used as an update value of the blue color element. The image processing apparatus according to claim 1, further comprising update value setting means. For each pixel constituting an image signal output from a solid-state image sensor having a Bayer array color filter provided on the light receiving surface, among the three primary color elements of red, blue, and green in the Bayer array, its own pixel is Image processing that causes a computer to perform image processing on three primary color signals that have undergone color demosaicing, which is a process of interpolating the values of two other color elements whose values other than the value of one color element that is unknown A program for
The computer,
The three primary color signals obtained by the color demosaicing process for each pixel in a predetermined pixel region unit composed of a pixel to be processed in the three primary color signals and surrounding pixels centering on the pixel to be processed Color difference signal calculating means for calculating a first color difference signal that is a color difference signal between red and green and a second color difference signal that is a color difference signal between blue and green from the values of the three primary color elements;
Each pixel in the predetermined pixel area unit is sorted in the order of the magnitudes of the values of the first and second color difference signals, and the sorting order is a middle range that includes a middle order. Sorting processing means for outputting a plurality of values of the first color difference signal and a plurality of values of the second color difference signal respectively located within
Of the plurality of values of the first color difference signal and the plurality of values of the second color difference signal output from the sorting processing means, when the pixel to be processed is a central pixel, the central pixel is the Bayer A pixel type classification means for classifying into a plurality of types of pixel types according to which color element value of the three primary color elements of the array is a pixel;
When the pixel type classification unit classifies the center pixel as a pixel type having a red color element value, the known red value in the center pixel is output from the sorting processing unit. A pixel type in which an updated value of a green color element is calculated from the value of the first color difference signal of which the sorting order is the middle order, and the color element of the central pixel has a value of a blue color element Is classified from the value of the second primary color signal which is known in the center pixel and the value of the second color difference signal having the sorting order output from the sorting processing means as the center order. First update value calculation means for calculating an update value of a green color element;
When the pixel type classification unit classifies the color element of the central pixel as a pixel type having a value other than the red color element, based on a plurality of values of the first color difference signal, A plurality of candidate values for updating the value of the red color element are calculated, and the pixel type classification unit classifies the central pixel as a pixel type having a value other than the blue color element. And a candidate value calculation means for calculating a plurality of candidate values for updating the value of the blue color element based on a plurality of values of the second color difference signal.
A plurality of candidate values for updating the value of the red color element calculated by the candidate value calculating means and a plurality of candidate values for updating the value of the blue color element are each set as the center of the processing target. Compared with the value of the color element of the pixel surrounding the pixel and having the same color element value as the color element of the central pixel to be processed, the difference value is calculated. A difference calculating means;
One or more red color element values whose difference values calculated using a plurality of candidate values for updating the red color element values by the difference calculation means are less than a predetermined threshold are updated. The update value of the red color element is calculated by the cumulative average value of the candidate values for performing the calculation, and the difference calculation means calculates the plurality of candidate values for updating the value of the blue color element. Second update value calculation for calculating an update value of the blue color element based on a cumulative average value of candidate values for updating the value of one or more blue color elements whose difference value is less than a predetermined threshold value Means,
An image processing program characterized by being made to function. The computer,
When all the difference values calculated by using the plurality of candidate values for updating the value of the red color element by the difference calculation means are equal to or larger than the threshold value, the difference is output from the sorting processing means. A first update value in which a value obtained by adding the first color difference value of the central sorting order and the update value calculated by the first update value calculation means is an update value of the red color element Setting means;
When all the difference values calculated by the difference calculation means using a plurality of candidate values for updating the value of the blue color element are equal to or greater than the threshold value, the difference is output from the sorting processing means. A value obtained by adding a value of the second color difference signal having a sorting order of the middle order and the update value calculated by the second update value calculation means is used as an update value of the blue color element. 4. The image processing program according to claim 3, wherein the program further functions as update value setting means.

Images