JP2014082782A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress false colors and moire generated in a high-frequency region.SOLUTION: An image processing device extracts a plurality of image signals from an image signal so that frequency components contained in the respective image signals are at least partially different from each other, determines whether a region of interest of the image signal is a high-frequency region, and when the region of interest is not determined as a high-frequency region, outputs an image signal containing a higher-frequency component or an image signal obtained by combining the plurality of image signals while more largely weighting an image signal containing a high-frequency component than when the region of interest is determined as a high-frequency region.

Description

  The present invention relates to an image processing apparatus capable of reducing the influence of noise in a high frequency region.

  Conventionally, as a method of suppressing false colors generated in a high frequency region of a subject, color interpolation of a pixel of interest is performed using pixels around the pixel of interest to be subjected to color interpolation.

  As shown in FIG. 16, an example of processing for performing color interpolation on an image signal output from a Bayer array pixel in which red (R), green (G), and blue (B) filters are arranged will be described. .

  In FIG. 16, the G filter positioned in the horizontal direction of the R filter and in the vertical direction of the B filter is a G1 filter, and the G filter positioned in the vertical direction of the R filter and in the horizontal direction of the B filter is a G2 filter. . The signals output from the pixels corresponding to the G1, G2, R, and B filters are G1sig, G2sig, Rsig, and Bsig.

  This image processing device separates an image signal into an image signal composed of G1 sig, an image signal composed of G2 sig, an image signal composed of Rsig, and an image signal composed of Bsig. Interpolation processing is performed so that all pixels have signals in the image signal.

  Then, the image processing apparatus obtains the correlation degree between the G1 sig or G2 sig value of the pixel adjacent to the top and bottom of the target pixel that is the target of color interpolation and the correlation degree between the G1 sig or G2 sig value of the pixel adjacent to the left and right. When the correlation in the horizontal direction is higher than the correlation in the vertical direction, the image processing apparatus sets the value of Rsig-G1sig as the color difference signal RG and sets the value of Bsig-G2sig as the color difference signal BG. And On the contrary, when the correlation degree in the vertical direction is higher than the correlation degree in the horizontal direction, the value of Rsig-G2sig is set as the color difference signal RG, and the value of Bsig-G1sig is set as the color difference signal BG (patent) Reference 1).

  Alternatively, the image processing apparatus uses the weighting coefficient corresponding to the difference between the Rsig-G1 sig value and the Rsig-G2 sig value to add and average the value of the Rsig-G1 sig value and the Rsig-G2 sig value to the color difference signal R- G. Similarly, a color difference signal B-G is obtained by averaging the values of Bsig-G1sig and Bsig-G2sig using a weighting coefficient corresponding to the difference between the value of Bsig-G1sig and the value of Bsig-G2sig ( (See Patent Document 2).

  In this way, processing for reducing false colors has been performed by adaptively selecting pixels used for color interpolation and generating color difference signals.

  Similarly, when generating a luminance signal, moire is generated by adaptively selecting pixels to be used for interpolation according to the correlation between pixels in the horizontal direction and vertical direction of the pixel of interest, and generating an image signal. Has been performed so that the image is suppressed and the image does not feel uncomfortable (see Patent Document 3).

JP 2002-300590 A Japanese Patent Laid-Open No. 08-023541 JP 2007-336384 A

  However, even with the above processing, the correlation between the horizontal direction and the vertical direction cannot be obtained correctly for an area including an image signal near the Nyquist frequency or an image signal including aliasing noise. In some cases, false colors and moire remained. Therefore, it is considered that there is room for improvement for further reducing false colors and moire.

  In order to solve the above-described problems, the present invention differs from the image signal and the determination means for determining whether or not the region of interest of the image signal is a high frequency region, and at least a part of the frequency component included in each image signal is different from the image signal. In this way, the extraction means for extracting a plurality of image signals and the discriminating means, when the area of interest is not determined to be the high frequency area, the higher frequency component than when the area of interest is determined to be the high frequency area. The image processing apparatus includes a processing unit that outputs an image signal including a plurality of image signals by increasing the weighting of the image signal including the image signal including the high frequency component.

  According to the present invention, it is possible to provide an image processing apparatus capable of improving the effect of suppressing false colors and moire generated in a high frequency region.

1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment of the present invention. It is a figure which shows the one part area | region which consists of a 3 * 3 pixel among the image pick-up elements comprised by the Bayer arrangement | sequence. FIG. 2 is a block diagram illustrating a configuration of a color difference signal generation circuit 104 according to the first embodiment of the present invention. 5 is a diagram for explaining processing by a color interpolation circuit 300. FIG. It is a figure which shows an example of a digital filter. It is a figure which shows an example of the frequency characteristic of LPF301-LPF304, LPF351-LPF353. It is a figure which shows the to-be-photographed subject with a gradation. It is a figure which shows the value of G1sig and G2sig in the image signal obtained by image | photographing the to-be-photographed object of FIG. It is a figure which shows CZP (circular zone plate). It is a figure which shows the value of G1sig and G2sig after interpolation located on the axis | shaft extended horizontally from the center part of CZP. It is a flowchart which shows the selection process of the color difference signal by the selection circuit 320 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the color difference signal generation circuit 104 in the 2nd Embodiment of this invention. It is a flowchart which shows the selection process of the color difference signal by the selection circuit 320 in the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the color difference signal generation circuit 104 in the 3rd Embodiment of this invention. It is a flowchart which shows the synthesis | combination process of the color difference signal by the synthetic | combination circuit 380 in the 3rd Embodiment of this invention. It is a figure which shows a Bayer arrangement | sequence.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to the first embodiment of the present invention.

  The imaging unit 101 includes a photographic lens, an imaging device, and a drive circuit thereof (not shown), and converts an optical image formed by the photographic lens into an electrical signal by the imaging device. This image sensor is composed of a CCD or CMOS sensor, and is composed of a set of pixels in the Bayer array shown in FIG.

  An analog signal output from the imaging unit 101 is converted into a digital signal by the A / D converter 102. The image signal converted into a digital signal by the A / D converter 102 is subjected to a known white balance adjustment by a white balance (WB) circuit 103.

  The image signal output from the white balance circuit 103 is input to the color difference signal generation circuit 104, the correlation degree detection circuit 110, and the luminance signal generation circuit 111. The correlation level detection circuit 110 detects the correlation level between the vertical and horizontal signals for each pixel or region from the image signal. The image signal output from the white balance circuit 103 is subjected to luminance signal processing by the output of the correlation detection circuit 110, the luminance signal generation circuit 111, and the luminance gamma circuit 112, and the luminance signal Y is output for each pixel or region. .

  Further, the color difference signal generation circuit 104 generates and outputs color difference signals RG and BG for each pixel or region from the input image signal. The signal output from the color difference signal generation circuit 104 is converted by the color conversion matrix circuit 105 into color difference signals RG and BG into RY and BY. Thereafter, the color suppression gain is suppressed by the color suppression circuit 106. The output signal of the color suppression circuit 106 passes through a known color gamma circuit 107 and a known chroma knee circuit 108 for adjusting the saturation gain, and a color difference signal UV is output for each pixel or region.

  Here, the luminance interpolation processing performed in the luminance signal generation circuit 111 will be described. FIG. 2 shows a partial region composed of 3 × 3 pixels in the image sensor configured by the Bayer array shown in FIG. 16. In this luminance interpolation process, the signal G1sig output from the pixel corresponding to the G1 filter and the signal G2sig output from the pixel corresponding to the G2 filter are not distinguished from each other and are treated as Gsig.

  First, the luminance signal generation circuit 111 interpolates Gsig for all pixels corresponding to other than the G filter (G1 filter, G2 filter), that is, pixels corresponding to the R filter and the B filter. For example, P1 to P9 in FIG. 2 are codes for identifying pixels, and R, G1, G2, and B indicate color filters to which the pixels P1 to P9 correspond, respectively. P5 (Rsig) indicates the value of Rsig in the pixel P5, P5 (Gsig) indicates the value of Gsig in the pixel P5, and P5 (Bsig) indicates the value of Bsig in the pixel P5.

  When Gsig is interpolated in the pixel P5 corresponding to the B filter, the correlation degree of the upper and lower and left and right signals is detected, and the direction with the higher correlation degree is determined. That is, the absolute value of the difference between the signals of the pixels positioned above and below the pixel to be interpolated and the absolute value of the difference between the signals of the pixels positioned at the left and right of the pixel to be interpolated are obtained.

  The difference between Hdiff and Vdiff is taken as an interpolation direction discrimination signal MatSw.

  If this MatSW is negative, the horizontal correlation is strong, and interpolation is performed from the horizontal pixel signal. That is, the average value of Gsig of the pixel P4 and Gsig of the pixel P6 is set as Gsig of the pixel P5. Conversely, if the difference is positive, interpolation is performed from the pixel signal in the vertical direction. That is, the average value of Gsig of the pixel P2 and Gsig of the pixel P8 is set as Gsig of the pixel P5. In the same manner, Gsig interpolation is performed for pixels corresponding to all R filters and B filters.

  Next, using the interpolated Gsig, Rsig interpolation is performed for pixels corresponding to all G filters and B filters. For the pixels corresponding to the G filter (P2, P4, P6, P8), using the Rsig of the pixel corresponding to the R filter located on the left or right or top and bottom, and the Gsig obtained by interpolation for those pixels, Rsig interpolation is performed. For the pixel corresponding to the B filter, the following expression is used by using the Rsig of the pixel corresponding to the R filter located around the pixel and the Gsig obtained by interpolation for these pixels. Perform interpolation.

  Similarly, Bsig interpolation can also be performed for pixels corresponding to all G filters and R filters. Thus, Rsig, Gsig, and Bsig are obtained for the same pixel. Further, the luminance signal Y is obtained by performing the calculation of the following formula (7) in each pixel.

  In addition, the coefficient of each term of this formula (7) can be changed as appropriate.

  Next, color interpolation processing performed by the color difference signal generation circuit 104 will be described. FIG. 3 is a block diagram showing a configuration of the color difference signal generation circuit 104. The color difference signal generation circuit 104 includes color interpolation circuits 300 and 350, low pass filters (LPF) 301 to 304, 351 to 353, color difference signal calculation circuits 311 to 314, a selection circuit 320, and a false color determination circuit 360. ing.

  The image signal output from the white balance circuit 103 is input to the color interpolation circuit 300. FIG. 4 is a diagram for explaining the interpolation processing by the color interpolation circuit 300. As shown in FIG. 4, the color interpolation circuit 300 decomposes the image signal input in the Bayer array into image signals composed of signals corresponding to the colors of the R, G1, G2, and B filters. At this time, 0 is inserted as a signal value for a pixel that does not correspond to the separated color filter.

  Thereafter, the color interpolation circuit 500 performs interpolation processing by LPF using a known digital filter, and outputs R, G1, G2, and B image signals having signals in all pixels.

  The image signals of the respective colors output from the color interpolation circuit 300 are input to LPFs 301 to 304 having different cutoff frequencies. FIG. 6A shows an example of the frequency characteristics of the LPFs 301 to 304. The LPF 301 has the frequency characteristic 601 in FIG. 6A, the LPF 302 has the frequency characteristic 602 in FIG. 6A, the LPF 303 has the frequency characteristic 603 in FIG. 6A, and the LPF 304 has the frequency characteristic 604 in FIG. As can be seen from FIG. 6A, the cutoff frequency increases in the order of LPF 301, LPF 302, LPF 303, and LPF 304. That is, higher frequency components are included in the image in the order of the image signal S01 output from the LPF 301, the image signal S02 output from the LPF 302, the image signal S03 output from the LPF 303, and the image signal S04 output from the LPF 304. . Each of these image signals S01 to S04 has an image signal of each color of R, G1, G2, and B in which frequency components are limited.

  The image signal S01 is input to the color difference signal calculation circuit 311, the image signal S02 is input to the color difference signal calculation circuit 312, the image signal S03 is input to the color difference signal calculation circuit 313, and the image signal S04 is input to the color difference signal calculation circuit 314. The color difference signal calculation circuits 311 to 314 obtain color difference signals by the method described above for the image signals S01 to S04.

  In other words, the color difference signal calculation circuit 311 refers to the vertical and horizontal correlation degrees of the pixel of interest obtained by the correlation level detection circuit 110, and generates color difference signals RG and BG. Specifically, when the correlation detection circuit 110 determines that the horizontal correlation is higher than the vertical correlation, the color difference signal calculation circuit 311 uses the value of Rsig−G1sig as the color difference signal. Let RG be the value of Bsig-G2sig as the color difference signal BG. On the other hand, when the correlation detection circuit 110 determines that the correlation in the vertical direction is higher than the correlation in the horizontal direction, the color difference signal calculation circuit 311 uses the value of Rsig-G2sig as the color difference signal R−. Let G be the value of Bsig-G1sig as the color difference signal BG.

  Alternatively, the color difference signal calculation circuit 311 uses a weighting coefficient corresponding to the difference between the Rsig-G1 sig value and the Rsig-G2 sig value, and calculates a color difference signal by averaging the Rsig-G1 sig value and the Rsig-G2 sig value. RG. Similarly, using a weighting coefficient corresponding to the difference between the Bsig-G1 sig value and the Bsig-G2 sig value, a value obtained by adding and averaging the Bsig-G1 sig value and the Bsig-G2 sig value is defined as a color difference signal BG.

  These processes are also performed in the color difference signal calculation circuits 312 to 314.

  The color difference signals RG and BG obtained from the color difference signal calculation circuits 311 to 314 are input to the selection circuit 320 described later as the color difference signals S11, S12, S13, and S14, respectively.

  Next, the false color discrimination operation will be described. Image data output from the white balance circuit 103 is input to the color interpolation circuit 350. Similar to the color interpolation circuit 300, the color interpolation circuit 350 outputs image signals of R, G1, G2, and B colors having signals in all pixels.

  Here, in the Bayer array shown in FIG. 16, the number of pixels corresponding to the G filter is twice as large as the number of pixels corresponding to the R filter and the number of pixels corresponding to the B filter. By disassembling this G filter into a G1 filter and a G2 filter, the pixel intervals (sampling intervals) of the R, G1, G2, and B filters become equal. Therefore, if a region where aliasing noise is generated is detected using the G1 and G2 image signals, the region where aliasing noise occurs in the R and B image signals can be specified.

  The image signals of the respective colors output from the color interpolation circuit 350 are input to LPFs 351 to 353 having different cutoff frequencies. FIG. 6B shows an example of frequency characteristics of the LPFs 351 to 353. The LPF 351 has the frequency characteristic 611 in FIG. 6B, the LPF 352 has the frequency characteristic 612 in FIG. 6B, and the LPF 353 has the frequency characteristic 613 in FIG. 6B. As can be seen from FIG. 6B, the cutoff frequency increases in the order of LPF 351, LPF 352, and LPF 353. That is, a higher frequency component is included in the image in the order of the image signal T01 output from the LPF 351, the image signal T02 output from the LPF 352, and the image signal T03 output from the LPF 353. In this embodiment, it is assumed that the LPF 301 and the LPF 351, the LPF 302 and the LPF 352, and the LPF 303 and the LPF 353 have the same frequency characteristics.

  The image signals T01 to T03 output from the LPFs 351 to 353 are input to the false color discrimination circuit 360.

  Here, a method for the false color discrimination circuit 360 to discriminate an area where the false color signal is generated will be described with reference to FIGS. FIG. 7 shows a vertically striped subject having gradation, and FIG. 8 shows values of G1sig and G2sig in an image signal obtained by photographing the subject of FIG.

  The horizontal axis in FIGS. 8A to 8I indicates the position of the pixel in the horizontal direction. FIGS. 8A, 8D, and 8G show G1sig obtained from a pixel corresponding to the G1 filter in one row and G2sig obtained from a pixel corresponding to the G2 filter in another row. Are arranged side by side. In the horizontal direction and the vertical direction, the sampling interval of the pixel corresponding to the G1 filter is equal to the sampling interval of the pixel corresponding to the G2 filter, and the spatial phase of the pixel corresponding to the G1 filter and the pixel corresponding to the G2 filter is the sampling interval. Is shifted by half.

  Since FIG. 7 is a vertical stripe, the subject image is equally incident on all the rows. Therefore, when these G1sig and G2sig are arranged in accordance with the pixel position in the horizontal direction, the image signal indicating the gradation of the subject shown in FIG. Is obtained. Here, in FIG. 8A, it is assumed that the frequency of the vertical stripes shown in FIG. B is sufficiently lower than the Nyquist frequency of the G1 filter and the G2 filter. In FIG. 8D, it is assumed that the frequency of the vertical stripes shown in FIG. B is higher than the frequency of the vertical stripes shown in FIG. A and slightly lower than the Nyquist frequency of the G1 filter and the G2 filter. In FIG. 8G, the frequency of vertical stripes shown in FIG. B is higher than the Nyquist frequency of the G1 filter and the G2 filter.

  FIGS. 8B, 8E, and 8H extract only G1sig of FIGS. 8A, 8D, and 8G, respectively, and correspond to the G1 filter. G1 sig interpolation is performed on a non-existing pixel using surrounding G1 sig. 8 (c), 8 (f), and 8 (i) extract only G2sig of FIGS. 8 (a), 8 (d), and 8 (g), and do not correspond to the G2 filter. G2sig interpolation is performed on the pixel using surrounding G2sig.

  As can be seen from FIGS. 8A to 8I, if the spatial frequency of the subject is sufficiently lower than the Nyquist frequency of the G1 filter and the G2 filter, the phase shift between G1sig and G2sig after interpolation becomes small. . Then, as the spatial frequency of the subject approaches the Nyquist frequency of the G1 filter and the G2 filter, the phase shift between G1sig and G2sig after interpolation increases, and when the Nyquist frequency is exceeded, the phase between G1sig and G2sig after interpolation is almost inverted. It will be in the state. That is, by detecting the difference in phase between G1sig and G2sig after interpolation, it is possible to determine whether or not the region is a high frequency region, that is, whether or not it is a region where a false color is generated.

  Here, the operation of the false color discrimination circuit 360 will be described with reference to FIGS.

  FIG. 9 is a diagram showing a CZP (circular zone plate). Concentric circles with the center of the image as the origin are arranged in layers, and the spatial frequency increases from the center toward the outside. FIG. 10 is a plot of G1sig and G2sig values after interpolation located on an axis extending horizontally from the center of CZP.

  In FIG. 10A, a curve 1001 indicates G1sig after interpolation of the image signal T01, and a curve 1002 indicates G2sig after interpolation of the image signal T01. In FIG. 10B, a curve 1003 indicates G1sig after interpolation of the image signal T02, and a curve 1004 indicates G2sig after interpolation of the image signal T02. In FIG. 10C, a curve 1005 indicates G1sig after interpolation of the image signal T03, and a curve 1006 indicates G2sig after interpolation of the image signal T03. Note that F0 to F4 indicate CZP spatial frequencies, where the spatial frequency at the center of the image is F0, and the spatial frequencies gradually increase in the order of F1, F2, F3, and F4 toward the outside of the image.

  The false color discriminating circuit 360 uses the difference in characteristics between the image signal composed of G1 sig after interpolation and the image signal composed of G2 sig after interpolation to determine whether or not it is a high frequency region where a false color signal is generated. Specifically, focusing on the inclination phase of the image signal composed of G1sig after interpolation and the image signal composed of G2sig after interpolation, this is a false color region where a false color signal is generated when the following equation (8) is satisfied. Is determined.

  Here, ΔG1h represents a horizontal inclination in the image signal composed of G1sig after interpolation, and ΔG2h represents a horizontal inclination in the image signal composed of G2sig after interpolation. Further, ΔG1v represents a vertical inclination in the image signal composed of G1sig after interpolation, and ΔG2v represents a vertical inclination in the image signal composed of G2sig after interpolation. For example, the horizontal inclination can be obtained with a known digital filter as shown in FIG. 5A and the vertical direction as shown in FIG. 5B. Note that the filter for obtaining the slope is not limited to this.

  Note that the false color region may be determined not in units of pixels but in units of a region of interest composed of a plurality of pixels. When discriminating by the region of interest, when the ratio of the image signal inclination obtained using each pixel included in the region satisfies the expression (8) is equal to or greater than the threshold, the region is false color You may make it discriminate | determine that it is an area | region. In the following description, the region of interest includes both a region composed of a plurality of pixels and a region composed of a single pixel.

  In this embodiment, from FIG. 10A, in the image signal T01, it can be determined that regions 2 to 4 in which the spatial frequency is higher than F1 in FIG. 9 are false color regions. Further, from FIG. 10B, in the image signal T02, it can be determined that the region 3 and the region 4 in which the spatial frequency is higher than F2 in FIG. 9 are false color regions. Further, from FIG. 10C, in the image signal T03, it can be determined that the region 4 in which the spatial frequency is higher than F3 in FIG. 9 is a false color region.

  The selection circuit 320 selects the color difference signal S11 generated from the image signal having the same frequency as the image signal T01 for the region 1 determined not to be a false color region in the image signal T01. In addition, the selection circuit 320 is determined from the image signal having the same frequency as that of the image signal T02 for the area 2 that is determined to be a false color area in the image signal T01 but is not determined to be a false color area in the image signal T02. The selected color difference signal S12 is selected. In addition, the selection circuit 320 is determined to be a false color area in the image signal T02, but the area 3 determined to be not a false color area in the image signal T03 is generated from an image signal having the same frequency as the image signal T03. The selected color difference signal S13 is selected. Then, the selection circuit 320 selects the color difference signal S14 generated from the image signal having a frequency lower than that of the image signal T03 for the region 4 determined to be a false color region in the image signal T03. In this way, the selection circuit 320 can select a color difference signal that does not include a false color for each pixel.

  In the present embodiment, the LPF 301 and the LPF 351, the LPF 302 and the LPF 352, and the LPF 303 and the LPF 353 have the same frequency characteristics, but the present invention is not limited to this.

  For example, in order to increase the probability that a false color does not occur in a region of the image signal S01 that is determined not to be a false color region in the image signal T01, the upper limit of the frequency included in the image signal S01 is set to the image signal T01. What is necessary is just to make it lower than the upper limit of the frequency contained in. That is, the cutoff frequency of the LPF 351 is set slightly higher than the cutoff frequency of the LPF 301. Similarly, the cutoff frequency of LPF 352 is set higher than the cutoff frequency of LPF 302, and the cutoff frequency of LPF 353 is set slightly higher than the cutoff frequency of LPF 303. However, in such a case, there is a possibility that the area where the color resolution is lowered may increase as compared with the case where the frequency characteristics of LPF 301 and LPF 351, LPF 302 and LPF 352, and LPF 303 and LPF 353 are matched.

  On the other hand, if it is desired to reduce the area where the color resolution is lowered, the cutoff frequencies of the LPF 301, LPF 302, and LPF 303 may be set slightly lower than the cutoff frequencies of the corresponding LPF 351, LPF 352, and LPF 353. However, in this case, for example, there is a possibility that the image signal S01 includes a higher frequency than the image signal T01. Therefore, even in an area that is determined not to be a false color area in the image signal T01, a false color area in the image signal S01. Is likely to be.

  Whether the cut-off frequencies of LPF 301 to LPF 303 and the corresponding LPF 351 to LPF 353 coincide with each other or whether a difference is provided may be adaptively changed depending on what kind of image is desired to be generated. Alternatively, since the degree of influence by reducing the resolution varies depending on the pattern or pattern of the subject, the cutoff frequency of these LPFs may be adaptively changed according to the subject.

  As described above, in the present embodiment, the color difference signal generation circuit 104 generates a plurality of image signals S01 to S04 that are transmitted through the LPFs 301 to 304 having different cutoff frequencies, and hierarchically forms the plurality of image signals S01 to S04. Color difference signals S11 to S14 are generated. Similarly, the color difference signal generation circuit 104 generates a plurality of image signals T01 to T03 that have passed through LPFs 351 to 353 having different cutoff frequencies, and detects a false color region for each of these different image signals. The selection circuit 320 selects any one of the color difference signals S11 to S14 according to which resolution of the image signals T01 to T03 the false color area is detected from. Therefore, the selection circuit 320 can select, for each pixel or region, a color difference signal generated from an image signal of a hierarchy having the highest frequency in a range that does not become a false color.

  By doing this, the color resolution is reduced by the amount necessary to suppress the false color only in the area where the false color occurs, and the color resolution does not need to be lowered in the area where the false color does not occur. It is possible to achieve both an improvement in resolution and suppression of false colors with an excellent balance.

  FIG. 11 is a flowchart showing a color difference signal selection process by the selection circuit 320. The selection circuit 320 performs this selection process in units of pixels or regions. Here, it is assumed that the selection circuit 320 performs this selection process in units of pixels.

  In step S1101, the selection circuit 320 determines whether or not the pixel of the image signal T01 corresponding to the pixel of interest has been determined to be a false color region by the false color determination circuit 360. If it is determined that the target pixel is not a false color region, the selection circuit 320 proceeds to step S1104, and selects the color difference signal S11 of the pixel corresponding to the target pixel as the color difference signal of the target pixel.

  If it is determined in step S1101 that the target pixel is not a false color area, the selection circuit 320 proceeds to step S1102. In step S1102, the selection circuit 320 determines whether or not the pixel of the image signal T02 corresponding to the pixel of interest has been determined to be a false color region by the false color determination circuit 360. If it is determined that the target pixel is not a false color area, the selection circuit 320 proceeds to step S1105, and selects the color difference signal S12 of the pixel corresponding to the target pixel as the color difference signal of the target pixel.

  If it is determined in step S1102 that the pixel of interest is not a false color area, the selection circuit 320 proceeds to step S1103. In step S1103, the selection circuit 320 determines whether or not the pixel of the image signal T03 corresponding to the pixel of interest has been determined to be a false color region by the false color determination circuit 360. If it is determined that the pixel of interest is not a false color region, the selection circuit 320 proceeds to step S1106, and selects the color difference signal S13 of the pixel corresponding to the pixel of interest as the color difference signal of the pixel of interest.

  If it is determined in step S1103 that the pixel of interest is not a false color area, the selection circuit 320 proceeds to step S1107, and selects the color difference signal S14 of the pixel corresponding to the pixel of interest as the color difference signal of the pixel of interest.

  Then, the selection circuit 320 performs color difference signal selection processing for all the pixels of the output image signal, and the color difference signals RG included in any of the color difference signals S11 to S14 selected for each pixel. BG is output. The output color difference signals RG and BG are input to the color conversion matrix circuit 105 and converted into signals RY and BY.

  In the present embodiment, Expression (8) is used to determine whether the area is a false color area, but the present invention is not limited to this. For example, paying attention to the magnitudes of ΔG1h, ΔG1v, ΔG2h, and ΔG2v, if any of the following formulas (9) to (12) is satisfied, it is determined that the area is a false color area. Also good. Further, the threshold value TH in these equations may be different from each other in the image signals T01 to T03. By adjusting these values, it is possible to change the standard for determining the false color region and adjust the balance between the improvement of the color resolution and the suppression of the false color.

  Further, in order to determine whether the region is a false color area, attention is paid to the inclination in the horizontal direction and the inclination in the vertical direction, but the present invention is not limited to this, and the inclination in the oblique direction may be noted.

  Further, in order to determine whether the region is a false color region, the gradient in the image signal composed of G1 sig after interpolation and the gradient in the image signal composed of G2 sig after interpolation are used, but the difference between the values of G1 sig and G2 sig May be used. That is, if the difference between the values of G1sig and G2sig is large, it can be determined that the phase is greatly shifted. Specifically, when none of the above formulas (8) to (12) is satisfied, as shown in formula (13), the absolute value of the difference between G1sig and G2sig in the pixel of interest exceeds the threshold value. Alternatively, it may be determined that the region is a false color region.

  However, since the values of G1sig and G2sig are both small when the luminance signal is small, these threshold values may be changed according to the luminance signal of the corresponding pixel generated by the luminance signal generation circuit 111. Good.

  Alternatively, in addition to the determination based on the gradient in the image signal composed of G1 sig after interpolation and the gradient in the image signal composed of G2 sig after interpolation, the determination based on the difference between the values of G1 sig and G2 sig may be combined.

  Further, these determination conditions may be made different between the image signals T01 to T03.

  Furthermore, it may be determined whether or not the pixel of interest is a false color region in consideration of the determination result of not only the pixel of interest but also the false color region of the pixel located in the peripheral region.

  Further, in this embodiment, a color difference signal is generated from an image signal of four layers and a false color area is determined in the image signal of three layers. However, the number of layers is not limited to this. Any color-difference signal may be generated from image signals having different frequencies in the (N + 1) -th layer, and false color regions may be determined in image signals having different frequencies in the N-th layer.

  In the present embodiment, the color interpolation circuit 300 and the color interpolation circuit 350 are described separately. However, as long as the characteristics are common, a single circuit may be used. Similarly, the LPFs 301 to 303 and the LPFs 351 to 353 may be shared by one circuit as long as the characteristics are common.

  In this embodiment, an example of selecting one of color difference signals generated from image signals having different frequencies in a plurality of hierarchies has been described in order to suppress false colors. However, the present invention is not limited to this. is not. By generating a luminance signal from image signals having different frequencies of the N + 1 layer by the same method as described above, and detecting a moire region instead of a false color region by the same method for image signals having different frequencies of the N layer, It is also possible to suppress moire by the luminance signal. That is, this embodiment can obtain an effect not only for false colors due to color signals but also for moire due to luminance. In addition, if the processing of this embodiment is performed on an image signal that has been subjected to resizing processing that reduces the number of pixels, it is also possible to suppress moire and false colors that are generated by resizing processing.

  As described above, the image processing apparatus according to the present embodiment, first, from the image signal obtained from the Bayer array image sensor, the image signal of the G1 filter that is the first color filter and the G2 that is the second color filter. The image signal of the filter is extracted and interpolated. Since the G1 filter and the G2 filter have pixels arranged in the same period and their spatial phases are shifted, the image signal after interpolation of the G1 filter and the image signal after interpolation of the G2 filter are in a high frequency region. Then, as shown in FIGS. 8 and 10, the phase is inverted. Therefore, the image processing apparatus determines whether the region of interest (the pixel of interest) is a high-frequency region from at least one of the inclination of the image signals of the G1 filter and the G2 filter and the difference between the image signals of the G1 filter and the G2 filter. Can be determined. When it is determined that the pixel of interest is in the high frequency region, the image processing apparatus uses a predetermined signal such as a color difference signal or a luminance signal from an image signal in which the cutoff frequency is lowered to a level at which the pixel of interest is not determined to be in the high frequency region. By generating a signal, it is possible to suppress false colors and moire.

  In addition, the image processing device leaves the generation of predetermined signals such as the color difference signal and the luminance signal to the subsequent device, and a flag indicating the determination result as to whether or not the target pixel is in the high frequency region is included in the image signal. You may make it set.

(Second Embodiment)
The second embodiment is different from the first embodiment in that, when the selection circuit 320 selects a color difference signal, whether or not the pixel of interest is an edge region is added to the determination. In the first embodiment, the selection circuit 320 selects a color difference signal generated from an image signal having a lower frequency when the false color determination circuit 360 determines that it is a false color region. As a result, if the area determined to be a false color area is an edge area, the color of the edge area may appear blurred.

  Since the configuration of the image processing apparatus in the present embodiment is the same as that of the image processing apparatus in the first embodiment, description thereof is omitted. In this embodiment, a part of the configuration of the color difference signal generation circuit 104 of FIG. 1 is different from that of the first embodiment.

  FIG. 12 is a block diagram showing a configuration of the color difference signal generation circuit 104 in the present embodiment. In the color difference signal generation circuit 104 according to the present embodiment, an edge determination circuit 370 to which image signals T01 to T03 output from the LPFs 351 to 353 are input is added to the configuration of the color difference signal generation circuit shown in FIG. . The edge region discrimination result by the edge discrimination circuit 370 is input to the selection circuit 320.

  The edge discriminating circuit 370 obtains Gsig for all pixels from the G1 image signal and the G2 image signal included in the image signal T01 output from the LPF 351 using Expression (14).

  Then, paying attention to the horizontal and vertical inclinations of the image signal composed of Gsig, when the following equation (15) is satisfied, it is determined that the target pixel is an edge region, and the determination result is output to the selection circuit 320. To do.

  Here, ΔGh represents a horizontal inclination in the image signal composed of Gsig after interpolation, and ΔGv represents a horizontal inclination in the image signal composed of G2 sig after interpolation. These inclinations can be obtained by a known digital filter as shown in FIG. 5A and the vertical direction as shown in FIG. 5B. Note that the filter for obtaining the slope is not limited to this. Further, the edge region may be determined not in units of pixels but in units of regions composed of a plurality of pixels. When discriminating by region unit, when the ratio of the image signal inclination obtained using each pixel included in the region that satisfies Equation (15) is equal to or greater than the threshold value, the region is an edge region. You may make it discriminate | determine that there exists.

  Similarly, the edge discriminating circuit 370 discriminates the edge region for the image signal T02 output from the LPF 352 and the image signal T03 output from the LPF 353.

  Here, although an example using Gsig from G1sig and G2sig has been described as a method for discriminating the edge region, the present invention is not limited to this method, and a luminance signal generated from Rsig, Gsig, and Bsig using Equation (7) is used. It may be used.

  The false color discriminating circuit 360 discriminates the false color area by the same processing as that of the first embodiment, and outputs the discrimination result to the selection circuit 320.

  FIG. 13 is a flowchart showing the color difference signal selection processing by the selection circuit 320 in this embodiment. The selection circuit 320 performs this selection process in units of pixels or regions. Here, it is assumed that the selection circuit 320 performs this selection process in units of pixels.

  In step S1301, the selection circuit 320 determines that the pixel of the image signal T01 corresponding to the pixel of interest is a false color area by the false color determination circuit 360, and determines that the pixel is not an edge area by the edge determination circuit 370. Judge whether or not. If the false color discriminating circuit 360 discriminates the false color region, the selection circuit 320 proceeds to step S1304 and selects the color difference signal S11 of the pixel corresponding to this pixel of interest as the color difference signal of the pixel of interest. To do. Alternatively, if the edge determination circuit 370 determines that the region is an edge region, the selection circuit 320 proceeds to step S1304 and selects the color difference signal S11 of the pixel corresponding to the target pixel as the color difference signal of the target pixel. To do. That is, even if it is determined that the pixel of interest is a false color area, the selection circuit 320, if it is determined that the target pixel is an edge area, gives priority to suppressing the color blur of the edge area. Select S11.

  If the selection circuit 320 determines that the area is a false color area in the false color determination circuit 360 and the edge determination circuit 370 determines that the area is not an edge area, the process proceeds to step S1302. In step S1302, the selection circuit 320 determines that the pixel of the image signal T02 corresponding to the pixel of interest is a false color area by the false color determination circuit 360, and determines that the pixel is not an edge area by the edge determination circuit 370. Judge whether or not. If the false color discriminating circuit 360 discriminates that it is a false color area, the selection circuit 320 proceeds to step S1305 and selects the color difference signal S12 of the pixel corresponding to this pixel of interest as the color difference signal of the pixel of interest. To do. Alternatively, if the edge determination circuit 370 determines that the region is an edge region, the selection circuit 320 proceeds to step S1305 and selects the color difference signal S12 of the pixel corresponding to the target pixel as the color difference signal of the target pixel. To do. That is, even if it is determined that the pixel of interest is a false color area, the selection circuit 320 gives priority to suppressing the color blur of the edge area if it is determined that it is an edge area. Select.

  If the selection circuit 320 determines that the area is a false color area in the false color determination circuit 360 and the edge determination circuit 370 determines that the area is not an edge area, the process proceeds to step S1303. In step S1303, the selection circuit 320 determines that the pixel of the image signal T03 corresponding to the pixel of interest is a false color area by the false color determination circuit 360, and determines that the pixel is not an edge area by the edge determination circuit 370. Judge whether or not. If it is determined by the false color determination circuit 360 that the region is a false color area, the selection circuit 320 proceeds to step S1306, and selects the color difference signal S13 of the pixel corresponding to the target pixel as the color difference signal of the target pixel. To do. Alternatively, if the edge determination circuit 370 determines that the region is an edge region, the selection circuit 320 proceeds to step S1306 and selects the color difference signal S13 of the pixel corresponding to the target pixel as the color difference signal of the target pixel. To do. That is, even if it is determined that the pixel of interest is a false color region, the selection circuit 320 gives priority to suppressing the color blur of the edge region if it is determined that it is an edge region. Select.

  On the other hand, if the selection circuit 320 determines that the area is a false color area in the false color determination circuit 360 and determines that the area is not an edge area in the edge determination circuit 370, the selection circuit 320 proceeds to step S1307 and outputs the color difference signal S14. select.

  As described above, in this embodiment, even when the false color determination circuit 360 determines that the area is a false color area, if the area is an edge area, the area is determined not to be an edge area. A color difference signal generated from an image signal of a higher hierarchy than the frequency is selected.

  By doing in this way, in addition to the effect acquired by the structure of 1st Embodiment, it also becomes possible to suppress that the color looks blurred in an edge area | region.

(Third embodiment)
In the third embodiment, the color difference signal generation circuit 104 does not output any one of the color difference signals S11 to S14, but outputs a color difference signal obtained by synthesizing these color difference signals S11 to S14. Different from the embodiment.

  Since the configuration of the image processing apparatus in the present embodiment is the same as that of the image processing apparatus in the first embodiment, description thereof is omitted. In this embodiment, a part of the configuration of the color difference signal generation circuit 104 of FIG. 1 is different from that of the first embodiment.

  FIG. 14 is a block diagram showing the configuration of the color difference signal generation circuit 104 in this embodiment. The color difference signal generation circuit 104 according to the present embodiment includes a synthesis circuit 380 instead of the selection circuit 320 illustrated in FIG.

  FIG. 15 is a flowchart showing the color difference signal synthesis processing by the synthesis circuit 380 in this embodiment. The synthesis circuit 380 performs this selection process in units of pixels or areas. Here, it is assumed that the synthesis circuit 380 performs this selection process in units of pixels. The combining circuit 380 combines the color difference signals S11 to S14 by changing the weighting of the color difference signals S11 to S14 according to which of the image signals T01 to T03 the false color area is detected.

  In step S1501, the synthesis circuit 380 determines whether the pixel of the image signal T01 corresponding to the pixel of interest is determined to be a false color area by the false color determination circuit 360. If it is determined by the false color determination circuit 360 that the region is not a false color region, the composition circuit 380 proceeds to step S1504, and assigns the weight of the color difference signal S11 of the pixel corresponding to this pixel of interest to the weight of another color difference signal. Set to be the maximum compared to.

  If it is determined in step S1501 that the false color determination circuit 360 determines that the area is a false color area, the synthesis circuit 380 proceeds to step S1502. In step S1502, the synthesis circuit 380 determines whether the pixel of the image signal T02 corresponding to the pixel of interest has been determined to be a false color area by the false color determination circuit 360. If it is determined by the false color determination circuit 360 that the region is not a false color region, the composition circuit 380 proceeds to step S1505 and assigns the weight of the color difference signal S12 of the pixel corresponding to this pixel of interest to the weight of other color difference signals. Set to be the maximum compared to.

  If it is determined in step S1502 that the false color determination circuit 360 determines that the area is a false color area, the composition circuit 380 proceeds to step S1503. In step S1503, the synthesis circuit 380 determines whether the pixel of the image signal T03 corresponding to the pixel of interest has been determined to be a false color area by the false color determination circuit 360. If it is determined by the false color determination circuit 360 that the region is not a false color region, the composition circuit 380 proceeds to step S1506 and assigns the weight of the color difference signal S13 of the pixel corresponding to this pixel of interest to the weight of other color difference signals. Set to be the maximum compared to

  On the other hand, if the false color discriminating circuit 360 discriminates that it is a false color area, the composition circuit 380 proceeds to step S1507, and assigns the weight of the color difference signal S14 of the pixel corresponding to this pixel of interest to other values. The maximum value is set in comparison with the weighting of the color difference signal.

  The synthesizing circuit 380 synthesizes the color difference signals S11 to S14 according to any of the following formulas according to the weighting set in steps S1504 to S1507.

  The combining circuit 380 selects Expression (16) when the weight of the color difference signal S11 is set to the maximum. Similarly, the synthesis circuit 380 selects Expression (17), Expression (18), and Expression (19), respectively, when the weights of the color difference signals S12, S13, and S14 are set to the maximum. Then, the synthesis circuit 380 outputs the color difference signals RG and BG obtained by this synthesis processing to the color conversion matrix circuit 105.

  Note that, similarly to the second embodiment, it may be considered whether the target pixel is an edge region. That is, even if the target pixel is determined to be a false color region, if the target pixel is determined to be an edge region, the image having the highest cutoff frequency among the image signals determined to be the edge region The weighting of the color difference signal generated from the signal may be maximized.

  The combining circuit 380 may change the weighting of the color difference signals S11 to S14 according to the false chromaticity detected from the image signals T01 to T03. The false chromaticity is obtained from ΔG1h, ΔG1v, ΔG2h, ΔG2v obtained from G1sig after interpolation and G2sig after interpolation, and a difference between values of G1sig and G2sig. That is, the composition circuit 380 sets the false chromaticity so that the value becomes larger as the phase shift between the interpolated G1sig and the interpolated G2sig is estimated to be close to 180 degrees. Then, the synthesis circuit 380 can set the weights of the color difference signals S11 to S14 according to the false chromaticity of the image signals T01 to T03.

(Other embodiments)
The object of the present invention can also be achieved by the following method. That is, a storage medium (or recording medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but the present invention includes the following cases. That is, based on the instruction of the program code, an operating system (OS) or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, the following cases are also included in the present invention. That is, the program code read from the storage medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, a CPU or the like provided in the function expansion card or function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the above storage medium, the storage medium stores program codes corresponding to the procedure described above.

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 A / D converter 103 White balance (WB) circuit 104 Color difference signal production circuit 105 Color conversion matrix circuit 106 Color suppression circuit 107 Color gamma circuit 108 Chromany circuit 110 Correlation detector 111 Luminance signal generation circuit 112 Luminance gamma Circuit 300, 350 Color interpolation circuit 301, 302, 303, 304, 351, 352, 353 Low-pass filter (LPF)
311, 312, 313, 314 color difference signal calculation circuit 320 selection circuit 360 false color discrimination circuit 370 edge discrimination circuit 380 synthesis circuit

Claims (12)

A discriminating means for discriminating whether or not the region of interest of the image signal is a high-frequency region;
Extraction means for extracting a plurality of image signals from the image signals so that at least some of the frequency components included in the respective image signals are different;
When the discriminating means does not discriminate that the region of interest is a high frequency region, it has processing means for outputting an image signal containing a higher frequency component than when the region of interest is discriminated to be a high frequency region. An image processing apparatus.   The processing unit determines, for each of the plurality of image signals extracted by the extracting unit, whether or not the region of interest is determined to be a high-frequency region by the determining unit. The image processing apparatus according to claim 1, wherein an image signal including the highest frequency component among the image signals that are not determined to be in a high frequency region is output. Detecting means for determining whether the region of interest is an edge region;
The processing means determines, for each of the plurality of image signals extracted by the extraction means, whether or not the region of interest is determined to be a high-frequency region by the determination means, and the detection means It is determined whether or not the region of interest is determined to be an edge region. If the region of interest is determined to be an edge region in any of the plurality of image signals, the region of interest is an edge region. The image processing apparatus according to claim 1, wherein an image signal including the highest frequency component among the image signals determined as is output. A discriminating means for discriminating whether or not the region of interest of the image signal is a high-frequency region;
Extraction means for extracting a plurality of image signals from the image signals so that at least some of the frequency components included in the respective image signals are different;
When the discriminating unit does not discriminate that the region of interest is a high frequency region, the weight of an image signal including a higher frequency component is increased than when it is discriminated that the region of interest is a high frequency region. An image processing apparatus comprising processing means for combining and outputting a plurality of image signals.   The processing unit determines, for each of the plurality of image signals extracted by the extracting unit, whether or not the region of interest is determined to be a high-frequency region by the determining unit. Among the image signals that are not determined to be in the high frequency region, the weighting of the image signal including the highest frequency component is set larger than the image signal of another cutoff frequency, and the plurality of image signals are synthesized. The image processing apparatus according to claim 4. Detecting means for determining whether the region of interest is an edge region;
The processing means determines, for each of the plurality of image signals extracted by the extraction means, whether or not the region of interest is determined to be a high-frequency region by the determination means, and the detection means It is determined whether or not the region of interest is determined to be an edge region. If the region of interest is determined to be an edge region in any of the plurality of image signals, the region of interest is an edge region. 5. The plurality of image signals are synthesized by weighting an image signal including the highest frequency component among image signals determined to be larger than an image signal having another cutoff frequency. Image processing apparatus.   The image signal is obtained from an image pickup device in which a plurality of color filters are arranged in a predetermined arrangement and pixels corresponding to the respective color filters are provided. The image processing apparatus according to any one of the above. The image sensor that has generated the image signal is one in which red, blue, and green color filters are arranged in a Bayer array,
The determination means includes an image signal obtained from a first green color filter arranged in a horizontal direction of the red color filter and arranged in a vertical direction of the blue color filter, and the red color Using the image signal from the second green color filter arranged in the vertical direction of the filter and in the horizontal direction of the blue color filter, it is determined whether or not the region of interest is a high frequency region. The image processing apparatus according to claim 7. A determination step of determining whether or not the region of interest of the image signal is a high-frequency region;
An extraction step of extracting a plurality of image signals from the image signals so that at least some of the frequency components included in the respective image signals are different;
And a processing step of outputting an image signal including a higher frequency component when the region of interest is not determined to be a high frequency region in the determination step than when the region of interest is determined to be a high frequency region. An image processing method. A determination step of determining whether or not the region of interest of the image signal is a high-frequency region;
An extraction step of extracting a plurality of image signals from the image signals so that at least some of the frequency components included in the respective image signals are different;
When the region of interest is not determined to be a high frequency region in the determination step, the weight of an image signal including a higher frequency component is increased than when the region of interest is determined to be a high frequency region. An image processing method comprising a processing step of combining and outputting a plurality of image signals.   A program for causing a computer of an image processing apparatus to execute the image processing method according to claim 9 or 10. A computer-readable recording medium in which the program according to claim 11 is stored.