JP2012100215A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of performing correction processing of defective pixels and processing for improving image quality with high accuracy at a high speed.SOLUTION: An image processing device comprises: an image processing unit that performs weighted addition of color information of a first image in which each pixel has color information of any one color component of plural color components based on the pixel position of a target pixel to calculate the value of a color component different from the color component of the target pixel, thereby generating a second image in which the each pixel has plural color components; and a storage unit that stores information of the pixel positions of defective pixels in the first image. The image processing unit prepares plural coefficient patterns for weighted addition of the color information of the first image based on the pixel position of the target pixel, and based on the color component of the target pixel and the pixel positions of the defective pixels, selects a coefficient pattern and performs weighted addition.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  2. Description of the Related Art Conventionally, an image sensor such as a CCD or CMOS has a light receiving element that does not operate normally as a defective pixel due to scratches generated during or after manufacture. An image picked up by an image pickup device having such a defective pixel deteriorates in image quality due to the defective pixel. Thus, various techniques have been developed to avoid image quality degradation due to defective pixels.

  For example, by detecting a change in the pixel value in the reference area centered on the defective pixel, the optimum direction for correcting the pixel value of the defective pixel is determined, and the pixel pair arranged in that direction is determined in the reference area. There is a technique of selecting from the above and correcting the pixel value of the defective pixel (see Patent Document 1, etc.).

JP 2005-175547 A

  By the way, each pixel of an image captured by an image sensor in which three color filters of red (R), green (G), and blue (B) are arranged at predetermined positions (for example, a Bayer array) is Or only one color component color information. Therefore, in the prior art, in order to improve the image quality, after performing interpolation processing of defective pixels, processing such as empty grid interpolation and color conversion is performed so that each pixel has color information of all three color components. There is a problem that processing takes time.

  In view of the above-described problems of the conventional technology, an object of the present invention is to provide a technology capable of performing defective pixel correction processing and high image quality processing with high accuracy and high speed.

  In order to solve the above problems, the image processing apparatus according to the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the target pixel. And calculating a value of a color component different from the color component of the pixel of interest and generating a second image in which each pixel has a plurality of color components, and storing information on the pixel position of the defective pixel in the first image And a storage unit, wherein the image processing unit prepares a plurality of coefficient patterns for weighted addition of the color information of the first image at the pixel position of the target pixel, and based on the color component of the target pixel and the pixel position of the defective pixel The coefficient pattern is selected and weighted.

  The image processing apparatus of the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the pixel of interest, The image processing unit calculates a value of different color components and generates a second image in which each pixel has a plurality of color components. The first image is added with pixel position information of defective pixels, and the image processing unit A plurality of coefficient patterns for weighting and adding the color information of the first image at the pixel position of the target pixel are prepared, and the coefficient pattern is selected and weighted and added based on the color component of the target pixel and the pixel position of the defective pixel.

  In addition, the image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel, and selects a coefficient pattern based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. May be.

  In addition, when the pixel adjacent to the target pixel is a defective pixel, the image processing unit may determine the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. .

  In addition, the coefficient pattern may include a value of zero or more that weights and adds the color information of the pixels in the minimum region including the color information of a plurality of color components with the pixel position of the target pixel as the center.

  The image processing apparatus according to the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the target pixel by a variable coefficient of zero or more. And calculating a value of a color component different from the color information of the target pixel, generating an image processing unit that generates a second image in which each pixel has a plurality of color components, and storing information on the pixel position of the defective pixel in the first image And a storage unit, wherein the image processing unit changes a coefficient based on the color component of the target pixel and the pixel position of the defective pixel, and performs weighted addition of the color information of the target pixel and the color information of the pixel adjacent to the target pixel. .

  The image processing apparatus according to the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the target pixel by a variable coefficient of zero or more. A color component value different from the color information of the pixel of interest, and an image processing unit that generates a second image in which each pixel has a plurality of color components. The first image is provided with position information of a defective pixel. The image processing unit changes the coefficient based on the color component of the target pixel and the pixel position of the defective pixel, and weights and adds the color information of the target pixel and the color information of the pixel adjacent to the target pixel.

  Further, the image processing unit may determine the strength of similarity in a plurality of directions at the pixel position of the target pixel, and change the coefficient based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. Good.

  In addition, when the pixel adjacent to the target pixel is a defective pixel, the image processing unit may determine the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. .

  Further, the image processing unit may always add the weight information of the color information of the first image at a constant color component ratio in all the pixels of the first image.

  In addition, when the first image includes a first color component having a high pixel density, a second color component having a low pixel density, and a third color component, the image processing unit includes the color information and the third color of the second color component. The component color information may be weighted and added at the same color component ratio.

  The image processing apparatus according to the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the target pixel by a variable coefficient of zero or more. Second color component values different from the color information of the pixel of interest are calculated, and the filter processing is performed with a predetermined fixed filter coefficient to correct different color components, and each pixel has a plurality of color components. An image processing unit that generates an image, and a storage unit that stores information on the pixel position of the defective pixel in the first image, and the image processing unit calculates a coefficient based on the color component of the target pixel and the pixel position of the defective pixel. change.

  The image processing apparatus according to the present invention weights and adds the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the target pixel by a variable coefficient of zero or more. By calculating a value of a color component different from the color information of the target pixel and performing a filtering process with a predetermined fixed filter coefficient, the value of the different color component is corrected, and each pixel has a plurality of color components. An image processing unit that generates a second image is provided. Position information of the defective pixel is added to the first image, and the image processing unit changes the coefficient based on the color component of the pixel of interest and the pixel position of the defective pixel.

  Further, the image processing unit may determine the strength of similarity in a plurality of directions at the pixel position of the target pixel, and change the coefficient based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. Good.

  In addition, when the pixel adjacent to the target pixel is a defective pixel, the image processing unit may determine the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. .

  The image processing unit may use a filter coefficient including positive and negative values as a predetermined fixed filter coefficient.

  Further, the different color components may be a luminance component and a color difference component.

  Further, the image processing unit, when the first image has a first color component, a second color component, and a third color component, a first similarity component composed of the first color component and the second color component, A second similarity component composed of the second color component and the third color component, a third similarity component composed of the third color component and the first color component, and a first similarity component composed of only the first color component. Similarity in a plurality of directions using at least one similarity component of a fourth similarity component, a fifth similarity component composed of only the second color component, and a sixth similarity component composed of only the third color component The degree of similarity may be determined for a plurality of directions based on the degree of similarity.

  The imaging device of the present invention includes an imaging element and the image processing device of the present invention.

  An image processing program according to the present invention includes an input procedure for reading a first image in which each pixel has color information of any one of a plurality of color components and information on the pixel position of a defective pixel is added; An image processing procedure for calculating the value of a color component different from the color component of the target pixel by weight-adding the color information of one image at the pixel position of the target pixel, and generating a second image in which each pixel has a plurality of color components; The image processing procedure prepares a plurality of coefficient patterns for weighted addition of the color information of the first image at the pixel position of the target pixel, and based on the positional relationship between the color component of the target pixel and the defective pixel. The coefficient pattern is selected and weighted.

  According to the present invention, defective pixel correction processing and high image quality processing can be performed with high accuracy and high speed.

Functional block diagram of electronic camera 1 The figure which shows the arrangement of the color component of Bayer arrangement type image data Flowchart of image restoration processing by image processing unit in one embodiment Operation Flowchart of Luminance Component Calculation Processing by Image Processing Unit of One Embodiment (Part 1) Operation Flowchart of Luminance Component Calculation Processing by Image Processing Unit of One Embodiment (Part 2) Operation Flowchart of Luminance Component Calculation Processing by Image Processing Unit of One Embodiment (Part 3) Flowchart of Color Difference Component Calculation Processing by Image Processing Unit of One Embodiment Diagram explaining peripheral addition Diagram showing an example of coefficient pattern The figure which shows the modification of each coefficient pattern of FIG. 9 when an attention pixel is a defective pixel Diagram showing an example of coefficient pattern The figure which shows an example of a band pass filter Diagram showing an example of coefficient pattern The figure which shows the modification of the coefficient pattern of FIG. The figure which shows the modification of the coefficient pattern 11 of FIG. 12 when there exists a defective pixel The figure which shows the modification of the coefficient pattern 12 of FIG. 12 when there exists a defective pixel The figure which shows the modification of the coefficient pattern 13 of FIG. 12 when there exists a defective pixel The figure which shows an example of the filter which interpolates a color difference component The figure which shows another example of the filter which interpolates a color difference component The figure which shows the data flow of one Embodiment Operational Flowchart of Luminance Component Calculation Processing by Image Processing Unit in Modification of Embodiment (Part 1) Operation flow chart of luminance component calculation processing by image processing unit in modified example of embodiment (No. 2)

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  However, in the embodiment of the present invention, the description will be made using an electronic camera having a function of improving the image quality of an image performed by the image processing apparatus of the present invention.

  FIG. 1 is a functional block diagram of an electronic camera 1 corresponding to an embodiment of the present invention.

  In FIG. 1, the electronic camera 1 includes a DFE 10, an image processing unit (for example, a one-chip microprocessor dedicated to image processing) 11, a CPU 12, a memory 13, and a display interface (display I / F) 14. The electronic camera 1 is connected to an external device such as a media I / F 17 that realizes an interface with a memory card (card-like removable memory) 16, a PC (personal computer) 18 via a predetermined cable or a wireless transmission path. An input / output I / F 19 that implements an interface is provided. These components of the electronic camera 1 are connected via a bus so as to be able to transmit information to each other.

  The electronic camera 1 also includes an imaging optical system 20, an imaging element 21, and a timing generator (TG) 22. The image sensor 21 captures an optical image formed through the imaging optical system 20 and outputs an image signal to the DEF 10. The DFE 10 is a digital front-end circuit that performs predetermined signal processing such as A / D conversion on an image signal output from the image sensor 21. The DEF 10 outputs the digitized image signal to the image processing unit 11 via the bus. The TG 22 controls the operation timing for each of the image sensor 21, the DEF 10, and the image processing unit 11 based on a control instruction from the CPU 12.

  The electronic camera 1 also includes an operation unit 23 having a release button, a mode switching selection button, and the like, and outputs an operation instruction from the operator received by the operation unit 23 to the CPU 12. The electronic camera 1 includes a monitor 15 that displays a display image generated and output by the display I / F 14.

  A monitor 24, a printer 25, and the like are connected to the PC 18, and an application program recorded on the CD-ROM 26 is installed in advance. In addition to the CPU, memory, and hard disk (not shown), the PC 18 is connected to the electronic camera 1 via a memory card I / F (not shown) that implements an interface with the memory card 16, a predetermined cable, and a wireless transmission path. And an external I / F (not shown) that implements an interface with an external device.

  In the electronic camera 1 having the configuration shown in FIG. 1, when the shooting mode is selected by the operator via the operation unit 23 and the release button is pressed, the CPU 12 performs timing control on the image sensor 21 and the DEF 10 via the TG 22. Do. The image sensor 21 generates an image signal corresponding to the optical image. The image signal is subjected to predetermined signal processing such as A / D conversion in the DEF 10 and is supplied to the image processing unit 11 as Raw image data. The raw image data is digital image data in which the relationship between the output signal value and the amount of incident light is linear.

  A plurality of light receiving elements are arranged in a matrix on the image pickup surface of the image pickup element 21 of the present embodiment, and RGB color filters are arranged in each light receiving element according to a known Bayer array. Raw image data captured by the image sensor 21 and supplied to the image processing unit 11 is represented by an RGB color system, and each pixel has an R component (second color component or third color component), G component (first color component). Color information of any one of the color component) and the B component (third color component or second color component). Such Raw image data is referred to as “Bayer array image data”.

  In addition, the imaging element 21 of the present embodiment is assumed to have a light receiving element that does not operate normally as a defective pixel due to a scratch or the like generated during or after manufacture. Defective pixels include “white defects” that output a large value image signal and “black defects” that do not output an image signal. The defective pixels in Bayer array image data correspond to white defects or black defects. It shall have a pixel value. In the present embodiment, it is assumed that pixel position information (address, etc.) of defective pixels in the image sensor 21 is stored in advance in the memory 13 (storage unit) as pixel position information of defective pixels in the image.

  The image processing unit 11 inputs such Bayer array image data (first image), and converts luminance components and color difference components, which are color components of the YCbCr color system different from the RGB color system, into all pixels. And an image (second image) composed of the color components of the luminance component and the color difference component is generated. The image processing unit 11 generates a luminance plane in which luminance components are associated with all pixels and a color difference plane in which chrominance components are associated with all pixels by sky lattice processing to improve image quality. Realize. Further, the image processing unit 11 performs a color conversion process for calculating an RGB plane from the luminance plane and the color difference plane generated as described above, as necessary. As will be described later, the image processing unit 11 according to the present embodiment calculates the luminance component and the color difference component in each pixel (target pixel) while considering the pixel position of the defective pixel. Such a series of processing is referred to as “image restoration processing”.

  Note that the image processing unit 11 performs image processing such as white balance processing and gradation conversion when performing image restoration processing on the Bayer array type image data output from the DFE 10. In addition, the image processing unit 11 performs compression processing on the Bayer array type image data that has undergone image restoration processing or the like as necessary, and records the compressed data in the memory card 16 via the media I / F 17.

  When the playback mode is selected by the operator via the operation unit 23, the image data recorded on the memory card 16 is read out to the image processing unit 11 via the media I / F 17, and the image processing unit 11 Is displayed on the monitor 15 via the display I / F 14.

  The decompressed image data is not displayed on the monitor 15 but is converted to the color system adopted by the monitor 24 or the printer 26 on the PC 18 side, and is sent to the PC 18 via the input / output I / F 19. It may be supplied. However, the process of converting the RGB plane generated by the image restoration process into the color system adopted by the monitors 15 and 24 and the printer 25 can be realized by a known technique and will not be described.

  Further, the image processing unit 11 records the Bayer array type image data for which image processing has been completed on the memory card 16 without performing compression processing, or uses the color system adopted by the monitor 24 or the printer 25 on the PC 18 side. It may be converted and supplied to the PC 18 via the input / output I / F 19.

  FIG. 2 is a diagram showing an arrangement of color components of Bayer array type image data. That is, in FIG. 2, the types of color components are indicated using R, G, and B, and the positions of pixels corresponding to the respective color components are indicated using the values of coordinates [i, j].

  FIG. 2 (1) shows a case where a pixel at coordinates [i, j] is an R component target pixel, and FIG. 2 (2) shows a pixel at coordinates [i, j] as a B component target pixel. FIG. 2 (3) shows a case where the pixel at coordinates [i, j] is the target pixel of the G component. Among the G component pixels, a pixel in which the R component is present in the horizontal direction is denoted as “Gr”, and a pixel in which the B component is present in the horizontal direction is denoted as “Gb”.

In the arithmetic expression described later, when the color information in each pixel is distinguished from the G component and other color components, R and B in FIG. 2 are replaced with Z, and the color information of the R component or B component pixel is displayed. Is expressed as Z [i, j], and the color information of the G component pixel is expressed as G [i, j]. Further, when the color information in each pixel is not distinguished by the color component, R, G, and B in FIG. 2 are replaced with A, and the color information of an arbitrary pixel is expressed as A [i, j].
<< One Embodiment >>
3 to 7 are operation flowcharts of the image processing unit 11 according to the embodiment of the present invention.

3 to 7 show the operation of the image restoration process among the image processes performed by the image processing unit 11. 3 shows a rough operation of the image restoration process, FIGS. 4 to 6 show an operation of “luminance component calculation process” included in the image restoration process, and FIG. 7 shows a “color difference” included in the image restoration process. Operation of “component calculation process” will be described. In the following, among the image processing performed by the image processing unit 11, the operation of the image restoration processing will be described, and description of other operations will be omitted.
1. Setting processing of index HV indicating vertical / horizontal similarity The image processing unit 11 performs similarity in the vertical direction and horizontal direction of pixels lacking the G component (hereinafter referred to as “vertical / horizontal similarity”) with respect to Bayer array image data. And an index HV indicating vertical and horizontal similarity is set based on the determination result (S1 in FIG. 3).

That is, the image processing unit 11 calculates the same-color similarity Cv0 in the vertical direction and the same-color similarity Ch0 in the horizontal direction defined by the following expressions 1 and 2 for the target pixel Z from which the G component is missing. Further, the image processing unit 11 calculates the vertical different-color similarity CvN0 and the horizontal different-color similarity ChN0 defined by the following equations 3 and 4 for the target pixel Z from which the G component is missing. Note that the image processing unit 11 according to the present embodiment outputs a defective pixel if any of the pixel of interest and the pixel in the range shown in FIG. The pixel value to be used is assumed to be used as it is.
・ Similarity between same colors:

・異色間類似度：

・ Similarity between different colors:

なお、本実施形態の画像処理部１１は、注目画素だけでなく、注目画素の周辺に位置する画素に対しても同様の値を算出し、各値を方向別に加重加算（以下、「周辺加算」と称する）して、注目画素の縦方向および横方向に対する最終的な同色間類似度および異色間類似度を算出する。すなわち、画像処理部１１は、座標［ｉ，ｊ］，［ｉ−１，ｊ−１］，［ｉ＋１，ｊ−１］，［ｉ−１，ｊ＋１］，［ｉ＋１，ｊ＋１］，［ｉ，ｊ−２］，［ｉ，ｊ＋２］，［ｉ−２，ｊ］，［ｉ＋２，ｊ］に位置する画素に対して、式１〜式４の演算を行う。そして、画像処理部１１は、算出した値を用いて、次式５〜式８による周辺加算を行い、縦方向に対する同色間類似度Ｃｖ［ｉ，ｊ］、横方向に対する同色間類似度Ｃｈ［ｉ，ｊ］、縦方向に対する異色間類似度ＣｖＮ［ｉ，ｊ］、横方向に対する異色間類似度ＣｈＮ［ｉ，ｊ］をそれぞれ算出する。

Note that the image processing unit 11 according to the present embodiment calculates similar values not only for the target pixel but also for pixels located around the target pixel, and performs weighted addition of each value for each direction (hereinafter, “peripheral addition”). The final similarity between same colors and similarity between different colors in the vertical and horizontal directions of the target pixel are calculated. That is, the image processing unit 11 has coordinates [i, j], [i−1, j−1], [i + 1, j−1], [i−1, j + 1], [i + 1, j + 1], [i, Calculations of Expressions 1 to 4 are performed on the pixels located at j-2], [i, j + 2], [i-2, j], and [i + 2, j]. Then, the image processing unit 11 uses the calculated values to perform peripheral addition according to the following formulas 5 to 8, and the same-color similarity Cv [i, j] in the vertical direction and the same-color similarity Ch [ i, j], the different color similarity CvN [i, j] in the vertical direction, and the different color similarity ChN [i, j] in the horizontal direction are calculated.

However, the similarity calculated by the following formulas 5 to 8 indicates that the smaller the value, the stronger the similarity. Further, the following Expressions 5 to 8 correspond to performing peripheral addition using the coefficient pattern shown in FIG.
・ Similarity between same colors:

・異色間類似度：

・ Similarity between different colors:

ここで、図２（１）のように、座標［ｉ，ｊ］に位置する注目画素がＲ成分の場合、同色間類似度Ｃｖ０，Ｃｈ０は、Ｇ成分の色情報のみから成るＧＧ間類似度成分（第４類似度成分）とＢ成分の色情報のみから成るＢＢ間類似度成分（第６類似度成分または第５類似度成分）とから算出される。一方、周辺加算後に得られる同色間類似度Ｃｖ，Ｃｈには、ＧＧ間類似度成分とＢＢ間類似度成分との他に、Ｒ成分の色情報のみから成るＲＲ間類似度成分（第５類似度成分または第６類似度成分）が含まれる。また、座標［ｉ，ｊ］に位置する注目画素がＲ成分の場合、異色間類似度ＣｖＮ０，ＣｈＮ０は、Ｇ成分の色情報とＲ成分の色情報とから成るＧＲ間類似度成分（第１類似度成分または第３類似度成分）を用いて算出される。一方、周辺加算後に得られる異色間類似度ＣｖＮ，ＣｈＮには、ＧＲ間類似度成分の他に、Ｇ成分の色情報とＢ成分の色情報とから成るＧＢ間類似度成分（第３類似度成分または第１類似度成分）が含まれる。すなわち、式５〜式８の周辺加算により、複数の色成分を考慮し、かつ、周辺の画素との連続性を考慮しつつ類似度が算出され、類似度の精度が高められる。

Here, as shown in FIG. 2A, when the target pixel located at the coordinates [i, j] is the R component, the same-color similarity Cv0, Ch0 is the GG similarity composed only of the G-component color information. It is calculated from the component (fourth similarity component) and the BB similarity component (sixth similarity component or fifth similarity component) consisting only of the B component color information. On the other hand, in the same-color similarity Cv and Ch obtained after the peripheral addition, in addition to the GG similarity component and the BB similarity component, an RR similarity component (fifth similarity) including only R-component color information is included. Degree component or sixth similarity component). Further, when the pixel of interest located at the coordinates [i, j] is an R component, the different color similarity CvN0, ChN0 is an inter-GR similarity component (first component) composed of G component color information and R component color information. (Similarity component or third similarity component). On the other hand, the similarity between colors CvN and ChN obtained after the peripheral addition includes, in addition to the similarity component between GRs, a similarity component between GBs (third similarity) composed of G component color information and B component color information. Component or first similarity component). That is, by the peripheral addition of Equations 5 to 8, the similarity is calculated in consideration of a plurality of color components and the continuity with surrounding pixels, and the accuracy of the similarity is improved.

  By the way, in Expression 3 and Expression 4, the term enclosed by the absolute value is composed of color information of two adjacent pixels. Therefore, the similarity between different colors calculated by Equation 3 and Equation 4 enables similarity determination in the fine structure of the Nyquist frequency level that changes at one pixel pitch. In addition, since the similarity between different colors is calculated on the assumption that all the color information of different color components represents the same luminance information, the determination of the strength of similarity using the similarity between different colors is performed. High reliability in achromatic parts.

  On the other hand, the determination of the strength of similarity using the similarity between the same colors is generally reliable for both the chromatic and achromatic parts, but when the similarity between the different colors is used in the detailed part of the image Compared to reliability. Therefore, in order to determine the similarity with high reliability with respect to the entire image, it is preferable to divide the entire image into an achromatic portion and a chromatic portion and use a similarity suitable for each portion.

  In order to determine whether or not the image near the pixel of interest is an achromatic color part, a color index indicating the presence or absence of local color is required. As such a color index, local color difference information is considered. It is done. Since the similarity between different colors calculated as described above reflects local color difference information as well as the strength of similarity, the similarity between different colors can be directly used as a color index.

  However, the similarity between different colors indicates that the smaller the value is, the stronger the similarity is. Therefore, if the similarity between different colors in the vertical and horizontal directions is a large value, the achromatic part is similar in both vertical and horizontal directions. This means that the image is weak or the image in the vicinity of the target pixel is a colored portion. On the other hand, if the similarity between different colors in at least one of the vertical direction and the horizontal direction is a relatively small value, the image in the vicinity of the target pixel is an achromatic color portion, and there is a direction with strong similarity. Means that

Therefore, the image processing unit 11 determines the threshold values ThNv and ThNh.
CvN [i, j] ≦ ThNv or ChN [i, j] ≦ ThNh ... Condition 1
Is satisfied, it is determined that the image in the vicinity of the target pixel is an achromatic portion, and if the condition 1 is not satisfied, the image in the vicinity of the target pixel is determined to be a chromatic portion. However, the threshold values ThNv and ThNh are set to values of about 10 or less when the gradation is 0 to 255.

When the image near the target pixel is an achromatic part, the image processing unit 11
| CvN [i, j] −ChN [i, j] | ≦ Th0 ... Condition 2
It is determined whether or not holds. This condition 2 is a condition for determining whether or not the vertical different-color similarity CvN [i, j] and the horizontal different-color similarity ChN [i, j] are approximately the same. When the difference between the vertical different-color similarity CvN [i, j] and the horizontal different-color similarity ChN [i, j] is very small, the threshold Th0 is strongly similar to one another due to the influence of noise. It plays a role of avoiding being erroneously determined. Therefore, for a color image with a lot of noise, the accuracy of similarity determination can be further improved by setting the threshold Th0 to a large value.

Therefore, when the condition 2 is satisfied, the image processing unit 11 determines that the target pixel is weak (or strong) in both vertical and horizontal directions, and sets “0” to the index HV [i, j] indicating the vertical and horizontal similarity. "Is set. On the other hand, when the condition 2 is not satisfied, the image processing unit 11
CvN [i, j] <ChN [i, j] ... Condition 3
It is determined whether or not holds. Then, when the condition 3 is satisfied, the image processing unit 11 determines that the target pixel is highly similar in the vertical direction, and sets “1” to the index HV [i, j] indicating the vertical and horizontal similarity. On the other hand, if the condition 3 does not hold, the image processing unit 11 determines that the pixel of interest has strong similarity in the horizontal direction, and sets “−1” to the index HV [i, j] indicating the vertical and horizontal similarity. To do.

In addition, when the image in the vicinity of the target pixel is a coloring portion, the image processing unit 11
| Cv [i, j] −Ch [i, j] | ≦ Th1 Condition 4
It is determined whether or not holds. Condition 4 is a condition for determining whether or not the same-color similarity Cv [i, j] in the vertical direction and the similar-color similarity Ch [i, j] in the horizontal direction are approximately the same. When the difference between the same-color similarity Cv [i, j] in the vertical direction and the similar-color similarity Ch [i, j] in the horizontal direction is very small, the threshold Th1 is strongly similar to one another due to the influence of noise. It plays a role of avoiding being erroneously determined. Similar to the threshold Th0, the threshold Th1 is set to a large value for a noisy color image, thereby improving the accuracy of similarity determination.

Therefore, when the condition 4 is satisfied, the image processing unit 11 determines that the target pixel is weak (or strong) in both vertical and horizontal directions, and sets “0” to the index HV [i, j] indicating the vertical and horizontal similarity. "Is set. On the other hand, if the condition 4 does not hold, the image processing unit 11
Cv [i, j] <Ch [i, j] ... Condition 5
It is determined whether or not holds. Then, when the condition 5 is satisfied, the image processing unit 11 determines that the target pixel has a strong similarity in the vertical direction, and sets “1” to the index HV [i, j] indicating the vertical and horizontal similarity. On the other hand, if the condition 5 does not hold, the image processing unit 11 determines that the pixel of interest has strong similarity in the horizontal direction, and sets “−1” to the index HV [i, j] indicating the vertical and horizontal similarity. To do.

  Note that the image processing unit 11 of the present embodiment does not consider whether each pixel including the target pixel is a defective pixel or not in the calculations of Expressions 1 to 8 when setting the index HV. You may go. For example, the image processing unit 11 reads the pixel position information of the defective pixel from the memory 13 and performs the calculation of Expressions 1 to 4, and the pixel G [i, j−1] or G [i, j + 1], or It is determined whether the pixel G [i−1, j] or G [i + 1, j] is a defective pixel. When the pixel G [i, j−1] or G [i, j + 1] is a defective pixel, the image processing unit 11 performs the index HV [i indicating the vertical and horizontal similarity without performing the calculations of Formulas 1 to 4. , J] is set to “1”. On the other hand, when the pixel G [i−1, j] or G [i + 1, j] is a defective pixel, the image processing unit 11 performs an index HV indicating the vertical and horizontal similarity without performing the calculations of Expressions 1 to 4. “−1” is set in [i, j].

  In addition, the image processing unit 11 calculates the pixel position of the defective pixel in the memory 13 among the same-color similarity Cv0 and Ch0 and the different-color similarity CvN0 and ChN0 at the pixel position of each pixel calculated by Expressions 1 to 4. Based on the above information, the peripheral addition of Expressions 5 to 8 may be performed without using the similarity between the same colors and the similarity between different colors including the defective pixel.

As a result, the accuracy of similarity can be improved and the processing speed can be increased.
2. Setting processing of index DN indicating oblique similarity Next, the image processing unit 11 performs similarity to the 45 ° oblique direction and the oblique 135 ° direction (hereinafter referred to as “belonging to the oblique 135 ° direction” in pixels lacking the G component with respect to the Bayer array type image data. The diagonal similarity ") is determined, and an index DN indicating the diagonal similarity is set according to the determination result (S2 in FIG. 3).

  That is, the image processing unit 11 calculates the similarity C45_0 in the oblique 45 degree direction and the similarity C35_0 in the oblique 135 degree direction defined by the following expressions 9 and 10 for the target pixel Z from which the G component is missing.

例えば、座標［ｉ，ｊ］に位置する注目画素がＲ成分の場合、類似度Ｃ４５_０および類似度Ｃ３５_０は、順に、ＧＧ間類似度成分、ＢＢ間類似度成分、Ｇ成分の色情報とＲ成分の色情報とから成るＲＢ間類似度成分（第２類似度成分）で構成される。

For example, when the target pixel located at the coordinates [i, j] is an R component, the similarity C45_0 and the similarity C35_0 are, in order, the GG similarity component, the BB similarity component, the G component color information, and the R component. RB similarity component (second similarity component) consisting of the color information.

  Then, the image processing unit 11 performs peripheral addition according to the following equations 11 and 12, and calculates the similarity C45 [i, j] with respect to the oblique 45 ° direction and the similarity C135 [i, j] with respect to the oblique 135 ° direction. . However, the following expressions 11 and 12 correspond to performing peripheral addition using the coefficient pattern shown in FIG.

これにより、縦横方向の場合と同様に、複数の色成分が考慮され、かつ周辺の画素との連続性が考慮されるため、類似度の精度が高められる。

As a result, as in the case of the vertical and horizontal directions, a plurality of color components are taken into account and continuity with surrounding pixels is taken into account, so that the accuracy of similarity is increased.

Next, the image processing unit 11 determines the threshold Th2.
| C45 [i, j] −C135 [i, j] | ≦ Th2 Condition 6
It is determined whether or not holds. When the condition 6 is satisfied, the image processing unit 11 determines that the target pixel has low (or strong) similarity in the diagonal direction, and sets “0” to the index DN [i, j] indicating the diagonal similarity. Set. On the other hand, when the condition 6 does not hold, the image processing unit 11
C45 [i, j] <C135 [i, j] ... Condition 7
It is determined whether or not holds.

  When the condition 7 is satisfied, the image processing unit 11 determines that the target pixel has a strong similarity with respect to the 45-degree oblique direction, and sets “1” to the index DN [i, j] indicating the diagonal similarity. On the other hand, when the condition 7 does not hold, the image processing unit 11 determines that the target pixel has a strong similarity with respect to the oblique 135 ° direction, and sets “−1” to the index DN [i, j] indicating the oblique similarity. Set.

Note that, in setting the index DN, the image processing unit 11 of the present embodiment determines whether each pixel including the target pixel is a defective pixel or not in the calculations of Expressions 9 to 12, as in the case of the index HV. You may consider it. In addition, the image processing unit 11 includes defective pixels based on information on the pixel positions of the defective pixels in the memory 13 among the similarities C45_0 and C135_0 at the pixel positions of the pixels calculated by Expression 9 and Expression 10. However, it is also possible to exclude the similarity and perform the peripheral addition of Expression 11 and Expression 12. Thereby, the accuracy of similarity can be improved, and the processing speed can be increased.
3. Luminance Component Calculation Processing Next, the image processing unit 11 performs luminance component calculation processing on each pixel based on the flowcharts shown in FIGS. 4 to 6 to calculate a luminance plane (S3 in FIG. 3).

  In the luminance component calculation processing of the present embodiment, the luminance component is calculated by weight-adding the color information of a plurality of pixels in the target region centered on the target pixel in the Bayer array image data. The color information used for this weighted addition is determined based on the strength of similarity in the target pixel, and the weighted addition coefficient is set so that the ratio of RGB constituting the luminance component is constant. In the present embodiment, in order for the image processing unit 11 to perform such luminance component calculation processing, a plurality of types of coefficients whose coefficients are arranged and their values are changed according to the strength of similarity and the pixel position of the defective pixel. Prepare a pattern.

9 and 10 show an example of a coefficient pattern for generating a luminance component. In each coefficient pattern, the position of color information and the value of the coefficient to be subjected to weighted addition are changed according to the similarity and the pixel position of the defective pixel. That is, the color information of the pixel of interest and the color information of the pixels adjacent to the pixel of interest (including pixels adjacent in the diagonal direction) are weighted and added within the narrowest region including the color information of the three color components of RGB. It has become so. Here, α, β, u ₁ , u ₂ , v ₁ , v ₂ , s ₁ , s ₂ , t ₁ , t ₂ are all zero or more and satisfy the following condition.

α + β = 1, u ₁ + u ₂ = 1, v ₁ + v ₂ = 1, s ₁ + s ₂ = 1, t ₁ + t ₂ = 1
The coefficient patterns shown in FIGS. 9 and 10 are set such that the ratio of RGB constituting the luminance component is “β / 2: α: β / 2”, and the color information of the R component and the B component The color information is weighted and added at the same ratio. That is, the luminance component Y is
Y = α · G + β · (R + B) / 2
Are in a relationship. Accordingly, a common coefficient pattern can be used for the Gr component pixel and the Gb component pixel, and a common coefficient pattern can be used for the R component pixel and the B component pixel.

  Examples of generally preferred constant settings include the following examples.

u ₁ ≈u ₂ , v ₁ ≈v ₂ , s ₁ ≈s ₂ , t ₁ ≈t ₂
(Α, β) = (1/3, 2/3), (4/9, 5/9), (5/11, 6/11), (1/2, 1/2), (5/9 , 4/9), (3/5, 2/5), (2/3, 1/3)
The ratio of RGB constituting the luminance component is not limited to “β / 2: α: β / 2”, but “α: β: γ” (α + β + γ = 1, α ≧ 0, β ≧ 0, γ ≧ 0) or the like. For example, α = 0.3, β = 0.6, and γ = 0.1. In addition, when the RGB ratio is “α: β: γ”, it is necessary to set coefficient patterns individually for the Gr component pixel and the Gb component pixel, and for the R component pixel and the B component pixel, respectively. It is necessary to set the coefficient pattern individually.

  As described above, the image processing unit 11 determines whether or not the target pixel is the G component in the luminance component calculation processing (S11 in FIG. 4).

  When the target pixel is not the G component (NO side in S11 in FIG. 4), the image processing unit 11 includes an index HV [i, j] indicating vertical and horizontal similarity in the target pixel and an index DN [i, j] indicating diagonal similarity. In accordance with the value of FIG. 4 (S12 in FIG. 4), the strength of similarity of the target pixel is classified into one of case1 to case9.

  case1: (HV [i, j], DN [i, j]) = (0,0): The similarity in all directions is strong or the similarity in all directions is weak (the direction with strong similarity is Unknown).

  case2: (HV [i, j], DN [i, j]) = (0,1): The similarity in the oblique 45 degree direction is strong.

  case3: (HV [i, j], DN [i, j]) = (0, −1): Similarity in the oblique 135 degree direction is strong.

  case4: (HV [i, j], DN [i, j]) = (1, 0): Similarity in the vertical direction is strong.

  case5: (HV [i, j], DN [i, j]) = (1,1): Strong similarity in the vertical and oblique 45 degree directions.

  case6: (HV [i, j], DN [i, j]) = (1, -1): Similarity in the vertical and oblique 135 degrees directions is strong.

  case7: (HV [i, j], DN [i, j]) = (− 1, 0): The horizontal similarity is strong.

  case8: (HV [i, j], DN [i, j]) = (− 1,1): The similarity in the horizontal and oblique 45-degree directions is strong.

  case9: (HV [i, j], DN [i, j]) = (− 1, −1): Similar in the horizontal and oblique 135 degree directions.

  In accordance with the classification result, the image processing unit 11 calculates a luminance component. That is, when the target pixel is classified as case 1 and the target pixel is not a defective pixel (NO side in FIG. 4S13), the image processing unit 11 calculates a luminance component using the coefficient pattern 1 (S14 in FIG. 4). On the other hand, when the target pixel is classified as case1 and the target pixel is a defective pixel (YES side in FIG. 4S13), the image processing unit 11 calculates a luminance component using the coefficient pattern 1a (S15 in FIG. 4).

  When the target pixel is classified as case 2 and the target pixel is not a defective pixel (NO side in FIG. 4 S16), the image processing unit 11 calculates a luminance component using the coefficient pattern 2 (S17 in FIG. 4). On the other hand, when the target pixel is classified as case 2 and the target pixel is a defective pixel (YES side in FIG. 4 S16), the image processing unit 11 calculates a luminance component using the coefficient pattern 2a (S18 in FIG. 4).

  When the target pixel is classified as case3 and the target pixel is not a defective pixel (NO side in FIG. 4S19), the image processing unit 11 calculates a luminance component using the coefficient pattern 3 (S20 in FIG. 4). On the other hand, when the target pixel is classified as case 3 and the target pixel is a defective pixel (YES side in FIG. 4 S19), the image processing unit 11 calculates a luminance component using the coefficient pattern 3a (FIG. 4 S21).

  When the target pixel is classified as case4 and the target pixel is not a defective pixel (NO side in FIG. 4S22), the image processing unit 11 calculates a luminance component using the coefficient pattern 4 (S23 in FIG. 4). On the other hand, when the target pixel is classified as case4 and the target pixel is a defective pixel (YES side in FIG. 4S22), the image processing unit 11 calculates a luminance component using the coefficient pattern 4a (S24 in FIG. 4).

  When the target pixel is classified as case 5 and the target pixel is not a defective pixel (NO side in FIG. 4 S25), the image processing unit 11 calculates a luminance component using the coefficient pattern 5 (S26 in FIG. 4). On the other hand, when the target pixel is classified as case5 and the target pixel is a defective pixel (YES side in FIG. 4S25), the image processing unit 11 calculates a luminance component using the coefficient pattern 5a (S27 in FIG. 4).

  When the target pixel is classified as case 6 and the target pixel is not a defective pixel (NO side in FIG. 5 S28), the image processing unit 11 calculates a luminance component using the coefficient pattern 6 (S29 in FIG. 5). On the other hand, when the target pixel is classified as case 6 and the target pixel is a defective pixel (YES side in FIG. 5 S28), the image processing unit 11 calculates a luminance component using the coefficient pattern 6a (S30 in FIG. 5).

  When the target pixel is classified as case 7 and the target pixel is not a defective pixel (NO side in FIG. 5S31), the image processing unit 11 calculates a luminance component using the coefficient pattern 7 (S32 in FIG. 5). On the other hand, when the target pixel is classified as case 7 and the target pixel is a defective pixel (YES side in FIG. 5S31), the image processing unit 11 calculates a luminance component using the coefficient pattern 7a (S33 in FIG. 5).

  When the target pixel is classified as case 8, and the target pixel is not a defective pixel (NO side in FIG. 5S34), the image processing unit 11 calculates a luminance component using the coefficient pattern 8 (S35 in FIG. 5). On the other hand, when the target pixel is classified as case 8 and the target pixel is a defective pixel (YES side in FIG. 5S34), the image processing unit 11 calculates a luminance component using the coefficient pattern 8a (S36 in FIG. 5).

  When the target pixel is classified as case 9, and the target pixel is not a defective pixel (NO side in FIG. 5S37), the image processing unit 11 calculates a luminance component using the coefficient pattern 9 (S38 in FIG. 5). On the other hand, when the target pixel is classified as case 9, and the target pixel is a defective pixel (YES side in FIG. 5S37), the image processing unit 11 calculates a luminance component using the coefficient pattern 9a (S39 in FIG. 5).

  Here, for example, when the coefficient pattern 1 is selected from the 18 kinds of coefficient patterns, the image processing unit 11 calculates the luminance component Y [i, j] as in the following Expression 13.

同様に、他の係数パターンが選択された場合についても、画像処理部１１は輝度成分Ｙ［ｉ，ｊ］を算出する。

Similarly, even when another coefficient pattern is selected, the image processing unit 11 calculates the luminance component Y [i, j].

  Next, the image processing unit 11 shows the luminance component Y [i, j] of the target pixel of the G component according to whether or not it is a defective pixel regardless of the similarity strength as described above. Calculation is performed using the coefficient pattern 10 or 10a.

  Incidentally, the index HV [i−1, j] and the index DN [i−1, j] of the pixel Z [i−1, j] adjacent to the left side of the pixel of interest are the G component and the index DN [i−1, j] is “ When the luminance component is calculated using the coefficient pattern 10 or 10a, a check pattern-like false structure that does not actually exist may occur. Therefore, in order to suppress the occurrence of such a false structure, when calculating the luminance component in the G component pixel, the image processing unit 11 of the present embodiment determines that the target pixel is the G component ( It is determined whether or not the index HV and the index DN of the pixel Z [i−1, j] adjacent to the left side of the target pixel are “0” (YES side in FIG. 4S11) (S40 in FIG. 6).

  That is, when the index HV [i−1, j] and the index DN [i−1, j] are “0” (YES side in FIG. 6 S40), the image processing unit 11 Based on whether the pixel or any of the pixels Z adjacent thereto is a defective pixel, one of coefficient patterns 0 and 0a to 0e for applying a low-pass filter to the G component is selected, and the luminance component Y [i, j] is selected. Is calculated (S46 to S51 in FIG. 6).

Here, in FIG. 11, α, β, e ₁ , e ₂ , e ₃ , e ₄ , e ₅ , f ₁ , f ₂ , f ₃ , and f ₄ are zero or more values, and e ₁ + e ₂ + E ₃ + e ₄ + e ₅ = f ₁ + f ₂ + f ₃ + f ₄ = 1 and e ₃ > e ₁ , e ₂ , e ₄ , e ₅ are satisfied. By changing the values of e ₁ , e ₂ , e ₃ , e ₄ , and e ₅ , the strength of the low-pass filter can be set. For example, the values of e ₁ , e ₂ , e ₃ , e ₄ , e ₅ are set to “1, 1, 6, 1, 1” when applying a strong low-pass filter, and “1” when applying a weak low-pass filter. , 1, 14, 1, 1 "and the like.

In addition, the pixels G [i−1, j−1], G [i−1, j + 1], and G [i + 1] are determined according to the relationship of e ₃ > e ₁ , e ₂ , e ₄ , and e _{5 in} the coefficient patterns 0a to 0e. , J−1] and G [i + 1, j + 1] can be reduced. Therefore, the determination of whether or not these pixels are defective pixels can be omitted, and the speed of image processing by the image processing unit 11 can be increased.

  Also, the coefficient patterns 0a to 0e are set so that when the target pixel or a pixel adjacent thereto is a defective pixel, the image value of the defective pixel is an average value of the pixel values of neighboring pixels having the same color component. Yes.

  For example, when the pixel of interest and the pixel adjacent thereto are not defective pixels (S46 in FIG. 6), the image processing unit 11 selects the coefficient pattern 0 and calculates the luminance component Y [i, j] as the following Expression 14.

同様に、他の係数パターンが選択された場合でも、画像処理部１１は輝度成分Ｙ［ｉ，ｊ］を算出する。

Similarly, even when another coefficient pattern is selected, the image processing unit 11 calculates the luminance component Y [i, j].

On the other hand, when at least one of the index HV [i−1, j] and the index DN [i−1, j] is not “0” (NO side in FIG. 6), the image processing unit 11 determines that the target pixel is a defective pixel. If not (NO side in FIG. 6A41), the luminance component is calculated using the coefficient pattern 10 (S42 in FIG. 6). If the target pixel is a defective pixel (YES side in FIG. 6A41), the luminance component is calculated using the coefficient pattern 10a. Is calculated (S43 in FIG. 6).
4). Luminance Surface Correction Processing Next, the image processing unit 11 performs edge enhancement processing of filter processing with a predetermined fixed filter coefficient on the calculated luminance surface to correct the luminance surface (S4 in FIG. 3).

  For example, as shown in FIG. 12, the image processing unit 11 adds the result obtained by applying a bandpass filter including positive and negative values as filter coefficients to the luminance plane to the original luminance plane, thereby performing edge enhancement processing. I do. That is, the correction of the luminance surface is performed by the calculation according to the following equations 15 and 16.

ただし、式１６において、Ｋは、エッジ強調の強さを可変するための係数であり、１オーダーの値をとる。

However, in Expression 16, K is a coefficient for changing the strength of edge enhancement and takes a value of one order.

Note that the image processing unit 11 may output the luminance plane calculated in this way as a monochrome image.
5. Color Difference Component Calculation Processing Next, the image processing unit 11 performs the color difference component calculation processing shown in FIG. 7 on the R component pixel and the B component pixel, calculates the Cr component at the R component pixel position, and B The Cb component is calculated at the pixel position of the component (S5 in FIG. 3).

  In the color difference component calculation processing of the present embodiment, the color difference component is calculated by directly weight-adding the color information of a plurality of pixels located in the target region centered on the target pixel in the Bayer array image data. The color information used for the weighted addition and the weighted addition coefficient are determined based on the strength of similarity in the pixel of interest. In the present embodiment, a coefficient pattern for performing this color difference component calculation process is prepared in advance and stored in the memory 13.

  13 and 14 show an example of a coefficient pattern used for calculating the color difference component. The coefficient pattern shown in FIG. 14 has a greater effect of suppressing the occurrence of false colors than the coefficient pattern shown in FIG. In the present embodiment, in addition to the coefficient pattern shown in FIG. 13, the coefficient pattern shown in FIGS. 15 to 17 in the case where the pixel of interest or a pixel adjacent thereto is a defective pixel is used.

  The image processing unit 11 according to the present embodiment classifies the following A to C according to the value of the index HV [i, j] indicating the vertical and horizontal similarity of the target pixel (R component pixel or B component pixel). (S60 in FIG. 7), the color difference component is calculated.

A) When HV [i, j] = 1 The image processing unit 11 determines that the target pixel Z [i, j] and the vertically adjacent pixel G [i, j ± 1] are not defective pixels (in FIG. NO side), the color difference component is calculated using the coefficient pattern 11 (S62 in FIG. 7). On the other hand, when the pixel G [i, j-1] is a defective pixel, the image processing unit 11 calculates a color difference component using the coefficient pattern 11a (S63 in FIG. 7), and the pixel G [i, j + 1] is a defective pixel. If it is, the color difference component is calculated using the coefficient pattern 11b (S64 in FIG. 7). If the target pixel Z [i, j] is a defective pixel, the color difference component is calculated using the coefficient pattern 11c (S65 in FIG. 7). .

B) When HV [i, j] = − 1 The image processing unit 11 determines that the target pixel Z [i, j] and the laterally adjacent pixel G [i ± 1, j] are not defective pixels (FIG. 7 S66). And the color difference component is calculated using the coefficient pattern 12 (S67 in FIG. 7). On the other hand, when the pixel G [i−1, j] is a defective pixel, the image processing unit 11 calculates a color difference component using the coefficient pattern 12a (S68 in FIG. 7), and the pixel G [i + 1, j] is a defective pixel. If it is, the color difference component is calculated using the coefficient pattern 12b (S69 in FIG. 7), and if the target pixel Z [i, j] is a defective pixel, the color difference component is calculated using the coefficient pattern 12c (S70 in FIG. 7). .

C) In the case of HV [i, j] = 0 The image processing unit 11 reads the pixel of interest Z [i, j] and the pixels G [i ± 1, j] and G [i, j ± 1] adjacent in the vertical and horizontal directions. Is not a defective pixel (NO side in FIG. 7 S71), a color difference component is calculated using the coefficient pattern 13 (S72 in FIG. 7). On the other hand, when the pixel G [i, j-1] is a defective pixel, the image processing unit 11 calculates a color difference component using the coefficient pattern 13a (S73 in FIG. 7), and the pixel G [i-1, j] If it is a defective pixel, the color difference component is calculated using the coefficient pattern 13b (FIG. 7S74), and if the pixel G [i + 1, j] is a defective pixel, the color difference component is calculated using the coefficient pattern 13c (FIG. 7S75). ) When the pixel G [i, j + 1] is a defective pixel, a color difference component is calculated using the coefficient pattern 13d (S76 in FIG. 7). When the pixel of interest Z [i, j] is a defective pixel, the coefficient pattern 13e is calculated. Is used to calculate the color difference component (S77 in FIG. 7).

  For example, when HV [i, j] = 1, the target pixel and the adjacent pixel are not defective pixels, and the target pixel is an R component, the image processing unit 11 selects the coefficient pattern 11 and selects the color difference component Cr [ i, j] is calculated as in the following equation 17 (S62 in FIG. 7).

また、注目画素がＢ成分である場合、色差成分Ｃｂ［ｉ，ｊ］は、式１７と同様に生成される。

Further, when the target pixel is the B component, the color difference component Cb [i, j] is generated in the same manner as Expression 17.

  When the image processing unit 11 sets the index HV in the target pixel Z [i, j] of the R component or the B component based on the pixel position information of the defective pixel stored in the memory 13, the color difference The component calculation process can be simplified. That is, in FIG. 3S1, the pixel G [i, j−1] or G [i, j + 1], or the pixel G [i−1, j] or G [i + 1, j] is determined as a defective pixel, and the target pixel Z When the index HV [i, j] is set to 1 or −1, the image processing unit 11 only needs to determine whether or not the target pixel Z is a defective pixel in FIG. 7 S61, S66, or S71.

  For example, when HV [i, j] = 1, the image processing unit 11 determines that the pixel of interest Z [i, j] is not a defective pixel based on information on the pixel position of the defective pixel in the memory 13 ( The color difference component is calculated using the coefficient pattern 11 (S62 in FIG. 7) (S62 in FIG. 7). On the other hand, when the image processing unit 11 determines that the target pixel Z [i, j] is a defective pixel (YES side in FIG. 7S61), the image processing unit 11 calculates a color difference component using the coefficient pattern 11c (FIG. 7S65).

  When HV [i, j] = − 1, the image processing unit 11 determines that the target pixel Z [i, j] is not a defective pixel based on the pixel position information of the defective pixel (S66 in FIG. 7). And the color difference component is calculated using the coefficient pattern 12 (S67 in FIG. 7). On the other hand, when the image processing unit 11 determines that the target pixel Z [i, j] is a defective pixel (YES side in FIG. 7 S66), the image processing unit 11 calculates a color difference component using the coefficient pattern 12c (FIG. 7 S70).

When HV [i, j] = 0, the image processing unit 11 determines that the target pixel Z [i, j] is not a defective pixel based on the information on the pixel position of the defective pixel (in FIG. 7, S71). NO side), a color difference component is calculated using the coefficient pattern 13 (S72 in FIG. 7). On the other hand, when the image processing unit 11 determines that the target pixel Z [i, j] is a defective pixel (YES side in FIG. 7S71), the image processing unit 11 calculates a color difference component using the coefficient pattern 13e (FIG. 7S77).
6). CrCb Component Interpolation Processing Next, the image processing unit 11 performs the above-described color difference component calculation processing on all R component pixels and B component pixels, and then performs the processing on pixels that lack Cr components and Cb components. The color difference plane is calculated by interpolating the Cr component and the Cb component (S6 in FIG. 3). In addition, since the process which interpolates Cr component and the process which interpolates Cb component can be performed similarly, description of the process which interpolates Cr component is performed here, and description of the process which interpolates Cb component is abbreviate | omitted.

The image processing unit 11 obtains an average value of Cr components in a plurality of R component pixels around the pixel as a Cr component in a pixel lacking the Cr component, and interpolates the Cr component. That is, the image processing unit 11 calculates the interpolation value of the Cr component at each pixel position of the B component, the Gr component, and the Gb component based on the following equations 18-20.
-B component pixels:

・Ｇｒ成分の画素：

Gr component pixels:

・Ｇｂ成分の画素：

Gb component pixels:

なお、画像処理部１１は、次式２１〜式２４のようにローパスフィルタを掛けて、画像の周期的な構造で発生する色モアレによる偽色を低減しつつ、Ｃｒ成分の補間値を算出してもよい。
・Ｒ成分の画素：

The image processing unit 11 applies a low-pass filter as in the following formulas 21 to 24 to calculate an interpolated value of the Cr component while reducing false colors due to color moire generated in the periodic structure of the image. May be.
-R component pixels:

・Ｂ成分の画素：

-B component pixels:

・Ｇｒ成分の画素：

Gr component pixels:

・Ｇｂ成分の画素：

Gb component pixels:

なお、Ｇ成分の画素におけるＣｒ成分と、Ｂ成分の画素におけるＣｒ成分とが「０」で初期化されている場合、式１８〜式２０の演算は、図１８に示すようなフィルタを掛けることによって行うことができ、式２１〜式２４の演算は、図１９に示すようなフィルタを掛けることによって行うことができる。

Note that when the Cr component in the G component pixel and the Cr component in the B component pixel are initialized to “0”, the calculations of Expressions 18 to 20 are performed with a filter as shown in FIG. The operations of Expressions 21 to 24 can be performed by applying a filter as shown in FIG.

As described above, the image processing unit 11 interpolates the Cr component and the Cb component to calculate the color difference plane, and outputs it as a YCbCr color image.
7). RGB Surface Calculation Processing The image processing unit 11 performs color system conversion processing on the luminance surface and the color difference surface to calculate the RGB surface (S7 in FIG. 3).

  That is, as the color system conversion process, the image processing unit 11 calculates, for example, an RGB plane by performing calculations according to the following formulas 25 to 27.

図２０は、本実施形態のデータの流れを示す。

FIG. 20 shows the data flow of this embodiment.

  In this embodiment, 18 kinds of coefficient patterns are prepared for the R component pixel and the B component pixel, the coefficient pattern is selected according to the similarity and whether the target pixel is a defective pixel, and the luminance component is Calculated. Therefore, it is possible to generate a very smooth luminance surface up to a fine edge portion.

  In the present embodiment, the colors of the three color components of RGB located in the narrowest range are considered in consideration of similarity in various directions and pixel positions of defective pixels among the target pixel and the pixels adjacent to the target pixel. Since weighted addition using information is performed, it is compatible with the luminance component calculation processing for Bayer array image data, and a smooth luminance surface can be calculated.

  In this embodiment, the coefficient pattern used in the luminance component calculation processing is set so that the ratio of RGB constituting the luminance component is constant, and the pixel value of the defective pixel is based on the direction of similarity. Thus, the pixel values of neighboring peripheral pixels having the same color component as the defective pixel are set to average values. As a result, it is possible to suppress the appearance of a structure that cannot exist in the subject (a structure resulting from the Bayer array) that changes for each pixel, and the ratio of RGB that constitutes a luminance value in each pixel. Compared with the prior art in which is non-uniform, more accurate luminance plane and color difference plane can be calculated.

  In the present embodiment, the edge enhancement process is performed on the luminance surface having a fine structure, so that the correction of the high frequency component can be realized, and the high definition is higher than the conventional technology in which the correction of the high frequency component is insufficient. Brightness surface can be obtained.

  In the present embodiment, the color difference plane is calculated directly from the Bayer array type image data independently of the luminance plane, and is not affected by the frequency characteristics of the luminance plane, which is a cause of false colors. The surface can be calculated.

  As shown in FIG. 20, in this embodiment, an index indicating similarity is directly calculated from Bayer array image data or pixel position information of defective pixels. In other words, unlike the conventional technique in which the similar directions in each pixel are classified with reference to “blurred luminance values” in which high-frequency components related to structural factors are uniformly destroyed, a Bayer array image is used. Since similarity is determined by directly using data or pixel position information of defective pixels, a luminance plane with high resolution performance in a fine structure portion can be calculated.

  As described above, according to the present embodiment, the defective pixel correction process and the high image quality process are performed at the same time, so that it can be performed with high accuracy and high speed.

  Further, in the present embodiment, since similarity is taken into account when calculating the luminance plane, the correction of the high frequency component can be realized by a band-pass filter having a fixed coefficient. Therefore, it is not necessary to perform correction using different correction values according to the similar direction of each pixel, which is performed in the prior art, and the number of reference planes for performing correction processing is small. Therefore, when the luminance plane is calculated by hardware, the hardware structure can be simplified and the cost can be reduced as compared with the prior art.

  In the present embodiment, the chrominance plane is calculated by interpolating the Cr component and the Cb component in FIG. 3S6. However, correction processing (for example, median processing) for further reducing the occurrence of false colors on such a chrominance plane is performed. Processing by a filter or the like may be applied.

In the luminance component calculation processing of the present embodiment, when the target pixel is the G component, different coefficient patterns 0 and 0a for selectively applying a low-pass filter according to the direction similarity in the pixel adjacent to the left side of the target pixel. ˜0e, 10 and 10a are prepared, but in the case of only normal image restoration processing, the luminance component in the target pixel of the G component is calculated using the coefficient patterns 10 and 10a regardless of such direction similarity. May be. That is, it is preferable that only the processes of FIGS. 6S41 to S43 are performed among the processes of S40 to S51 of FIG.
<< Modification of one embodiment >>
The operation of the modification of one embodiment will be described below.

  The difference between the present embodiment and the first embodiment is that the process of setting the index DN (FIG. 3S2) indicating the diagonal similarity performed in the first embodiment is omitted, and accordingly, FIGS. The luminance component calculation process shown is simplified as shown in FIGS. That is, in FIG. 21S12 'and FIG. 22S40', the image processing unit 11 of the present embodiment selects a coefficient pattern based only on the index HV. In the operation flowchart of the image processing unit 11 shown in FIGS. 21 and 22, steps that perform the same operations as those in FIGS. 4 to 6 are given the same step numbers and description thereof is omitted.

  As described above, in the present embodiment, the setting of the index DN indicating the diagonal similarity performed in the first embodiment is omitted, and the number of coefficient patterns is reduced. Therefore, according to the present embodiment, it is possible to generate a very smooth luminance surface up to fine edge portions in various directions with a simpler processing procedure than that of the first embodiment, High-quality processing can be performed with high accuracy and high speed.

In the luminance component calculation processing of the present embodiment, when the target pixel is the G component, different coefficient patterns 0 and 0a for selectively applying a low-pass filter according to the direction similarity in the pixel adjacent to the left side of the target pixel. ˜0e, 10 and 10a are prepared, but in the case of only normal image restoration processing, the luminance component in the target pixel of the G component is calculated using the coefficient patterns 10 and 10a regardless of such direction similarity. May be. That is, it is preferable that only the processes in FIGS. 22S41 to 43 are performed among the processes in FIGS. 22S40 ′ to S51.
<< Other embodiments >>
The operation of another embodiment of the present invention will be described below.

  In the present embodiment, image processing is performed by the PC 18 shown in FIG.

  The PC 18 is preinstalled with an image processing program recorded on a recording medium such as the CD-ROM 26 (an image processing program for realizing image restoration processing by the image processing unit 11 of the above-described embodiment). To do. That is, an image processing program is stored in a hard disk (not shown) in the PC 18 so that it can be executed by a CPU (not shown).

  Next, the operation of this embodiment will be described with reference to FIG.

  The image data subjected to the image restoration process by the PC 18 is assumed to be image data generated by being imaged by the image sensor 21 of the electronic camera 1 and subjected to predetermined signal processing by the DFE 10. In addition, it is assumed that the image data is subjected to image processing such as white balance processing, gradation conversion, and γ correction in advance by the image processing unit 11. Further, it is assumed that the image data is generated as an image file by adding information on the pixel position of the defective pixel by the image processing unit 11 and stored in the memory card 16 via the card I / F 17.

  It is assumed that the image file has a format conforming to, for example, the Exif (Exchangeable image file format for digital still cameras) standard or the TIFF / EP standard. The pixel value of the defective pixel in the image data may be corrected in advance by a known defective pixel correction process.

  In the electronic camera 1, when the PC communication mode is selected by the operator via the operation unit 23 and the transfer of image data is instructed, the data is stored in the card memory 16 via the card I / F 17 and the input / output I / F 19. The processed image file is transferred to the PC 18. When a CPU (not shown) of the PC 18 reads an image file from the electronic camera 1, it executes the above-described image processing program. A CPU (not shown) of the PC 18 reads pixel position information of defective pixels added to the image file based on an image processing program, and performs image restoration processing similar to that of the embodiment on the image data. .

  By executing the image processing program, the image data subjected to the image restoration process is compressed as necessary and recorded on a hard disk (not shown), or converted into a color system adopted by the monitor 24 or the printer 25. It is preferable to be displayed and output.

  As described above, in the present embodiment, the image restoration process similar to that in the above-described embodiment is performed by the PC 18, and the defective pixel correction process and the image quality improving process can be performed with high accuracy and at high speed.

  Further, the CPU (not shown) of the PC 18 may read image data from the memory card 16 and execute an image processing program when the memory card 16 storing the image file is directly attached to the PC 18.

  In the present embodiment, the image data to which the pixel position information of the defective pixel is added is used in the image restoration processing. However, the PC 18 together with the image file, the pixel position of the defective pixel of the image sensor 21 from the electronic camera 1. By reading this information and recording it in a memory (not shown) or the like, it is possible to perform image restoration processing on the image data of the image file to which the pixel position information of the defective pixel is not added.

  The image processing program may be downloaded and installed on the PC 18 by accessing a predetermined home page via the Internet.

Further, the image processing program may not be executed by the PC 18 but may be executed by a remote server connected via the Internet or the like. That is, the PC 18 may perform image restoration processing on the image data by transferring the image data read from the electronic camera 1 to a server or the like that can execute the image processing program via the Internet or the like.
<< Additional items of embodiment >>
In the above-described embodiment, the processing for an image that is represented in the RGB color system and in which each pixel has color information of any one of R, G, and B color components has been described. The present invention can also be applied to an image shown in another color system.

  In the above-described embodiment, image data in which color components are arranged as shown in FIG. 2 is a processing target. However, the present invention is not limited to this, and image data having various color component arrangements is a processing target. Can do.

  In the above embodiment, the image data is subjected to image processing such as white balance processing before the image restoration processing, but the present invention is not limited to this, and image processing such as white balance processing is performed after the image restoration processing. You may go.

  From the above detailed description, features and advantages of the embodiment will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and changes, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents included in the scope disclosed in.

1 electronic camera, 10 DFE, 11 image processing unit, 12 CPU, 13 memory, 14 display I / F, 15, 24 monitor, 16 memory card, 17 media I / F, 18 PC, 19 input / output I / F, 20 Imaging optical system, 21 imaging device, 22 TG, 23 operation unit, 25 printer, 26 CD-ROM

Claims (20)

The color information of the first image in which each pixel has color information of any one of a plurality of color components is weighted and added at the pixel position of the pixel of interest to obtain a value of a color component different from the color component of the pixel of interest. An image processing unit that calculates and generates a second image in which each pixel has a plurality of color components;
A storage unit that stores information on pixel positions of defective pixels in the first image,
The image processing unit prepares a plurality of coefficient patterns for weight-adding the color information of the first image at the pixel position of the target pixel, and based on the color component of the target pixel and the pixel position of the defective pixel, An image processing apparatus characterized by selecting a coefficient pattern and performing weighted addition. The color information of the first image in which each pixel has color information of any one of a plurality of color components is weighted and added at the pixel position of the pixel of interest to obtain a value of a color component different from the color component of the pixel of interest. An image processing unit that calculates and generates a second image in which each pixel has a plurality of color components;
In the first image, information on the pixel position of the defective pixel is added,
The image processing unit prepares a plurality of coefficient patterns for weight-adding the color information of the first image at the pixel position of the target pixel, and based on the color component of the target pixel and the pixel position of the defective pixel, An image processing apparatus characterized by selecting a coefficient pattern and performing weighted addition. The image processing apparatus according to claim 1 or 2,
The image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel, and calculates the coefficient pattern based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. An image processing apparatus characterized by selecting. The image processing apparatus according to claim 3.
When the pixel adjacent to the target pixel is the defective pixel, the image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
The coefficient pattern is an image having a value of zero or more that weights and adds color information of pixels in a minimum area including color information of the plurality of color components centered on a pixel position of the target pixel. Processing equipment. The color information of the pixel of interest by weight-adding the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the pixel of interest using a variable coefficient of zero or more. An image processing unit that calculates a value of a different color component and generates a second image in which each pixel has a plurality of color components;
A storage unit that stores information on pixel positions of defective pixels in the first image,
The image processing unit changes the coefficient based on a color component of the target pixel and a pixel position of the defective pixel, and performs weighted addition of the color information of the target pixel and the color information of a pixel adjacent to the target pixel. An image processing apparatus. The color information of the pixel of interest by weight-adding the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the pixel of interest using a variable coefficient of zero or more. An image processing unit that calculates a value of a different color component and generates a second image in which each pixel has a plurality of color components,
In the first image, position information of defective pixels is added,
The image processing unit changes the coefficient based on a color component of the target pixel and a pixel position of the defective pixel, and performs weighted addition of the color information of the target pixel and the color information of a pixel adjacent to the target pixel. An image processing apparatus. The image processing apparatus according to claim 6 or 7,
The image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel, and calculates the coefficient based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. An image processing apparatus characterized by changing. The image processing apparatus according to claim 8.
When the pixel adjacent to the target pixel is the defective pixel, the image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. An image processing apparatus. The image processing apparatus according to any one of claims 6 to 9,
The image processing device, wherein all the pixels of the first image weight-add the color information of the first image at a constant color component ratio at all times. The image processing apparatus according to claim 10.
When the first image is composed of a first color component having a high pixel density, a second color component having a low pixel density, and a third color component, the image processing unit is configured to display color information of the second color component and the first color component. An image processing apparatus characterized by performing weighted addition of color information of three color components at the same color component ratio. The color information of the pixel of interest by weight-adding the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the pixel of interest using a variable coefficient of zero or more. By calculating a value of a different color component and performing filter processing with a predetermined fixed filter coefficient, the different color component is corrected, and a second image in which each pixel has a plurality of color components is generated. An image processing unit;
A storage unit that stores information on pixel positions of defective pixels in the first image,
The image processing device, wherein the image processing unit changes the coefficient based on a color component of the pixel of interest and a pixel position of the defective pixel. The color information of the pixel of interest by weight-adding the color information of the first image in which each pixel has color information of any one of a plurality of color components at the pixel position of the pixel of interest using a variable coefficient of zero or more. By calculating a value of a different color component and performing filter processing with a predetermined fixed filter coefficient, the different color component is corrected, and a second image in which each pixel has a plurality of color components is generated. An image processing unit,
In the first image, position information of defective pixels is added,
The image processing device, wherein the image processing unit changes the coefficient based on a color component of the pixel of interest and a pixel position of the defective pixel. The image processing apparatus according to claim 12 or 13,
The image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel, and calculates the coefficient based on the determination result together with the color component of the target pixel and the pixel position of the defective pixel. An image processing apparatus characterized by changing. The image processing apparatus according to claim 14.
When the pixel adjacent to the target pixel is the defective pixel, the image processing unit determines the strength of similarity in a plurality of directions at the pixel position of the target pixel based on the pixel position of the adjacent pixel. An image processing apparatus. The image processing apparatus according to any one of claims 12 to 15,
The image processing apparatus, wherein the image processing unit uses a filter coefficient including positive and negative values as the predetermined fixed filter coefficient. The image processing apparatus according to any one of claims 12 to 16,
The image processing apparatus according to claim 1, wherein the different color components are a luminance component and a color difference component. The image processing apparatus according to claim 17.
The image processing unit, when the first image has a first color component, a second color component, and a third color component,
A first similarity component composed of the first color component and the second color component;
A second similarity component composed of the second color component and the third color component;
A third similarity component composed of the third color component and the first color component;
A fourth similarity component composed of only the first color component;
A fifth similarity component composed only of the second color component;
The similarity with respect to a plurality of directions is calculated using at least one similarity component of the sixth similarity component composed of only the third color component, and the similarity with respect to the plurality of directions is calculated based on the similarity. An image processing apparatus characterized by determining the strength of the image. An image sensor;
An image processing device according to any one of claims 1 to 18,
An imaging apparatus comprising: An input procedure for reading a first image in which each pixel has color information of any one of a plurality of color components and information on the pixel position of a defective pixel is added;
The color information of the first image is weighted and added at the pixel position of the target pixel to calculate a value of a color component different from the color component of the target pixel, and a second image in which each pixel has a plurality of color components is generated. Image processing procedure to be executed by a computer,
The image processing procedure prepares a plurality of coefficient patterns for weighted addition of the color information of the first image at the pixel position of the target pixel, and based on the positional relationship between the color component of the target pixel and the defective pixel, An image processing program, wherein the coefficient pattern is selected and weighted.

Images