JP2004364201A - Interpolated pixel formation instrument and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To find a correlation of a direction comparatively correctly and prevent the generation of a false color when performing pixel interpolation. <P>SOLUTION: Photodiode groups S1, S2 and S3 are regulated. The photodiode group S1 contains photodiodes B<SB>2,2</SB>which correspond to view pixels, and is constituted of photodiodes G<SB>1,1</SB>which are positioned at upper lefts of the photodiodes B<SB>2,2</SB>, and photodiodes R<SB>3,3</SB>which are positioned at lower rights of the photodiodes B<SB>2,2</SB>. The photodiode groups S2 and S3 adjoin the photodiode group S1 and are parallel to the photodiode group S1. In the groups S2 and S3, the proportion of a color of a color filter which is formed on a constituting photodiode PD is the same as that of the photodiode group S1. A correlation of the direction of the view pixels is found by using image data obtained based on the photodiode groups S1, S2, and S3, and the pixel interpolation (formation of R image data and G image data) of the view pixels is performed by using a filter corresponding to the correlation of the direction. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
【Technical field】
The present invention relates to an interpolation pixel generation apparatus and method, and a correlation direction detection apparatus and method.
[0002]
BACKGROUND OF THE INVENTION
Some digital cameras have an image pickup device having a three-plate CCD and others have a single-plate CCD.
[0003]
The three-plate CCD has an R CCD in which a filter that transmits red light components is formed on the light receiving surface of all photodiodes, and a G filter in which filters that transmit green light components are formed in all photodiodes. The CCD is composed of a CCD for B on which a photodiode for B is formed, and a filter for transmitting a blue light component is formed on all photodiodes. In the three-plate CCD, there are three photodiodes corresponding to one pixel in order to obtain image data representing a red component, a green component, and a blue component of the pixel.
[0004]
On the other hand, in the single-plate CCD, unlike the three-plate CCD, there is one photodiode corresponding to one pixel, and image data representing a red component, a green component, or a blue component of the pixel is obtained. Image data representing one color component among the red, green and blue components of the pixel can be obtained, but image data representing two color components other than the one color cannot be obtained. For this purpose, for example, an interpolation process is performed to obtain image data representing these two color components.
[0005]
As an example of the interpolation processing, there is one in which directional correlation is determined, and filter processing is performed in accordance with the direction determined to have correlation (see Patent Document 1). Since filter processing is performed according to the correlation direction, generation of a false signal can be prevented in advance.
[0006]
[Patent Document 1]
JP-A-8-298669
[0007]
When the correlation direction is discriminated using the same color image data, the distance between the photodiodes that give the same color image data may be long, so that the relative direction discrimination may not be performed. When the correlation direction is determined using image data of different colors, false colors may be generated due to erroneous determination.
[0008]
DISCLOSURE OF THE INVENTION
An object of the present invention is to make it possible to determine the direction relatively accurately and to prevent the occurrence of false colors.
[0009]
The first invention provides serial three-primary-color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any one of the three primary colors is formed on the light-receiving surface. In the interpolated pixel generation device that generates data representing the interpolated pixel by inputting the signal, a plurality of photoelectric conversion elements including a photoelectric conversion element adjacent to the target photoelectric conversion element corresponding to the target pixel for which the direction of high correlation should be detected is detected. The first three primary color image data obtained on the basis of the first photoelectric conversion element group and the transmission color component of the color filter formed on the light receiving surface adjacent to the first photoelectric conversion element group. Second three-primary-color image data obtained based on a second photoelectric conversion element group composed of a plurality of photoelectric conversion elements having a proportion equal to the plurality of photoelectric conversion elements constituting the first photoelectric conversion element group. And detecting means for detecting a direction in which the pixel of interest is highly correlated, and determining a filter for generating data representing the pixel to which the pixel of interest is to be interpolated according to the direction detected by the detecting means. And an interpolation pixel generation unit that generates data representing an interpolation pixel corresponding to the pixel of interest using the filter determined by the filter determination unit.
[0010]
The first invention also provides a method suitable for the interpolation pixel generation apparatus. That is, in this method, serial three-primary-color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any one of the three primary colors is formed on the light-receiving surface. In the interpolated pixel generation device that generates data representing the interpolated pixel by inputting the signal, a plurality of photoelectric conversion elements including a photoelectric conversion element adjacent to the target photoelectric conversion element corresponding to the target pixel for which the direction of high correlation should be detected is detected. The first three primary color image data obtained on the basis of the first photoelectric conversion element group and the transmission color component of the color filter formed on the light receiving surface adjacent to the first photoelectric conversion element group. The second three primary colors obtained based on the second photoelectric conversion element group composed of a plurality of photoelectric conversion elements whose ratio is equal to the plurality of photoelectric conversion elements constituting the first photoelectric conversion element group Based on the image data, a direction in which the pixel of interest is highly correlated is detected, and a filter for generating a pixel to interpolate the pixel of interest is determined according to the detected edge directionality. An interpolation pixel corresponding to the target pixel is generated using a filter.
[0011]
According to the first invention, based on the first photoelectric conversion element group including a plurality of photoelectric conversion elements including the photoelectric conversion elements adjacent to the target photoelectric conversion element corresponding to the target pixel for which a highly correlated direction is to be detected. The obtained first three primary color image data and the ratio of the transmitted color component of the color filter that is adjacent to the first photoelectric conversion element group and is formed on the light receiving surface are the first photoelectric conversion element group. The second three primary color image data obtained based on the second photoelectric conversion element group composed of a plurality of photoelectric conversion elements equal to the plurality of photoelectric conversion elements to be formed is obtained. Based on the obtained first image data, the direction in which the pixel of interest is highly correlated is detected. A filter that generates a pixel to be interpolated is determined according to the detected direction. An interpolation pixel is generated using the determined filter.
[0012]
The first photoelectric conversion element group and the second photoelectric conversion element group are not relatively separated from each other. The direction in which the pixel of interest is highly correlated can be known relatively accurately, and an interpolation pixel can be generated using an appropriate filter corresponding to the direction. Further, the plurality of photoelectric conversion elements constituting the first photoelectric conversion element group and the plurality of photoelectric conversion elements constituting the second photoelectric conversion element group have the same ratio of the color filters formed on the light receiving surface. Therefore, it is possible to prevent erroneous determination of the direction of correlation of the pixel of interest. Generation of false colors can be prevented.
[0013]
The photoelectric conversion element of interest may or may not be included in the first photoelectric conversion element group.
[0014]
The direction of the first photoelectric conversion element and the direction of the second photoelectric conversion element group are, for example, the vertical direction of the highly correlated direction to be detected.
[0015]
The first photoelectric conversion element group includes, for example, a first plurality of photoelectric conversion element groups having a common photoelectric conversion element, and the second photoelectric conversion element group includes, for example, a common photoelectric conversion element. It consists of a second plurality of photoelectric conversion element groups. In this case, the first plurality of photoelectric conversion element groups and the second plurality of photoelectric conversion element groups will be arranged at target positions with the focused photoelectric conversion element as the center.
[0016]
The correlation direction detection apparatus according to the second invention is output from a solid-state electronic image sensor including a large number of photoelectric conversion elements having a color filter that transmits a light component of any of the three primary colors formed on a light receiving surface. A plurality of input means for inputting serial three primary color image data, and a plurality of photoelectric conversion elements adjacent to the target photoelectric conversion element corresponding to the target pixel for detecting a direction having high correlation among the three primary color image data input to the input means; Of the three primary color image data inputted to the input means, a first calculation means for calculating the level of the first three primary color image data obtained based on the first photoelectric conversion element group consisting of the first photoelectric conversion elements, the first primary color image data inputted to the input means. A plurality of photoelectric conversions in which the ratio of the transmitted color component of the color filter that is adjacent to the photoelectric conversion element group and formed on the light receiving surface constitutes the first photoelectric conversion element group Calculated by the second calculation means for calculating the level of the second three-primary color image data obtained based on the second photoelectric conversion element group composed of a plurality of photoelectric conversion elements equal to the child, and the first calculation means. According to another aspect of the invention, there is provided detection means for detecting a direction in which the pixel of interest is highly correlated based on the level and the level calculated by the second calculation means.
[0017]
The second invention also provides a method suitable for the correlation direction detection apparatus. That is, in this method, serial three-primary-color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any one of the three primary colors is formed on the light-receiving surface. To the first photoelectric conversion element group including a plurality of photoelectric conversion elements including photoelectric conversion elements adjacent to the target photoelectric conversion element corresponding to the target pixel whose correlation direction is to be detected among the input three primary color image data. A first level of the first three primary color image data obtained based on the first three primary color image data is calculated, and among the input three primary color image data, a color, which is adjacent to the first photoelectric conversion element group and formed on the light receiving surface Based on the second photoelectric conversion element group composed of a plurality of photoelectric conversion elements in which the ratio of the transmission color component of the filter is equal to the plurality of photoelectric conversion elements constituting the first photoelectric conversion element group. It is second to calculate the second level of the three primary image data, based on the first level calculated and the second level, and detects a direction of high correlation of the pixel of interest.
[0018]
Also in the second invention, the first photoelectric conversion element group and the second photoelectric conversion element group are not relatively separated from each other. The direction in which the pixel of interest is highly correlated can be known relatively accurately. Therefore, the interpolation pixel can be generated using an appropriate filter corresponding to the direction. Further, the plurality of photoelectric conversion elements constituting the first photoelectric conversion element group and the plurality of photoelectric conversion elements constituting the second photoelectric conversion element group have the same ratio of the color filters formed on the light receiving surface. Therefore, it is possible to prevent erroneous determination of the correlation direction of the pixel of interest.
[0019]
[Explanation of Examples]
FIG. 1 shows an embodiment of the present invention and shows a part of a light receiving surface of a CCD.
[0020]
The CCD is provided with a large number of photodiodes (photoelectric conversion elements) PD in the horizontal direction and the vertical direction. The CCD according to this embodiment has a so-called honeycomb array, and photodiodes PD are arranged in even rows in odd columns, and photodiodes PD are arranged in odd rows in even columns. However, the photodiodes PD may be arranged in the odd rows in the odd columns, and the photodiodes PD may be arranged in the even rows in the even columns.
[0021]
On the light receiving surface of the photodiode PD, a red filter that transmits red light component, a green filter that transmits green light component, or a blue filter that transmits blue light component is formed. Has been.
[0022]
In FIG. 1, the red filter is indicated by “R”, the green filter is indicated by “G”, and the blue filter is indicated by “B”. Further, the position of the photodiode PD in the CCD is shown with a suffix “R”, “G” or “B”. In the CCD according to this embodiment, red filters, green filters, or blue filters are alternately formed on the photodiode PD for each column.
[0023]
In this embodiment, the correlation direction of a specific pixel (pixel of interest) constituting the image is detected from the right diagonal direction, left diagonal direction, vertical direction, and horizontal direction, and the characteristics according to the detected correlation directionality Pixel interpolation is performed using a filter having An evaluation value in the right oblique direction, an evaluation value in the left oblique direction, an evaluation value in the vertical direction, and an evaluation value in the horizontal direction are calculated as described below. The directionality of the pixel of interest is detected using the calculated evaluation value.
[0024]
Photodiode corresponding to the target pixel (referred to as the target photodiode) B _{2, 2} Is indicated by hatching.
[0025]
2 to 4 show a part of the light receiving surface of the CCD, and a photodiode B as a pixel of interest. _{2, 2} Pixels corresponding to are taken up. As an identification code of the photodiode PD, a code of a color filter formed on the photodiode PD is used.
[0026]
FIG. 2 shows a combination of photodiodes PD used when calculating an evaluation value in the right oblique direction.
[0027]
Attention photodiode B _{2, 2} A first central photodiode group (first photoelectric conversion element group) S1 is defined in an obliquely leftward direction with respect to. The first central photodiode group S1 includes the target photodiode B _{2, 2} In addition to the target photodiode B _{2, 2} Upper left photodiode G _1,1 And noticed photodiode B _{2, 2} Photodiode R diagonally below right _{3, 3} It is included. The ratio of the color filters formed in the three photodiodes constituting the first central photodiode group S1 is 1: 1 for the red filter, the green filter, and the blue filter, respectively.
[0028]
An external photodiode group (second photoelectric conversion element group) S2 and a diagonally lower left and upper right side of the first central photodiode group S1, parallel to and adjacent to the first central photodiode group S1, and S3 is defined respectively. The first external photodiode group S2 is a photodiode G constituting the first central photodiode group S1. _1,1 , B _{2, 2} And R _{3, 3} Photodiodes R located diagonally to the left of each _0,2 , G _1,3 And B _{2, 4} It is composed of The ratio of the color filters formed in the three photodiodes constituting the first external photodiode group S2 is 1: 1 as in the first central photodiode group S1. The second photodiode group S3 is a photodiode G constituting the first central photodiode group S1. _1,1 , B _{2, 2} And R _{3, 3} Photodiodes B located diagonally to the upper right _2,0 , R _3,1 And G _4,2 It is composed of In the second external photodiode group S3, the ratio of the color filters formed in the three photodiodes constituting the second external photodiode group S3 is 1 as in the first central photodiode group S1. One-on-one.
[0029]
The directions of the first central photodiode group S1 and the external photodiode groups S2 and S3 are perpendicular to the direction in which the evaluation value of the pixel of interest is to be detected (right oblique direction).
[0030]
The right oblique direction evaluation value Vr is obtained based on Equation 1.
[0031]
Vr = | S3-S1 | + | S1-S2 |
However, S1 = G _1,1 + B _{2, 2} + R _{3, 3} ... Formula 2
S2 = R _0,2 + G _1,3 + B _{2, 4} ... Formula 3
S3 = B _2,0 + R _3,1 + G _4,2 ... Formula 4
[0032]
In Equations 1 to 4, S1, S2, and S3 are based on the signal charges accumulated in the first central photodiode group S1, the first external photodiode group S2, and the second external photodiode group S3, respectively. The obtained image data is added. Further, the same code as that of the photodiode is used for the image data obtained based on the signal charge accumulated in each photodiode.
[0033]
The pixel of interest B based on the right oblique direction evaluation value Vr obtained based on Expressions 1 to 4 _{2, 2} Of the right diagonal direction is determined.
[0034]
Since the evaluation value Vr in the diagonally right direction is calculated from the image data obtained based on the photodiode groups that are adjacent to each other and are close to each other, the calculated evaluation value Vr is relatively reliable.
[0035]
FIG. 3 shows a combination of photodiodes PD used when calculating an evaluation value in the diagonally left direction.
[0036]
Attention photodiode B _{2, 2} A second central photodiode group (first photoelectric conversion element group) S4 is defined in an obliquely rightward direction with respect to. The second central photodiode group S4 includes a target photodiode B. _{2, 2} In addition to the target photodiode B _{2, 2} Photodiode R on the upper right of _3,1 And noticed photodiode B _{2, 2} Lower left photodiode G _1,3 It is included. The ratio of the color filters formed in the three photodiodes constituting the second central photodiode group S4 is 1: 1 for the red filter, the green filter, and the blue filter, respectively.
[0037]
The third and fourth external photodiode groups (second conversion) are arranged diagonally to the lower right and upper left of the second central photodiode group S4, parallel to and adjacent to the second central photodiode group S4. Element group) S5 and S6 are defined. The third photodiode group S5 includes a photodiode R that constitutes the second central photodiode group S4. _3,1 , B _{2, 2} And G _1,3 Photodiodes G located diagonally below and to the right _4,2 , R _{3, 3} And B _{2, 4} It is composed of The ratio of the color filters formed in the three photodiodes constituting the third external photodiode group S5 is 1: 1 as in the second central photodiode group S2. The fourth photodiode group S6 includes a photodiode R that constitutes the second central photodiode group S2. _3,1 , B _{2, 2} And G _1,3 Photodiode B located diagonally above _2,0 , G _1,1 And R _0,2 It is composed of In the fourth photodiode group S6 as well, the ratio of the color filters formed in the three photodiodes constituting the fourth photodiode group S6 is 1: 1 as in the second central photodiode group S4. Pair one.
[0038]
The directions of the second central photodiode group S4 and the external photodiode groups S5 and S6 are perpendicular to the direction in which the evaluation value of the pixel of interest is to be detected (left oblique direction).
[0039]
The left diagonal direction evaluation value Vl is obtained based on Equation 5.
[0040]
Vl = | S6-S4 | + | S5-S4 |
However, S4 = G _1,3 + B _{2, 2} + R _3,1 ... Formula 6
S5 = B _{2, 4} + R _{3, 3} + G _4,2 ... Formula 7
S6 = R _0,2 + G _1,1 + B _2,0 ... Formula 8
[0041]
In Equations 5 to 8, S4, S5, and S6 are based on the signal charges accumulated in the second central photodiode group S4, the third external photodiode group S5, and the fourth external photodiode group S6, respectively. The obtained image data is added. Further, the same code as that of the photodiode is used for the image data obtained based on the signal charge accumulated in each photodiode.
[0042]
The pixel of interest B based on the left diagonal direction evaluation value Vl obtained based on Expressions 5 to 8 _{2, 2} Of the left diagonal direction is determined.
[0043]
Since the evaluation value Vl in the diagonally left direction is calculated from image data obtained on the basis of photodiode groups that are adjacent to each other and close to each other, the calculated evaluation value Vl is relatively reliable.
[0044]
FIG. 4 shows combinations of photodiodes PD used when calculating the evaluation value in the vertical direction.
[0045]
Pixel of interest B _{2, 2} And upper pixel group S7 including pixel B _{2, 2} And a lower photodiode group S8 including The upper photodiode group S7 includes a target photodiode B _{2, 2} And this focused photodiode B _{2, 2} Photodiode B located above _2,0 It consists of and. The lower photodiode group S8 includes a target photodiode B _{2, 2} And this focused photodiode B _{2, 2} Photodiode B located below _{2, 4} It consists of and.
[0046]
The respective directions of the upper photodiode group S7 and the lower photodiode group S8 are the same as the direction in which the evaluation value is to be calculated (vertical direction).
[0047]
The vertical direction evaluation value Vv is obtained based on Equation 9.
[0048]
Vv = | 2 × B _2,0 -2 x B _{2, 2} | + | 2 × B _{2, 4} -2 x B _{2, 2} ｜ ・ ・ ・ Formula 9
In Equation 9, B _2,0 , B _{2, 2} , B _{2, 4} Indicates the level of image data obtained on the basis of the signal charges accumulated in the photodiodes with the corresponding reference numerals. In Expression 9, the factor of 2 is to adjust the level difference from the value obtained in the calculation of the horizontal direction evaluation value, as will be described later.
[0049]
The pixel of interest B based on the evaluation value Vv obtained based on Equation 9 _{2, 2} Are correlated in the vertical direction.
[0050]
FIG. 5 shows combinations of photodiodes PD used when calculating the evaluation value in the horizontal direction.
[0051]
Attention photodiode B _{2, 2} Around the first photodiode group (first photoelectric conversion element group) S9, second photodiode group (second photoelectric conversion element group) S10, third photodiode group (first photoelectric conversion element) Group) S11 and a fourth photodiode group (second photoelectric conversion element group) S12 are defined.
[0052]
The first photodiode group S9 includes the target photodiode B _{2, 2} Photodiode G located in the upper left of _1,1 And this photodiode G _1,1 Photodiode R located at the lower left of _0,2 And is composed of.
[0053]
The second photodiode group S10 includes a target photodiode B _{2, 2} Photodiode R located in the upper right of _3,1 And this photodiode R _3,1 Photodiode G located in the lower right of _4,2 And is composed of.
[0054]
The third photodiode group S11 includes the target photodiode B _{2, 2} Photodiode G located in the lower left of _1,3 And this photodiode G _1,3 Photodiode R located in the upper left of _0,2 And is composed of.
[0055]
The fourth photodiode group S12 includes a target photodiode B _{2, 2} Photodiode R deserving to the lower right of _{3, 3} And this photodiode R _{3, 3} Photodiode G located in the upper right of _4,2 And is composed of.
[0056]
The color filters formed in the photodiodes constituting these photodiode groups S9 to S12 have a ratio of R and G of 1: 1.
[0057]
The first photodiode group S9 and the third photodiode group S11 (a plurality of first photoelectric conversion element groups) are connected to a common photodiode R. _0,2 Is included. In addition, the second photodiode S10, the fourth photodiode group S12 (a plurality of second photoelectric conversion element groups) are connected to a common photodiode G. _4,2 Is included.
[0058]
The first photodiode group S9 and the fourth photodiode group S12 are parallel, and the target photodiode B _{2, 2} It is arranged in a symmetrical position with respect to the center. Also, the second photodiode group S10 and the third photodiode group S11 are parallel to each other, and the target photodiode B _{2, 2} It is arranged in a symmetrical position with respect to the center.
[0059]
The horizontal direction evaluation value Vh is obtained based on Equation 10.
[0060]
Vh = | S9−S10 | + | S11−S12 |
However, S9 = R _0,2 + G _1,1 ... Formula 11
S10 = R _3,1 + G _4,2 ... Formula 12
S11 = R _0,2 + G _1,3 ... Formula 13
S12 = R _{3, 3} + G _4,2 ... Formula 14
[0061]
In Expressions 10 to 14, S9, S10, S11, and S12 are obtained based on the signal charges accumulated in the first, second, third, and fourth photodiode groups S1, S10, S11, and S12, respectively. The image data is added.
[0062]
Based on the horizontal direction evaluation value Vh obtained based on Expressions 10 to 14, the target pixel B _{2, 2} Are correlated in the horizontal direction.
[0063]
All of the photodiodes constituting the photodiode groups S1 to S12 are the target photodiode B. _{2, 2} Adjacent to.
[0064]
FIG. 6 is a flowchart showing pixel interpolation processing.
[0065]
First, a target pixel to be interpolated is designated (step 1). As described above, the evaluation values Vr, Vl, Vv, and Vh are calculated for the designated pixel of interest based on Expressions 1, 5, 9 and 10 (step 2). Based on the calculated evaluation value, interpolation processing is performed using a filter suitable for the correlation of the pixel of interest.
[0066]
If the value obtained by subtracting the horizontal direction evaluation value Vh from the vertical direction evaluation value Vv is larger than the predetermined threshold value TH (YES in step 3), it is determined that the pixel of interest has a high horizontal correlation. The Therefore, as will be described later, an interpolation pixel is generated using a horizontal interpolation filter (step 4).
[0067]
If the value obtained by subtracting the vertical direction evaluation value Vv from the horizontal direction evaluation value Vh is greater than a predetermined threshold value TH (YES in step 5), it is determined that the pixel of interest has a high vertical correlation. The For this purpose, as will be described later, an interpolation pixel is generated using a vertical interpolation filter (step 6).
[0068]
If the value obtained by subtracting the direction evaluation value Vl in the left diagonal direction from the direction evaluation value Vr in the right diagonal direction is larger than the predetermined threshold value TH (YES in Step 7), the pixel of interest has a large correlation in the right diagonal direction. It is judged. For this purpose, as will be described later, an interpolation pixel is generated using a right diagonal interpolation filter (step 8).
[0069]
If the value obtained by subtracting the direction evaluation value VR in the diagonally right direction from the direction evaluation value Vl in the diagonally left direction is larger than the predetermined threshold value TH (YES in step 9), the pixel of interest has a large correlation in the diagonally left direction. It is judged. Therefore, as will be described later, an interpolation pixel is generated using a left diagonal interpolation filter (step 10).
[0070]
If none of the subtraction values is greater than the predetermined threshold value (NO in step 9), the interpolation process is not performed and the image data corresponding to the target pixel is output (step 11).
[0071]
Until the interpolation processing for the image data for one frame is completed, the processing from step 1 to step 11 is repeated while changing the target pixel (step 12).
[0072]
The interpolation filter used according to the correlation of the pixel of interest is as follows.
[0073]
Filter used to generate red component interpolation pixels
Horizontal interpolation filter Frh = (R _3,1 + R _{3, 3} + R _0,2 ) / 3 Equation 15
Vertical interpolation filter Frv = (R _3,1 + R _{3, 3} + R _0,2 ) / 3 ... Formula 16
Right diagonal interpolation filter Frr = (2 × R _3,1 + R _0,4 ) / 3 ... Formula 17
Left diagonal interpolation filter Frl = (2 × R _{3, 3} + R _0,0 ) / 3 ... Equation 18
[0074]
Filter used to generate interpolated pixels for green component
Horizontal interpolation filter Fgh = (G _1,1 + G _1,3 + G _4,2 ) / 3 ... Equation 19
Vertical interpolation filter Fgv = (G _1,1 + G _1,3 + G _4,2 ) / 3 ... Equation 20
Right diagonal interpolation filter Fgr = (2 × G _1,3 + G _4,0 ) / 3 ... Formula 21
Left diagonal interpolation filter Fgl = (2 × G _1,1 + G _{4, 4} ) / 3 ... Formula 22
[0075]
A filter used to generate interpolated pixels for the blue component
Horizontal interpolation filter Fbh = (4 × B _{2, 2} + B _-1,1 + B _-1,3 + B _5,1 + B _5,3 ) / 8 ... Formula 23
Vertical interpolation filter Fbv = (2 × B _{2, 2} + B _2,0 + B _{2, 4} ) / 4 ... Equation 24
Right diagonal interpolation filter Fbr = (4 × B _{2, 2} + B _2,0 + B _{2, 4} + B _-1,3 + B _5,1 ) / 8 ... Equation 25
Left diagonal interpolation filter Fbl = (4 × B _{2, 2} + B _2,0 + B _{2, 4} + B _-1,1 + B _5,3 ) / 8 ... Equation 26
[0076]
Interpolated pixel (pixel of interest B) for a position (imaginary pixel) where the photodiode PD does not exist _{2, 2} Interpolated pixel B generated at a position above _2,1 ) Is generated based on Equation 27.
[0077]
B _2,1 = (B _2,0 + B _1,1 + B _{2, 2} + B _3,1 ) / 4 ... Formula 27
[0078]
It goes without saying that interpolation pixels can be generated for imaginary pixels of other positions and color components in the same manner as in Expression 27.
[0079]
FIG. 7 is a block diagram showing an electrical configuration of a digital still camera in which the above-described interpolation processing is performed.
[0080]
RGB serial image data representing a subject image is output from a CCD in which the color filter shown in FIG. 1 is formed on the photodiode PD, and the offset correction circuit 21 performs offset correction. The RGB serial image data output from the offset correction circuit 21 is subjected to matrix correction in the linear matrix circuit 22 and color correction is performed so that the original subject color is obtained. The RGB serial image data output from the linear matrix circuit 22 is gain-corrected by the gain correction circuit 23 and gamma-corrected by the gamma correction circuit 24. Image data that has been gamma corrected in the gamma correction circuit 24 is input to the RGB interpolation circuit 25.
[0081]
The RGB interpolation circuit 25 is given a threshold value TH, and the above-described interpolation processing is performed. Details of the RGB interpolation circuit 25 will be described later.
[0082]
By performing the interpolation process in the RGB interpolation circuit 25, parallel RGB image data is obtained from the serial RGB image data. The RGB parallel image data is input to the RGB / YC conversion circuit 26, and luminance data Y and color difference data C are generated. The generated luminance data Y and color difference data C are subjected to noise reduction processing in the noise reduction circuit 27.
[0083]
The luminance data Y subjected to the noise reduction process is subjected to contour correction in the contour correction circuit 28. The color difference data C subjected to the noise reduction process is subjected to color correction in the color difference matrix 29. The luminance data Y output from the contour correction circuit 28 and the color difference data C output from the color difference matrix circuit 29 are given to the display device, whereby the subject image is displayed on the display screen of the display device. Further, the luminance data Y and the color difference data C output from the contour correction circuit 28 and the color difference data C output from the color difference matrix circuit 29 are applied to the memory card, whereby the luminance data Y and the color difference data C are recorded on the memory card. .
[0084]
FIG. 8 is a block diagram showing an electrical configuration of the RGB interpolation circuit 25.
[0085]
When serial RGB image data is supplied to the RGB interpolation circuit 25, the RGB image data is input to the delay device 30. Four 1H delay circuits 31, 32, 33 and 34 are connected in series to the delay device 30. The 1H delay circuit outputs a delay for one horizontal scanning line. From the delay device 30, serial RGB image data not delayed, serial RGB image data delayed by 1H (1H is one horizontal scanning period), serial RGB image data delayed by 2H, serial RGB image data delayed by 3H Serial RGB image data delayed by 4H is output. Therefore, serial RGB image data for 5 lines is output from the delay circuit 30. The serial RGB image data for five lines output from the delay circuit 30 is input to the adaptive interpolation filter processing circuit 60 and the direction evaluation value calculation circuit 40.
[0086]
The serial RGB image data for five lines input to the direction evaluation value calculation circuit 40 is input to the adder 41. By performing addition processing in the adder 41, as described above, the addition data S1 to S12 for each of the photodiode groups S1 to S12 are obtained.
[0087]
The addition data S1, S2, and S3 are input to the right oblique evaluation value calculation circuit 42. The right oblique evaluation value calculation circuit 42 calculates a right oblique direction evaluation value Vr. The addition data S4, S5 and S6 are input to the left oblique evaluation value calculation circuit 43. The left oblique direction evaluation value calculation circuit 43 calculates the left oblique direction evaluation value Vl. Image data obtained based on the photodiodes constituting the upper photodiode group S7 and the lower photodiode group S8 (the target pixel is B _{2, 2} If so, photodiode B _2,0 , B _{2, 2} And B _{2, 4} Image data obtained on the basis of the vertical direction evaluation value calculation circuit 44. The vertical direction evaluation value calculation circuit 44 calculates the vertical direction evaluation value Vv. The addition data S9, S10, S11, and S12 are input to the horizontal evaluation value calculation circuit 45. The horizontal evaluation value calculation circuit 45 calculates a horizontal evaluation value Vh.
[0088]
Data representing the evaluation value calculated by the direction evaluation value calculation circuit 40 is input to the determination circuit 50. The discrimination circuit 50 includes a first comparator 51, a second comparator 52, a third comparator 53, and a fourth comparator 54 to which data indicating the threshold value TH is given. .
[0089]
Data representing the right oblique evaluation value Vr and data representing the left oblique evaluation value Vl are input to the first comparator 51, and a process of subtracting the left oblique evaluation value Vl from the right oblique evaluation value Vr is performed. Is compared with the threshold value TH (step 7 in FIG. 6).
[0090]
Data representing the right oblique evaluation value Vr and data representing the left oblique evaluation value Vl are also input to the second comparator 52, and processing for subtracting the right oblique evaluation value Vr from the left oblique evaluation value Vl is performed. A comparison process between the value and the threshold value TH is performed (the process of step 8 in FIG. 6).
[0091]
Data representing the vertical direction evaluation value Vv and data representing the horizontal direction evaluation value Vh are input to the third comparator 53 and the fourth comparator 54. In the third comparator 53, processing for subtracting the horizontal direction evaluation value Vh from the vertical direction evaluation value Vv is performed, and comparison processing between the subtraction value and the threshold value TH is performed (processing in Step 3 in FIG. 6). Further, the fourth comparator 54 performs a process of subtracting the vertical direction evaluation value Vv from the horizontal direction evaluation value Vh, and performs a comparison process between the subtraction value and the threshold value TH (step 4 in FIG. 6).
[0092]
Data representing the comparison result of the first comparator 51 to the fourth comparator 54 is input to the comparator 55. In the comparator 55, the data indicating the input comparison result is compared, and data indicating the discrimination result is generated. The generated discrimination result data is given to the adaptive interpolation filter processing circuit 60. In the comparator 55, the following processing is performed with the same priority as the processing shown in steps 3, 5, 7, and 9 of FIG.
[0093]
In other words, if the first value obtained by subtracting the horizontal direction evaluation value Vh from the vertical direction evaluation value Vv is larger than the threshold value TH, the determination result data is generated so that the interpolation is performed using the horizontal interpolation filter. When the first value is less than or equal to the threshold value, if the second value obtained by subtracting the vertical direction evaluation value Vv from the horizontal direction evaluation value Vh is greater than the threshold value TH, interpolation is performed using the vertical interpolation filter. As a result, discrimination result data is generated. When both the first value and the second value are equal to or less than the threshold value TH, if the third value obtained by subtracting the left oblique evaluation value Vl from the right oblique evaluation value Vr is greater than the threshold value TH, Discrimination result data is generated so that interpolation is performed using an interpolation filter. The fourth value obtained by subtracting the right diagonal evaluation value Vr from the left diagonal evaluation value Vl when all of the first value, the second value, and the third value are equal to or less than the threshold value TH. If it is larger than the value TH, discrimination result data is generated so that interpolation is performed using the left diagonal interpolation filter. If any of the first value, the second value, the third value, and the fourth value is equal to or less than the threshold value TH, the image data representing the pixel of interest is obtained without performing the interpolation process. The discrimination result data is generated so as to be output.
[0094]
The adaptive interpolation filter processing circuit 60 includes a filter corresponding to the above-described Expression 15 to Expression 26 (if necessary, a filter corresponding to Expression 27 is also included). Based on the discrimination result data given from the discrimination circuit 50, an interpolation process is performed using a corresponding filter (or a process of outputting image data representing the pixel of interest without performing the interpolation process). Parallel RGB image data subjected to the interpolation processing is output from the adaptive interpolation filter 60 and is supplied to the RGB / YC conversion circuit 26 as described above.
[Brief description of the drawings]
FIG. 1 shows a part of a light receiving surface of a CCD.
FIG. 2 shows a combination of photodiodes when calculating an evaluation value in the right oblique direction.
FIG. 3 shows a combination of photodiodes when calculating a left oblique direction evaluation value.
FIG. 4 shows a combination of photodiodes when calculating a vertical direction evaluation value.
FIG. 5 shows a combination of photodiodes when a horizontal direction evaluation value is calculated.
FIG. 6 is a flowchart illustrating pixel interpolation processing.
FIG. 7 is a block diagram showing a part of the electrical configuration of a digital still camera.
FIG. 8 is a block diagram showing an electrical configuration of an RGB interpolation circuit.
[Explanation of symbols]
25 RGB interpolation circuit
30 delay circuit
40 direction evaluation value calculation circuit
50 discrimination circuit
60 Adaptive interpolation filter processing circuit
PD photodiode
S1, S2, S3, S4, S5, S6, S7, S8, S9, S10, S11, S12
Photodiode group

Claims (7)

Interpolated pixels by inputting serial three primary color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any of the three primary colors is formed on the light receiving surface In an interpolated pixel generation device that generates data representing
A first primary color image obtained based on a first photoelectric conversion element group including a plurality of photoelectric conversion elements including a photoelectric conversion element adjacent to a target photoelectric conversion element corresponding to a target pixel whose high correlation direction is to be detected. A plurality of photoelectric conversion elements in which the ratio of data and transmitted color components of color filters adjacent to the first photoelectric conversion element group and formed on the light receiving surface constitute the first photoelectric conversion element group Detecting means for detecting a direction in which the pixel of interest is highly correlated based on the second three-primary-color image data obtained based on the second photoelectric conversion element group composed of a plurality of photoelectric conversion elements equal to
According to the direction detected by the detecting means, a filter determining means for determining a filter that generates data representing a pixel to which the target pixel is to be interpolated, and the target pixel using the filter determined by the filter determining means Interpolation pixel generation means for generating data representing an interpolation pixel corresponding to
An interpolated pixel generation apparatus comprising: The interpolation pixel generation device according to claim 1, wherein the photoelectric conversion element of interest is included in the first photoelectric conversion element group. The interpolation pixel generation device according to claim 1, wherein a direction of the first photoelectric conversion element group and a direction of the second photoelectric conversion element group are perpendicular to a direction to be detected. The first photoelectric conversion element group includes a first plurality of photoelectric conversion element groups having a common photoelectric conversion element, and the second photoelectric conversion element group includes a second plurality of photoelectric conversion elements. Of photoelectric conversion elements,
The first plurality of photoelectric conversion element groups and the second plurality of photoelectric conversion element groups are arranged at target positions around the photoelectric conversion element of interest.
The interpolated pixel generation device according to claim 1. Input means for inputting serial three-primary-color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any one of the three primary colors is formed on the light-receiving surface;
A first photoelectric conversion comprising a plurality of photoelectric conversion elements including a photoelectric conversion element adjacent to a target photoelectric conversion element corresponding to a target pixel for detecting a direction having high correlation among the three primary color image data input to the input means. A first calculating means for calculating the level of the first three primary color image data obtained based on the element group;
Of the three primary color image data input to the input means, the ratio of the transmission color component of the color filter adjacent to the first photoelectric conversion element group and formed on the light receiving surface is the first photoelectric conversion element. Second calculation means for calculating the level of second primary color image data obtained based on a second photoelectric conversion element group composed of a plurality of photoelectric conversion elements equal to the plurality of photoelectric conversion elements constituting the group; Detecting means for detecting a direction in which the pixel of interest is highly correlated based on the level calculated by the first calculating means and the level calculated by the second calculating means;
A correlation direction detection apparatus comprising: Interpolated pixels by inputting serial three primary color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits a light component of any of the three primary colors is formed on the light receiving surface In an interpolated pixel generation device that generates data representing
A first primary color image obtained based on a first photoelectric conversion element group including a plurality of photoelectric conversion elements including a photoelectric conversion element adjacent to a target photoelectric conversion element corresponding to a target pixel whose high correlation direction is to be detected. A plurality of photoelectric conversion elements in which the ratio of data and transmitted color components of color filters adjacent to the first photoelectric conversion element group and formed on the light receiving surface constitute the first photoelectric conversion element group Based on the second three-primary-color image data obtained based on the second photoelectric conversion element group consisting of a plurality of photoelectric conversion elements equal to, the direction in which the pixel of interest is highly correlated is detected,
According to the detected direction, determine a filter that generates a pixel to interpolate the pixel of interest,
Generating an interpolated pixel corresponding to the pixel of interest using the determined filter;
Interpolated pixel generation method. Input serial three-primary-color image data output from a solid-state electronic image sensor including a large number of photoelectric conversion elements in which a color filter that transmits light components of any one of the three primary colors is formed on the light-receiving surface;
Based on the first photoelectric conversion element group composed of a plurality of photoelectric conversion elements including the photoelectric conversion elements adjacent to the target photoelectric conversion element corresponding to the target pixel whose detected direction is highly correlated among the input three primary color image data. Calculating the first level of the first three primary color image data obtained by
Of the input three primary color image data, the ratio of the transmitted color components of the color filter adjacent to the first photoelectric conversion element group and formed on the light receiving surface constitutes the first photoelectric conversion element group. Calculating a second level of second primary color image data obtained based on a second photoelectric conversion element group composed of a plurality of photoelectric conversion elements equal to the plurality of photoelectric conversion elements;
Based on the calculated first level and second level, a direction in which the pixel of interest is highly correlated is detected.
Correlation direction detection method.

Images