JP2018006881A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of acquiring satisfactory image data by using only an acquired signal value to perform appropriate correction on a function pixel.SOLUTION: An image processing apparatus comprises: imaging means 200 including multiple imaging pixels that are disposed in a two-dimensional shape, and multiple function pixels having sensitivity which is different from that of the imaging pixels; a boundary discrimination part for discriminating whether a function pixel is present in a boundary part; and a gain value calculation part for calculating a gain value with respect to the function pixel. The boundary part is a location where a signal value is changed discontinuously. The gain value calculation part uses signal values outputted from the imaging pixels disposed around a function pixel for which the boundary discrimination part discriminates that the function pixel is not present in the boundary part, to calculate a gain value with respect to a function pixel for which the boundary discrimination part discriminates that the function pixel is present in the boundary part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus and an image processing method for an imaging signal output from a functional pixel position in an imaging device mounted on a digital still camera or the like.

  The digital still camera has an image pickup device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor in order to photoelectrically convert light incident from a lens system. An imaging element is generally configured by two-dimensionally arranging imaging pixels. In addition, a technique for mounting a functional pixel having a specific function is widely known. Here, the functional pixel in this document refers to a pixel having different sensitivity with respect to the imaging pixel.

  For example, Patent Document 1 discloses an imaging element having a focus detection pixel having a pupil division function as a functional pixel. There are two types of focus detection pixels in which the sensitivity region of the light receiving unit is decentered, and focus detection is performed by detecting an image shift amount from a signal output from each pixel.

  As a photoelectric conversion characteristic of a functional pixel such as a focus detection pixel, there is a characteristic that the responsiveness of the signal value with respect to the light amount is linearly lower than that of the imaging pixel. Further, the signal attenuation rate varies greatly depending on the combination of the characteristics of the lens and the image sensor. Therefore, in order to handle a signal output from the functional pixel as an image pickup signal for image data generation, it is necessary to perform appropriate correction.

  As one of the correction methods for functional pixels, gain correction for performing signal amplification is known. The gain correction functions as an appropriate correction if the signal value obtained from the functional pixel and the gain value for the signal value are reliable values.

  However, on the other hand, when noise due to an image sensor or an electronic circuit occurs, a reliable signal value cannot be obtained, so that a suitable correction value cannot be obtained by gain correction. In addition, a suitable correction value cannot be acquired even when the gain value is not appropriate due to some factors such as the state of the lens and the shooting environment. In such a case, if a subject with no significant change in contrast is imaged, a correction value that deviates from the signal value output from the surrounding imaging pixels is calculated, so that a significant correction error appears.

  On the other hand, interpolation correction is known as another correction method for functional pixels. Interpolation correction is correction performed by referring to signal values of imaging pixels located around functional pixels, and functions as appropriate correction in many cases.

  However, characteristic information is lost because interpolation correction is a kind of averaging process. Therefore, when interpolation correction is performed on all functional pixels, the resolution of the image data is impaired.

  For example, when a functional pixel is located on the edge of a subject whose contrast changes greatly, it is conceivable that the signal value to be obtained is significantly different from the signal values of surrounding imaging pixels. In such a case, interpolation from surrounding pixels does not work favorably, and the sharpness of the edge portion may be reduced. The same applies when the imaging pixel to be referenced is located on the edge.

  Therefore, for functional pixels that exist in the non-edge region, interpolation correction is performed to avoid an error case due to some cause, and for functional pixels that exist near the edge, the signal value obtained from the functional pixel is used as much as possible. A correction method using both corrections, in which gain correction is performed in order to utilize, is effective.

  When performing such correction, it is necessary to set an appropriate gain value for a functional pixel existing near the edge.

  In this document, the focus detection pixel located near the edge hereinafter refers to a focus detection pixel located on the edge. In addition, focus detection pixels in which an imaging pixel to be referred to when performing interpolation correction is located on the edge are also shown. The focus detection pixel located in the non-edge region indicates a focus detection pixel excluding the focus detection pixel located near the edge.

  As a method for setting the gain value, for example, Patent Document 2 can be cited. According to the invention disclosed in Patent Document 2, the imaging apparatus sets a gain amount that seems to be appropriate from information obtained by communication from the optical apparatus, and gives linear correction to the focus detection pixels.

  As a result, it is possible to accurately estimate the output of the imaging pixel that should originally be at the position of the focus detection pixel.

Japanese Patent No. 3592147 JP 2006-108955 A

  However, in the invention described in Patent Document 2, it is essential to obtain information from the optical device. Accordingly, there is a problem that an appropriate gain value cannot be set when communication with the optical device is impossible or when the optical device is incompatible as a result of communication.

  In particular, since the gain value changes depending on the characteristics of the optical device, the characteristics of the image sensor, and the optical information at the time of shooting, it is difficult to set an appropriate gain value when information cannot be obtained from the optical device. It becomes. Accordingly, there is a need for a means for setting an appropriate gain value using only the signal value acquired from the image sensor.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus and an image processing method capable of appropriately correcting a functional pixel by using only acquired signal values and acquiring good image data. To do.

  According to the first aspect of the present invention, there is provided an imaging means having a plurality of imaging pixels arranged two-dimensionally and a plurality of functional pixels having different sensitivities from the imaging pixels, and the functional pixels are boundary portions. A boundary determination unit that determines whether or not the function pixel exists, and a gain value calculation unit that calculates a gain value for the functional pixel. The boundary unit is a portion where the signal value changes discontinuously, and the gain value calculation Determined that the boundary determination unit exists in the boundary part using a signal value output from the imaging pixels arranged around the functional pixel determined that the boundary determination unit does not exist in the boundary part An image processing apparatus that calculates a gain value for the functional pixel.

  In the invention according to claim 2, the gain calculation unit outputs from the imaging pixels arranged around a predetermined functional pixel with respect to the functional pixel determined that the boundary determination unit does not exist in the boundary. The estimated gain value is calculated by dividing the representative value of the signal value to be divided by the signal value output from the predetermined functional pixel, and the estimated gain value is spatially interpolated to calculate the estimated gain value. The image processing apparatus according to claim 1, wherein a gain value is calculated.

  According to a third aspect of the present invention, in the spatial interpolation of the estimated gain value, the gain value for each predetermined group is calculated using the estimated gain value, and the gain value for each predetermined group is spatially interpolated. The image processing apparatus according to claim 2, wherein the image processing apparatus is performed.

  According to a fourth aspect of the present invention, the gain correction is performed by performing gain correction using the gain value calculated by the gain value calculation unit with respect to the functional pixel determined that the boundary determination unit exists in the boundary part. 4. The method according to claim 1, further comprising: an interpolation correction unit configured to perform interpolation correction on the functional pixel that is determined not to exist in the boundary part by the boundary determination unit. This is an image processing apparatus.

  The invention described in claim 5 is characterized in that the boundary determination unit determines whether or not the functional pixel exists in the boundary by calculating a variation of the imaging pixels arranged around the functional pixel. An image processing apparatus according to claim 1.

  6. The image processing apparatus according to claim 1, wherein the imaging unit includes a functional pixel group in which a plurality of functional pixels are arranged adjacent to each other. It is.

  According to a seventh aspect of the present invention, the functional pixel is a focus detection pixel that photoelectrically converts a divided light beam among the light beams from the imaging optical system that guides the light beam to the imaging unit. An image processing apparatus according to claim 6.

  The invention described in claim 8 is characterized in that the functional pixel is a saturation prevention pixel having different sensitivity by adopting a different structure or a filter with respect to the imaging pixel. 6. The image processing apparatus according to claim 6.

  The invention described in claim 9 is characterized in that the imaging means is a stacked solid-state imaging device in which a plurality of light receiving elements each having a plurality of layers of light receiving portions stacked in the depth direction are arranged on a semiconductor substrate. An image processing apparatus according to any one of claims 1 to 8.

  According to a tenth aspect of the present invention, there is provided an imaging device having the image processing device, wherein the gain value for the functional pixel is obtained when the imaging device cannot discriminate optical device information from an optical device mounted on the imaging device. The imaging device according to claim 1, wherein the imaging device is calculated.

  The invention described in claim 11 is an image processing method of an imaging means having a plurality of imaging pixels arranged two-dimensionally and a plurality of functional pixels having different sensitivities from the imaging pixels, The determination unit determines whether the functional pixel is present at a boundary portion where the signal value changes discontinuously, the gain calculation unit calculates a gain value for the functional pixel, and the boundary determination unit faces the boundary. A gain value for the functional pixel is calculated using a signal value output from the imaging pixels arranged around the functional pixel determined not to be.

  According to the present invention, it is possible to obtain an image processing apparatus and an image processing method capable of appropriately correcting a functional pixel by using only the acquired signal value and acquiring good image data.

It is the block diagram which showed the main structures of the camera system which is one Embodiment of this invention. It is sectional drawing which simplified the single pixel of the image pick-up element mounted in the imaging device shown in FIG. It is the schematic diagram which showed the pixel arrangement | sequence row | line of the image pick-up element which has arrange | positioned the focus detection pixel. It is the schematic diagram which showed the structure of the focus detection pixel. It is the schematic diagram which showed the relationship between X coordinate and a gain value. It is the block diagram which showed the structure of the correction process part. It is the schematic diagram which extracted 3 * 4 pixel centering on the focus detection pixel group. It is a schematic diagram which shows gain value calculation in an image sensor. It is the schematic diagram shown about the spatial interpolation of the gain value for every X coordinate. It is a schematic diagram which shows replacement of a value. It is a flowchart of the signal value correction process of a focus detection pixel. It is the schematic diagram which showed the pixel arrangement | sequence row | line of the image pick-up element which has arrange | positioned the saturation prevention pixel. It is the schematic diagram which extracted 4x4 pixels centering on the saturation prevention pixel group.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

  The block diagram shown in FIG. 1 shows the main configuration of a camera system according to an embodiment of the present invention. The camera system shown in the figure includes an imaging device 100, an optical device 101, a photographing optical system 110, an imaging device 200, a signal processing unit 150, a correction processing unit 152, an image processing unit 154, and a memory unit 160. A camera CPU 170, a user interface (I / F) 171, a recording medium interface (I / F) 172, and an image display unit 190.

  The optical device 101 can be exchanged for the imaging device 100 and includes a lens CPU (not shown). The photographing optical system 110 includes a plurality of lens groups (not shown) including a focus lens group and a zoom lens group. In this figure, only one lens is shown for simplicity.

  The image sensor 200 receives the light beam collected by the photographing optical system 110, photoelectrically converts it, and outputs it as a signal value. A CMOS image sensor is used as the imaging device 200 of the present embodiment.

  The light receiving surface of the image sensor 200 is composed of a large number of pixels. These pixels are vertical color separation type image sensors that can output RGB color component signals from a single pixel by using the difference in depth that is photoelectrically converted depending on the wavelength of incident light. Details of the vertical color separation type image sensor will be described later.

  Focus detection pixels are periodically arranged in the photodiode 210 located on the uppermost surface of the image sensor 200. Details of the focus detection pixel will be described later.

  The image sensor 200 of this embodiment incorporates a variable gain amplifier that amplifies a color component signal read from a pixel and an A / D converter that digitally converts an analog image signal.

  The signal processing unit 150 performs normal signal processing on the signal output from the image sensor 200. Normal signal processing includes dark processing, linearization, and the like, which are the same as known techniques, and thus description thereof is omitted. The signal output from the image sensor 200 is subjected to various signal processing and then temporarily recorded as raw data in the memory unit 160. In addition, the memory unit 160 functions as a buffer for various data.

  The correction processing unit 152 performs processing for correcting the signal value output from the functional pixel with respect to the Raw data read from the memory unit 160. Details of the signal value correction processing of the functional pixel will be described later.

  The image processing unit 154 performs normal image processing on the image data corrected by the correction processing unit 152. Normal image processing includes white balance adjustment, color tone correction processing, gradation conversion processing, and the like, which are the same as those in the publicly known technology, and thus description thereof is omitted.

  The camera CPU 170 performs comprehensive control of the entire imaging apparatus 100. In particular, the camera CPU 170 controls the signal processing unit 150, the correction processing unit 152, and the image processing unit 154.

  For example, the user I / F 171 includes operation members such as a release button, a power button, a command dial, and a cross key. When the user operates these operation members, the camera CPU 170 issues an instruction to perform a predetermined operation. .

  The recording medium I / F 172 records or reads RAW data and developed image data with a recording medium (not shown). This recording medium is a detachable recording medium such as a semiconductor memory.

  The image display unit 190 displays image data processed by the image processing unit 154, image data read from a recording medium (not shown), and the like.

  Note that when the imaging element 200 that does not include the above-described variable gain amplifier, gain correction circuit, and A / D converter is employed, these devices may be mounted individually.

  FIG. 2 is a cross-sectional view schematically showing a single pixel of the image sensor 200 mounted on the image pickup apparatus 100. As described above, the image sensor 200 of the present embodiment is a so-called vertical color separation type image sensor, and each pixel is formed by stacking three photodiodes in the depth direction.

  When light enters a certain pixel, the blue (B) component in the incident light is photoelectrically converted mainly by the photodiode 210 located on the uppermost surface. Similarly, the green (G) component in the incident light is mainly photoelectrically converted by the photodiode 230 located at the intermediate depth, and the red (R) component is mainly photoelectrically converted by the photodiode 250 located at the lowermost layer. . These vertical color separations utilize the characteristics of silicon (Si) used as the material of the image sensor 200.

  These three photodiodes are formed by performing a predetermined doping process at different depths inside the Si substrate. Specifically, the B component photodiode 210 is formed to a depth of about 0.2 to 0.5 μm, and the G component photodiode 230 is formed to a depth of about 0.5 to 1.5 μm. The R component photodiode 250 is formed to a depth of about 1.5 to 3.0 μm.

  Therefore, the image sensor 200 according to the present embodiment can acquire color component signals of three colors of RGB with one pixel, while the color filter essential for the Bayer image sensor is unnecessary. Since each pixel can photoelectrically convert all three color wavelength components, there is a merit that it is not necessary to perform essential pixel interpolation in the Bayer image sensor.

  Note that the present invention is not limited to the above configuration, and a plurality of photoelectric conversion films having specific absorption characteristics formed of an organic material, an inorganic material, or the like may be stacked.

  Further, a configuration using an image sensor having a standard structure in which a color filter is mounted on a single-layer sensor may be used.

Example 1
FIG. 3 is a schematic diagram showing a pixel arrangement row of the image sensor 200 in which focus detection pixels are arranged in a part of the photodiode matrix located on the uppermost surface. In the following embodiments, an imaging pixel refers to a pixel that is not a focus detection pixel. Ln and Rn are left-eye and right-eye focus detection pixels that are paired with each other, and are disposed adjacent to each other.

  For example, the left eye focus detection pixel L1 and the right eye focus detection pixel R1 are paired with each other to form a focus detection pixel group S1. Similarly, the left eye focus detection pixel Ln is paired with the right eye focus detection pixel Rn to form a focus detection pixel group Sn. Here, n is an arbitrary natural number, and is set according to the necessity of the image sensor 200. It is assumed that the detection pixel group Sn is periodically arranged.

  Each focus detection pixel is also represented by XY coordinates with the origin at the upper left of the image sensor 200. The XY coordinates are coordinates for the focus detection pixel. The left-eye focus detection pixel and the right-eye focus detection pixel are not distinguished, and the imaging pixels are excluded and counted.

  Next, the focus detection pixel will be described. The focus detection pixel is a pixel arranged to read out distance measurement data for autofocus (AF) in the digital camera. By performing phase difference detection using a signal output from the image sensor 200, highly accurate AF is possible.

  The structure of the left eye focus detection pixel Ln and the right eye focus detection pixel Rn will be described with reference to FIG. FIG. 4A is a schematic diagram showing the structure of the left eye focus detection pixel Ln. A microlens 270 is formed on the light incident side of the left eye focus detection pixel Ln. The left-eye focus detection pixel Ln has an opening 290 that is eccentric, and has a structure in which the amount of incident light is limited by shielding approximately half of the area of the pixel.

FIG. 4B is a schematic diagram showing the structure of the right eye focus detection pixel Rn. The right eye focus detection pixel Rn has a structure having an opening 290 at a symmetrical position across the optical axis of each microlens 270, as compared with the left eye focus detection pixel Ln.

  The focus detection process by the focus detection pixel is a known technique disclosed in, for example, Patent Document 1, and description thereof in this embodiment is omitted.

  The signal output from the focus detection pixel is used for focus detection, but at the same time, it is also used for generating image data as an image signal. Here, the focus detection pixel has a structure in which approximately half of the entire pixel is coated and the amount of incident light is limited. Therefore, the focus detection pixel has approximately half the sensitivity compared to the imaging pixel, and has the same light intensity as the imaging pixel. On the other hand, the signal value decreases. Therefore, in order to use the signal value output from the focus detection pixel for the image data, it is necessary to perform a correction process on the signal value.

  In the correction method for the focus detection pixel, interpolation correction is performed on the focus detection pixel located in the non-edge region from the imaging pixels located in the periphery, and on the other hand, the interpolation detection is located near the edge which is an error part. It is effective to perform gain correction on the focus detection pixels.

  In order to perform appropriate gain correction on the focus detection pixels located near the edge, it is necessary to set an appropriate gain value. The gain value for the focus detection pixel is determined by the structure of the pixel and the incident ray angle. Further, the incident ray angle is determined by the coordinates of the focus detection pixel in the image sensor 200.

  Here, in the focus detection pixel according to the present embodiment, the pupil division direction is the X direction. Therefore, the difference in incident light angle in the Y direction does not substantially affect the amount of incident light, and pixels having the same X coordinate are incident light. The amount difference is negligible. Therefore, in the present embodiment, it is assumed that the gain value is the same if the X coordinate is the same.

  FIG. 5 is a schematic diagram showing the relationship between the X coordinate and the gain value for the left-eye focus detection pixel of this embodiment. In the schematic diagram of FIG. 5, the vertical axis represents the gain value, the horizontal axis represents the X coordinate, the curves a and b represent the relationship curve between the X coordinate and the gain value.

  As described above, the focus detection pixel is pupil-divided in the X direction, and the right half region of the pixel is shielded from light. In the case of such a structure, the focus detection pixel located to the left of the center of the image sensor 200 has a reduced amount of incident light compared to the vicinity of the center due to the incident light angle. Accordingly, since it is necessary to apply a strong gain, the gain value takes a high value. Conversely, the focus detection pixel located to the right of the center of the image sensor 200 has a low gain value because the amount of incident light increases.

  Further, the gain value with respect to the central X coordinate in the horizontal direction of the image sensor 200 is approximately 2.0 times. This is due to the fact that light rays ideally enter the image sensor 200 perpendicularly so that only the light rays in the coating portion are blocked. In this embodiment, the film portion has a substantially half area with respect to the entire pixel, so that the gain value is approximately 2.0 times.

  FIG. 5 is a schematic diagram for the left-eye focus detection pixel. In the case of the right-eye focus detection pixel, a figure having a substantially inverted shape across the X coordinate axis is obtained.

  Curves a and b in FIG. 5 indicate the relationship between the X coordinate and the gain value under different conditions. The incident ray angle is determined by the imaging conditions and the characteristics of the optical device, and the relationship curve between the X coordinate and the gain value takes various shapes depending on the conditions. If such a relationship curve between the X coordinate and the gain value is calculated, a suitable gain value for all focus detection pixels can also be calculated.

  Therefore, conventionally, when the imaging apparatus acquires information from the optical apparatus, a relationship curve between the X coordinate and the gain value is calculated, and a suitable gain value for the focus detection pixel is calculated. However, in the present invention, by using a group of focus detection pixels having the same gain value, a suitable gain value is set without using information necessary for calculation.

  That is, the correction processing unit 152 in the present embodiment first calculates an estimated gain value for the focus detection pixels existing in the non-edge region from the output ratio between the focus detection pixels and the imaging pixels located in the vicinity thereof. Subsequently, the individual estimated gain values calculated are grouped for each value with the common X coordinate of the focus detection pixel, and a representative value for each group is taken. The relationship curve between the X coordinate and the gain value is estimated by performing spatial interpolation using the representative value for each identical X coordinate group thus obtained as an element. By calculating the gain value for each identical X coordinate group as described above and applying it to the focus detection pixels located near the edge, a suitable gain value is set using only the signal value acquired from the image sensor 200. Is possible.

  The main configuration of the correction processing unit 152 is shown in the block diagram shown in FIG. The correction processing unit 152 includes a gain value calculation unit, a gain value interpolation unit, a gain correction unit, and an interpolation correction unit. Hereinafter, the signal value correction processing of the focus detection pixel in the correction processing unit 152 will be described.

  The raw data read from the memory unit 160 to the correction processing unit 152 is first subjected to determination (hereinafter referred to as edge determination) as to whether the focus detection pixel is located near the edge by the gain value calculation unit. The above determination is performed for all focus detection pixels, and the determination result is recorded in the memory unit 160.

  The edge determination is performed by calculating and evaluating the variation of the imaging pixels located around the target focus detection pixel. In this embodiment, the difference between the maximum value and the minimum value of imaging pixels located in the vicinity is used as the variation. However, it is possible to use dispersion as a variation, and the present invention is not limited to this.

  FIG. 7 is a diagram in which 3 × 4 pixels are extracted from the pixel arrangement row of the image sensor 200 illustrated in FIG. 3 with the focus detection pixel group S1 as the center. The left-eye focus detection pixel L1 and the right-eye focus detection pixel R1 use TL, TUL, TC, TR, TUR, BL, BLL, BC, BR, and BLR as imaging pixels located in the vicinity. Further, since the left eye focus detection pixel L1 and the right eye focus detection pixel R1 are arranged adjacent to each other, if one is located near the edge, the other is also located near the edge.

ただし、ｓ＿ＴＬ、ｓ＿ＴＵＬ、ｓ＿ＴＣ、ｓ＿ＴＲ、ｓ＿ＴＵＲ、ｓ＿ＢＬ、ｓ＿ＢＬＬ、ｓ＿ＢＣ、ｓ＿ＢＲ、ｓ＿ＢＬＲ、ｓ＿Ｌ１、ｓ＿Ｒ１、は各画素から出力される信号値である。 In this embodiment, the variation is calculated by normalizing the difference between the maximum value and the minimum value of imaging pixels located in the vicinity with the signal value of the focus detection pixel group. When the maximum value of imaging pixels located in the periphery is defined as M, the minimum value is defined as m, and the variation is defined as V, M, m, and V are expressed by the following equations.

However, s_TL, s_TUL, s_TC, s_TR, s_TUR, s_BL, s_BLL, s_BC, s_BR, s_BLR, s_L1, and s_R1 are signal values output from each pixel.

  In the edge determination, it is evaluated whether or not the calculated variation exceeds the boundary value. The magnitude of the variation indicates the magnitude of the signal difference between the imaging pixels located in the vicinity, that is, if the variation is greater than or equal to the boundary value, it means that the focus detection pixel is located near the edge, and if less than the boundary value This means that the focus detection pixel is located in the non-edge region. Here, the boundary value is a numerical value that can be freely set, and may be set as appropriate according to the needs of the practitioner. The edge determination is performed as described above.

  Next, for the focus detection pixel determined to be located in the non-edge region, the gain value calculation unit calculates an interpolation estimated value from the signal value of the imaging pixel located in the vicinity thereof. The interpolation estimated value is a representative value with the signal value of the imaging pixel located in the vicinity as an element. In the present embodiment, the median is calculated as the representative value thereafter, but the present invention is not limited to this and may be an average value or the like.

The calculated interpolation estimated value is a value estimated in an interpolated manner based on the signal value of the surrounding pixels with respect to the signal value of the predetermined focus detection pixel. By dividing this interpolation estimated value by the actual signal value of the focus detection pixel, an estimated gain value which is a virtual gain value of the focus detection pixel is calculated. The calculated estimated gain value is recorded in the memory unit 160. When the interpolation estimated value is I and the estimated gain value is G_E, for example, the estimated gain value of the left-eye focus detection pixel L1 is expressed by the following equation.

  The gain value interpolation unit reads the recorded estimated gain value and performs spatial interpolation of the gain value. By performing such interpolation, it is possible to calculate the gain value related to the focus detection pixel determined to be located near the edge. Since interpolation of gain values needs to be performed between pixels having the same structure, it is performed separately for the right eye focus detection pixel and the left eye focus detection pixel.

  FIG. 8 is a schematic diagram showing gain value calculation in the image sensor 200. Hereinafter, the spatial interpolation method of the gain value in the left eye focus detection pixel will be described with reference to FIG. The set of numbers in FIG. 8 is the coordinates of the left eye focus detection pixel, and the Y coordinate takes only an odd number. Here, m and n are arbitrary natural numbers, and are appropriately set according to the needs of the practitioner. In addition, a square frame covering the coordinates (2.3) or the like means that the focus detection pixel is located on the edge.

  The gain value interpolation unit first calculates a representative value for each X coordinate. As described above, the gain value is a numerical value determined for each X coordinate by the amount of incident light. Therefore, the estimated gain value for each focus detection pixel at the same X coordinate calculated by the gain value calculation unit should be the same. However, the estimated gain value is only a value estimated from the signal values of the surrounding pixels of the focus detection pixel, and since the accuracy thereof is low, the same numerical value cannot actually be obtained. Therefore, by obtaining representative values using these as elements, a gain value with higher accuracy is estimated in each identical X coordinate group. Although the median value is calculated as the representative value, the present invention is not limited to this.

  For example, the representative value of the same X coordinate group whose X coordinate is 1 is calculated using the estimated gain value of the left-eye focus detection pixel from the coordinates (1.1) to the coordinates (1.2n−1) as an element. The representative value of the same X coordinate group whose X coordinate is 2 is calculated in the same manner, but since the coordinate (2.3) is located on the edge, the variation in edge determination exceeds the boundary value. Therefore, the estimated gain value is not calculated. Accordingly, the estimated gain value from the coordinate (2.1) to the coordinate (2.2n-1) excluding the coordinate (2.3) is calculated as an element. It is assumed that all focus detection pixels are located on the edge for the same X coordinate group whose X coordinate is 3. Then, since there is no focus detection pixel for which the estimated gain value is calculated, the representative value in this group is not calculated.

  Here, for a pixel group that does not have an estimated gain value, such as the same X coordinate group having an X coordinate of 3, a gain value must be estimated to give a value. In addition, in the case of a pixel group with a small number of focus detection pixels determined to be located in the non-edge region, the accuracy of the representative value may be insufficient. Therefore, spatial gain interpolation for each X coordinate is performed using the fact that the gain value changes smoothly in space.

  FIG. 9 is a schematic diagram showing spatial interpolation of gain values for each X coordinate. In this embodiment, spatial interpolation is performed by polynomial approximation of representative values using the X coordinate as a parameter. In the schematic diagram of FIG. 9, the vertical axis represents the gain value, the horizontal axis represents the X coordinate, the black dot represents the representative value for each X coordinate, and the curve represents an approximate curve obtained by polynomial approximation of the representative value for each X coordinate. .

  The approximate curve calculated in this way is a curve estimated as a relationship curve between the X coordinate and the gain value under a specific condition. Since the approximate curve uses the signal values of all focus detection pixels existing in the non-edge region as elements, high-precision estimation is performed.

  FIG. 10 is a schematic diagram showing value replacement. A white circle in FIG. 10 indicates the gain value after replacement corresponding to the representative value for each X coordinate. The representative values for all the X coordinates are replaced with values obtained by substituting the X coordinates into the approximate polynomial. Also, the gain value is calculated by the approximate polynomial for the X coordinate for which no representative value exists.

  As described above, by replacing the representative value for each X coordinate with the gain value obtained by the approximate curve, a highly accurate gain value is calculated for all the representative values for each X coordinate. Further, an outlier that is a representative value with insufficient accuracy, such as a large deviation from the approximate curve, can be corrected. Further, it is possible to calculate a gain value for an X coordinate having no representative value such as the same X coordinate group having an X coordinate of 3.

  Next, gain correction processing in the gain correction unit will be described. The gain correction process is performed by multiplying the signal value obtained from the focus detection pixel located near the edge by the gain value at the X coordinate calculated by the gain value interpolation unit.

  Next, the interpolation correction process in the interpolation correction unit will be described. As described above, the interpolation correction works more favorably on the focus detection pixels located in the non-edge region. The interpolation correction unit performs an interpolation correction process on the focus detection pixels located in the non-edge region by using the imaging pixels located around the focus detection pixels. For example, a median value may be calculated for the signal values of the imaging pixels located in the vicinity and replaced with the signal values of the focus detection pixels.

  FIG. 11 shows a flowchart of the signal value correction process of the focus detection pixel in this embodiment. Hereinafter, the signal value correction processing of this embodiment will be described with reference to the drawings.

  When a signal value correction processing command is transmitted from the camera CPU 170 in step S101, raw data is read from the memory unit 160 to the correction processing unit 152 in step S102.

  In step S103, the gain value calculation unit determines whether a predetermined focus detection pixel is located near the edge. If it is determined that the predetermined focus detection pixel is located near the edge, flag = 1 is recorded in the memory unit 160 for the predetermined pixel in step S104.

  On the other hand, when it is determined that the predetermined focus detection pixel is located in the non-edge region, flag = 0 is recorded in the memory unit 160 for the predetermined pixel in step S105. Subsequently, an interpolation estimated value is calculated for a predetermined focus detection pixel, and an estimated gain value is calculated. The estimated gain value is recorded in the memory unit 160 similarly to the flag.

  In step S107, it is determined whether edge determination has been performed for all focus detection pixels. If edge determination has not been performed for all focus detection pixels, the process returns to step S103, and edge determination is performed for pixels for which edge determination has not been performed.

  When edge determination is performed for all focus detection pixels, gain value spatial interpolation is performed in the gain value interpolation unit in step S108. First, the calculated estimated gain value is read from the memory unit 160 for the focus detection pixel in which flag = 0 is recorded. A representative value for each X coordinate is calculated from the read estimated gain value, and further, a spatial correction of the gain value using the representative value for each X coordinate as an element is performed. The calculated gain values at all X coordinates are stored in the memory unit 160.

  In step S109, a determination is made as to whether a predetermined focus detection pixel is flag = 1. When flag = 1, since the focus detection pixel is located near the edge, a gain correction process is performed in the gain correction unit in step S110. At this time, a gain value corresponding to the X coordinate of a predetermined focus detection pixel is read from the memory unit 160 and applied.

  On the other hand, when the predetermined focus detection pixel is flag = 0, since it is a focus detection pixel located in the non-edge region, an interpolation correction process is performed in the interpolation correction unit in step S111.

  In step S112, it is determined whether correction processing has been performed for all focus detection pixels. If correction processing has not been performed for all focus detection pixels, the process returns to step S109, and correction processing is performed on pixels for which correction processing has not been performed.

  When the correction process is performed on all focus detection pixels, the signal value correction process ends. Here, this flow is performed separately for the right eye focus detection pixel and the left eye focus detection pixel. Therefore, this process is performed twice so that the signal value correction process is performed on both detection pixels.

  In steps S101 to S107, the processing of the right eye focus detection pixel and the left eye focus detection pixel may be shared. In that case, Step S108 to Step S113 are performed separately for the right eye focus detection pixel and the left eye focus detection pixel.

  As described above, since the imaging apparatus 100 according to the present embodiment can calculate the gain value from the focus detection pixels in the non-edge region for the focus detection pixels located near the edge, the signal value acquired from the image sensor 200 is obtained. Gain correction processing can be performed using only Therefore, even in the case where information on the mounted optical device cannot be acquired, a preferable correction process can be performed by using only the signal value acquired from the image sensor 200.

  Also, by performing polynomial approximation, the estimation error that can be held by the gain values of all focus detection pixels can be spatially removed, reducing the error cases caused by the estimation error in gain value setting for the final image data it can.

  In addition, gain correction is performed when the focus detection pixel is located near the edge, and interpolation correction is performed when the focus detection pixel is located in the non-edge region. Therefore, the final image is estimated without loss of sharpness at the edge. Correction processing with less error is possible.

(Example 2)
In the first embodiment, focus detection pixels are arranged as functional pixels. However, the present invention can also be applied to other functional pixels. The invention in the second embodiment is an image processing method of the imaging apparatus 100 including the imaging element 200 having a saturation prevention pixel as a functional pixel.

  The saturation prevention pixel is a functional pixel having a lower sensitivity than the imaging pixel by a method of shielding a part of the pixel or installing a filter for suppressing the transmittance of the incident light amount in the pixel. Since it has a lower sensitivity than the imaging pixel, it can output a signal value without saturation even when the signal value of the imaging pixel is saturated. Installed to make corrections.

  FIG. 12 is a schematic diagram showing a pixel arrangement row of the image sensor 200 in which saturation prevention pixels are arranged in a part of the photodiode matrix located on the uppermost surface. LTn, RTn, LBn, and RBn are saturation prevention pixels, and are arranged so that each pixel is adjacent to two other saturation prevention pixels. Here, n is an arbitrary natural number, and is set according to the necessity of the image sensor 200.

  For example, the upper left saturation prevention pixel LT1, the upper right saturation prevention pixel RT1, the lower left saturation prevention pixel LB1, and the lower right saturation prevention pixel RB1 constitute a saturation prevention pixel group U1.

  The saturation prevention pixel in this embodiment has a structure in which the amount of incident light is limited by shielding the periphery of the pixel. The central portion of the pixel is an opening, and the opening has an area that is approximately a quarter of the opening of the imaging pixel. In addition, all the saturation prevention pixels in the image sensor 200 have the same structure.

  The signal value output from the saturation prevention pixel contributes to signal processing for preventing saturation when the peripheral pixel is saturated. On the other hand, when the peripheral pixels are not saturated, they are used only for generating image data as image signals. Here, since the saturation prevention pixel has a structure in which the amount of incident light is limited, it has only about one-fourth the sensitivity as compared with the imaging pixel, and the signal value decreases for the same amount of light as compared with the imaging pixel. End up. Therefore, in order to use the signal value output from the saturation prevention pixel for the image data, it is necessary to perform a correction process on the signal value as in the first embodiment.

  Here, since the central portion of the saturation prevention pixel in this embodiment is an opening, the difference in the amount of vignetting is negligible between pixels having the same image height. Therefore, in this embodiment, a focus detection pixel group having a common gain value for each image height is formed.

  The method for the signal value correction processing is the same as that in the first embodiment, but the method for calculating the variation is different and will be described below. FIG. 13 is a diagram in which 4 × 4 pixels are extracted from the pixel arrangement column of the image sensor 200 illustrated in FIG. 12 with the saturation prevention pixel group U1 as the center.

ただし、ｓ＿ＬＴＴ、ｓ＿ＬＵＬ、ｓ＿ＬＴＬ、ｓ＿ＲＴＴ、ｓ＿ＲＵＲ、ｓ＿ＲＴＲ、ｓ＿ＬＢＬ、ｓ＿ＬＬＬ、ｓＬＢＢ、ｓ＿ＲＢＲ、ｓ＿ＲＬＲ、ｓ＿ＲＢＢ、ｓ＿ＬＴ１、ｓ＿ＲＴ１、ｓ＿ＬＢ１、ｓ＿ＲＢ１は各画素から出力される信号値である。 The variation in the second embodiment is calculated by normalizing the difference between the maximum value and the minimum value of the imaging pixels located around the saturation prevention pixel group U1 with the signal value of the focus detection pixel group. When the maximum value of imaging pixels located in the periphery is defined as M, the minimum value is defined as m, and the variation is defined as V, M, m, and V are expressed by the following equations.

However, s_LTT, s_LUL, s_LTL, s_RTT, s_RUR, s_RTR, s_LBL, s_LLL, sLBB, s_RBR, s_RLR, s_RBB, s_LT1, s_RT1, s_LB1, and s_RB1 are output from each pixel.

  The variation is calculated as described above. Other processes such as calculation of the estimated gain value may be performed in the same manner as in the first embodiment. However, unlike the first embodiment, since all the functional pixels have the same structure, the signal value correction process needs to be performed only once.

  According to the present embodiment, it is possible to perform a suitable signal value correction on a pixel having a sensitivity different from that of an imaging pixel such as a saturation prevention pixel.

  As described above, it is possible to obtain an image processing apparatus and an image processing method capable of appropriately correcting the functional pixel by using only the acquired signal value and acquiring good image data.

DESCRIPTION OF SYMBOLS 100 Image pick-up apparatus 101 Optical apparatus 110 Image pick-up optical system 130 Reading control part 150 Signal processing part 152 Correction processing part 154 Image processing part 160 Memory part 170 Camera CPU
171 User interface (I / F)
172 Recording medium interface (I / F)
190 Image display unit 200 Image sensor 210 Photodiode 230 located on the top surface Photodiode 250 located at the intermediate depth Photodiode 270 located on the bottom layer Microlens 290 Opening

Claims (11)

An imaging means having a plurality of imaging pixels arranged two-dimensionally and a plurality of functional pixels having different sensitivities from the imaging pixels;
A boundary determination unit that determines whether the functional pixel is present at the boundary;
A gain value calculation unit for calculating a gain value for the functional pixel,
The boundary is a point where the signal value changes discontinuously,
The gain value calculation unit uses the signal value output from the imaging pixels arranged around the functional pixel determined that the boundary determination unit does not exist in the boundary unit, and the boundary determination unit is used as the boundary unit. An image processing apparatus that calculates a gain value for the functional pixel determined to be present. The gain calculator is
The representative value of the signal value output from the imaging pixel arranged around the predetermined functional pixel is determined from the predetermined functional pixel with respect to the functional pixel determined that the boundary determination unit does not exist in the boundary portion. Calculate the estimated gain value by dividing by the output signal value,
The image processing apparatus according to claim 1, wherein the gain values for all the functional pixels are calculated by spatially interpolating the estimated gain values.   The spatial interpolation of the estimated gain value is performed by calculating the gain value for each predetermined group using the estimated gain value and spatially interpolating the gain value for each predetermined group. The image processing apparatus according to claim 2. A gain correction unit that performs gain correction using the gain value calculated by the gain value calculation unit with respect to the functional pixel that has been determined that the boundary determination unit exists in the boundary part;
4. The image processing according to claim 1, further comprising: an interpolation correction unit that performs interpolation correction on the functional pixel that is determined not to exist in the boundary by the boundary determination unit. 5. apparatus.   5. The boundary determination unit according to claim 1, wherein the boundary determination unit determines whether the functional pixel is present in the boundary part by calculating a variation of the imaging pixels arranged around the functional pixel. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 1, wherein the imaging unit includes a functional pixel group in which a plurality of the functional pixels are arranged adjacent to each other.   7. The focus detection pixel according to claim 1, wherein the functional pixel is a focus detection pixel that photoelectrically converts a divided light beam among light beams from an imaging optical system that guides the light beam to the imaging unit. Image processing apparatus.   The image according to any one of claims 1 to 6, wherein the functional pixel is a saturation prevention pixel having different sensitivity by adopting a different structure or a filter with respect to the imaging pixel. Processing equipment.   2. The stacked solid-state imaging device according to claim 1, wherein the imaging unit is a stacked solid-state imaging device in which a plurality of light receiving elements each having a plurality of layers of light receiving portions stacked in a depth direction are arranged on a semiconductor substrate. The image processing apparatus according to any one of 8. An imaging apparatus having the image processing apparatus,
The gain value for the functional pixel is calculated when the imaging apparatus cannot discriminate optical apparatus information from the optical apparatus mounted on the imaging apparatus. Imaging device. An image processing method of an imaging means having a plurality of imaging pixels arranged two-dimensionally and a plurality of functional pixels having different sensitivities from the imaging pixels,
The boundary determination unit determines whether or not the functional pixel is present at a boundary portion where the signal value changes discontinuously,
The gain calculation unit calculates a gain value for the functional pixel,
A gain value for the functional pixel is calculated using a signal value output from the imaging pixels arranged around the functional pixel determined by the boundary determination unit as not facing the boundary. Image processing method.

Images