JP6065395B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging equipment.

  For example, a method of generating a reduced image with a relatively low resolution by adding and thinning out pixels at the time of moving image shooting or high-speed continuous shooting of still images is conventionally known. Here, when a spatial frequency component higher than the sampling interval of the reduced image is included in the original image, moire or false color occurs in the reduced image after the addition thinning. As a countermeasure, a technique for expanding the range of pixels to be added before addition thinning and reducing aliasing distortion at the time of thinning readout (for example, Patent Document 1) has been proposed.

Japanese Patent No. 3877695

  However, with the conventional technology, it is still difficult to obtain an image with a high resolution feeling while suppressing generation of moire and false colors when generating a color reduced image. For example, in the case of Patent Document 1, moire and false color can be suppressed, but the resolution of the reduced image is reduced by the same effect as the low-pass filter.

An imaging device that is an example of the present invention includes a pixel that receives light of a first color, a pixel that receives light of a second color, a pixel that receives light of a third color, and a first color Signals from the two first pixels that receive the first light, and six second pixels that receive the light of the first color having an interval wider than the interval between the two first pixels A circuit that adds signals from a plurality of pixels that receive light of the second color, adds signals from a plurality of pixels that receive light of the third color, and two second A signal obtained by adding signals from two first pixels by a signal generated from a difference between a signal obtained by adding signals from one pixel and a signal obtained by adding signals from six second pixels; A signal obtained by adding signals from two second pixels, a signal obtained by adding signals from a plurality of pixels receiving light of the second color, and a third color And an image processing unit that performs edge enhancement processing on the image data generated from a signal obtained by adding signals from a plurality of pixels for receiving.

  According to the present invention, when generating a color reduced image, an image with a higher resolution can be obtained while suppressing the occurrence of moire and false colors.

The figure which shows the structural example of an electronic camera A diagram showing a configuration example of a solid-state imaging device 2 is a diagram illustrating a circuit configuration example of a pixel PX in FIG. Overview of Example 1 The figure which shows the correspondence of the pixel position in a solid-state image sensor, and each color pixel The figure which shows the pixel position in the solid-state image sensor in Example 1, and the sample gravity center of each color signal of a reduction image. (A)-(d): The figure which shows the example of arrangement | positioning of the pixel of the addition range in Example 1. FIG. Outline diagram of Example 2 The figure which shows the pixel position in the solid-state image sensor in Example 2, and the sample gravity center of each color signal of a reduction image. (A)-(d): The figure which shows the example of arrangement | positioning of the pixel of the addition range in Example 2. FIG. Outline diagram of Example 3 The figure which shows the pixel position in the solid-state image sensor in Example 3, and the sample gravity center of each color signal of a reduction image. (A)-(d): The figure which shows the example of arrangement | positioning of the pixel of the addition range in Example 3. FIG. The figure which shows the structural example of the image processing apparatus of 2nd Embodiment. A flowchart showing an operation example of the image processing apparatus according to the second embodiment.

<Description of First Embodiment>
FIG. 1 is a diagram illustrating a configuration example of an electronic camera according to the first embodiment which is an example of an imaging apparatus.

  The electronic camera 1 includes an imaging optical system 2, a solid-state imaging device 3, an image processing engine 4, a memory 5, a recording I / F 6, a monitor 7, and an operation unit 8. Here, the solid-state imaging device 3, the memory 5, the recording I / F 6, the monitor 7, and the operation unit 8 are each connected to the image processing engine 4.

  The imaging optical system 2 includes a plurality of lenses including, for example, a zoom lens and a focus lens. For the sake of simplicity, FIG. 1 shows the imaging optical system 2 with a single lens.

  The solid-state imaging device 3 is a device that images an image of a subject by a light beam that has passed through the imaging optical system 2. The solid-state imaging device 3 according to the first embodiment is an XY address type solid-state imaging device (CMOS image sensor) formed on a silicon substrate using a CMOS (complementary metal oxide semiconductor) process. A configuration example of the solid-state imaging element 3 will be described later.

  Here, in the shooting mode of the electronic camera 1, the solid-state imaging device 3 performs shooting of a still image and a moving image accompanied by recording in the nonvolatile storage medium 9 in accordance with an input from the operation unit 8. In addition, the solid-state imaging device 3 continuously captures images for observation (through images) at predetermined intervals even during standby for shooting. The through-image data (or the moving image data) acquired in time series is used for moving image display on the monitor 7 and various arithmetic processes by the image processing engine 4.

  The solid-state imaging device 3 according to the first embodiment also has an operation mode (normal readout mode) for reading out the electrical signal of each pixel by non-addition, and an operation mode (addition thinning-out readout) for reading out the electrical signal from a plurality of pixels by addition thinning. Mode). In the addition thinning-out readout mode, a reduced image having a smaller image size is read out from the solid-state imaging device 3 as compared with the case where all pixels are read out in the normal readout mode. The addition thinning readout mode is selected, for example, when shooting a through image, shooting a moving image, or high-speed continuous shooting of still images.

  The image processing engine 4 is a processor that comprehensively controls the operation of the electronic camera 1. For example, the image processing engine 4 controls autofocus (AF) and automatic exposure (AE) using a through image signal.

  Further, the image processing engine 4 is an example of an image processing unit, and performs various types of image processing (for example, color conversion processing, gradation conversion processing, white balance adjustment processing, noise removal processing, contour enhancement processing) on image data. Apply.

  The memory 5 temporarily stores image data in the pre-process and post-process of image processing. For example, the memory 5 is an SDRAM that is a volatile storage medium.

  The recording I / F 6 has a connector for connecting a nonvolatile storage medium 9. The recording I / F 6 executes data writing / reading with respect to the storage medium 9 connected to the connector. The storage medium 9 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 9.

  The monitor 7 is a display device that displays various images. For example, the monitor 7 displays a moving image of a through image (viewfinder display) under the shooting mode under the control of the image processing engine 4. In addition, the operation unit 8 receives an image capturing instruction, an instruction to switch various modes, and the like from the user.

  Next, a configuration example of the solid-state imaging device 3 of the first embodiment will be described with reference to FIG.

  The solid-state imaging device 3 includes a pixel unit 11, a plurality of horizontal control signal lines 12, a vertical scanning circuit 13, a plurality of vertical signal lines 14, a signal output circuit 15 which is an example of a signal output unit, and an imaging device control. Circuit 16. Here, the image sensor control circuit 16 supplies control signals to the vertical scanning circuit 13 and the signal output circuit 15. Note that the control signal may be supplied from the image processing engine 4 of the electronic camera 1. In the above case, the image sensor control circuit 16 can be omitted from the solid-state image sensor 3.

  The pixel unit 11 includes a plurality of pixels PX that convert incident light into electrical signals. The pixels PX of the pixel unit 11 are arranged in a matrix in the first direction D1 and the second direction D2 on the light receiving surface. Hereinafter, the first direction D1 and the second direction D2 are also referred to as a row direction D1 and a column direction D2. In FIG. 2, the arrangement of the pixels PX is shown in a simplified manner, but it goes without saying that a larger number of pixels are arranged on the light receiving surface of the actual solid-state imaging device.

  Here, on the front surface of each pixel PX, a plurality of types of color filters that transmit light of different color components are arranged in a predetermined color arrangement. Therefore, the pixel PX outputs an electrical signal corresponding to each color by color separation with a color filter. For example, in the first embodiment, red (R), green (Gr, Gb), and blue (B) color filters are arranged in each pixel PX according to a 2-by-2 Bayer array. Thereby, the pixel part 11 can acquire a color image at the time of imaging | photography. Hereinafter, the pixels PX having red (R), green (Gr, Gb), and blue (B) filters are also referred to as red pixels (R), blue pixels (B), and green pixels (Gr, Gb), respectively.

  When attention is paid to the row direction D1, for example, in the odd-numbered rows of the pixel unit 11, red pixels (R) and green pixels (Gr) are alternately arranged. Further, for example, in even rows of the pixel unit 11, green pixels (Gb) and blue pixels (B) are alternately arranged.

  When focusing on the column direction D2, for example, in the odd-numbered columns of the pixel unit 11, green pixels (Gb) and red pixels (R) are alternately arranged. Further, for example, in the even-numbered columns of the pixel unit 11, blue pixels (B) and green pixels (Gr) are alternately arranged. In this specification, Gr and Gb color filters are collectively referred to as green (G) filters, and green pixels (Gr, Gb) are sometimes collectively referred to as green pixels (G). .

  Further, a horizontal control signal line 12 connected to the vertical scanning circuit 13 is arranged in each row of the pixel unit 11. Each horizontal control signal line 12 supplies control signals (a selection signal φSEL, a reset signal φRST, and a transfer signal φTX, which will be described later) output from the vertical scanning circuit 13 to the pixels PX arranged in the row direction D1.

  A vertical signal line 14 is arranged in each column of the pixel array. The plurality of pixels PX arranged in the column direction D2 are connected to each other by a vertical signal line 14 provided for each column. That is, the pixel unit 11 outputs electric signals from the plurality of pixels PX arranged in the same column via the common vertical signal line 14. One end (the lower side in FIG. 2) of each vertical signal line 14 is connected to the signal output circuit 15.

  Here, a circuit configuration example of the pixel PX of FIG. 2 will be described with reference to FIG.

  The pixel PX includes a photodiode PD, a transfer transistor TX, a reset transistor RST, an amplification transistor AMI, an amplification unit AMP, a selection transistor SEL, and a floating diffusion FD.

  The photodiode PD generates a signal charge by photoelectric conversion according to the amount of incident light. The transfer transistor TX is turned on during the high level period of the transfer signal φTX, and transfers the signal charge accumulated in the photodiode PD to the floating diffusion FD.

  The source of the transfer transistor TX is a photodiode PD, and the drain of the transfer transistor TX is a floating diffusion FD. The floating diffusion FD is a diffusion region formed by introducing impurities into a semiconductor substrate, for example. The floating diffusion FD is connected to the gate of the amplification transistor AMI and the source of the reset transistor RST.

  The reset transistor RST is turned on during a high level period of the reset signal φRST, and resets the floating diffusion FD to the power supply voltage VDD. The amplification transistor AMI has a drain connected to the power supply voltage VDD, a gate connected to the floating diffusion FD, and a source electrode connected to the constant current source ISS as an output node. Since the source follower is configured by the amplification transistor AMI, a voltage corresponding to the potential of the floating diffusion FD is generated at the source of the amplification transistor AMI.

  The amplifying unit AMP is a variable gain amplifier for adjusting the gain for each pixel PX when performing weighted addition in the addition thinning readout mode. The input of the amplification unit AMP is connected to the source of the amplification transistor AMI, and the output of the amplification unit AMP is connected to the drain of the selection transistor SEL.

  The selection transistor SEL is turned on during a high level period of the selection signal φSEL, and connects the output of the amplifier AMP to the vertical signal line 14.

  Note that when the reset transistor RST of the pixel PX is turned on, a dark signal including a noise component is read from the pixel PX to the vertical signal line 14. Further, based on the charge transferred to the floating diffusion FD, a bright signal including a noise component and a light receiving component is read from the pixel PX to the vertical signal line 14.

  Returning to FIG. 2, the signal output circuit 15 is a circuit that reads the electrical signal of the pixel PX from the pixel unit 11 in the row direction (D1). The signal output circuit 15 includes a column capacitor 21, a horizontal adder 22, a column amplifier 23, a sample hold unit 24, a column ADC 25, a horizontal data bus 26, and a data register 27. One column capacitor 21 is provided for each vertical signal line 14.

  The horizontal adder 22 includes a plurality of addition control switches ADD1 that connect between the adjacent odd-numbered vertical signal lines 14, a plurality of addition control switches ADD2 that connect between the adjacent even-numbered vertical signal lines 14, Each of the vertical signal lines 14 includes a plurality of column switches LSW. By switching on / off the addition control switch ADD1, the electric signals of the pixels PX in the odd columns are added in the row direction (D1). Further, the electrical signals of the pixels PX in the even-numbered columns are added in the row direction (D1) by turning on / off the addition control switch ADD2. Note that when all pixel readout is performed, the addition control switches ADD1 and ADD2 are both turned off.

  One set of column amplifier 23, sample hold unit 24, and column ADC 25 are provided for each vertical signal line 14. In each set, the column amplifier 23, the sample hold unit 24, and the column ADC 25 are connected in series.

  The column amplifier 23 inverts and amplifies the electrical signal output from the pixel PX via the vertical signal line 14. The sample hold unit 24 samples the input analog signal (bright signal or dark signal) at a predetermined timing, holds the sampled analog signal for a predetermined period, and outputs the analog signal to the subsequent column ADC 25. The column ADC 25 A / D converts the input bright signal and dark signal.

  Only one horizontal data bus 26 is provided in the signal output circuit 15. The horizontal data bus 26 is connected to the output of each column ADC 25, and outputs an image signal after A / D conversion by the column ADC 25 to the data register 27. The data register 27 synthesizes the image signals after thinning read out from different columns as necessary. The output of the data register 27 is connected to the image processing engine 4.

  Here, a reading example when the solid-state imaging device 3 is operated in the addition thinning-out reading mode will be described. As an example, when the attention area (the range indicated by the two-dot chain line in FIG. 2) of the 4 × 4 pixels of the pixel unit 11 is read by addition thinning, the following operation is performed. Note that the signal value obtained by performing the thinning-out in the region of interest corresponds to the signal value of one pixel of the reduced image. Hereinafter, one pixel of the reduced image is also referred to as a sample point.

  When thinning out the four red pixels (R) in the attention area, the addition control switch ADD1 corresponding to the vertical signal lines 14 in the odd columns in the attention area is turned on. Further, one of the column switches LSW corresponding to the vertical signal line 14 is turned on, and the other column switches LSW are turned off. Then, the selection signals φSEL in the two odd rows in the region of interest are simultaneously set to the high level. As a result, the signals of the four red pixels in the attention area are read together.

  Similarly, when adding and thinning four blue pixels (B) in the attention area, the addition control switch ADD2 corresponding to the vertical signal lines 14 in the even columns in the attention area is turned on. Further, one of the column switches LSW corresponding to the vertical signal line 14 is turned on, and the other column switches LSW are turned off. Then, the selection signals φSEL in two even rows in the region of interest are simultaneously set to a high level. Thereby, the signals of the four blue pixels in the attention area are read out together.

  In the case where the eight green pixels (G) in the attention area are thinned out for addition, for example, after four green pixels (Gb) and four green pixels (Gr) are separately thinned out separately, the data register 27 finally The two may be synthesized.

  In addition, in addition thinning-out readout mode, the horizontal addition unit 22 may perform only horizontal addition for each row, and the data register 27 may perform addition in the column direction (D2) of the region of interest. Needless to say, the size of the region of interest is an example and can be changed as appropriate.

  Further, the gain of the image signal after thinning out may be adjusted by the column amplifier 23, or may be adjusted in advance by the amplifier AMP of the pixel PX before addition. In addition, when performing weighted addition in order to align the sample centroid of each color signal of the reduced image, it is necessary to adjust the gain by the amplification unit AMP of the pixel PX.

Example 1
Hereinafter, as an example 1, an operation example of the electronic camera 1 in the addition thinning readout mode will be described. In the first embodiment, four types of image signals R, Gr (G for odd rows), Gb (G for even rows), and B are added and thinned out from the solid-state imaging device 3 in which RGB color filters are arranged in a Bayer array. Read (see FIG. 4). When the image size at the time of reading all pixels is W × H, in the first embodiment, a reduced image having an image size of W / 4 × H / 4 is read.

  FIG. 5 is a diagram showing the pixel positions in the solid-state imaging device 3 and the correspondences between the color pixels. The coordinates of each color pixel of the solid-state imaging device 3 are expressed as R (i, j), G (i + 1, j), G (i, j + 1), B (i + 1, j + 1). In the first embodiment, the signal values corresponding to the sample point (x, y) of the reduced image are R ′ (x, y), Gr ′ (x, y), Gb ′ (x, y), B ′. It is expressed as (x, y).

  Addition thinning readout of the solid-state imaging device 3 in the first embodiment is performed as follows under the control of the imaging device control circuit 16. The four signal values corresponding to the sample point (x, y) of the reduced image are generated by averaging the signals of the same color pixels included in the 4 × 4 pixel range (addition range) corresponding to the sample point. The

For example, when the reference pixel (i, j) = (m, n) located at the upper left corner of the readout unit is a red pixel, the arrangement of the color pixels in the addition range is as shown in FIG. At this time, the signal values (R ′ _R (x, y), Gr ′ _R (x, y), Gb ′ _R (x, y), B ′ _R (x, y)) at each sample point are expressed by the formula ( 1).

以上、標本点（ｘ，ｙ）に注目して加算間引き読み出しの例を説明した。なお、他の標本点の加算間引き読み出しは、基準画素の位置をシフトさせて同様の読み出しを行えばよい。一例として、標本点（ｘ＋１，ｙ）に対応する信号値（Ｒ'_R（ｘ＋１，ｙ），Ｇｒ'_R（ｘ＋１，ｙ），Ｇｂ'_R（ｘ＋１，ｙ），Ｂ'_R（ｘ＋１，ｙ））は、基準画素（ｍ，ｎ）＝（ｉ＋４，ｊ）として、上記の式（１）により求めればよい。

Heretofore, an example of addition thinning readout has been described focusing on the sample point (x, y). In addition, the thinning-out readout of other sample points may be performed by shifting the position of the reference pixel. As an example, signal values (R ′ _R (x + 1, y), Gr ′ _R (x + 1, y), Gb ′ _R (x + 1, y), B ′ _R (x + 1, y) corresponding to the sample point (x + 1, y). )) May be obtained from the above equation (1) as the reference pixel (m, n) = (i + 4, j).

  In this embodiment, since the color filters are arranged in a cycle of 2 × 2 and the image reduction magnification is 1/4 × 1/4, which is the reference pixel in the addition range at each sample point of the reduced image Will be the same color.

  However, when the reduction magnification is an odd number (for example, 1/3 × 1/3), the color of the reference pixel is switched for each sample point. In this case, what is necessary is just to adjust suitably the pixel added by Formula (1). For example, when the reference pixel is Gr, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is Gb, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is B, the arrangement of each color pixel in the addition range is as shown in FIG.

  FIG. 6 is a diagram illustrating a pixel position in the solid-state imaging device according to the first embodiment and a sample centroid of each color signal of the reduced image. In FIG. 6, the sample centroid of the R signal is indicated by a white circle (◯) in the figure. In addition, the center of gravity of the sample of the Gr signal is indicated by a cross (x) in the figure. Further, the center of gravity of the sample of the Gb signal is indicated by a triangle (Δ) in the figure. Also, the sample centroid of the B signal is indicated by an asterisk (*) in the figure. Note that the example of FIG. 6 is a case where all of the reference pixels are red pixels. If the colors of the reference pixels are different, the sample centroid of each color signal in FIG. 6 is replaced.

  Next, the signal value of each sample point is input to the image processing engine 4. Then, the image processing engine 4 uses the Gr image signal and the Gb image signal to execute a color difference selection process corresponding to the gradient of the image (see FIG. 4).

  Here, a case is considered where the Gr image signal and the Gb image signal are added and averaged without being distinguished, and read with one G signal value. In this case, the sample centroid of the G signal at the sample point (x, y) is in the middle (i + 1.5, j + 1.5) between the x mark and the Δ mark in FIG. Deviation occurs. In addition, the distribution of R pixels and B pixels in the addition range is a 3 × 3 range, but the distribution of G pixels in the addition range is a 4 × 4 range. The deviation of the center of gravity of the sample and the distribution of the pixels to be added as described above may appear as a weak false color when, for example, an image containing a lot of high-frequency signals is reduced in a state close to focusing.

  On the other hand, when the Gr image signal and the Gb image signal are separated and read as in the first embodiment, the distribution of Gr pixels and Gb pixels in the addition range is a 3 × 3 range, and the distribution of R pixels and B pixels Matches. Also, the sample centroid of the R and B signals can be expected to take a value close to either of the sample centroids of the Gr and Gb signals according to the gradient direction of the image due to the subject's ring. Therefore, the image processing engine 4 suppresses the occurrence of false colors by performing the following color difference selection process.

  First, the image processing engine 4 obtains the horizontal gradient (ΔH) and the vertical gradient (ΔV) at the sample point (x, y) of interest by Expression (2).

第２に、画像処理エンジン４は、注目する標本点（ｘ，ｙ）で適用するＣｒ’（ｘ，ｙ），Ｃｂ’（ｘ，ｙ）を、上記のΔＨ，ΔＶの大小評価により式（３）で決定する。

Second, the image processing engine 4 calculates Cr ′ (x, y) and Cb ′ (x, y) to be applied at the sample point (x, y) of interest by the above-described evaluation of ΔH and ΔV ( Determine in 3).

上記の式（３）により、画像の勾配を考慮して、偽色の影響を受けにくいと予想される色差Ｃｒ’，Ｃｂ’が選択される。なお、式（３）の「ＴＨＥ＿ＧＲＡＤ」は、勾配の大きさを規定する閾値である。

According to the above equation (3), the color differences Cr ′ and Cb ′ that are expected to be hardly affected by the false color are selected in consideration of the gradient of the image. Note that “THE_GRAD” in Equation (3) is a threshold value that defines the magnitude of the gradient.

  Thirdly, the image processing engine 4 obtains the RGB value of the sample point (x, y) by the equation (4) using the color differences Cr ′ and Cb ′.

なお、画像処理エンジン４は、式（４）で求めたＲＧＢ値を用いて、式（５）によりＹＵＶ値への変換を行えばよい。

Note that the image processing engine 4 may perform the conversion to the YUV value according to the equation (5) using the RGB value obtained according to the equation (4).

以上の処理により、縮小画像の標本点（ｘ，ｙ）についてＹＵＶ画像情報が生成される。そして、画像処理エンジン４は、他の標本点にも同様の処理を行い、縮小画像全体のＹＵＶ画像情報を生成する。

Through the above processing, YUV image information is generated for the sample point (x, y) of the reduced image. Then, the image processing engine 4 performs similar processing on other sample points, and generates YUV image information of the entire reduced image.

  Thereafter, the image processing engine 4 records the YUV image information of the reduced image on the storage medium 9 via the recording I / F 6. Alternatively, the image processing engine 4 may display an image on the monitor 7 using the YUV image information of the reduced image.

Hereinafter, effects of the first embodiment will be described. In the configuration of the first embodiment, four image signals of R, Gr, Gb, and B are generated at each sample point at the time of addition thinning readout from the solid-state imaging device 3.
In the case of the first embodiment, since all the color signals necessary for each sample point of the reduced image are prepared, it is not necessary to perform the interpolation processing of the reduced image. Therefore, in Example 1, sufficient resolution can be secured at each sample point of the reduced image. Further, for example, when considering a case of reducing an image containing a lot of high-frequency signals in a state close to focusing, in the case of the first embodiment, compared with a case where a reduced image is generated by color interpolation of a thinned readout image of a Bayer array structure. In this case, the occurrence of moire and false colors in the reduced image due to the color interpolation process is greatly suppressed.

  In the first embodiment, Gr and Gb image signals are acquired for each sample point, whereby the intensity distribution of the green signal in the original image is maintained. Thereby, since an adaptive color difference selection process can be performed, false colors can be further suppressed.

  In the case of the first embodiment, four signal values are acquired for one sample point. Therefore, in the first embodiment, when the size of the reduced image is W / 4 × H / 4 with respect to the image size (W × H) at the time of reading all pixels, the read data amount is 1 as compared with the case of reading all pixels. / 4 × 1/4 × 4 = 1/4 times.

(Modification of Example 1)
In the first embodiment, each signal value of the reduced image is obtained by a simple average of each color pixel. However, each signal value of the reduced image may be obtained by weighted addition. As an example, the weighting coefficient may be set to 9: 3: 3: 1 as in Expression (6) instead of Expression (1).

上記の式（６）の場合には、４つの信号のサンプル重心を一致させることができる。なお、上記の式（６）の場合には、サンプル重心位置は一致するが、加算範囲は1画素分ずれるため実施例１の色差選択処理は有効である。

In the case of Equation (6) above, the sample centroids of the four signals can be matched. In the case of the above formula (6), the sample barycentric positions coincide with each other, but the addition range is shifted by one pixel, so that the color difference selection process of the first embodiment is effective.

(Example 2)
Next, an operation example of the electronic camera 1 in the addition thinning readout mode according to the second embodiment will be described with reference to FIGS. It is assumed that the apparatus configuration and operation in each of the following embodiments are the same as those in the first embodiment unless otherwise specified. Note that the size of the reduced image in each embodiment is the same as that in the first embodiment.

  In Example 2, four types of image signals R, Gh, Gl, and B are read out from the solid-state imaging device 3 by addition thinning (see FIG. 8). Gh is a signal including a green high frequency component in the addition range, and Gl is a signal including a green low frequency component in the addition range.

  As an example, Gh is generated by adding the outputs of two green pixels located in the center of the 4 × 4 pixel addition range. In addition, Gl is generated by adding the outputs of six green pixels located in the periphery of the 4 × 4 pixel addition range (see FIG. 10). The sampling interval of the pixels added by Gh is smaller than the sampling interval of the pixels added by Gl. Therefore, Gh includes a higher spatial frequency component than Gl. In the second embodiment, signal values corresponding to the sample point (x, y) of the reduced image are R ′ (x, y), Gh ′ (x, y), Gl ′ (x, y), B ′. It is expressed as (x, y).

  Addition thinning readout of the solid-state imaging device 3 in the second embodiment is performed as follows under the control of the imaging device control circuit 16.

For example, when the reference pixel (i, j) = (m, n) located at the upper left end of the readout unit is a red pixel, the arrangement of the color pixels in the addition range is as shown in FIG. At this time, the signal values (R ′ _R (x, y), Gh ′ _R (x, y), Gl ′ _R (x, y), B ′ _R (x, y)) at each sample point are expressed by the formula ( 7).

以上、標本点（ｘ，ｙ）に注目して加算間引き読み出しの例を説明した。なお、他の標本点の加算間引き読み出しは、基準画素の位置をシフトさせて同様の読み出しを行えばよい。一例として、標本点（ｘ＋１，ｙ）に対応する信号値（Ｒ'_R（ｘ＋１，ｙ），Ｇｈ'_R（ｘ＋１，ｙ），Ｇｌ'_R（ｘ＋１，ｙ），Ｂ'_R（ｘ＋１，ｙ））は、基準画素（ｍ，ｎ）＝（ｉ＋４，ｊ）として、上記の式（７）により求めればよい。

Heretofore, an example of addition thinning readout has been described focusing on the sample point (x, y). In addition, the thinning-out readout of other sample points may be performed by shifting the position of the reference pixel. As an example, signal values (R ′ _R (x + 1, y), Gh ′ _R (x + 1, y), Gl ′ _R (x + 1, y), B ′ _R (x + 1, y) corresponding to the sample point (x + 1, y). )) May be obtained from the above equation (7) as the reference pixel (m, n) = (i + 4, j).

  In this embodiment, since the color filters are arranged in a cycle of 2 × 2 and the image reduction magnification is 1/4 × 1/4, which is the reference pixel in the addition range at each sample point of the reduced image Will be the same color.

  However, when the reduction magnification is an odd number (for example, 1/3 × 1/3), the color of the reference pixel is switched for each sample point. In this case, what is necessary is just to adjust suitably the pixel added by Formula (7). For example, when the reference pixel is Gr, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is Gb, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is B, the arrangement of each color pixel in the addition range is as shown in FIG.

  FIG. 9 is a diagram illustrating a pixel position in the solid-state imaging device according to the second embodiment and a sample centroid of each color signal of the reduced image. In the case of the second embodiment, the sample centroids (indicated by ◯ in the figure) of the respective color signals coincide with each other.

  Next, the signal value of each sample point is input to the image processing engine 4. Here, the image processing engine 4 according to the second embodiment performs edge enhancement processing on the reduced image using the Gh and Gl image signals.

  First, the image processing engine 4 obtains the G signal value (G ′ (x, y)) of the sample point from the Gh and Gl image signals according to Expression (8).

上記の式（８）の演算により、縮小画像の標本点のＲＧＢ信号値が揃うこととなる。なお、式（７）および式（８）を合わせて考えると、上記のＧ信号値も、Ｒ，Ｂ信号値と同様に加重係数の比率が９：３：３：１となる。

The RGB signal values of the sample points of the reduced image are aligned by the calculation of the above equation (8). Note that when the equations (7) and (8) are considered together, the ratio of the weighting coefficients of the G signal value is 9: 3: 3: 1 as well as the R and B signal values.

  Next, the image processing engine 4 extracts a high-frequency component (δG ′) of the image from the difference between the Gh and Gl image signals according to Expression (9). Note that δG ′ reflects the unevenness of the G signal in the Gh addition range (3 × 3) with respect to the Gl addition range (5 × 5).

そして、画像処理エンジン４は、上記の式（５）によりＲＧＢ値からＹＵＶ値への変換を行う。ここで、輝度信号値Ｙに対してはＧ信号値の寄与が大きいため、標本点での輝度信号値Ｙの凹凸はＧ信号の凹凸で近似できる。そこで、実施例２の画像処理エンジン４は、輝度信号値Ｙに対して、式（１０）に示すようにδＧ’を加算してエッジ強調処理を施す。

Then, the image processing engine 4 performs conversion from RGB values to YUV values according to the above equation (5). Here, since the contribution of the G signal value to the luminance signal value Y is large, the unevenness of the luminance signal value Y at the sample point can be approximated by the unevenness of the G signal. Therefore, the image processing engine 4 according to the second embodiment performs edge enhancement processing by adding δG ′ to the luminance signal value Y as shown in Expression (10).

ここで、αはエッジ強調の効果を調節するための係数である。αは画像全体で単一の値でもよく、ＹやδＧ'の分布に応じて適応的に変化させてもよい。例えば、Ｙの分布が平坦な箇所や、１画素のみ孤立したδＧ'が存在する箇所ではαの値を小さくし、まとまったδＧ'の分布が見られる箇所ではαの値を大きく設定してもよい。このような適応的な処理により、画像のノイズを強調することなく、縮小画像のエッジ強調処理を行うことができる。

Here, α is a coefficient for adjusting the effect of edge enhancement. α may be a single value for the entire image, or may be adaptively changed according to the distribution of Y and δG ′. For example, if the distribution of Y is flat or where δG ′ isolated by only one pixel is present, the value of α may be reduced, and if a uniform distribution of δG ′ is seen, the value of α may be set larger. Good. By such adaptive processing, edge enhancement processing of a reduced image can be performed without enhancing image noise.

  Through the above processing, YUV image information is generated for the sample point (x, y) of the reduced image. Then, the image processing engine 4 performs similar processing on other sample points, and generates YUV image information of the entire reduced image.

  Thereafter, the image processing engine 4 records the YUV image information of the reduced image on the storage medium 9 via the recording I / F 6. Alternatively, the image processing engine 4 may display an image on the monitor 7 using the YUV image information of the reduced image.

  Even with the configuration of the second embodiment, since it is not necessary to perform interpolation processing at the sample point level of the reduced image, sufficient resolution can be secured at each sample point of the reduced image, and moire or false color in the reduced image can be secured. Occurrence can be greatly suppressed. In the case of the second embodiment, the resolution enhancement of the reduced image can be further improved by the edge enhancement process using the Gh and Gl image signals of the sample points.

  In the second embodiment, four signal values are acquired for one sample point. Therefore, in the first embodiment, when the size of the reduced image is W / 4 × H / 4 with respect to the image size (W × H) at the time of reading all pixels, the read data amount is 1 as compared with the case of reading all pixels. / 4 × 1/4 × 4 = 1/4 times.

(Example 3)
Next, an operation example of the electronic camera 1 in the addition thinning readout mode according to the third embodiment will be described with reference to FIGS.

  In the third embodiment, R and B image signals are read from the solid-state imaging device 3 by addition thinning, and three types of G signals (G1, G2, and G3) having different pixel arrangements in the addition range are generated. The In the third embodiment, the image processing engine 4 performs color difference selection processing and edge enhancement processing by combining the G1, G2, and G3 signals (see FIG. 11).

  As an example, G1 is generated by adding the outputs of two green pixels located in the center of the 4 × 4 pixel addition range. G2 and G3 are generated by adding the outputs of three green pixels located in the periphery of the 4 × 4 pixel addition range. The green pixels added by G2 are arranged in an L shape within the above addition range. Further, the green pixels added in G3 are arranged in an inverted L shape within the above addition range so as to face the arrangement of the G2 addition pixels (see FIG. 13). In the third embodiment, the signal values corresponding to the sample point (x, y) of the reduced image are R ′ (x, y), G1 ′ (x, y), G2 ′ (x, y), G3 ′. (X, y), B ′ (x, y).

  Addition thinning readout of the solid-state imaging device 3 in the third embodiment is performed as follows under the control of the imaging device control circuit 16.

For example, when the reference pixel (i, j) = (m, n) located at the upper left end of the readout unit is a red pixel, the arrangement of the color pixels in the addition range is as shown in FIG. At this time, the signal values (R ′ _R (x, y), G1 ′ _R (x, y), G2 ′ _R (x, y), G3 ′ _R (x, y), B ′ _R ( x, y)) can be obtained by equation (11).

以上、標本点（ｘ，ｙ）に注目して加算間引き読み出しの例を説明した。なお、他の標本点の加算間引き読み出しは、基準画素の位置をシフトさせて同様の読み出しを行えばよい。一例として、標本点（ｘ＋１，ｙ）に対応する信号値（Ｒ'_R（ｘ＋１，ｙ），Ｇ１'_R（ｘ＋１，ｙ），Ｇ２'_R（ｘ＋１，ｙ），Ｇ３'_R（ｘ＋１，ｙ），Ｂ'_R（ｘ＋１，ｙ））は、基準画素（ｍ，ｎ）＝（ｉ＋４，ｊ）として、上記の式（１１）により求めればよい。

Heretofore, an example of addition thinning readout has been described focusing on the sample point (x, y). In addition, the thinning-out readout of other sample points may be performed by shifting the position of the reference pixel. As an example, signal values (R ′ _R (x + 1, y), G1 ′ _R (x + 1, y), G2 ′ _R (x + 1, y), G3 ′ _R (x + 1, y) corresponding to the sample point (x + 1, y). ), B ′ _R (x + 1, y)) may be obtained from the above equation (11) as the reference pixel (m, n) = (i + 4, j).

  In this embodiment, since the color filters are arranged in a cycle of 2 × 2 and the image reduction magnification is 1/4 × 1/4, which is the reference pixel in the addition range at each sample point of the reduced image Will be the same color.

  However, when the reduction magnification is an odd number (for example, 1/3 × 1/3), the color of the reference pixel is switched for each sample point. In this case, what is necessary is just to adjust suitably the pixel added by Formula (11). For example, when the reference pixel is Gr, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is Gb, the arrangement of each color pixel in the addition range is as shown in FIG. When the reference pixel is B, the arrangement of each color pixel in the addition range is as shown in FIG.

  FIG. 12 is a diagram illustrating a pixel position in the solid-state imaging device according to the third embodiment and a sample centroid of each color signal of the reduced image. In FIG. 12, the sample centroid of the R signal is indicated by a white circle (◯) in the figure. Also, the sample centroid of the G1 signal is indicated by a black circle (●) in the figure. Further, the center of gravity of the sample of the G2 signal is indicated by a cross (x) in the figure. Further, the center of gravity of the sample of the G3 signal is indicated by a triangle (Δ) in the figure. Also, the sample centroid of the B signal is indicated by an asterisk (*) in the figure. Note that the example of FIG. 12 is a case where all the reference pixels are red pixels, and the sample centroid of each color signal of FIG. 12 changes when the colors of the reference pixels are different.

  Next, the signal value of each sample point is input to the image processing engine 4. Here, the image processing engine 4 according to the third embodiment performs color difference selection processing and edge enhancement processing.

  First, the image processing engine 4 obtains Gr and Gb signal values from the G1, G2, and G3 signal values according to Expression (12).

式（１１）と式（１２）の比較により、式（１２）で取得されるＧｒ，Ｇｂ信号値は、それぞれ５画素の単純平均となっていることが分かる。そのため、実施例３のＧｒ，Ｇｂ信号のサンプル重心は、実施例１のＧｒ，Ｇｂ信号のサンプル重心と一致する。

By comparing the equations (11) and (12), it can be seen that the Gr and Gb signal values obtained by the equation (12) are each a simple average of 5 pixels. Therefore, the sample centroids of the Gr and Gb signals of the third embodiment coincide with the sample centroids of the Gr and Gb signals of the first embodiment.

  Further, although the number of added pixels is different between the third embodiment and the first embodiment, G signal values in the same range are added. Therefore, the image processing engine 4 according to the third embodiment applies the equations (2) to (5) according to the first embodiment to perform color difference selection processing and generate YUV image information.

  Further, the image processing engine 4 can extract the high-frequency component (δG ′) of the image from the G1, G2, and G3 signals according to Expression (13).

式（１１）と式（１３）の比較により、式（１３）で取得されるδＧ’は、実施例２でＧｈ，Ｇｌ信号値から求めたδＧ’と同一であることが分かる。そのため、実施例３の画像処理エンジン４は、実施例２の式（１０）を適用して、輝度信号値Ｙに対して適応的なエッジ強調処理を施すことができる。

Comparison between Expression (11) and Expression (13) shows that δG ′ obtained by Expression (13) is the same as δG ′ obtained from the Gh and Gl signal values in the second embodiment. For this reason, the image processing engine 4 according to the third embodiment can perform adaptive edge enhancement processing on the luminance signal value Y by applying the expression (10) of the second embodiment.

  According to the configuration of the third embodiment, the same effects as those of the first and second embodiments can be obtained. In the case of the third embodiment, five signal values are acquired for one sample point. Therefore, in the first embodiment, when the size of the reduced image is W / 4 × H / 4 with respect to the image size (W × H) at the time of reading all pixels, the read data amount is 1 as compared with the case of reading all pixels. / 4 × 1/4 × 5 = 5/16 times.

<Description of Second Embodiment>
FIG. 14 is a diagram illustrating a configuration example of the image processing apparatus according to the second embodiment. The image processing apparatus according to the second embodiment is a personal computer in which a program for generating a reduced image by adding and thinning a color image to be processed is installed.

  A computer 31 illustrated in FIG. 14 includes a data reading unit 32, a storage device 33, a CPU 34, a memory 35, an input / output I / F 36, and a bus 37. The data reading unit 32, the storage device 33, the CPU 34, the memory 35, and the input / output I / F 36 are connected to each other via a bus 37. Furthermore, an input device 38 (keyboard, pointing device, etc.) and a monitor 39 are connected to the computer 31 via an input / output I / F 36. The input / output I / F 36 accepts various inputs from the input device 38 and outputs display data to the monitor 39.

  The data reading unit 32 is used when reading image data or a program from the outside. For example, the data reading unit 32 communicates with a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device in accordance with a known communication standard. A communication device to perform (USB interface, LAN module, wireless LAN module, etc.). The data reading unit 32 functions as an acquisition unit that acquires a color image to be processed.

  The storage device 33 is configured by a storage medium such as a hard disk or a nonvolatile semiconductor memory, for example. The storage device 33 stores an image processing program. The storage device 33 can also store image data read from the data reading unit 32 and reduced image data generated by a program.

  The CPU 34 is a processor that comprehensively controls each unit of the computer 31. The CPU 34 functions as a conversion unit that generates a reduced image by executing a program.

  The memory 35 temporarily stores various calculation results in the program. The memory 35 is, for example, a volatile SDRAM.

  Hereinafter, an operation example of the image processing apparatus according to the second embodiment will be described with reference to the flowchart of FIG. 15.

  Step # 101: The CPU 34 acquires the color image data to be processed from the data reading unit 32. Here, the color image may be a RAW image before color interpolation or a color image after color interpolation. In the second embodiment, an example will be described in which a RAW image that is captured by an electronic camera and in which a RGB color is arranged in a mosaic pattern in a Bayer array is a processing target.

  The color image data acquired in # 101 is recorded in the storage device 33 or the memory 35 under the control of the CPU. Note that when the image data to be processed is stored in the storage device 33 in advance, the CPU 34 may omit the process of # 101.

  Step # 102: The conversion unit of the CPU 34 thins out the color image to generate an RGB image signal of the reduced image. Specifically, the conversion unit # 102 performs addition decimation by any one of the methods of the first to third embodiments.

  Step # 103: The conversion unit of the CPU 34 generates YUV image information from the RGB signal values of the reduced image.

  Here, when the addition decimation in the first embodiment is performed in # 102, the conversion unit in # 103 performs the color difference selection process similar to that in the first embodiment. In addition, when the addition decimation in the second embodiment is performed in # 102, the conversion unit in # 103 performs the same edge enhancement processing as in the second embodiment. In addition, when the addition thinning of the third embodiment is performed in # 102, the conversion unit of # 103 performs the color difference selection process and the edge enhancement process similar to those of the third embodiment.

  Step # 104: The CPU 34 records the image information of the reduced image generated at # 104 in the storage device 33. Alternatively, the CPU 34 may display an image on the monitor 39 using the image information of the reduced image. Above, description of the flowchart of FIG. 15 is complete | finished.

  According to the second embodiment, substantially the same effect as that of the first embodiment can be obtained when a reduced image is generated in a post-processing step from a color image captured by an electronic camera.

<Supplementary items of the embodiment>
(Supplement 1): The image reduction magnification in the present invention can be appropriately changed without being limited to the example of the above embodiment. In the present invention, the image reduction magnification may be different in the horizontal direction and the vertical direction.

  (Supplement 2): If the speed of image reading is insufficient for the required frame rate even by the addition decimation in the above embodiment, the bit accuracy of the reduced image is reduced to compensate for the speed of reading. Also good.

  (Supplement 3): When the reduction magnification is large, the pixel size of the solid-state imaging device may be insufficient for the reduced image format. In such a case, a known up-conversion process such as a super-resolution process may be performed on the reduced image.

  For example, the G signal value calculated by the expressions (12) and (4) of the third embodiment corresponds to the range of 4 × 4 pixels of the image sensor, but the read G1 signal is 2 × 2 of the image sensor. Only the pixel range is supported. This corresponds to the pixel size when an image read out at a reduction ratio of 1/4 × 1/4 for speedup is upconverted 2 × 2 times by processing such as super-resolution or interpolation enlargement. This means that information to be held is discretely held. Such subpixel spatial distribution information may be used to improve the resolution of the up-converted image.

  (Supplement 4): In the above embodiment, an electronic camera as an example of an imaging apparatus has been described. However, the imaging apparatus of the present invention includes, for example, a configuration including only the solid-state imaging device 3 of the first embodiment or a device in which the solid-state imaging device 3 and the image processing engine 4 (image processing unit) are integrated on-chip. It is a concept.

  (Supplement 5): In the said embodiment, the color filter array of the solid-state image sensor 3 is not limited to a Bayer arrangement. For example, the present invention can be applied to a complementary color filter array using magenta, green, cyan, and yellow, or a color filter array in which Gr or Gb in the Bayer array is replaced with another color.

  (Supplement 6): In the second embodiment, the example in which the function of the conversion unit of the image processing apparatus is realized by software by a program has been described. However, in the present invention, the function of the conversion unit may of course be realized by hardware using, for example, an ASIC.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera, 2 ... Imaging optical system, 3 ... Solid-state image sensor, 4 ... Image processing engine, 5 ... Memory, 6 ... Recording I / F, 7 ... Monitor, 8 ... Operation part, 9 ... Storage medium, 11 ... Pixel unit, 12 ... horizontal control signal line, 13 ... vertical scanning circuit, 14 ... vertical signal line, 15 ... signal output circuit, 16 ... image sensor control circuit, 21 ... column capacitor, 22 ... horizontal adder, 23 ... column amplifier , 24 ... Sample hold unit, 25 ... Column ADC, 26 ... Horizontal data bus, 27 ... Data register, 31 ... Computer, 32 ... Data reading unit, 33 ... Storage device, 34 ... CPU, 35 ... Memory, 36 ... Input / output I / F, 37 ... bus, 38 ... input device, 39 ... monitor

Claims (3)

A pixel that receives light of the first color;
A pixel that receives light of the second color;
A pixel receiving light of the third color;
Signals from two first pixels that receive light of the first color are added, and the light of the first color is received having an interval wider than the interval between the two first pixels. Signals from a plurality of pixels that receive light of the third color are added by adding signals from six second pixels, add signals from a plurality of pixels that receive the light of the second color and a circuit for adding,
The signal generated from the difference between the signal obtained by adding the signals from the two first pixels and the signal obtained by adding the signals from the six second pixels is used as the signal from the two first pixels. The added signal, the signal obtained by adding the signals from the six second pixels, the signal obtained by adding the signals from the plurality of pixels receiving the light of the second color, and the light of the third color An image processing apparatus comprising: an image processing unit that performs edge enhancement processing on image data generated from a signal obtained by adding signals from a plurality of pixels that receive light . The imaging device according to claim 1 ,
An imaging apparatus in which the first color light is green light, the second color light is red light, and the third color light is blue light. The imaging device according to claim 2 ,
The imaging device, wherein the plurality of pixels are arranged in a Bayer array.

Images