JP4478934B2 - Image signal processing device - Google Patents

Description

  The present invention relates to an image signal processing apparatus, and more particularly to a technique for processing an image signal output from a solid-state imaging device in which pixels are arranged with a half-pitch shift for each column.

  FIG. 6 is a schematic configuration diagram of a solid-state imaging device having a pixel array called a honeycomb array. As shown in the figure, this solid-state imaging device 10 is arranged by shifting the center point of the geometric shape of the light receiving cell 11 every other half in the pixel pitch in the row direction and the column direction (1/2 pitch). It has been made to.

  The solid-state imaging device 10 is assigned a primary color filter of any one of R (red), G (green), and B (blue) corresponding to each light receiving cell 11. As shown in FIG. 6, the BRBR... Row is arranged in the next stage of the GGGG... Row in the horizontal direction, the GGGG... Row is arranged in the next stage, and the RBRB. In the column direction, the GGGG column, the BBBB column, the GGGG column, and the RRRR column are repeated in a cyclic manner.

  This type of honeycomb array solid-state image sensor 10 has R, G, and B pixels arranged at an angle of 45 degrees, and has a higher horizontal and vertical resolution than a square array solid-state image sensor.

  By the way, the solid-state image sensor 10 may be driven by a moving image (movie) or may be driven by a still image. In the case of movie driving, it is necessary to periodically read out signal charges accumulated in each light receiving cell 11 at a predetermined frame rate. It is difficult to read out signal charges within the frame rate, and therefore, horizontal lines are thinned out for reading.

On the other hand, Patent Document 1 proposes a solid-state imaging device device in which pixels of the same color in the horizontal direction are mixed to thereby achieve high image quality and high-speed signal charge readout.
JP 2003-264844 A

The reading order of information from the solid-state imaging device 10 shown in FIG. 6 is to read out pixel information of two lines of the G line and the RB line arranged at an angle of 45 degrees as shown in FIG. In movie driving, the read pixel information (... G _n B _n R _n, G _{n + 1} B _{n + 1} R _{n + 1,} ...) Is used as pixel information at each spatial position, or FIG. As shown in B), pixel information between adjacent lines read out is interpolated and pixel information with an increased number of lines in the vertical direction is used as pixel information at each spatial position. In other words, the information on the pixel space position is not used effectively, the vertical resolution is not sufficiently extracted, and a false signal is generated.

  In addition, if the same vertical number as that of a solid-state image sensor having a general square pixel arrangement is read out, pixel information of two lines arranged at 45 degrees is read out at the same time, and thus a drive frequency of 1.4 times is required. In addition, the required performance of the device including the periphery becomes severe, resulting in a cost increase and a system load due to an increase in power consumption.

  On the other hand, by reading out all pixel information from a solid-state image sensor with a honeycomb arrangement, and performing the same signal processing as a still image (such as signal processing that creates an insufficient pixel by interpolation processing so as to have the same pixel arrangement as a square pixel arrangement) If the vertical resolution is at least equivalent to the vertical resolution, there is a problem that high-speed signal processing and memory capacity increase. In addition, if the RB checkered pattern is not arranged in the vertical direction, image processing equivalent to a still image is impossible. On the other hand, the solid-state imaging device arranged in the RB checkered pattern can deal with only vertical odd thinning. There is.

  The present invention has been made in view of such circumstances, and can improve the vertical resolution and reduce the generation of false signals without increasing the driving frequency when driving a solid-state image pickup element having a honeycomb arrangement. An object is to provide an image signal processing apparatus.

In order to achieve the above object, the image signal processing apparatus of the present invention is a solid-state imaging device in which a plurality of pixels each assigned with a color filter of any one of R, G, and B are two-dimensionally arranged, The pixels are arranged in a positional relationship of 45 degrees obliquely to each other, and a G line consisting of only G pixels in the horizontal direction and an RB line in which R pixels and B pixels are alternately repeated in the horizontal direction are repeated in the vertical direction. comprising a solid-state imaging device, the solid the adjacent by the image pickup device to read out the pixel information alternately from the G line and the RB line for each pixel, the G line of pixel information and the RB lines adjacent to each other reading means for outputting the pixel information as pixel information for one read line, the other of either one of said G line and the RB line as the first line and second line, before According to the vertical spatial position of the second line in the one readout line and the other readout line that are different from each other in the vertical spatial position on the solid-state imaging device, the one readout line and the other readout line Correcting means for correcting the pixel information of the second line of each readout line by weighting and calculating the pixel information of the second line, wherein the second line after correction at each readout line Correction means for performing weighting so that pixel information becomes pixel information corresponding to a vertical spatial position of the first line .
For example, the correcting means may include pixel information of the second line in the one readout line as D1, pixel information of the second line in the other readout line as D2, and the first line in the one readout line as the first line. The difference in vertical spatial position between the first readout line and the second line is a, and the vertical spatial position between the first line in the one readout line and the second line in the other readout line. The pixel information D after correction of the second line in the one readout line is calculated by the equation {D = (D1 × a + D2 × b) / (a + b)} where b is the difference between the two.

  That is, when reading out the pixel information for one line from the solid-state image sensor, the reading unit alternately reads out the pixel information from the adjacent G line and RB line for each pixel, thereby generating point-sequential pixel information of GBGRGBGR. Output. Based on the vertical spatial position of one of the G line and RB line (for example, G line) of the pixel information of one line read out as described above, R pixels on the RB line, The spatial position of the B pixel in the vertical direction is shifted by an interval between the G line and the RB line. Therefore, the correction means corrects the pixel information of the other line with a weight corresponding to the deviation of the spatial position in the vertical direction. As a result, the information of the original pixel space position of each pixel is effectively used to improve the vertical resolution and to reduce the generation of false signals.

In the aspect of the invention, the reading unit may read out pixel information in the vertical direction from the solid-state imaging device when driving a movie. Thereby, the pixel information for one frame can be read out within a predetermined frame rate required for the movie without increasing the driving frequency for reading out the pixel information from the solid-state imaging device.

Further, in one embodiment of the present invention, the solid-state imaging device is characterized in that G pixels, R pixels, and B pixels are arranged in a line in the vertical direction. Since pixels of the same color are arranged in a row in the vertical direction, there is no limitation on the thinning interval, and both odd thinning and even thinning are possible.

In one aspect of the present invention, the correction unit is a pixel of an interpolation line that is located between the readout lines in the vertical direction of the solid-state imaging device and is inserted between the readout lines on data. information, the be one that interpolation operation based on the pixel information of each read line, a spatial position in the vertical direction of the interpolation line, the spatial position in the vertical direction of the same color of pixels in the vertical direction of the respective read line Based on the above, weighted interpolation is performed.

  That is, pixel information on an arbitrary line (interpolation line) different from the line read by the reading means is calculated by interpolation. Thus, the vertical resolution is improved while reducing the number of vertical lines read from the solid-state imaging device.

  According to the present invention, the pixel information for one line alternately read for each pixel from the adjacent G line and RB line from the solid-state image pickup device having the honeycomb arrangement is the vertical space between the G pixel and the RB pixel. Although the position is shifted, the vertical spatial position of one of the G and RB lines is used as a reference, and the pixel information of the other line is corrected with a weight proportional to the vertical spatial position. Thus, information on the original pixel space position of each pixel can be used effectively, so that the vertical resolution can be improved and the generation of false signals can be reduced.

  Hereinafter, preferred embodiments of an image signal processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing an embodiment of an imaging apparatus including an image signal processing apparatus according to the present invention.

  An image pickup apparatus (digital camera) 100 shown in FIG. 1 has functions for recording and reproducing still images and moving images, and the overall operation of the digital camera 100 is centrally controlled by a central processing unit (CPU) 12. The CPU 12 functions as a control means for controlling the camera system according to a predetermined program, and performs various calculations such as automatic exposure (AE) calculation, automatic focus adjustment (AF) calculation, white balance (WB) adjustment calculation, etc. It functions as a means, and further outputs data (coefficients) required for the interpolation calculation in the signal processing circuit 16 as will be described later.

  A memory 18 connected to the CPU 12 via the bus 14 includes a ROM and an SDRAM, and the ROM stores programs executed by the CPU 12, various data necessary for control, and the like. The SDRAM is used as a program development area and a calculation work area for the CPU 12 and as a temporary storage area for image data.

  The operation unit 20 includes a shooting button, a mode switching lever, a mode dial, a menu / OK key, a cross key, and the like. A signal generated by the operation on the operation unit 18 is input to the CPU 12, and the CPU 12 performs digital processing based on the input signal. Each circuit of the camera 100 is controlled, and for example, lens drive control, photographing operation control, image processing control, image data recording / reproduction control, LCD monitor display control, and the like are performed.

  The LCD monitor 22 can be used as an electronic viewfinder for checking the angle of view at the time of shooting, and is used as means for reproducing and displaying recorded images. The LCD monitor 22 is also used as a user interface display screen, and displays information such as menu information, selection items, and setting contents as necessary.

  The media controller 24 performs necessary signal conversion in order to deliver an input / output signal suitable for the recording medium 26 mounted in a media socket (not shown).

  Next, the shooting function of the digital camera 100 will be described.

  When the shooting mode is selected, power is supplied to an image pickup unit including a solid-state image pickup device 10 (hereinafter referred to as “CCD” in this embodiment) made of a CCD or the like, and the camera is ready for shooting. As shown in FIG. 6, the CCD 10 has a honeycomb arrangement, R, G, and B pixels are arranged at an angle of 45 degrees, and the GGGG... G line and the BRBR. The array pattern is alternately arranged, and the GGGG column, the BBBB column, the GGGG column, and the RRRR column are repeated in the vertical direction in a cyclic manner.

  The taking lens 30 is electrically driven by a lens driver 32 controlled by the CPU 12 to perform focus control. The light that has passed through the photographing lens 30 forms an image on the light receiving surface of the CCD 10. The CCD 10 has an electronic shutter function for controlling the charge accumulation time (shutter speed) of each light receiving cell 11. The CPU 12 performs control of charge accumulation time in the CCD 10 and readout control of signal charges from the CCD 10 via the timing generator 34 and the CCD driver 36.

  The subject image formed on the light receiving surface of the CCD 10 is converted into a signal charge of an amount corresponding to the amount of incident light by each light receiving cell 11. The signal charge accumulated in each light receiving cell 11 is sequentially read out as a voltage signal (image signal) corresponding to the signal charge based on a drive pulse given from the timing generator 34 and the CCD driver 36 in accordance with a command from the CPU 12.

  The signal output from the CCD 10 is sent to an analog processing unit (CDS / ADC) 38, where RGB signals for each pixel are sampled and held (correlated double sampling processing), and converted into a digital signal by an A / D converter. The The dot sequential RGB signal converted into the digital signal is applied to the signal processing circuit 16.

  As will be described later, the signal processing circuit 16 includes an interpolation processing unit that corrects the vertical spatial position shift of the RGB pixels in the CCD 10, and the RGB image data corrected here is temporarily stored in the memory 18. The signal processing circuit 16 processes the RGB image data stored in the memory 18 in accordance with a command from the CPU 12. That is, the signal processing circuit 16 functions as an image processing unit including a synchronization circuit, a white balance correction circuit, a gamma correction circuit, a contour correction circuit, a luminance / color difference signal generation circuit, and the like, and uses the memory 18 in accordance with a command from the CPU 12. While performing predetermined signal processing. The image data processed by the signal processing circuit 16 is stored in the memory 18 again.

  When the captured image is output to the LCD monitor 22, the image data is read from the memory 18 and sent to the LCD driver 40 via the bus 14. The LCD driver 40 converts the input image data into an LCD display signal and outputs it to the LCD monitor 22.

  When the CCD 10 is driven by a movie, the image being captured is displayed on the LCD monitor 22 in real time. The photographer can check the shooting angle of view from the video (movie image) displayed on the LCD monitor 22.

  When the shooting button of the operation unit 20 is half-pressed, the digital camera 100 starts AE and AF processing. After that, when the shooting button is fully pressed, a shooting operation for recording a still image starts. That is, the image data acquired in response to the full press of the shooting button is converted into a luminance / color difference signal (Y / C signal) in the signal processing circuit 16 and subjected to predetermined processing such as gamma correction, and then the memory. 18.

  The Y / C signal stored in the memory 18 is compressed according to a predetermined format by the compression / decompression circuit 42 and then recorded on the recording medium 26 via the media controller 24. For example, a still image is recorded in JPEG (Joint Photographic Experts Group) format.

  Next, image signal processing according to the present invention in the signal processing circuit 16 will be described.

  When the pixel information in the vertical direction is read from the CCD 10 by ½ decimation when driving the CCD 10 in a movie, the voltage is higher than the voltage at the time of transfer to the electrodes V1A and V3A corresponding to the CCD transfer gate as shown in FIG. A voltage reading pulse is applied, and the charge accumulated in the light receiving cell corresponding to the transfer gate is read out to the vertical transfer path. Thereafter, the vertical transfer path is driven in four phases by a vertical transfer timing signal, and charges read to the vertical transfer path are transferred toward the horizontal transfer path. The charges transferred to the horizontal transfer path are transferred in the horizontal direction by a horizontal drive pulse and sequentially read out as voltage signals corresponding to the signal charges.

  Here, the G line corresponding to the electrode V1A and the RB line corresponding to the electrode V3A are signal-processed as one line, and the above-described analog processing unit 38 outputs dot-sequential RGB signals in the order of GBGRGBGR. The

  FIG. 2A shows dot-sequential RGB signals on lines n and n + 1 read from the CCD 10.

  Here, since the vertical position of the G pixel (pixel on the G line) and the RB pixel (pixel on the RB line) in the line n are shifted, the image information is corrected according to this spatial position shift. . For example, using the G line as a reference, the RB pixel of the RB line is interpolated so as to become pixel information corresponding to the spatial position of the G line.

That is, as shown in FIG. 2 (B), RGB pixel information of the line n-1 the (RGB signal) R _n-1 G _n-1 B _n-1, the RGB signal of the line _n R n G n B n If the RGB signal of line n + 1 is R _{n + 1} G _{n + 1} B _{n + 1} , and the RGB signal of line n + 2 is R _{n + 2} G _{n + 2} B _{n + 2} , the actual data line n after interpolation processing, The n + 1 RGB signal is corrected as shown in the following equation.
[Equation 1]
Actual data line n
G = G _n
B = (B _n × 3 + B _{n + 1} × 1) / 4
R = (R _n × 3 + R _{n + 1} × 1) / 4
Actual data line n + 1
G = G _{n + 1}
B = (B _{n + 1} × 3 + B _{n + 2} × 1) / 4
R = (R _{n + 1} × 3 + R _{n + 2} × 1) / 4
Further, the RGB signals of the line (interpolation line) n between the line n-1 and the line n and the interpolation line n + 1 between the line n and the line n + 1 are calculated by the following equations.
[Equation 2]
Interpolation line n
G = (G _n-1 + G _n ) / 2
B = (B _n-1 × 1 + B _n × 3) / 4
R = (R _n-1 × 1 + R _n × 3) / 4
Interpolation line n + 1
G = (G + G _{n + 1} ) / 2
B = (B _n × 1 + B _{n + 1} × 3) / 4
R = (R _n × 1 + R _{n + 1} × 3) / 4
[Equation 1] will be described in detail with reference to FIG.

  FIG. 3 is a diagram showing the spatial positional relationship of the RGB pixels of the CCD 10.

Now, correction of the line n among the lines n-1, n, n + 1... Read out from the CCD 10 by 1/2 thinning will be described. If the G line is used as a reference, it is not necessary to correct G _n on the G line. On the other hand, when obtaining a B on G line, using a distance a in the vertical direction from the G line to B _n, and a distance b in the vertical direction from the G line to B _{n + 1,} B _n and B B on the G line is obtained by linearly interpolating _{n + 1} . This B can be obtained by the following equation.
[Equation 3]
B = ( _Bn * a + _{Bn + 1} * b) / (a + b)
= (B _n × a + B _{n + 1} × 3a) / (a + 3a) (∵b = 3a)
= (B _n × 3 + B _{n + 1} × 1) / 4
Similarly, R on the G line can be obtained.

  Next, the interpolation processing unit of the signal processing circuit 16 that performs the interpolation processing will be described.

  FIG. 4 is a principal block diagram showing an embodiment of the interpolation processing unit in the signal processing circuit 16. The interpolation processing unit includes line memories 50 and 52, multipliers 54 and 56, and an adder 58.

  Each of the line memories 50 and 52 holds one line of RGB signals (CCD data) output from the analog processing unit 38 in a dot-sequential manner, and simultaneously processes two lines of CCD data of line n and line n + 1 one pixel at a time. It outputs to the multipliers 54 and 56. Coefficients necessary for the interpolation operation are added to the other inputs of the multipliers 54 and 56 from the terminals a and b of the CPU 12, and the multipliers 54 and 56 multiply these two inputs. That is, when the G signal is input to the multipliers 54 and 56 as shown in [Equation 1], the CPU 12 outputs the coefficients 1 and 0 to the multipliers 54 and 56, respectively, and the RB signal is the multiplier. When the signals are input to 54 and 56, the coefficients 3/4 and 1/4 are output to the multipliers 54 and 56, respectively. The multiplication results in the multipliers 54 and 56 are respectively added to an adder 58, where they are added and output as interpolation data.

  FIG. 5 is a principal block diagram showing another embodiment of the interpolation processing unit of the signal processing circuit 16. Note that portions common to the interpolation processing unit shown in FIG. 4 are denoted by the same reference numerals, and detailed description thereof is omitted.

  The interpolation processing unit shown in FIG. 5 is configured by further adding a line memory 53, a multiplier 57, an adder 59, and a selector 60 to the interpolation processing unit shown in FIG.

  When outputting interpolation data in the actual data line, the CPU 12 outputs a selection signal to the selector 60 so that the selector 60 selects and outputs the input A. When outputting interpolation data in the interpolation line, the CPU 12 Outputs a selection signal to the selector 60 so as to select and output the input B.

  Further, when outputting the interpolation data in the actual data line as described with reference to FIG. 4, the CPU 12 uses the coefficient (1, 0) or (3/4, 1/4) necessary for the interpolation calculation from the terminals a and b. ) And the interpolation data in the interpolation line is output, as shown in [Equation 2], the coefficients (1/2, 1/2) or (1 / 4, 3/4) is output.

  That is, when the G signals on the lines n + 1 and n + 2 are input to the multipliers 56 and 57, the CPU 12 outputs the coefficients 1/2 and 1/2 to the multipliers 56 and 57, respectively. When the RB signals of lines n + 1 and n + 2 are input to 57, the coefficients 1/4 and 3/4 are output to the multipliers 56 and 57, respectively. The multiplication results in the multipliers 56 and 57 are respectively added to the adder 59, where they are added and output to the B input of the selector 60.

  In this interpolation processing unit, when CCD data for one line is fetched, the interpolation data for the actual data line and the interpolation data for the interpolation line are calculated, and the interpolation data for a plurality of lines (two lines) is output. Like to do.

  In this embodiment, the case where the pixel information in the vertical direction is read from the CCD 10 by 1/2 thinning when the CCD 10 is movie-driven has been described. However, the CCD 10 has a G pixel, an R pixel, and a B pixel in the vertical direction. Since the pixels are arranged in a line, the data may be read out by either an odd thinning system or an even thinning system, and thinning can be performed at an arbitrary thinning rate.

Now, assuming that the thinning is 1 / N and the G line is used as a reference, the correction data in the actual data line can be expressed by the following equation.
[Equation 4]
G = G _{n
R = (R n × (N} × 2-1) + R n + 1) / (N × 2)
B = ( _Bn * (N * 2-1) + _{Bn + 1} ) / (N * 2)
However, thinning: 1 / N
In addition, an arbitrary number of interpolation lines can be inserted between actual data lines. In this case, correction data in an arbitrary interpolation line can be expressed by the following equation.
[Equation 5]
G = ( _Gn * (ML) + _{Gn + 1} * L) / M
_{R = R n × {(N} -1) / N × 2 + L / M}
+ R _{n + 1} × {(N + 1) / N × 2 + (ML) / M}
B = _Bn * {(N-1) / N * 2 + L / M}
+ B _{n + 1} × {(N + 1) / N × 2 + (ML) / M}
However, thinning: 1 / N
M times line interpolation (M-1 line insertion)
L = 1 to M−1: When L ≦ N × 2 L = 1−in (M / N) to M−1 to int (M / N): When L> N × 2

FIG. 1 is a block diagram showing an embodiment of an imaging apparatus including an image signal processing apparatus according to the present invention. FIGS. 2A and 2B are diagrams showing dot-sequential RGB signals on lines n and n + 1 read from the CCD and RGB signals after interpolation processing, respectively. FIG. 3 is a diagram showing the spatial positional relationship of the RGB pixels of the CCD used for explaining the interpolation processing according to the present invention. FIG. 4 is a principal block diagram showing an embodiment of the interpolation processing unit in the signal processing circuit. FIG. 5 is a principal block diagram showing another embodiment of the interpolation processing unit in the signal processing circuit. FIG. 6 is a schematic configuration diagram of a solid-state imaging device having a honeycomb arrangement. FIGS. 7A and 7B are diagrams showing dot sequential RGB signals on lines n and n + 1 read from the CCD and RGB signals after conventional interpolation processing, respectively.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Solid-state image sensor (CCD), 11 ... Light receiving cell, 12 ... Central processing unit (CPU), 16 ... Signal processing circuit, 34 ... Timing generator, 36 ... CCD driver, 50, 52, 53 ... Line memory, 54 , 56, 57 ... multipliers, 58, 59 ... adders, 60 ... selectors, 100 ... imaging devices (digital cameras)

Claims (5)

A solid-state imaging device in which a plurality of pixels each assigned with a color filter of any one of R, G, and B are two-dimensionally arranged, and the pixels are arranged in a 45-degree oblique positional relationship with each other A solid-state imaging device in which a G line including only G pixels in the horizontal direction and an RB line in which R pixels and B pixels are alternately repeated in the horizontal direction are repeated in the vertical direction;
Wherein by alternately read out pixel information from the G line and the RB line for each pixel, the G line pixel information and the RB line pixel information one reading of the adjacent adjacent to each other in the solid- Reading means for outputting pixel information for a line ;
Wherein in the G line and the either of the RB line other was a second line as a first line, wherein the solid space position in the vertical direction on the imaging element are different from one each other read lines and other read line the According to the vertical spatial position of the two lines, the pixel information of the second line in the one readout line and the other readout line is weighted and calculated, thereby calculating the second line of each readout line. Correcting means for correcting pixel information, wherein the pixel information of the second line after correction in each readout line is weighted to become pixel information corresponding to a vertical spatial position of the first line. When,
An image signal processing apparatus comprising: The correction means includes pixel information of the second line in the one readout line as D1, pixel information of the second line in the other readout line as D2, and the first line and the one in the one readout line. The difference in vertical spatial position with respect to the second line in the read line is a, and the difference in vertical spatial position between the first line in the one read line and the second line in the other read line The pixel information D after the correction of the second line in the one readout line is calculated by the formula {D = (D1 × a + D2 × b) / (a + b)}, where b is. Item 2. The image signal processing device according to Item 1. 3. The image signal processing apparatus according to claim 1, wherein the reading unit reads out pixel information in the vertical direction from the solid-state image pickup device when driving a movie. 4. The solid-state imaging device, G pixels in the vertical direction, R pixels and B pixels image signal processing apparatus according to any one of 3 claims 1, characterized in that it is arranged in a row. Wherein the correction means, the pixel information of the interpolation line to be inserted between the reading lines the position to and at data on during each read line in the vertical direction of the solid-state imaging device, the pixels in each read line Interpolation calculation is performed based on information, and weighted interpolation is performed based on a vertical spatial position of the interpolation line and a vertical spatial position of pixels of the same color in the vertical direction of each readout line. The image signal processing apparatus according to claim 3 .

Images