JP2009033737A - Signal processing apparatus, image pickup apparatus and synchronization processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a signal processing apparatus which can perform color signal interpolating processing (a synchronizing processing) for the reduction of a false color and noise irrespective of a signal. <P>SOLUTION: A synchronization processing unit 12 which processes image pickup signals including R, G and B signals and luminance signals (W signals) outputted from an image pickup device 10 is used to interpolate other color signals than the respective color signals at pixel positions where the R, G and B signals exist. This color signal interpolating processing includes luminance use estimating processing which estimates color signals interpolated at the pixel positions using not only the same kinds of color signals as the color signals to be interpolated but also the luminance signals existing around the pixel positions. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a signal processing apparatus that processes an imaging signal including a plurality of types of color signals and luminance signals output from an imaging element.

  In a single-plate image sensor having a Bayer array color filter, only one color component signal can be obtained from each of a large number of photoelectric conversion elements included in the image sensor. Therefore, a pixel corresponding to each of the large number of photoelectric conversion elements. It is necessary to perform a so-called synchronization process for interpolating a color signal that cannot be obtained from the photoelectric conversion element at the position. Since the synchronization process is a process of interpolating a color signal that does not originally exist at the pixel position using surrounding color signals, generation of false colors becomes a problem. Patent Document 1 discloses a method that can reduce false colors with any signal.

FIG. 5 is a diagram illustrating an arrangement of color signals obtained from a single-plate image sensor having a color filter with a Bayer arrangement.
The circle shown in FIG. 5 is a color signal, the circle with “R” written inside indicates a red signal, the circle with “G” written inside shows a green signal, and “B” written inside. Circles indicate blue signals. On each color signal, a number (01 to 16) indicating a pixel position is attached.

  When the image signal as shown in FIG. 5 is subjected to the synchronization process by the method disclosed in Patent Document 1, it should be interpolated to the pixel position 10 having only the G signal in the line where the B and G signals are arranged. Assuming that the calculation of the R signal (R10) is correlated in the vertical direction, the difference between the R signal and the G signal in the local region of the image is equal, that is, R10−G10 = Based on the premise that the relationship of R09-G09 is established, R10 = G10 + (R09-G09) is performed. In this case, the G signal (G09) at the R09 pixel position is calculated by performing a one-dimensional interpolation operation on the G signal on the RG line where the pixel position 09 exists, such as G09 = (G05 + G13) / 2. To do.

  As described above, conventionally, for a color signal (for example, R10) to be interpolated at an arbitrary pixel position (for example, pixel position 10), the G signal (G10) at the pixel position and the line including the pixel position are adjacent to each other. By using the color signal (R09) of the same type as the color signal to be interpolated at the pixel position in the vicinity of this line and the G signal (G05, G13) in the vicinity of the color signal of the same type, The color is effectively reduced.

JP 2001-101398 A

  However, in the method disclosed in Patent Document 1, if G10, G05, and G13 contain a lot of noise, R10 generated by using these also contains a lot of noise, resulting in image quality degradation. I will. In addition, depending on the subject color, the correlation between the R signal and the G signal is lowered, and the false color reduction effect may be weakened. In addition, when the subject has a correlation in the oblique direction, the G and RB lines are alternately arranged, so there is no false color reduction effect in the correlation calculation using the G signal.

  The present invention has been made in view of the above circumstances, and provides a signal processing device capable of performing color signal interpolation processing (synchronization processing) to reduce false color and noise for any signal. The purpose is to do.

  The signal processing apparatus of the present invention is a signal processing apparatus that generates color image data by performing signal processing on an imaging signal including a plurality of types of color signals and luminance signals output from an imaging device, and the plurality of types of colors Each of the signals is arranged at a pixel position of pixel data constituting the color image data to be generated, the luminance signal is arranged around the pixel position, and each of the plurality of types of color signals is present. A synchronization processing means for performing a process for interpolating a color signal other than each of the color signals at a position, and the process for interpolating the color signal by the synchronization processing means should be interpolated at the pixel position; A luminance utilization estimation process for estimating a color signal by using the color signal of the same type as the color signal to be interpolated around the pixel position and the luminance signal.

  In the signal processing device according to the aspect of the invention, the imaging signal arranged at the pixel position includes three types of color signals, a first color signal, a second color signal, and a third color signal, and the luminance signal. A first column in which the luminance signal and the first color signal are arranged in a specific direction; and the luminance signal, the second color signal, and the third color signal are arranged in the specific direction. The second column and the second column are arranged alternately in an orthogonal direction orthogonal to the specific direction, the pixel position is a coordinate position where the color signal is arranged, and the synchronization processing means A first color signal to be interpolated between a pixel position where the second color signal exists and a pixel position where the third color signal exists, A second color signal and a third color signal to be interpolated at a pixel position where the first color signal exists; In this case, the luminance utilization estimation process performs interpolation. The luminance utilization estimation process includes a first color signal to be interpolated at the pixel position and a second color signal including the color signal present at the pixel position. The luminance signal in the vicinity of the pixel position included in the column, the luminance signal in the vicinity of the pixel position in the first column adjacent to the second column, and the vicinity of the luminance signal in the first column The second color signal to be interpolated at the pixel position is estimated using a first color signal, and the vicinity of the pixel position included in the first column including the color signal present at the pixel position A luminance signal, a second color signal in the vicinity of the pixel position in the second column next to the first column, and a luminance signal in the vicinity of the second color signal in the second column The third color signal to be interpolated at the pixel position is estimated at the pixel position. The luminance signal in the vicinity of the pixel position included in the first column including the existing color signal, and the third color signal in the vicinity of the pixel position in the second column adjacent to the first column; The estimation processing is performed using a luminance signal in the vicinity of the third color signal in the second column.

  In the luminance use estimation process, the signal processing apparatus according to the present invention is in the vicinity of the pixel position included in the second column including the color signal existing at the pixel position at the pixel position where the first color signal is to be interpolated. The luminance signal is interpolated using the luminance signal, and the first adjacent pixel position which is the pixel position adjacent to the pixel position in the orthogonal direction includes the luminance signal present at the first adjacent pixel position. The first color signal is interpolated using the first color signal in the vicinity of the first adjacent pixel position included in one column, and the luminance signal interpolated at the pixel position is converted into the first adjacent pixel position. The first color signal to be interpolated at the pixel position is estimated by adding the difference between the first color signal interpolated to the luminance signal and the luminance signal existing at the first adjacent pixel position, and the second color signal is estimated. The color signal present at the pixel position where the color signal should be interpolated The luminance signal is interpolated using the luminance signal in the vicinity of the pixel position included in the first column including the second color in the vicinity of the pixel position included in the second column adjacent to the first column. In the vicinity of the second adjacent pixel position included in the second column including the second color signal existing in the second adjacent pixel position at the second adjacent pixel position where the signal is present. The luminance signal is interpolated using the luminance signal, and the luminance signal interpolated at the second adjacent pixel position and the luminance signal interpolated at the second adjacent pixel position into the luminance signal interpolated at the pixel position. The second color signal to be interpolated at the pixel position is estimated, and the third color signal to be interpolated includes the color signal present at the pixel position. A luminance signal is obtained using the luminance signal in the vicinity of the pixel position included in one column. The third adjacent pixel is positioned at a third adjacent pixel position where the third color signal in the vicinity of the pixel position included in the second column adjacent to the first column is present. The luminance signal is interpolated using the luminance signal in the vicinity of the third adjacent pixel position included in the second column including the third color signal existing at the position, and the luminance signal interpolated at the pixel position is By adding the difference between the third color signal existing at the third adjacent pixel position and the luminance signal interpolated at the third adjacent pixel position, the third color signal to be interpolated at the pixel position is obtained. presume.

  In the signal processing device according to the aspect of the invention, the synchronization processing unit may interpolate a third color signal to be interpolated at a pixel position where the second color signal exists and a pixel position where the third color signal exists. The second color signal to be interpolated is performed by the luminance non-use estimation process, and the luminance non-use estimation process includes the color signal to be interpolated at the pixel position at the pixel position. This is estimation processing using the same type of color signal as the color signal to be interpolated in the vicinity of the color signal included in the second column including the color signal.

  In the signal processing apparatus according to the present invention, the color signal to be interpolated at the pixel position is the same type as the color signal to be interpolated around the pixel position. And a first color signal utilization estimation process for estimating using the first color signal and the first color signal.

  In the signal processing device of the present invention, the first color signal and the second color are obtained when the imaging signal arranged at the pixel position is viewed in a crossing direction intersecting the specific direction and the orthogonal direction. A third column in which signals are arranged in the cross direction, a fourth column in which the first color signals and the third color signals are arranged in the cross direction, and the luminance signals in the cross direction. The synchronization processing means is arranged so that the color signal correlated with the intersecting direction is present among the pixel positions having the fifth column sandwiched therebetween in a direction orthogonal to the intersecting direction. , A third color signal to be interpolated at a pixel position where the second color signal exists, a second color signal to be interpolated at a pixel position where the third color signal exists, and the third column A third color signal to be interpolated at a pixel position where the first color signal is included, The second color signal to be interpolated at the pixel position where the first color signal exists in the fourth column is interpolated by the first color signal use estimation process, The first color signal use estimation process includes the second color signal to be interpolated at the pixel position where the third color signal exists in the fourth column including the color signal present at the pixel position. The first color signal in the vicinity of the pixel position, the first color signal in the vicinity of the pixel position in the third column adjacent to the fourth column, and the first color signal in the third column A third color signal that is estimated using a second color signal in the vicinity of the color signal and is to be interpolated at a pixel position where the second color signal exists includes the color signal present at the pixel position The first color signal in the vicinity of the pixel position included in the third column and the fourth column adjacent to the third column Estimation is performed using the first color signal in the vicinity of the pixel position and the third color signal in the vicinity of the first color signal in the fourth column, and the third color signal included in the third column is included in the estimation. The third color signal to be interpolated at the pixel position where the one color signal exists is in the fourth column next to the first color signal and the third column including the first color signal. Estimation is performed using the third color signal in the vicinity of the pixel position and the first color signal in the vicinity of the third color signal in the fourth column, and the first color signal included in the fourth column is included in the fourth column. The second color signal to be interpolated at the pixel position where the one color signal exists is the second color signal near the pixel position in the third column adjacent to the first color signal and the fourth column. And the first color signal in the vicinity of the second color signal in the fourth column.

  In the signal processing device of the present invention, the synchronization processing means includes a first color signal to be interpolated at a pixel position where the second color signal exists and a pixel position where the third color signal exists, The second color signal to be interpolated at the pixel position where the first color signal included in the third column exists and the pixel position where the first color signal included in the fourth column exists. The third color signal to be interpolated is to be interpolated by the first color signal non-use estimation process, and the first color signal non-use estimation process is the presence of the second color signal. A first color signal to be interpolated at a pixel position to be estimated is estimated using a first color signal in the vicinity of the second color signal in the third column including the second color signal; The first color signal to be interpolated at the pixel position where the third color signal exists is the first color signal including the third color signal. The second color to be interpolated at the pixel position where the first color signal included in the third column exists and is estimated using the first color signal in the vicinity of the third color signal in the column A signal is estimated using a second color signal in the vicinity of the first color signal in the third column, and is interpolated at a pixel position where the first color signal included in the fourth column exists. This is a process of estimating the third color signal to be performed using the third color signal in the vicinity of the first color signal in the fourth column.

  In the signal processing device of the present invention, the first color signal is a green signal, the second color signal is a red signal, and the third color signal is a blue signal.

  In the signal processing device according to the aspect of the invention, the image pickup device detects color components of light corresponding to each of the first color signal, the second color signal, and the third color signal. Each photoelectric conversion element included in the first group includes a first group of conversion elements and a second group of luminance detection photoelectric conversion elements that detect a luminance component of light corresponding to the luminance signal. Is based on the position of each photoelectric conversion element included in the second group, and one photoelectric conversion element included in the first group is included in each photoelectric conversion element included in the second group. It is arranged at a position shifted in a predetermined direction from the reference position so as to be adjacent to each other.

  In the signal processing device according to the aspect of the invention, the synchronization processing unit may be configured to specify the specific direction, the orthogonal direction orthogonal to the specific direction, or the specific direction and the orthogonal so that the total number of the three types of color signals is reduced. After adding the color signals of the same color adjacent in the direction, the imaging signal is arranged at the coordinate position.

  In the signal processing apparatus of the present invention, the image pickup device includes a large number of photoelectric conversion elements for obtaining the color signal, and the three kinds of color signals reduce the total number of charges read from the photoelectric conversion element. As described above, after adding charges of the same color component adjacent to the specific direction, the orthogonal direction orthogonal to the specific direction, or adjacent to the specific direction and the orthogonal direction, a color signal corresponding to the added charge is output. Obtained by driving.

  In the signal processing apparatus of the present invention, the three types of color signals are a green signal, a red signal, and a blue signal, and the synchronization processing unit adds only the red signal and the blue signal.

  In the signal processing apparatus of the present invention, the three kinds of color signals are a green signal, a red signal, and a blue signal, and only the red signal and the blue signal are obtained by the addition.

  The imaging device of the present invention includes the signal processing device and the imaging device.

  The synchronization processing program of the present invention is a program for causing a computer to function as the synchronization processing means.

  According to the present invention, it is possible to provide a signal processing apparatus capable of performing color signal interpolation processing (synchronization processing) that reduces false color and noise for any signal.

  Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a schematic plan view of an image pickup element mounted on an image pickup apparatus that is an embodiment of the present invention.
The image pickup device 10 shown in FIG. 1 includes a first group including a large number of photoelectric conversion elements (1, 2, 3) arranged in a square lattice pattern in a row direction X on a semiconductor substrate and a column direction Y orthogonal thereto. And a second group consisting of a large number of photoelectric conversion elements (4) arranged in a square lattice pattern in the row direction X and the column direction Y on the semiconductor substrate. The number of photoelectric conversion elements constituting the first group and the number of photoelectric conversion elements constituting the second group are the same.

  When each photoelectric conversion element included in the first group is based on the position of each photoelectric conversion element 4 included in the second group, the first photoelectric conversion element 4 included in the second group All the photoelectric conversion elements included in the group are arranged at positions deviated from the reference position in a predetermined direction (in the example shown in FIG. 1, obliquely in the upper left direction of 45 degrees) so that the photoelectric conversion elements are adjacent to each other. The conversion elements are arranged in a so-called honeycomb shape. That is, a large number of photoelectric conversion elements included in the imaging element 10 are arranged such that a plurality of photoelectric conversion element arrays each including a plurality of photoelectric conversion elements arranged at a constant pitch in the Y direction are arranged at a constant pitch in the X direction. The two adjacent photoelectric conversion element arrays are arranged at about 1/2 of the arrangement pitch in the Y direction of the photoelectric conversion elements included in the photoelectric conversion element array and shifted from each other in the Y direction. The configuration is as described in US Pat.

  The photoelectric conversion elements 1 to 4 have the same structure, and the detected light components are different depending on the filters formed above the respective light receiving surfaces.

  The photoelectric conversion element 1 has an R color filter that transmits a red (R) component of light formed on the light receiving surface, thereby functioning as a photoelectric conversion element that detects the R component of light. The photoelectric conversion element 2 has a G color filter that transmits the green (G) component of light formed on the light receiving surface, thereby functioning as a photoelectric conversion element that detects the G component of light. The photoelectric conversion element 3 has a B color filter that transmits the blue (B) component of light formed on the light receiving surface, thereby functioning as a photoelectric conversion element that detects the B component of light. In the photoelectric conversion element 4, a luminance filter having spectral characteristics correlated with the luminance component of light is formed on the light receiving surface, thereby functioning as a photoelectric conversion element for detecting the luminance component of light.

  The brightness filter corresponds to an ND filter, a transparent filter, a white filter, a gray filter, or the like, but the configuration in which light is directly incident on the light receiving surface without providing anything above the light receiving surface of the photoelectric conversion element 4 is also possible. It can be said that a filter is provided.

  The color filters formed above the light receiving surfaces of the photoelectric conversion elements of the first group are in a Bayer array. That is, each photoelectric conversion element of the first group includes an RG photoelectric conversion element row in which the photoelectric conversion elements 1 and the photoelectric conversion elements 2 are alternately arranged in this order in the X direction, and the photoelectric conversion element 2 and the photoelectric conversion element. 3 and GB photoelectric conversion element rows arranged alternately in the X direction in this order are arranged alternately in the Y direction.

  In the following description, the red component imaging signal obtained from the photoelectric conversion element 1 is an R signal, the green component imaging signal obtained from the photoelectric conversion element 2 is a G signal, and the blue component imaging signal obtained from the photoelectric conversion element 3 is a B signal. An imaging signal of a luminance component obtained from the photoelectric conversion element 4 is referred to as a W signal.

FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus according to the embodiment of the present invention.
The image pickup apparatus shown in FIG. 2 includes the image pickup element 10 shown in FIG. 1, an AFE 11 that performs predetermined analog signal processing and digital conversion processing on analog signals output from the image pickup element 10, and color image data to be generated. A synchronization processing unit 12 that performs a synchronization process for giving a RGB color signal to a pixel position that is a coordinate position of pixel data to be processed, and R, G, and B 3 generated at one pixel position by the synchronization process YC conversion unit 13 that converts one color signal into luminance signal Y and color difference signal C, a synthesis processing unit 14 that combines the YC signal and the image pickup signal obtained from photoelectric conversion element 4, a compression processing unit 15, and a recording Media 16.

  The synchronization processing unit 12 arranges the imaging signal output from the AFE 11 at the coordinate position on the memory corresponding to the photoelectric conversion element from which the imaging signal is obtained, and each of the photoelectric conversion elements 1 to 3 on the memory. A color signal that cannot be obtained from each of these is obtained from the surrounding photoelectric conversion elements at the coordinate positions corresponding to (this is the pixel position of the pixel data constituting the color image data to be generated). A synchronization process for generating an interpolation using a color signal is performed. The synchronization processing unit 12 is a function realized by, for example, a DSP executing a predetermined program. Hereinafter, a specific method of this synchronization processing will be described.

  FIG. 3 is a diagram showing an arrangement of image pickup signals arranged on the memory. In FIG. 3, a circle with “R” inside shows the R signal obtained from the photoelectric conversion element 1, and a circle with “G” inside shows the G signal obtained from the photoelectric conversion element 2, The circle with “B” inside shows the B signal obtained from the photoelectric conversion element 3, and the circle with “W” inside shows the W signal obtained from the photoelectric conversion element 4. The position where each signal exists corresponds to the position of the photoelectric conversion element from which the signal is output, and this position is defined as the pixel position. A number (01 to 32) indicating the position is given above each pixel position. The G signal corresponds to the first color signal in the claims, the R signal corresponds to the second color signal in the claims, and the B signal corresponds to the third color signal in the claims. Corresponding to

  As shown in FIG. 3, when the imaging signal arranged on the memory is viewed in the Z direction intersecting the X direction and the Y direction at an angle of 45 degrees, the W signal and the G signal are alternately arranged in the Z direction. The arrangement is such that the arranged first column and the second column in which the R signal and the B signal are alternately arranged in the Z direction with the W signal interposed therebetween are alternately arranged in the orthogonal direction orthogonal to the Z direction. Yes. Further, when viewed in the Y direction, a third column in which G signals and R signals are alternately arranged in the Y direction and a fourth column in which G signals and B signals are alternately arranged in the Y direction are represented by W In this configuration, the signals are alternately arranged in the X direction across the fifth column in which the signals are arranged in the Y direction. The Z direction corresponds to a specific direction in the claims, and the Y direction corresponds to an intersecting direction in the claims.

  The synchronization processing unit 12 determines in which direction each of a large number of imaging signals output from the AFE 11 has a correlation by using, for example, a W signal adjacent to the imaging signal, For imaging signals that are correlated in a direction orthogonal to this, the color signal to be interpolated at the pixel position where the imaging signal exists is the same type of color signal as the color signal to be interpolated around the pixel position. And luminance use estimation processing that is estimated using the W signal, and a luminance signal that estimates the color signal to be interpolated at the pixel position using the same type of color signal as the color signal to be interpolated around the pixel position. By the non-use estimation process, the color signal that does not exist is interpolated at the pixel position where each of the R signal, the G signal, and the B signal exists.

  Further, the synchronization processing unit 12 interpolates a color signal to be interpolated at a pixel position where the image pickup signal exists for an image signal correlated in the X direction or the Y direction, around the pixel position. G-use estimation processing that estimates using the same color signal and G signal as the power color signal, and the color signal to be interpolated at the pixel position is the same type as the color signal to be interpolated around the pixel position The color signal that does not exist is interpolated at the pixel position where each of the R signal, the G signal, and the B signal exists by the G non-use estimation process that estimates using the color signal.

  For example, when determining in which direction G10 shown in FIG. 3 is correlated, the synchronization processing unit 12 uses the Y direction: (W05 + W13) / 2− (W06 + W14) / 2, the X direction: (W05 + W06). / 2- (W13 + W14) / 2, Z direction: (W06-W13), direction orthogonal to the Z direction: (W05-W14), and the direction in which the absolute value is the smallest among the calculation results , It is determined that the direction has a correlation with G10.

  In the following description, for convenience, the R signal existing at the pixel position ** is described as “R **”, the G signal existing at the pixel position ** is described as “G **”, and the pixel The B signal existing at the position ** is described as “B **”, and the W signal existing at the pixel position ** is described as “W **”.

  Hereinafter, for example, a method of interpolating a color signal at a pixel position where an imaging signal correlated in the Z direction exists (Method 1) and an interpolation of a color signal at a pixel position where an imaging signal correlated in the Y direction exists, for example The method (method 2) to be performed will be specifically described.

(Method 1)
(A) When R signal is interpolated at the pixel position where the B signal exists In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by the luminance non-use estimation process. That is, the synchronization processing unit 12 uses the R signal to be interpolated at the pixel position where the B signal exists, and the R signal near the pixel position included in the second column including the B signal existing at the pixel position. To estimate.
For example, when the R signal (R11) is interpolated at the pixel position 11, the synchronization processing unit 12 calculates the average of the R signals (R04, R18) in the vicinity of B11 included in the second column including B11. R11 is estimated from (1). R11 = (R04 + R18) / 2 (1)

(B) When B signal is interpolated at a pixel position where the R signal exists In this case, the synchronization processing unit 12 estimates a B signal to be interpolated by luminance non-use estimation processing. That is, the synchronization processing unit 12 uses the B signal to be interpolated at the pixel position where the R signal exists, and the B signal near the pixel position included in the second column including the R signal existing at the pixel position. To estimate.
For example, when the B signal (B18) is interpolated at the pixel position 18, the synchronization processing unit 12 calculates the average of the B signals (B11, B25) in the vicinity of R18 included in the second column including R18. B18 is estimated from (2). B18 = (B11 + B25) / 2 (2)

(C) When G signal is interpolated to each pixel position where B signal is present and pixel position where R signal is present In this case, the synchronization processing unit 12 estimates the G signal to be interpolated by the luminance use estimation process. . That is, the G signal to be interpolated to the pixel position where the B signal exists and the pixel position where the R signal exists respectively, and the W near the pixel position included in the second column including the color signal existing at the pixel position. The estimation is performed using the signal, the W signal in the vicinity of the pixel position in the first column adjacent to the second column, and the G signal in the vicinity of the W signal in the first column.

  For example, when the G signal (G11) is interpolated at the pixel position 11, it is assumed that the difference between the W signal and the G signal in the local region of the image is equal in an object correlated in the Z direction, that is, G11−. Based on the premise that the relationship of W11 = {(G06-W06) + (G15-W15)} / 2 is established, G11 = W11 + {(G06-W06) + (G15-W15)} / 2. G11 is estimated by the calculation of 3).

  Specifically, the synchronization processing unit 12 first uses the luminance signal (W07, W14) in the vicinity of the pixel position 11 included in the second column including B11 existing at the pixel position 11 at the pixel position 11. The signal (W11) is interpolated and the vicinity of the pixel position 06 included in the first column including W06 existing at the pixel position 06 at the pixel position 06 that is the pixel position adjacent to the pixel position 11 in the direction orthogonal to the Z direction. G06 is interpolated using the G signal (G03, G10) of the first pixel, and the pixel position 15 that is the pixel position adjacent to the pixel position 11 in the direction orthogonal to the Z direction includes the first W15 existing at the pixel position 15. G15 is interpolated using G signals (G12, G19) in the vicinity of the pixel position 15 included in the column.

  W11 is obtained, for example, by calculating the average of W07 and W14, G06 is obtained, for example, by calculating the average of G03 and G10, and G15 is obtained, for example, by calculating the average of G12 and G19. . After obtaining W11, G06, and G15, G11 is estimated by performing an operation of W11 + {(G06-W06) + (G15−W15)} / 2 according to the equation (3). In the same way, the G signal is also interpolated at the pixel position where the R signal exists.

  Note that Equation (3) estimates G11 based on the correlation between the pixel position 11 and the pixel positions 06 and 15, but the correlation between the pixel position 11 and the pixel position 06 or the pixel position 11 and the pixel. It is also possible to estimate G11 based on the correlation with the position 15. In this case, the above equation (3) may be used as modified as the following equation (4) or (5). G11 = W11 + (G06-W06) (4), G11 = W11 + (G15-W15) (5).

(D) When R signal and B signal are interpolated at the pixel position where the G signal exists In this case, the synchronization processing unit 12 estimates the R signal and the B signal to be interpolated by the luminance use estimation process. That is, the R signal to be interpolated at the pixel position where the G signal exists, the W signal in the vicinity of the pixel position included in the first column including the G signal existing at the pixel position, and the first column. The estimation is performed using the R signal in the vicinity of the pixel position in the second column and the W signal in the vicinity of the R signal in the second column. Also, the B signal to be interpolated at the pixel position where the G signal exists, the W signal near the pixel position included in the first column including the G signal present at the pixel position, and the first column adjacent to the W signal. Is estimated using the B signal in the vicinity of the pixel position in the second column and the W signal in the vicinity of the B signal in the second column.

For example, when the R signal (R19) is interpolated at the pixel position 19, in the subject having a correlation in the Z direction, the difference between the W signal and the R signal in the local region of the image is equal, that is, R19−. Based on the premise that the relationship of W19 = {(R18−W18) + (R20−W20)} / 2 is established, R19 = W19 + {(R18−W18) + (R20−W20)} / 2. R19 is estimated by the calculation of 6).
Similarly, based on the premise that the relationship B19−W19 = {(B11−W11) + (B27−W27)} / 2 is established, B19 = W19 + {(B11−W11) + (B27−W27)} / 2... B19 is estimated by the calculation of (7).

Specifically, the synchronization processing unit 12 uses the luminance signal (W15, W22) in the vicinity of the pixel position 19 included in the first column including G19 present at the pixel position 19 as the luminance signal (W15, W22). W19) is interpolated, and a W signal (W14, W14, near the pixel position 18 included in the second column including R18 existing at the pixel position 18 is added to the pixel position 18 that is the pixel position adjacent to the pixel position 19 in the X direction. W18) is interpolated using W21), and the pixel position 20 that is the pixel position adjacent to the pixel position 19 in the X direction is replaced with the W in the vicinity of the pixel position 20 included in the second column including R20 that exists at the pixel position 20. W20 is interpolated using signals (W16, W23).
Further, the synchronization processing unit 12 generates a W signal in the vicinity of the pixel position 11 included in the second column including B11 existing at the pixel position 11 at the pixel position 11 that is the pixel position adjacent to the pixel position 19 in the Y direction. W11 is interpolated using (W07, W14), and the pixel position 27 included in the second column including B27 existing at the pixel position 27 is added to the pixel position 27 that is the pixel position adjacent to the pixel position 19 in the Y direction. Interpolation of W27 is performed using neighboring W signals (W23, W30).

  For example, W19 is obtained by calculating the average of W15 and W22, W18 is obtained by calculating the average of W14 and W21, and W20 is obtained by calculating the average of W16 and W23, for example. , W11 is obtained, for example, by calculating the average of W07 and W14, and W27 is obtained, for example, by calculating the average of W23 and W30. After obtaining W18, W19, W20, W11, and W27, R19 is estimated by calculating W19 + {(R18−W18) + (R20−W20)} / 2 according to Equation (6), B19 is estimated by calculating W19 + {(B11-W11) + (B27-W27)} / 2 according to (7).

  Note that Equation (6) estimates R19 based on the correlation between the pixel position 19 and the pixel positions 18 and 20, but the correlation between the pixel position 19 and the pixel position 18 or the pixel position 19 and the pixel. It is also possible to estimate R19 based on the correlation with the position 20. In this case, the above formula (6) may be used as modified as the following formula (8) or (9). R19 = W19 + (R18−W18) (8), R19 = W19 + (R20−W20) (9).

  Expression (7) estimates B19 based on the correlation between the pixel position 19 and the pixel positions 11 and 27, but the correlation between the pixel position 19 and the pixel position 11 or the pixel position 19 and the pixel. It is also possible to estimate B19 based on the correlation with the position 27. In this case, the above equation (7) may be used as modified as the following equation (10) or (11). B19 = W19 + (B11−W11) (10), B19 = W19 + (B27−W27) (11).

  By performing the processes (a) to (d), it is possible to create a state in which the R signal, the G signal, and the B signal exist at each pixel position.

(Method 2)
(A) When G signal is interpolated at a pixel position where B signal exists In this case, the synchronization processing unit 12 estimates a G signal to be interpolated by G non-use estimation processing. That is, the synchronization processing unit 12 estimates the G signal to be interpolated at the pixel position where the B signal exists, using the G signal in the vicinity of the B signal in the fourth column including the B signal.
For example, when the G signal (G11) is interpolated at the pixel position 11, the synchronization processing unit 12 calculates the average of the G signals (G03, G19) in the vicinity of B11 included in the fourth column including G11 ( 12), G11 is estimated. G11 = (G03 + G19) / 2 (12)

(B) When a G signal is interpolated at a pixel position where an R signal exists In this case, the synchronization processing unit 12 estimates a G signal to be interpolated by a G non-use estimation process. That is, the synchronization processing unit 12 estimates the G signal to be interpolated at the pixel position where the R signal exists using the G signal in the vicinity of the R signal in the third column including the R signal.
For example, when the G signal (G18) is interpolated at the pixel position 18, the synchronization processing unit 12 calculates the average of the G signals (G10, G26) in the vicinity of R18 included in the third column including R18 ( 13), G18 is estimated. G18 = (G10 + G26) / 2 (13)

(C) When R signal is interpolated at pixel position where B signal exists In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by the G use estimation process. That is, the R signal to be interpolated at the pixel position where the B signal exists, the G signal near the pixel position included in the fourth column including the color signal present at the pixel position, and the adjacent to the fourth column. Is estimated using the G signal near the pixel position in the third column and the R signal near the G signal in the third column.

  For example, when the R signal (R11) is interpolated at the pixel position 11, it is assumed that the difference between the R signal and the G signal in the local region of the image is equal in the subject having a correlation in the Y direction, that is, R11−. Based on the premise that the relationship of G11 = {(R10−G10) + (R12−G12)} / 2 is established, R11 = G11 + {(R10−G10) + (R12−G12)} / 2. R11 is estimated by the calculation of 14).

  Specifically, the synchronization processing unit 12 first uses the G signal (G03, G19) in the vicinity of the pixel position 11 included in the fourth column including B11 existing at the pixel position 11 at the pixel position 11 as G. The signal (G11) is interpolated, and an R signal (R02) in the vicinity of the pixel position 10 included in the third column including G10 existing at the pixel position 10 is added to the pixel position 10 which is the pixel position adjacent to the pixel position 11 in the X direction. , R18) is used to interpolate R10, and R in the vicinity of the pixel position 12 included in the third column including G12 existing at the pixel position 12 is added to the pixel position 12 that is adjacent to the pixel position 11 in the X direction. R12 is interpolated using the signals (R04, R20).

  G11 is obtained, for example, by calculating the average of G03 and G19, R10 is obtained, for example, by calculating the average of R02 and R18, and R12 is obtained, for example, by calculating the average of R04 and R20. . After obtaining G11, R10, and R12, R11 is estimated by calculating G11 + {(R10−G10) + (R12−G12)} / 2 according to Equation (14).

  The equation (14) estimates R11 based on the correlation between the pixel position 11 and the pixel positions 10 and 12, but the correlation between the pixel position 11 and the pixel position 10 or the pixel position 11 and the pixel. It is also possible to estimate R11 based on the correlation with the position 12. In this case, the above equation (14) may be used as modified as the following equation (15) or (16). R11 = G11 + (R10−G10) (15), R11 = G11 + (R12−G12) (16).

(D) When B signal is interpolated at a pixel position where R signal exists In this case, the synchronization processing unit 12 estimates the B signal to be interpolated by the G use estimation process. That is, the B signal to be interpolated at the pixel position where the R signal exists, the G signal near the pixel position included in the third column including the color signal present at the pixel position, and the third column adjacent to the G signal. The G signal in the vicinity of the pixel position in the fourth column and the B signal in the vicinity of the G signal in the fourth column are used for estimation.

  For example, when the B signal (B18) is interpolated at the pixel position 18, it is assumed that the difference between the B signal and the G signal in the local region of the image is the same in the subject correlated in the Y direction, that is, B18− Based on the premise that the relationship of G18 = {(B17−G17) + (B19−G19)} / 2 is established, R18 = G18 + {(B17−G17) + (B19−G19)} / 2. B18 is estimated by the calculation of 17).

  Specifically, the synchronization processing unit 12 first uses the G signal (G10, G26) in the vicinity of the pixel position 18 included in the third column including R18 existing at the pixel position 18 at the pixel position 18 as G. The signal (G18) is interpolated, and the B signal (B09) in the vicinity of the pixel position 17 included in the fourth column including G17 existing at the pixel position 17 is added to the pixel position 17 that is the pixel position adjacent to the pixel position 18 in the X direction. , B25) is used to interpolate B17, and the pixel position 19 that is the pixel position adjacent to the pixel position 18 in the X direction is replaced with B near the pixel position 19 included in the fourth column including G19 existing at the pixel position 19. B19 is interpolated using the signals (B11, B27).

  For example, G18 is obtained by calculating the average of G10 and G26, B17 is obtained by calculating the average of B09 and B25, and B19 is obtained by calculating the average of B11 and B27, for example. . After obtaining G18, B17, and B19, B18 is estimated by calculating G18 + {(B17−G17) + (B19−G19)} / 2 according to Equation (17).

  Note that Equation (17) estimates B18 based on the correlation between the pixel position 18 and the pixel positions 17 and 19, but the correlation between the pixel position 18 and the pixel position 17 or the pixel position 18 and the pixel. It is also possible to estimate B18 based on the correlation with the position 19. In this case, the above equation (17) may be used as modified as the following equation (18) or (19). B18 = G18 + (B17−G17) (18), B18 = G18 + (B19−G19) (19).

(E) When R signal is interpolated at the pixel position where the G signal in the fourth column exists In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by the G use estimation process. That is, the R signal to be interpolated at the pixel position where the G signal in the fourth column exists, the G signal, and the R signal near the pixel position in the third column adjacent to the fourth column, Estimation is performed using a G signal in the vicinity of the R signal in the fourth column.

  For example, when the R signal (R19) is interpolated at the pixel position 19, it is assumed that the difference between the R signal and the G signal in the local region of the image is equal in the subject correlated in the Y direction, that is, R19−. Based on the premise that the relationship of G19 = {(R18−G18) + (R20−G20)} / 2 is established, R19 = G19 + {(R18−G18) + (R20−G20)} / 2. R19 is estimated by the calculation of 20).

  Specifically, the synchronization processing unit 12 first sets the vicinity of the pixel position 18 included in the third column including R18 existing at the pixel position 18 to the pixel position 18 that is the pixel position adjacent to the pixel position 19 in the X direction. G18 is interpolated using the G signal (G10, G26) of the pixel, and the pixel included in the third column including R20 present at the pixel position 20 at the pixel position 20 that is the pixel position adjacent to the pixel position 18 in the X direction. G20 is interpolated using G signals (G12, G28) in the vicinity of position 20.

  G18 is obtained, for example, by calculating the average of G10 and G26, and G20 is obtained, for example, by calculating the average of G12 and G28. After obtaining G18 and G20, R19 is estimated by calculating G19 + {(R18−G18) + (R20−G20)} / 2 according to equation (20).

  Equation (20) estimates R19 based on the correlation between the pixel position 19 and the pixel positions 18 and 20, but the correlation between the pixel position 19 and the pixel position 18 or the pixel position 19 and the pixel. It is also possible to estimate R19 based on the correlation with the position 20. In this case, the above equation (20) may be used as modified as the following equation (21) or (22). R19 = G19 + (R18−G18) (21), R19 = G19 + (R20−G20) (22).

(F) When interpolating the B signal at the pixel position where the G signal in the third column exists In this case, the synchronization processing unit 12 estimates the B signal to be interpolated by the G use estimation process. That is, the B signal to be interpolated at the pixel position where the G signal in the third column exists is changed to the vicinity of the pixel position in the fourth column adjacent to the G signal and the third column including the G signal. Estimation is performed using the B signal and the G signal in the vicinity of the B signal in the fourth column.

  For example, when the B signal (B10) is interpolated at the pixel position 10, it is assumed that the difference between the B signal and the G signal in the local region of the image is the same in an object correlated in the Y direction, that is, B10−. Based on the premise that the relationship of G10 = {(B09−G09) + (B11−G11)} / 2 is established, B10 = G10 + {(B09−G09) + (B11−G11)} / 2. B10 is estimated by the calculation of 23).

  Specifically, the synchronization processing unit 12 first has a vicinity of the pixel position 09 included in the fourth column including B09 existing at the pixel position 09 at the pixel position 09 that is the pixel position adjacent to the pixel position 10 in the X direction. G09 is interpolated using the G signal (G01, G17), and the pixel included in the fourth column including B11 existing at the pixel position 11 at the pixel position 11 that is the pixel position adjacent to the pixel position 10 in the X direction. G11 is interpolated using G signals (G03, G19) in the vicinity of position 11.

  G09 is obtained, for example, by calculating the average of G01 and G17, and G11 is obtained, for example, by calculating the average of G03 and G19. After obtaining G01 and G11, B10 is estimated by calculating G10 + {(B09−G09) + (B11−G11)} / 2 according to equation (23).

  Note that Equation (23) estimates B10 based on the correlation between the pixel position 10 and the pixel positions 09 and 11, but the correlation between the pixel position 10 and the pixel position 09 or the pixel position 10 and the pixel. It is also possible to estimate B10 based on the correlation with the position 11. In this case, the above equation (23) may be used as modified as the following equation (24) or (25). B10 = G10 + (B09−G09) (24), B10 = G10 + (B11−G11) (25).

(G) When R signal is interpolated at a pixel position where the G signal in the third column exists In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by the G non-use estimation process. That is, the synchronization processing unit 12 estimates the R signal to be interpolated at the pixel position where the G signal in the third column exists using the R signal in the vicinity of the G signal in the third column.
For example, when the R signal (R10) is interpolated at the pixel position 10, the synchronization processing unit 12 calculates the average of the R signals (R02, R18) in the vicinity of G10 included in the third column including G10 ( 26), R10 is estimated. R10 = (R02 + R18) / 2 (26).

(H) When interpolating a B signal at a pixel position where a G signal in the fourth column exists In this case, the synchronization processing unit 12 estimates a B signal to be interpolated by a G non-use estimation process. That is, the synchronization processing unit 12 estimates the B signal to be interpolated at the pixel position where the G signal in the fourth column exists using the B signal in the vicinity of the G signal in the fourth column.
For example, when the B signal (B17) is interpolated at the pixel position 17, the synchronization processing unit 12 calculates the average of the B signals (B09, B25) in the vicinity of G17 included in the fourth column including G17 ( 27), B17 is estimated. B17 = (B09 + B25) / 2 (27).

  By performing the processes (a) to (h), it is possible to create a state in which an R signal, a G signal, and a B signal exist at each pixel position.

  The synchronization processing unit 12 performs the processing of the method 1 or the method 2 as described above according to the subject, so that color signals of R, G, and B colors are obtained at pixel positions corresponding to the photoelectric conversion elements 1, 2, and 3. Is generated. The YC conversion unit 13 generates a YC signal corresponding to each pixel position from the signals at the pixel positions corresponding to the photoelectric conversion elements 1, 2, and 3, and the synthesis processing unit 14 generates the luminance signal generated at each pixel position. Are combined with the W signal at the pixel position adjacent to the pixel position. Thereby, color image data having the same number of resolutions as the photoelectric conversion elements constituting the first group is completed. Although the synthesis process may not be performed, the photoelectric conversion element 4 has higher sensitivity than the other photoelectric conversion elements, and thus the dynamic range can be expanded by performing the synthesis process. The combined color image data is compressed by the compression processing unit 15 and recorded on the recording medium 16.

  As described above, for an oblique subject having a correlation in the Z direction (for example, an image like an arrow extending in the Z direction), the synchronization processing is performed by the method 1, and a vertical subject having a correlation in the Y direction (for example, As for the image (such as an arrow extending in the Y direction), by performing the synchronization process by the method 2, the diagonal object is included as compared to the conventional case where the synchronization process is also performed by the method 2 for the diagonal object. False colors when the subject is imaged can be reduced, and high image quality can be realized.

  In addition, since the photoelectric conversion element 4 has a wider wavelength range of incident light than other photoelectric conversion elements and has higher sensitivity, the S / N of the W signal is better than the S / N of the G signal. . When performing luminance utilization estimation processing as shown in equations (3) to (11), it is possible to estimate a signal to be interpolated based on a low-noise W signal, so that an image with low noise can be generated. It becomes.

  For example, when a transparent filter or the like is used as a luminance filter, the W signal includes all components of the R signal, the G signal, and the B signal. Therefore, the correlation between the G signal and the R signal, Compared with the correlation with the B signal, the correlation between the W signal and each of the R signal, the G signal, and the B signal is high. In the method 1 described above, the G signal that is interpolated at the pixel position where the R signal exists and the pixel position where the B signal exists, and the R signal and B signal that are interpolated at the pixel position where the G signal exists are W It is obtained based on the correlation between the signal and the G signal or the correlation between the W signal and the R signal or the B signal. Thus, since the signal to be complemented by the highly correlated signal is estimated, the false color reduction effect can be improved.

  Further, the synchronization processing described above is not limited to an image pickup apparatus equipped with the image pickup element 10 having the configuration shown in FIG. 1, for example, an image pickup element having a first group of photoelectric conversion elements and a second group. The present invention can also be applied to a two-plate type image pickup apparatus in which an image pickup element having a photoelectric conversion element is mounted and these are arranged by shifting sampling points. The synchronization processing may be performed by inputting RAW data obtained from the image sensor 10 to a computer and executing a predetermined program in the computer.

  In the present embodiment, the image sensor 10 is an image sensor, but it may be a line sensor. In this case, a photoelectric conversion element that outputs an R signal, a photoelectric conversion element that outputs a G signal, a photoelectric conversion element that outputs a B signal is arranged in a predetermined direction, and a photoelectric conversion element that outputs a W signal is in a predetermined direction. An image sensor having an arrangement in which the lines are arranged in a direction orthogonal to the predetermined direction may be used.

  In the above description, the color signals obtained from the photoelectric conversion elements 1 to 3 are arranged as they are at the pixel positions of the pixel data constituting the color image data to be generated, and then the synchronization processing is performed. However, the color signals obtained from the photoelectric conversion elements 1 to 3 are arranged at pixel positions after adding the color signals of the same color so that the total number thereof is reduced. Synchronization processing may be performed.

  FIG. 4 is a diagram showing an array of imaging signals arranged on the memory when such addition processing is performed. In FIG. 4, a square with “R +” inside shows an R + signal obtained by adding together R signals obtained from the photoelectric conversion element 1, and a square with “G +” inside is a photoelectric conversion element. 2 represents a G + signal obtained by adding together the G signals obtained from 2, and a circle with “B +” inside is a B + signal obtained by adding together the B signals obtained from the photoelectric conversion element 3 Is shown.

  For example, as shown in FIG. 4, G01, G03, G17, and G19 shown in FIG. 3 are added, and the added G + signal is arranged at the pixel position 10 at the center position. Further, R02, R04, R18, and R20 shown in FIG. 3 are added, and the R + signal after the addition is arranged at the pixel position 11 at the center position thereof. Further, B09, B11, B25, and B27 shown in FIG. 3 are added, and the B + signal after the addition is arranged at the pixel position 18 at the center position thereof. Further, G10, G12, G26, and G28 shown in FIG. 3 are added, and the G + signal after the addition is arranged at the pixel position 19 at the center position thereof.

  In this way, by adding the color signals of the same color components adjacent to each other in the X direction and the Y direction, two G + signals are present in a range surrounded by the pixel positions 01 to 04 to 28 to 25 in FIG. And the B + signal are generated. Hereinafter, the synchronization processing in the imaging signal array as shown in FIG. 4 will be described.

  The synchronization processing unit 12 determines in which direction each of the color signals arranged on the memory has a correlation by using, for example, a W signal adjacent to the imaging signal, and the Z direction or For a color signal correlated in the orthogonal direction, the color signal to be interpolated at the pixel position where the color signal exists, the color signal of the same type as the color signal to be interpolated around the pixel position, and W The R signal, the G signal, and the B signal are interpolated using a luminance utilization estimation process that estimates using the signal.

  Also, the synchronization processing unit 12 interpolates the color signal to be interpolated at the pixel position where the color signal exists for the color signal correlated in the X direction or the Y direction, around the pixel position. The R signal, the G signal, and the B signal are interpolated using a G use estimation process that estimates using the same kind of color signal and G signal as the power color signal. Hereinafter, a method for interpolating the R signal, the G signal, and the B signal at the pixel position 10 in FIG. 4 will be described in detail.

  First, when determining in which direction G + 10 shown in FIG. 4 is correlated, the synchronization processing unit 12 performs the Y direction: (W05 + W13) / 2− (W06 + W14) / 2, the X direction: (W05 + W06). / 2- (W13 + W14) / 2, Z direction: (W06-W13), direction orthogonal to the Z direction: (W05-W14), and the direction in which the absolute value is the smallest among the calculation results , G + 10 is determined to be correlated.

(Method 1: When there is a correlation in the Z direction)
(A) When R signal is interpolated at pixel position 10 In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by luminance use estimation processing.
When the R signal (R10) is interpolated at the pixel position 10, the premise that the difference between the W signal and the R signal in the local region of the image is equal in the subject correlated in the Z direction is the R signal before addition. Using,
R10−W10 = {(R02−W02) + (R04−W04) + (R18−W18) + (R20−W20)} / 4 (28)
Can be expressed as
Further, since R + 11 = (R02 + R04 + R18 + R20), R10 is estimated by the calculation of Expression 29 obtained by modifying Expression 28 below.
R10 = W10 + {“R + 11” −W02−W04−W18−W20} / 4 (29)

Specifically, the synchronization processing unit 12 estimates W02, W04, W18, W20, and W10 from neighboring pixels in the Z direction.
For example, W02 = (W05 + Wb) / 2, W04 = (W07 + Wd) / 2, W18 = (W21 + W14) / 2, W20 = (W23 + W16) / 2, and W10 = (W13 + W06) / 2.

Therefore, since Equation 29 can be transformed into the following Equation 30, R10 can be obtained from Equation 30.
R10 = (W13 + W06) / 2 + {“R + 11” − (W05 + Wb) / 2− (W07 + Wd) / 2− (W21 + W14) / 2− (W23 + W16) / 2} / 4 (30)

(B) When B Signal is Interpolated at Pixel Position 10 In this case, the synchronization processing unit 12 estimates the B signal to be interpolated by the luminance use estimation process.
When the B signal (B10) is interpolated at the pixel position 10, the premise that the difference between the W signal and the B signal in the local region of the image is equal in the subject correlated in the Z direction is the B signal before addition. Using,
B10−W10 = {(B09−W09) + (B11−W11) + (B25−W25) + (B27−W27)} / 4 (31)
Can be expressed as
Furthermore, since “B + 18” = (B09 + B11 + B25 + B27), B10 is estimated by the calculation of Expression 32 obtained by modifying Expression 31.
B10 = W10 + {“B + 18” −W09−W11−W25−W27} / 4 (32)

Since Expression 32 can be transformed into the following Expression 33 similarly to the processing of (a) above, B10 can be obtained from this Expression 33.
B10 = (W13 + W06) / 2 + {“B + 18” − (Wf + W05) / 2− (W14 + W07) / 2− (Wh + W21) / 2− (W30 + W23) / 2} / 4 (33)

(C) When G Signal is Interpolated at Pixel Position 10 In this case, the synchronization processing unit 12 estimates the G signal to be interpolated by the luminance use estimation process.
When the G signal (G10) is interpolated at the pixel position 10, the premise that the difference between the W signal and the G signal in the local region of the image is equal in the subject correlated in the Z direction is the G signal before addition. Using,
G10−W10 = {(G01−W01) + (G03−W03) + (G17−W17) + (G19−W19)} / 4 (34)
Can be expressed as
Furthermore, since “G + 10” = (G01 + G03 + G17 + G19), G10 is estimated by the calculation of Expression 35 obtained by modifying Expression 34.
G10 = W10 + {“G + 10” −W01−W03−W17−W19} / 4 (35)

Since Expression 35 can be transformed into the following Expression 36 similarly to the processing of (a) above, G10 can be obtained from Expression 36.
G10 = (W13 + W06) / 2 + {"G + 10"-(We + Wa) / 2- (W06 + Wc) / 2- (Wg + W13) / 2- (W22 + W15) / 2} / 4 (36)

(Method 2: When there is a correlation in the Y direction)
(A) When R signal is interpolated at pixel position 10 In this case, the synchronization processing unit 12 estimates the R signal to be interpolated by the G use estimation process.
For example, when the R signal (R10) is interpolated at the pixel position 10, the premise that the difference between the G signal and the R signal in the local region of the image is equal in the subject having a correlation in the Y direction is the G before the addition. Using signal and R signal,
R10−G10 = {(R02−G02) + (R04−G04) + (R18−G18) + (R20−G20} / 4 (37)
Can be expressed as
Furthermore, since “R + 11” = (R02 + R04 + R18 + R20), R10 is estimated by the calculation of Expression 38 obtained by modifying Expression 37.
R10 = G10 + {“R + 11” −G02−G04−G18−G20} / 4 (38)

  Specifically, the synchronization processing unit 12 first obtains G02 from neighboring pixels in the Y direction. For example, G02 = G10. Similarly, G04 = G12, G18 = G26, and G20 = G28. Next, G10 is obtained as “G + 10” / 4 as the addition center.

Furthermore, since “G + 19” = (G10 + G12 + G26 + G28), R10 can be estimated by the calculation of Expression 39 obtained by modifying Expression 38.
R10 = “G + 10” / 4 + {“R + 11” − “G + 19”} / 4 (39)

(B) When B Signal is Interpolated at Pixel Position 10 In this case, the synchronization processing unit 12 estimates the B signal to be interpolated by the G use estimation process.
For example, when the B signal (B10) is interpolated at the pixel position 10, the premise that the difference between the G signal and the B signal in the local region of the image is equal in the subject correlated in the Y direction is the G before the addition. Using signal and B signal,
B10−G10 = {(B09−G09) + (B11−G11) + (B25−G25) + (B27−G27)} / 4 (40)
Can be expressed as
Furthermore, since “B + 18” = (B09 + B11 + B25 + B27), B10 is estimated by the calculation of Expression 41 obtained by modifying Expression 40.
B10 = G10 + {“B + 18” −G09−G11−G25−G27} / 4 (41)

Since Expression 41 can be transformed into the following Expression 42 in the same manner as the processing of (a) above, B10 can be obtained from this Expression 42.
B10 = "G + 10" / 4 + {"B + 18"-"G + 10"} / 4 (42)

(C) When G Signal is Interpolated at Pixel Position 10 In this case, the synchronization processing unit 12 sets “G + 10” / 4 as the G signal.

  As described above, even if color signals or charges of the same color are added, false color reduction can be performed by performing correlation calculation in the vertical and horizontal directions and in the diagonal direction. In addition, since the color resolution is less likely to be a problem even when the resolution is lower than the luminance resolution, the readout rate can be reduced by adding charges and reading and having little influence on the image quality. Also, the number of color pixels to be calculated can be reduced by adding signals after signal output. In the example shown in FIG. 4, since the color signals are added in the vertical and horizontal directions, it is possible to obtain a low resolution image of equal resolution in the vertical and horizontal directions.

  As a method of addition, a method of adding color signals of the same color component adjacent in the X direction or a method of adding color signals of the same color component adjacent in the Y direction may be employed. For example, G01 and G17 shown in FIG. 3 are added, the G signal is arranged at the pixel position 09, and B09 and B25 are added and the B signal is arranged at the pixel position 17, thereby performing FIG. In comparison, color image data in which the number of pixel data is halved may be generated. Alternatively, G01 and G03 shown in FIG. 3 are added, the G signal is arranged at the pixel position 02, and R02 and R04 are added and the R signal is arranged at the pixel position 03, so that FIG. In comparison, color image data in which the number of pixel data is halved may be generated.

  Further, only the R signal and the B signal may be added without adding the G signal. In this case, since the G signal having a high correlation with the luminance is not added, the W signal and the G signal are arranged in a diamond shape, and high resolution can be obtained in the vertical and horizontal directions. Further, by adding the R signal and the B signal, the number of pixels can be reduced, the cost of calculation and readout can be reduced, and the sensitivity can be improved.

  The R signal and B signal interpolation processing is equivalent to the above-described interpolation processing at the time of addition, and the G signal interpolation processing is equivalent to the above-described interpolation processing at the time of non-addition. Thereby, the R signal and the B signal are interpolated at the pixel position obtained by adding the R signal and the B signal, and the G signal is interpolated at all the pixel positions before the addition. When the R signal and the B signal are interpolated at all pixel positions before addition, G utilization estimation processing using the G signal interpolated at all pixel positions is used. That is, the G signal at the pixel position where the added R signal and B signal are arranged is subtracted from the R signal and B signal to obtain a difference, and the difference values are averaged and interpolated to obtain the G signal at each pixel position. Can be obtained by adding.

  Here, the color signals are added on the memory. However, when the image sensor 10 is driven, the added color signal is output from the image sensor 10 by adding the charges read from the photoelectric conversion elements. You may make it let it.

  In this case, the imaging device 10 is a CCD (Charge Coupled Device) type. In other words, the charge is read from the photoelectric conversion element to the vertical charge transfer path, transferred in the Y direction, the transferred charge is transferred to the X direction in the horizontal charge transfer path, and the transferred charge is converted into a signal corresponding to the charge amount. A configuration having an output unit for converting and outputting. And the drive part which drives the image pick-up element 10 WHEREIN: The charge of the same color component adjacent to a X direction, a Y direction, or a Y direction and a X direction so that the total of the charge read from the photoelectric conversion element may decrease. Is added in the vertical charge transfer path or the horizontal charge transfer path, and then driving for outputting a color signal corresponding to the added charge is performed.

Schematic plan view of an image sensor mounted on an image pickup apparatus according to an embodiment of the present invention 1 is a block diagram showing a schematic configuration of an imaging apparatus that is an embodiment of the present invention. The figure which showed the arrangement | sequence of the imaging signal output from AFE of FIG. 2, and arrange | positioned on the memory The figure which showed the arrangement | sequence after the addition of the imaging signal output from AFE of FIG. 2 and arrange | positioned on the memory The figure for demonstrating the conventional synchronous processing method

Explanation of symbols

1 to 3 Color detection photoelectric conversion element 4 Luminance detection photoelectric conversion element 12 Synchronization processing unit

Claims (15)

A signal processing apparatus that generates color image data by performing signal processing on an imaging signal including three types of color signals and luminance signals output from an imaging element,
Each signal included in the imaging signal is arranged at a coordinate position, and processing for interpolating the three types of color signals is performed at the pixel position that is the coordinate position corresponding to one pixel data of the color image data to be generated. A synchronization processing means,
The process for interpolating the color signal by the synchronization processing means is to convert the color signal to be interpolated at the pixel position into the color signal of the same type as the color signal to be interpolated around the pixel position and the luminance signal. A signal processing apparatus including luminance use estimation processing to be estimated by using. The signal processing device according to claim 1,
The imaging signal disposed at the coordinate position includes three types of color signals, a first color signal, a second color signal, and a third color signal, and the luminance signal, and the luminance signal and the first color signal. A first column in which one color signal is arranged in a specific direction; and a second column in which the luminance signal, the second color signal, and the third color signal are arranged in the specific direction. It is a configuration arranged alternately in the orthogonal direction orthogonal to the direction,
The pixel position is a coordinate position where the color signal is arranged,
The synchronization processing means includes a pixel position where the second color signal exists and a pixel position where the third color signal exists among pixel positions where a color signal correlated in the specific direction exists. The first color signal to be interpolated and the second color signal and the third color signal to be interpolated at the pixel position where the first color signal exists are interpolated by the luminance use estimation process. And
The luminance use estimation processing includes the first color signal to be interpolated at the pixel position, the luminance signal in the vicinity of the pixel position included in the second column including the color signal existing at the pixel position, Estimating using the luminance signal in the vicinity of the pixel position in the first column next to the second column and the first color signal in the vicinity of the luminance signal in the first column; A second color signal to be interpolated at the position, the luminance signal in the vicinity of the pixel position included in the first column including the color signal existing at the pixel position, and the second color signal adjacent to the first column. A third color signal to be estimated by using the second color signal in the vicinity of the pixel position in the column and the luminance signal in the vicinity of the second color signal in the second column and to be interpolated to the pixel position. For the luminance signal in the vicinity of the pixel position included in the first column including the color signal present at the pixel position. A signal, a third color signal in the vicinity of the pixel position in the second column adjacent to the first column, and a luminance signal in the vicinity of the third color signal in the second column. A signal processing apparatus which is a process to be estimated using. The signal processing apparatus according to claim 2,
In the luminance use estimation process, luminance is used by using the luminance signal in the vicinity of the pixel position included in the second column including the color signal existing at the pixel position at the pixel position where the first color signal is to be interpolated. Interpolate the signal, and the first adjacent pixel position, which is the pixel position adjacent to the pixel position in the orthogonal direction, includes the luminance signal present in the first adjacent pixel position, the first column included in the first column The first color signal interpolated at the first adjacent pixel position by interpolating the first color signal using the first color signal in the vicinity of one adjacent pixel position and interpolating at the pixel position. And the difference between the luminance signal existing at the first adjacent pixel position and the first color signal to be interpolated at the pixel position are estimated,
The luminance signal is interpolated at the pixel position where the second color signal is to be interpolated using the luminance signal in the vicinity of the pixel position included in the first column including the color signal existing at the pixel position, and the second color signal is interpolated. The second adjacent pixel position in the second adjacent pixel position, which is the pixel position where the second color signal exists in the vicinity of the pixel position and is included in the second column adjacent to the first column, is present in the second adjacent pixel position. The luminance signal is interpolated using the luminance signal in the vicinity of the second adjacent pixel position included in the second column including the second color signal, and the second adjacent signal is added to the luminance signal interpolated at the pixel position. Estimating the second color signal to be interpolated at the pixel position by adding the difference between the second color signal present at the pixel position and the luminance signal interpolated at the second adjacent pixel position;
The luminance signal is interpolated at the pixel position where the third color signal is to be interpolated using the luminance signal in the vicinity of the pixel position included in the first column including the color signal existing at the pixel position, and the second color signal is interpolated. The third adjacent pixel position, which is the pixel position where the third color signal exists in the vicinity of the pixel position and is included in the second column adjacent to the first column, exists at the third adjacent pixel position. A luminance signal is interpolated using the luminance signal in the vicinity of the third adjacent pixel position included in the second column including the third color signal, and the luminance signal interpolated at the pixel position is interpolated with the third adjacent signal. A signal processing device that estimates the third color signal to be interpolated at the pixel position by adding the difference between the third color signal present at the pixel position and the luminance signal interpolated at the third adjacent pixel position . The signal processing device according to claim 2 or 3,
The synchronization processing means includes a third color signal to be interpolated at a pixel position where the second color signal exists, and a second color signal to be interpolated at a pixel position where the third color signal exists. Is interpolated by luminance non-use estimation processing,
In the luminance non-use estimation process, the color signal to be interpolated at the pixel position is the same as the color signal to be interpolated in the vicinity of the color signal included in the second column including the color signal existing at the pixel position. A signal processing apparatus that is a process that estimates using various types of color signals. The signal processing device according to any one of claims 2 to 4,
In the process for interpolating the color signal by the synchronization processing means, the color signal to be interpolated at the pixel position is the same type of color signal as the color signal to be interpolated around the pixel position, and the first A signal processing apparatus including a first color signal use estimation process for estimation using a color signal. The signal processing device according to claim 5,
The first color signal and the second color signal are arranged in the intersecting direction when the image pickup signal arranged at the pixel position is viewed in the intersecting direction intersecting the specific direction and the orthogonal direction. A third column, a fourth column in which the first color signal and the third color signal are arranged in the crossing direction, and a fifth column in which the luminance signal is arranged in the crossing direction are sandwiched. It is configured to be alternately arranged in a direction orthogonal to the intersecting direction,
The synchronization processing means includes a third color signal to be interpolated at a pixel position where the second color signal is present among pixel positions where a correlated color signal exists in the intersecting direction, and the third color signal. A second color signal to be interpolated at a pixel position where the color signal of the second color signal is present, a third color signal to be interpolated at a pixel position where the first color signal included in the third column is present, and The second color signal to be interpolated at the pixel position where the first color signal exists in the fourth column is to be interpolated by the first color signal use estimation process,
The first color signal use estimation process includes a second color signal to be interpolated at a pixel position where the third color signal exists in a fourth column including the color signal present at the pixel position. The first color signal in the vicinity of the pixel position, the first color signal in the vicinity of the pixel position in the third column adjacent to the fourth column, and the first color signal in the third column. And a second color signal in the vicinity of the color signal of
The third color signal to be interpolated at the pixel position where the second color signal exists, and the first color signal near the pixel position included in the third column including the color signal present at the pixel position. A first color signal in the vicinity of the pixel position in the fourth column next to the third column, and a third color signal in the vicinity of the first color signal in the fourth column And using
A third color signal to be interpolated at a pixel position where the first color signal included in the third column is present, a third column including the first color signal and the first color signal. Using the third color signal in the vicinity of the pixel position in the fourth column next to the first color signal and the first color signal in the vicinity of the third color signal in the fourth column,
The second color signal to be interpolated at the pixel position where the first color signal included in the fourth column is present, the first color signal, and the third color signal adjacent to the fourth column. A signal processing apparatus which is a process of estimating using a second color signal in the vicinity of the pixel position in the column and a first color signal in the vicinity of the second color signal in the fourth column. The signal processing device according to claim 6,
The synchronization processing means is included in the third column and the first color signal to be interpolated at the pixel position where the second color signal exists and the pixel position where the third color signal exists, respectively. The second color signal to be interpolated at the pixel position where the first color signal exists and the third color signal to be interpolated at the pixel position where the first color signal included in the fourth column exists. Are interpolated by the first color signal non-use estimation process,
In the first color signal non-use estimation process, the first color signal to be interpolated at the pixel position where the second color signal exists is replaced with the first column of the third column including the second color signal. The first color signal that is estimated using the first color signal in the vicinity of the second color signal and is to be interpolated at the pixel position where the third color signal exists includes the third color signal. The second color to be estimated by using the first color signal in the vicinity of the third color signal in the fourth column and to be interpolated at the pixel position where the first color signal included in the third column exists. The pixel position at which the first color signal included in the fourth column exists is estimated using the second color signal in the vicinity of the first color signal in the third column. A signal processing device that is a process for estimating a third color signal to be interpolated using a third color signal in the vicinity of the first color signal in the fourth column The signal processing device according to any one of claims 2 to 7,
The signal processing device, wherein the first color signal is a green signal, the second color signal is a red signal, and the third color signal is a blue signal. The signal processing device according to any one of claims 2 to 8,
A first group in which the image sensor includes a color detection photoelectric conversion element that detects a color component of light corresponding to each of the first color signal, the second color signal, and the third color signal. And a second group of luminance detecting photoelectric conversion elements for detecting a luminance component of light according to the luminance signal,
When the photoelectric conversion elements included in the first group are based on the positions of the photoelectric conversion elements included in the second group, the photoelectric conversion elements included in the second group A signal processing device arranged at a position shifted in a predetermined direction from the reference position so that one photoelectric conversion element included in one group is adjacent. The signal processing device according to claim 1,
The synchronization processing unit is configured to reduce the total number of the three kinds of color signals in a specific direction, an orthogonal direction orthogonal to the specific direction, or the same color adjacent to the specific direction and the orthogonal direction. A signal processing device that adds the signals and arranges the imaging signal at the coordinate position. The signal processing device according to claim 1,
The image sensor includes a number of photoelectric conversion elements for obtaining the color signal,
The three types of color signals are adjacent to the specific direction, the orthogonal direction orthogonal to the specific direction, or adjacent to the specific direction and the orthogonal direction so that the total number of charges read from the photoelectric conversion element decreases. A signal processing apparatus obtained by driving to add charges of the same color component and then outputting a color signal corresponding to the added charge. The signal processing device according to claim 10,
The three kinds of color signals are a green signal, a red signal and a blue signal,
The synchronization processing means is a signal processing device that adds only the red signal and the blue signal. The signal processing device according to claim 11,
The three kinds of color signals are a green signal, a red signal and a blue signal,
A signal processing apparatus in which only the red signal and the blue signal are obtained by the addition. A signal processing device according to any one of claims 1 to 13,
An imaging apparatus comprising the imaging element.   A synchronization processing program for causing a computer to function as the synchronization processing means of the signal processing apparatus according to claim 1.

Images