JP2007097203A - Method of correcting light volume difference - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To suppress the deterioration of a composite image, based on the light volume difference between a plurality of images, in the light volume correcting method which obtains high solution images by obtaining one composite image from a plurality of images produced using image shifting. <P>SOLUTION: An image memory 4 stores a plurality of images photographed by a photographing element, while composing them. An LPF (low-pass filter) 5 cuts a spatial frequency component less than a predetermined frequency in a predetermined direction from a composite image data stored in the image memory 4, whereby the image data in which the light volume difference between the plurality of images is corrected, is produced to rewrite the data stored in the image memory 4. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a light amount difference correction method capable of obtaining a high-resolution image, and in particular, takes a plurality of images by moving image light from a subject relative to an imaging means, and combines them. The present invention relates to a light amount difference correction method for obtaining one high-resolution image.

  In recent years, a still image imaging device called an electronic still camera has been put into practical use as an imaging device for capturing a still image. A home video camera has been put to practical use as a moving image capturing apparatus for capturing a moving image. In these imaging apparatuses, a solid-state imaging device is used as means for receiving image light from a subject and capturing an image. This solid-state imaging device is a so-called two-dimensional CCD (Charge Coupled Device) image sensor. In this solid-state imaging device, a plurality of light receiving regions are arranged in a matrix on an image plane that is a two-dimensional plane.

  The imaging device forms image light from the subject on the imaging surface of the solid-state imaging device and receives the light in each light receiving region. The image light is photoelectrically converted into an electric signal indicating the amount of light received in the light receiving area, and then recorded on the recording medium as an image signal. When this image signal is visually displayed on a display device alone, a captured still image is obtained. Further, when the image signals are continuously visually displayed in the order in which the image signals are captured, a captured moving image is obtained.

  An image picked up by such an image pickup apparatus is composed of pixels corresponding to the light receiving region of the solid-state image pickup device. That is, each image sensor is equivalent to sampling the amount of image light that varies spatially continuously at a spatial sampling frequency. This spatial sampling frequency is the reciprocal of the pixel arrangement period. Thereby, the light quantity change of the image light is smoothed in units of pixels. Therefore, the larger the number of pixels, the better the image resolution.

  As a technique for increasing the resolution of an image, there is a solid-state imaging device using image shift. Image shift is a technique for moving the light receiving position of image light incident on a solid-state imaging device. In a solid-state imaging device using image shift, imaging is performed a plurality of times with different light receiving positions on the imaging plane of image light from a subject. The output images are generated by superimposing the captured images so that the light receiving positions of the images coincide with each other.

  As a solid-state imaging device using this image shift mechanism, the one described in cited document 1 (Japanese Patent Laid-Open No. 63-284980) is known. In this solid-state imaging device, a parallel plate that transmits light is interposed between a condensing lens that collects light from a subject and the solid-state imaging device. The parallel plate takes a first state arranged perpendicular to the optical axis and a second state inclined about an oblique direction of 45 degrees with respect to the horizontal and vertical directions of the field of view. The solid-state imaging device captures the first image when the parallel plate is in the first state, and then tilts the parallel plate to capture the second image in the second state.

  FIG. 32 is a diagram illustrating an equivalent pixel arrangement of an output image. This output image is a monochrome image. The light receiving regions of the solid-state imaging device are arranged in a matrix with a horizontal array period PH and a vertical array period PV, and the first image and the second image halve light from the subject. It is assumed that the image is picked up by shifting each pixel in an oblique direction. At this time, in the output image obtained by combining the two images, the pixels are arranged in the horizontal direction arrangement period (PH / 2) and the vertical direction arrangement period (PV / 2). That is, the number of pixels in the entire image is quadrupled. In FIG. 32, a pixel s1 is an actual pixel from which pixel data is obtained from the first original image captured in the first state. The pixel s2 is an actual pixel from which pixel data is obtained from the second original image captured in the second state. In FIG. 32, the actual pixels are shown with hatching. Thereby, in the output image, the actual pixels from which the data of each pixel is obtained are arranged in a checkered pattern.

  A pixel (virtual pixel) that does not have pixel data is adjacent to two real pixels along each arrangement direction. The pixel data of these virtual pixels can be obtained, for example, by interpolating values obtained by averaging pixel data of four adjacent real pixels. In this way, in the solid-state imaging device according to the conventional technique, a high-resolution image including four times as many pixels as the number of light-receiving regions of the solid-state imaging device can be obtained.

  In the solid-state imaging device described above, image shift processing is performed to generate an image signal of one output image from two original images that are successively captured. Therefore, in this apparatus, when capturing two original images, it is desirable that each exposure time be equal in order not to cause a light amount difference between the two original images. However, even when the exposure times are the same, there may be a difference in the amount of light between the two original images due to flickering of fluorescent lamps.

  Thus, when a light quantity difference occurs between two original images, a checkered pattern that does not originally appear in the image light appears in the output image, resulting in a problem that the image quality is significantly impaired. FIG. 33 is a diagram illustrating an example of a checkered pattern that appears when a flat plain pattern is imaged. In the figure, the number written in each pixel indicates the value of the pixel data in that pixel, and the pixel s1 in the first image is assumed to have pixel data, that is, the light quantity is 100. Further, it is assumed that the pixel data s2 in the second image originally has pixel data of 100, but becomes pixel data 90 due to the above-described phenomenon, and the amount of light is reduced. At this time, if the image data of the virtual pixel is obtained by averaging the pixel data of the four adjacent real pixels, the pixel data is calculated as 95. In this way, when a light amount difference occurs between the two images, a checkered pattern appears as shown in FIG. 33 where the pixel data of all pixels is originally a plain pattern.

  The present applicant has previously proposed an imaging apparatus having a light amount difference correcting means for solving this problem (Cited document 2 (Japanese Patent Application No. 8-267552; title of the invention “imaging apparatus”)). Below, the light quantity difference correction method proposed by the present applicant will be described.

  First, one light quantity difference correction method (light quantity difference correction method 1) will be described with reference to FIGS. FIG. 34 shows a histogram of one block obtained by dividing the first original image into predetermined blocks, and FIG. 35 shows a histogram of one block obtained by dividing the second original image into predetermined blocks. The light amount difference correction means is a minimum value or average value among pixel data values of pixels corresponding to each block of two original images (for example, an integer value of 0 or more and 255 or less when recorded in 8 bits). The light quantity difference between the two screens is corrected by adjusting the value and the maximum value in the following manner.

  Here, the minimum value, the average value, and the maximum value of the pixel data of the pixels corresponding to the first image are α, β, and γ, respectively. On the other hand, the minimum value, the average value, and the maximum value of the pixel data of the pixels corresponding to the second image are δ, ε, and ζ, respectively.

  As shown in FIG. 36, the horizontal axis represents the pixel data value of the pixel corresponding to the second image before the light amount difference correction, and the vertical axis represents the pixel data corresponding to the second image after the light amount difference correction. The pixel data values are taken, and point η (δ, α), point θ (ε, β), and point ι (ζ, γ) are taken. Then, projection is performed by connecting the above points η, θ, ι with straight lines, and pixel data values δ, ε, ζ of the pixels of the second image are respectively converted into pixel data values α of the pixels of the first image. , Β, γ.

  According to the above configuration, the predetermined pixel value (that is, the minimum value, the average value, and the maximum value) of the pixel data of the pixel of the second image is converted into the predetermined pixel data of the first image by the light amount difference correction unit. Since each value is converted to a value, the values of predetermined pixel data match between the two original images.

  As a result, the difference in the amount of imaging light between the two original images can be corrected, and the deterioration in the image quality of the output image due to the difference in the amount of light can be prevented.

  The above proposal also shows a method of correcting the light amount difference by obtaining a moving average (light amount difference correcting method 2). This light quantity difference correction method 2 obtains a moving average for every two original images at this pixel position in a pixel of a second image to be subjected to light quantity difference correction, and obtains a difference between the moving averages, that is, a light quantity difference. This light amount difference is added to the pixel data of a certain pixel in the second image. By performing this operation for each pixel of the second image, the light amount difference is corrected.

The above proposal also shows a method (light amount difference correction method 3) in which a sensor for detecting the amount of light for each imaging is separately provided and light amount difference correction is performed based on the light amount value of each original image obtained from the sensor output. Yes.
JP-A 63-284980 Japanese Patent Application No. 8-267552 (May 6, 1998)

  In the light quantity difference correction methods 1 and 2 proposed by the applicant, the light quantity difference correction can be performed without degrading the image quality in a plain pattern as shown in FIG.

  However, the light amount difference correction method 1 has a problem that a block boundary appears in the output image (similar to block distortion known in image compression technology) in order to perform light amount difference correction for each block. . Further, since the light amount difference correction is performed based on the three light amounts of the minimum value, the average value, and the maximum value, the influence of the error of these three parameters is large. For example, when the maximum value of the second original image signal becomes larger than the original light amount due to the influence of noise or the like, the second original image signal after the light amount difference correction becomes darker than the first original image signal. The checkerboard pattern will remain and will not disappear.

  Further, in the light amount difference correction method 2, there arises a problem that a false edge occurs at the edge portion and the edges are doubled, or the light amount difference correction in the peripheral portion of the edge is not sufficiently performed.

  Further, the light amount difference correction method 3 in which a sensor for detecting the light amount is separately provided performs light amount difference correction uniformly on the entire original image. For this reason, when the light amount difference varies depending on areas such as the optical axis portion and the peripheral portion of the original image, the light amount difference correction cannot be performed correctly. In particular, when the exposure time is controlled using a mechanical shutter, the exposure time around the optical axis is longer than the exposure time around the periphery, and the image quality is significantly degraded.

  The present invention has been made in order to solve the above-described problem. A light amount difference is generated between two original images, and even when the light amount is partially different from the original image, the output image is corrected by the light amount difference correction. It is an object of the present invention to provide a light amount difference correction method capable of preventing image quality deterioration.

  In the light amount difference correction method according to the present invention, image light from a subject is imaged, monochrome image data is generated by the imaging unit, the imaging unit is moved to a plurality of relative positions with respect to the recording image light, Generating image data in the imaging unit at a relative position, combining a plurality of image data generated by the imaging unit, and removing a spatial frequency component generated based on a light amount difference between the plurality of image data; It is a feature.

  In the light amount difference correction method according to the present invention, colored image light from a subject is imaged, u types of image data per image light are generated by an imaging unit, and v types of image capturing units with respect to the image light. (V ≧ 2) is moved to a relative position, image data is generated in the imaging means at each relative position, and u × v pieces of image data generated by the imaging means are combined for each type of image data, A feature is that a spatial frequency component generated based on a light amount difference between image data at v relative positions is removed.

  In the light amount difference correction method, colored image light from the subject is imaged through a color filter, u is 3, v is 2, and a basic array pattern of a transmission region of the color filter is It is preferable to consist of a Bayer array.

  In the light amount difference correction method, pixel data that cannot have data by combining the image data is obtained by interpolation and added, and after the image data is generated, the image data before being obtained and added by interpolation Thus, it is preferable to remove the spatial frequency component.

  In the light quantity difference correction method, pixel data that cannot have data by combining the image data is obtained by interpolation and the pixel data is obtained by interpolation and the image data after addition is added to the image data. It is preferable to remove the spatial frequency component.

  In the light amount difference correction method, the image pickup means is moved with respect to the image light by a first position, a half of a horizontal pixel interval in the horizontal direction, and a half of a vertical pixel interval in the vertical direction. It is preferable to move relative to the second position.

  In the light amount difference correction method, the imaging unit is moved relative to the image light to a first position and a second position moved from the first position in the horizontal direction by a horizontal pixel interval. Is preferred.

  In the light amount difference correction method, the image pickup unit is moved relative to the image light to a first position and a second position moved vertically from the first position by a vertical pixel interval. Is preferred.

  In the light amount difference correction method, it is preferable that the spatial frequency component is removed using a two-dimensional filter having a base band of a pixel array of image data obtained by combining the plurality or u types of image data as a pass band.

  In the light amount difference correction method, when the amount of movement for moving the image pickup means to a plurality of relative positions is X pixels (X> 0), 2 × (X−INT (X)) is a period in the movement direction. It is preferable to remove the spatial frequency component using a one-dimensional filter having a spatial frequency as a cutoff frequency and a spatial frequency lower than that as a passband.

  Here, INT (y) represents the maximum integer not exceeding y.

  In the light amount difference correction method, the highest spatial frequency component in the horizontal direction and the highest spatial frequency component in the vertical direction in the base band of the pixel array of the image data obtained by synthesizing the plurality or u types of image data, It is preferable to remove the spatial frequency component using at least an attenuation filter.

  In the light quantity difference correction method, a horizontal one-dimensional low-pass filter that uses a maximum spatial frequency in the horizontal direction of the baseband as a cutoff frequency, and a maximum spatial frequency in the vertical direction of the baseband is used as a cutoff frequency. It is preferable to remove the spatial frequency component using a filter in which a vertical one-dimensional low-pass filter is cascade-connected.

  As described above, in the light amount difference correction method of the present invention, the spatial frequency component generated based on the light amount difference generated between the plurality of original image signals is removed by moving the imaging unit to a plurality of relative positions with respect to the image light. Therefore, a striped pattern based on the light amount difference can be suppressed, and the image quality can be improved.

  Further, by removing the spatial frequency component before the interpolation process after combining a plurality of original images, the processing amount in the light amount difference correction (filtering) can be reduced, and high-speed processing is possible.

  Further, by removing the spatial frequency component after interpolation processing of a plurality of original images, it is possible to suppress the stripe pattern generated by the interpolation process simultaneously with the stripe pattern generated by the light amount difference, and the image quality can be improved. Also, the horizontal or vertical edge can be accurately corrected.

  Further, by setting the image shift direction to an oblique direction, a high-quality image can be obtained with a small number of image shifts.

  Further, by setting the image shift direction to the horizontal direction or the vertical direction, the resolution in the horizontal direction or the vertical direction can be improved.

  Further, the filter is a two-dimensional filter that uses the base band of the pixel array of the image data obtained by combining a plurality of original images as a pass band, or the frequency component of the boundary line of the base band is cut off in the image shift moving direction. By removing the spatial frequency component using a one-dimensional filter having a frequency, a striped pattern based on a light amount difference can be reliably suppressed.

  Further, by removing the spatial frequency component using a filter that cuts (attenuates) at least the highest frequency component in the horizontal and vertical directions of the baseband of the pixel array of the image data obtained by combining a plurality of original images, the most fringe is obtained. It is possible to cut the frequency component of the portion where the pattern is conspicuous and to prevent the frequency component in the base band from being cut more than necessary.

  In addition, a horizontal one-dimensional low-pass filter and a vertical one-dimensional low-pass filter are cascaded with the highest frequency in the horizontal and vertical directions of the baseband of the pixel array of the image data composed of a plurality of original images as the cutoff frequency. By removing the spatial frequency component using a filter, it is possible to cut the frequency component of the most conspicuous stripe pattern, and to suppress cutting the frequency component in the baseband more than necessary, thereby improving the image quality. I can plan. Further, the circuit configuration used for the light quantity difference correction method can be simplified.

  (First Embodiment) FIG. 1 shows a configuration of an image pickup apparatus according to the present embodiment. As shown in the figure, the imaging apparatus according to the present embodiment includes an optical system 1, a solid-state imaging device 2, an A / D converter (Analog to Digital Converter) 3, an image memory 4, an LPF 5, a driving unit 6, and a synchronization signal. The generator 7, the memory controller 8, the signal processor 9, and the recording medium 10 are provided.

  Image light from the subject is condensed by the condenser lens of the optical system 1 and then imaged on the imaging surface of the image sensor 2. A plurality of light receiving regions PD are arranged in a matrix on the imaging surface of the image sensor 2.

  The optical system 1 includes an image shift mechanism in addition to a condensing lens that condenses image light. The image shift mechanism is controlled by the drive unit 6 and shifts the imaging position of the image light on the imaging plane to the first and second imaging positions, respectively, at predetermined time intervals. This operation is referred to as an image shift operation.

  The image sensor 2 captures image light by causing each light receiving area PD to receive image light from the optical system 2 for a predetermined exposure time. When the exposure time elapses, the image sensor 2 derives the light reception data from each light reception region PD as the first or second original image signal to the A / D converter 3 at predetermined time intervals. The first and second original image signals are image signals obtained by imaging the image light when the image light is imaged at the first and second imaging positions, respectively. Each original image signal is composed of received light data corresponding to the amount of light received in each light receiving area PD.

  The A / D converter 3 converts the first and second original image signals, which are analog signals from the image sensor 2, into digital signals and stores them in the image memory 4.

  The synchronization signal generation unit 7 generates a synchronization signal corresponding to the imaging operation of the two original image signals, and supplies it to the drive unit 6 and the memory control unit 8. The drive unit 6 performs an image shift operation using an image shift mechanism in the optical system 1. Thereby, in the image sensor 2, the image light received by each light receiving area PD is shifted from the image light before movement in the image of the subject. The memory control unit 8 (image compositing means in the claims) stores the received light data in the image memory 4 in association with two original image signals having different imaging positions.

  A low-pass filter (hereinafter referred to as “LPF”) 5 (light quantity difference correction means in the claims) inputs an original image signal from the image memory 4, performs light quantity difference correction described later on the original image signal, and then again the image memory 4. To store. At this time, the original image signal before the light amount difference correction in the image memory 4 is overwritten and erased by the original image signal after the light amount difference correction.

  The original image signal after the light amount difference correction stored in the image memory 4 is given to the signal processing unit 9 (interpolating means in claims). The signal processing unit 9 generates image data that cannot be obtained from imaging from two original image signals by interpolation (generates image data of pixels that do not have data when the two original image signals are combined) ), The interpolated image signal is stored in the recording medium 10 as an output image signal of the output image.

  Next, a specific example of the imaging device according to the present embodiment will be described. FIG. 2 is a plan view showing a specific pixel arrangement on the imaging surface of the image sensor 2. In the imaging device 2, N × M light receiving areas PD are arranged in a matrix with arrangement periods PH and PV along the horizontal and vertical directions H and V. The horizontal and vertical directions H and V are orthogonal to each other. Here, a direction passing through PD (1, 1), PD (2, 2), PD (3, 3),... Is referred to as a first oblique direction U1. In addition, a direction passing through PD (1, 3), PD (2, 2), PD (3, 1),... Is referred to as a second oblique direction U2. Furthermore, when the first and second oblique directions U1 and U2 are collectively referred to, and when any one of the oblique directions is arbitrarily selected, it is abbreviated as “oblique direction U”.

In this pixel array, the sampling frequencies fH and fV in the horizontal and vertical directions H and V with respect to the image light are reciprocals of the array periods in the horizontal and vertical directions H and V, and are expressed by the following equations.
fH = 1 / PH (1)
fV = 1 / PV (2)
Further, hereinafter, a group of components arranged on a straight line along the horizontal direction H will be referred to as “rows”. Similarly, a group of components arranged in a straight line along the vertical direction V is referred to as a “column”. In a group of components arranged in a matrix, each row is defined as a first row, a second row,..., An Nth row from the upper side to the lower side of the drawing. In addition, each column is defined as a first column, a second column,. Among these constituent elements, when a single constituent element belonging to the nth row and the mth column is represented, a reference numeral (n, m) is attached together with a reference symbol indicating the constituent element generically. n and m are arbitrary integers of 1 or more and N or M or less. In FIG. 2, the arrangement pattern of the light receiving areas PD is represented by 12 light receiving areas PD (1, 1) to PD (4, 3) in 4 rows and 3 columns. The structure shown in FIG. 2 is periodically repeated in the horizontal and vertical directions H and V on the actual imaging surface of the image sensor 2.

  FIG. 3 is a diagram illustrating a positional relationship between the first and second imaging positions of the image light. With the first image formation position A1 as a reference, the second image formation position B1 is only half the length of the arrangement period PH in the horizontal direction H of the light receiving area PD from the first image formation position A1 and in the vertical direction V. The position is moved by the length of half of the arrangement period PV.

  When the image light is imaged at the first and second imaging positions A1 and B1, the imaging device 2 captures the image light and obtains first and second original image signals.

  FIG. 4 is a diagram illustrating a composite image including the first and second original images indicating the above-described original image. The pixels of the first original image are represented with right-down hatching, and the pixels of the second original image are represented with right-up hatching. The composite image is generated by superimposing the first and second original images so as to match the spatial position at the time of imaging. This compositing operation is performed by using the image memory 4 and determining and storing addresses at which the pixel data is stored so that the pixel data of the first and second original images are alternately stored. That is, the composite image is represented as an image obtained by superimposing the first and second original images so as to be shifted by the same shift length in a direction parallel to and opposite to the shift direction of the image formation position described above.

  In the composite image pixel s corresponding to the pixel of the first original image, the subscripts (n, m) are both n and m odd. In addition, in the composite image pixel s corresponding to the pixel of the second original image, the subscripts (n, m) are both n and m are even. Hereinafter, a pixel having pixel data corresponding to any original image is referred to as an actual pixel.

  As the entire composite image, real pixels having pixels corresponding to one of the first and second original images are arranged in a checkered pattern. There is a virtual pixel having no pixel data between two real pixels adjacent in the horizontal and vertical directions H and V. This virtual pixel does not correspond to the pixels of the first and second original images. In the virtual pixel, subscripts (n, m) of the pixel s are either n or m, and one of them is an even number and the other is an odd number.

  The signal processing unit 9 generates the pixel data of the virtual pixel by interpolating and substituting the pixel data of the surrounding real pixels.

  In this interpolation processing, for example, linear interpolation or cubic convolution interpolation is used. The linear interpolation method is an interpolation method that interpolates pixel data of a virtual pixel as an average value of pixel data of four real pixels around the virtual pixel in a synthesized image, for example. The cubic convolution interpolation method is an interpolation method in which pixel data of virtual pixels is interpolated using pixel data of 16 real pixels around the virtual pixels.

  Further, the signal processing unit 9 performs gamma correction. The electric-photoelectric conversion characteristics of a cathode ray tube (CRT) have non-linearity. For this reason, the received light data corresponding to the received light amount is corrected so that the received light amount of the imaging device is proportional to the emission intensity of the cathode ray tube. This correction is called gamma correction.

  The arrangement period of the output image is halved in both horizontal and vertical directions compared to the pixel arrangement of the first and second original images. Therefore, if the resolution of the first and second original images is the standard resolution, the resolution of the output image is higher than the standard resolution in each direction H and V. Further, the arrangement period of the pixels in the oblique direction U of the output image is half of the arrangement period of the composite image. However, the output image signal is obtained by interpolating the composite image signal. At this time, even if the pixel arrangement period of the output image becomes smaller than that of the synthesized image due to the pixels increased by the interpolation processing, the resolution is not different from that of the synthesized image. Therefore, the resolution in the oblique direction U of the output image is lower than the resolution that should be obtained from the pixel arrangement period.

  Next, the LPF 5 that performs light amount difference correction, which is a characteristic part of the present invention, will be described. The LPF 5 applies a low-pass filter to a composite image composed of actual pixels in order to remove a checkerboard pattern that occurs when there is a light amount difference between two original images.

FIG. 5 is a diagram showing the low-pass characteristics of the LPF 5 and the base band of the composite image on the spatial frequency coordinates. The base band 11 is a rectangular area (hatching area in the figure) having the following four points as vertices.
(FH, 0)
(0, fV)
(-FH, 0)
(0, -fV)
Here, for example, (fH, 0) represents a spatial frequency which is a spatial frequency fH in the horizontal direction and a spatial frequency 0 in the vertical direction. A checkered frequency component generated when there is a light quantity difference between two original images has a period twice as large as the shift amount by the image shift, and the direction coincides with the image shift direction. Accordingly, the frequency component exists on the boundary line connecting the point (fH, 0) and the point (0, fV) and the boundary line connecting the point (−fH, 0) and the point (0, −fV) in FIG. For this reason, it is desirable for the LPF 5 to have zero gain on this boundary line, and it is desirable to use the base band 11 as a pass band. Further, in order not to attenuate the frequency component originally held in the original image (in order not to blur the image), it is desirable that the gain of the LPF 5 is 1 in a region within the rectangular region excluding the boundary line. Further, it is desirable that the LPF 5 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted.

  A low-pass filter that satisfies the above requirements tends to have a very large order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter r1 is raised.

  The low-pass filter r1 is a two-dimensional low-pass filter expressed as a 7-row 7-column matrix. When the element in the p-th row and the q-th column is expressed as r1 (p, q), the calculation for correcting the light amount difference of the composite image by the low-pass filter r1 is calculated by the following equation.

  Here, s (n, m) represents a pixel in the nth row and mth column of the composite image before the light amount difference correction. Further, t (n, m) represents a pixel in the nth row and mth column of the composite image after the light amount difference correction. As shown in FIG. 4, the virtual pixel has no pixel data. In the calculation of Expression (4), the pixel data of the virtual pixel is assumed to be zero. A low-pass filter r2 that is more simplified than the low-pass filter r1 is shown below.

  The low-pass filter r2 is a two-dimensional low-pass filter expressed as a 5 × 5 matrix. The element in the pth row and the qth column is expressed as r2 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r2 is calculated by the following equation.

  The low-pass filter r2 has the advantage of requiring less calculation amount than the low-pass filter r1. However, the low-pass filter r2 has a strong tendency to become smaller than 1 as the gain of the pass band 11 approaches the boundary line. As a result, the pass band characteristics are not as good as those of the low-pass filter r1, and after the light amount difference correction is performed. There is a drawback that the composite image is blurred.

  A low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r3 is a one-dimensional low-pass filter represented as a 4 × 4 matrix. The element in the p-th row and the q-th column is expressed as r3 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r3 is calculated by the following equation.

  The low-pass filter r3 performs low-pass filtering of the composite image in the image shift direction shown in FIG. As described above, the frequency component of the checkered pattern that occurs when there is a light amount difference between the two original images is a spatial frequency having a cycle of twice the directed line segment A1B1 representing the image shift amount shown in FIG. Equal to Fd. The direction of the spatial frequency Fd is equal to the first oblique direction U1, and is a spatial frequency having the following magnitude in this direction.

  Unlike the above-described two-dimensional low-pass filter, the low-pass filter r3 is limited to cut off only on the broken line 12 including the spatial frequency Fd. Even such a one-dimensional low-pass filter r3 for the first oblique direction U1 can sufficiently correct the light amount difference.

  Furthermore, a low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r4 is a one-dimensional low-pass filter represented as a 3 × 3 matrix. The element in the pth row and the qth column is expressed as r4 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r4 is calculated by the following equation.

  The low-pass filter r4 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The low-pass filter r4 has an advantage that the calculation amount is smaller than that of the low-pass filter r3. However, the low-pass filter r4 tends to be smaller than 1 as the gain of the pass band 11 approaches the broken line 12, and as a result, the pass band characteristics are not better than the low-pass filter r3, and the composite image after the light amount difference correction is performed. Has the disadvantage of blurring.

  As described above, according to the imaging apparatus of the present embodiment, the light amount difference correction is performed by the low-pass filter, so that the processing can be performed more easily than in the past, and the low-pass processing is performed for each pixel. A partial light amount difference between the original images can be accurately corrected, and deterioration of the output image can be prevented.

  In the present embodiment, (1) a composite image is generated by combining two original image signals, (2) low-pass filtering is performed on the composite image, and then (3) a signal processing unit 9 However, the processing order is not limited to this, and low-pass filtering may be performed after the virtual pixel image data is obtained (see the fifth embodiment). Further, low-pass filtering and interpolation processing may be performed simultaneously.

  In addition, although an example in which the low-pass filters r1 to r4 are used as the LPF 5 is shown here, the present invention is not limited to this example. For example, as shown in the fifth embodiment, the two opposite base bands 11 shown in FIG. Two horizontal and vertical one-dimensional low-pass filters each having a vertex as a cutoff frequency may be used.

  (Second Embodiment) An image pickup apparatus according to a second embodiment of the present invention will be described below. The imaging device of the present embodiment has a configuration similar to that of the imaging device of the first embodiment. For this reason, the same parts as those of the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted. In the imaging apparatus according to the present embodiment, unlike the first embodiment, image shift is performed in the horizontal direction H.

  FIG. 6 is a diagram illustrating a positional relationship between the first and second imaging positions of the image light. With reference to the first imaging position A2, the second imaging position B2 moves from the first imaging position A2 by a length (PH / 2) that is half the arrangement period PH in the horizontal direction H of the light receiving area PD. Is the position.

  When the image light is imaged at the first and second imaging positions A2 and B2, the image sensor 2 captures the image light and obtains first and second original image signals.

  FIG. 7 is a diagram illustrating a composite image including the first and second original images indicating the above-described original image. The pixels of the first original image are represented with right-down hatching, and the pixels of the second original image are represented with right-up hatching. The composite image is generated by superimposing the first and second original images so as to match the spatial position at the time of imaging. This compositing operation is performed by using the image memory 4 and determining and storing addresses at which the pixel data is stored so that the pixel data of the first and second original images are alternately stored. That is, the composite image is represented as an image obtained by superimposing the first and second original images so as to be shifted by the same shift length in a direction parallel to and opposite to the shift direction of the image formation position described above.

  In the pixel s (n, m) of the composite image corresponding to the pixel of the first original image, m is an odd number. Further, m is an even number in the pixel s (n, m) of the composite image corresponding to the pixel of the second original image.

  As the entire composite image, real pixels having pixels corresponding to one of the first and second original images are arranged in a vertical stripe pattern. In the second embodiment, the pixel configuration of the composite image and the output image is the same.

  The signal processing unit 9 performs gamma correction.

  The arrangement period of the output image is halved in the horizontal direction H as compared with the pixel arrangement of the first and second original images. Therefore, if the resolution of the first and second original images is the standard resolution, the resolution of the output image is higher than the standard resolution in the horizontal direction H.

Here, the LPF 5 in the second embodiment will be described. The LPF 5 applies a low-pass filter to the composite image in order to remove a vertical stripe pattern that occurs when there is a light amount difference between the two original images. FIG. 8 is a diagram showing the base band of the composite image and the low-pass characteristics of the LPF 5 on the spatial frequency coordinates. The base band 13 of the composite image is a rectangular area (hatching area in the figure) having the following four points as vertices.
(FH, fV / 2)
(-FH, fV / 2)
(-FH, -fV / 2)
(FH, -fV / 2)
The frequency components of the vertical stripe pattern generated when there is a light quantity difference between the two original images are the boundary line connecting (fH, fV / 2), (fH, -fV / 2) and (-fH, fV / 2), on the boundary line connecting (−fH, −fV / 2). Accordingly, the LPF 5 to be used preferably has a zero gain on this boundary line, and preferably has the base band 15 as a pass band. Further, it is desirable that the LPF 5 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted. A low-pass filter that satisfies such requirements tends to be very large in order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter r5 is raised.

  The low-pass filter r5 is a two-dimensional low-pass filter expressed as a 4 × 4 matrix. The element in the p-th row and the q-th column is expressed as r5 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r5 is calculated by the following equation.

  Here, s (n, m) represents a pixel in the nth row and mth column of the composite image before the light amount difference correction. Further, t (n, m) represents a pixel in the nth row and mth column of the composite image after the light amount difference correction.

  A low-pass filter r6 simplified more than the low-pass filter r5 is shown below.

  The low-pass filter r6 is a two-dimensional low-pass filter represented as a 3 × 3 matrix. The element in the p-th row and the q-th column is expressed as r6 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r6 is calculated by the following equation.

  The low-pass filter r6 has an advantage that the calculation amount is smaller than that of the low-pass filter r5. However, the low-pass filter r6 has a strong tendency to become smaller than 1 as the gain of the pass band 13 approaches the boundary line. As a result, the pass band characteristics are not better than the low-pass filter r5, and after the light amount difference correction is performed. There is a drawback that the composite image is blurred.

  A low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r7 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r7 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r7 is calculated by the following equation.

  The low-pass filter r7 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The frequency component of the vertical stripe pattern generated when there is a light amount difference between the two original images is equal to the spatial frequency Fc2 having a cycle of twice the directed line segment A2B2 representing the image shift amount shown in FIG. . The direction of the spatial frequency Fc2 is equal to the horizontal direction H, and is a spatial frequency having a magnitude of fH in this direction.

  Unlike the above-described two-dimensional low-pass filter, the low-pass filter r7 is limited to cut off the broken line 14 including the spatial frequency Fc2. Even such a one-dimensional low-pass filter r7 for the horizontal direction H can sufficiently correct the light amount difference.

  A low-pass filter with a further reduced amount of calculation is shown below.

  The low-pass filter r8 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r8 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r8 is calculated by the following equation.

  The low-pass filter r8 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The low-pass filter r8 has an advantage that the calculation amount is smaller than that of the low-pass filter r7. However, the low-pass filter r8 tends to be smaller than 1 as the gain of the passband 13 approaches the broken line 14, and as a result, the passband characteristics are not as good as those of the low-pass filter r7. There is a drawback that the composite image is blurred.

  As described above, according to the imaging apparatus of the present embodiment, the light amount difference correction is performed by the low-pass filter, so that the processing can be performed more easily than in the past, and the low-pass processing is performed for each pixel. A partial light amount difference between the original images can be accurately corrected, and deterioration of the output image can be prevented.

  (Third Embodiment) An image pickup apparatus according to a third embodiment of the present invention will be described below. Since the imaging apparatus according to the present embodiment has a configuration similar to that of the imaging apparatuses according to the first and second embodiments, the same parts are denoted by the same reference numerals and detailed description thereof is omitted. In the imaging apparatus according to the present embodiment, the image shift is performed in the vertical direction V, unlike the first and second embodiments.

  FIG. 9 is a diagram showing a positional relationship between the first and second imaging positions of the image light in the present embodiment. With reference to the first imaging position A3, the second imaging position B3 is a position moved from the first imaging position A3 by a length that is half the arrangement period PV in the vertical direction V of the light receiving area PD.

  When the image light is imaged at the first and second imaging positions A3 and B3, the image sensor 2 captures the image light and obtains first and second original image signals.

  FIG. 10 is a diagram illustrating a composite image including the first and second original images indicating the above-described original image. The pixels of the first original image are represented with right-down hatching, and the pixels of the second original image are represented with right-up hatching. The composite image is generated by superimposing the first and second original images so as to match the spatial position at the time of imaging. This compositing operation is performed by using the image memory 4 and determining and storing addresses at which the pixel data is stored so that the pixel data of the first and second original images are alternately stored. That is, the composite image is represented as an image obtained by superimposing the first and second original images so as to be shifted by the same shift length in a direction parallel to and opposite to the shift direction of the image formation position described above.

  In the synthesized image pixel s (n, m) corresponding to the first original image pixel, n is an odd number. Further, n is an even number in the pixel s (n, m) of the composite image corresponding to the pixel of the second original image.

  As the entire synthesized image, real pixels having pixels corresponding to one of the first and second original images are arranged in horizontal stripes. In the third embodiment, the pixel configuration of the composite image and the output image is the same.

  The signal processing unit 9 performs gamma correction.

  The arrangement period of the output image is halved in the vertical direction V as compared with the pixel arrangement of the first and second original images. Therefore, when the resolution of the first and second original images is the standard resolution, the resolution of the output image is higher than the standard resolution in the vertical direction V.

The LPF 5 in the third embodiment will be described. The LPF 5 applies a low-pass filter to the composite image in order to remove a horizontal stripe pattern that occurs when there is a light amount difference between the two original images. FIG. 11 is a diagram showing the base band of the composite image and the low pass characteristics of the LPF 5 on the spatial frequency coordinates. The base band 15 of the composite image is a rectangular area (hatching area in the figure) having the following four points as vertices.
(FH / 2, fV)
(-FH / 2, fV)
(-FH / 2, -fV)
(FH / 2, -fV)
The frequency component of the horizontal stripe pattern generated when there is a light amount difference between two original images has a period of twice the image shift amount in the image shift direction. Therefore, it exists on the boundary line connecting the points (fH / 2, fV) and (−fH / 2, fV) and on the boundary line connecting the points (−fH / 2, −fV) and (fH / 2, −fV). . Therefore, the LPF 5 desirably has zero gain on this boundary line, and preferably has the base band 15 as a pass band. Further, it is desirable that the LPF 5 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted. A low-pass filter that satisfies such requirements tends to be very large in order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter r9 is raised.

  The low-pass filter r9 is a two-dimensional low-pass filter represented as a 4 × 4 matrix. The element in the p-th row and the q-th column is expressed as r9 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r9 is calculated by the following equation.

  Here, s (n, m) represents a pixel in the nth row and mth column of the composite image before the light amount difference correction. Further, t (n, m) represents a pixel in the nth row and mth column of the composite image after the light amount difference correction.

  A low-pass filter r10 that is more simplified than the low-pass filter r9 is shown below.

  The low-pass filter r10 is a two-dimensional low-pass filter represented as a 3 × 3 matrix. The element in the pth row and the qth column is expressed as r10 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r10 is calculated by the following equation.

  The low-pass filter r10 has an advantage that the calculation amount is smaller than that of the low-pass filter r9. However, the low-pass filter r10 tends to be smaller than 1 as the gain of the passband 15 approaches the boundary, and as a result, the passband characteristics are not as good as those of the low-pass filter r9, and after the light amount difference correction. There is a drawback that the composite image is blurred.

  A low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r11 is a one-dimensional low-pass filter expressed as a column vector. The p-th element is represented as r11 (p). The calculation for correcting the light amount difference of the composite image by the low-pass filter r11 is calculated by the following equation.

  The low-pass filter r11 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The frequency component of the horizontal stripe pattern generated when there is a light amount difference between the two original images is equal to the spatial frequency Fc3 having a cycle of twice the directed line segment A3B3 representing the image shift amount shown in FIG. . The direction of the spatial frequency Fc3 is equal to the vertical direction V, and is a spatial frequency having a magnitude of fV in this direction.

  Unlike the above-described two-dimensional low-pass filter, the low-pass filter r11 is limited to cut off the broken line 16 including the spatial frequency Fc3. Even such a low-pass filter r11 with respect to the vertical direction V can sufficiently correct the light amount difference.

  Furthermore, a low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r12 is a one-dimensional low-pass filter expressed as a column vector. The p-th element is represented as r12 (p). The calculation for correcting the light amount difference of the composite image by the low-pass filter r12 is calculated by the following equation.

  The low-pass filter r12 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The low-pass filter r12 has an advantage that the calculation amount is smaller than that of the low-pass filter r11. However, the low-pass filter r12 has a strong tendency to become smaller than 1 as the gain of the passband 15 approaches the broken line 16, and as a result, the passband characteristics are not as good as those of the low-pass filter r11. There is a drawback that the composite image is blurred.

  Also in the present embodiment, it is possible to correct a light amount difference between two original images by a simple process. Further, since the low-pass filtering is performed for each pixel, even when a light amount difference between images partially occurs, the light amount difference correction can be accurately performed, and deterioration of the image based on the light amount difference can be suppressed. .

  (Fourth Embodiment) Although the imaging devices of the first to third embodiments described above have been described as imaging devices that capture monochrome images, the same applies to the color imaging device that captures color images. Correction can be performed. For example, color imaging devices include single-plate and three-plate imaging devices. In a single-plate imaging device, a color filter that restricts the wavelength of light received by each light receiving region is provided on the light incident side of the imaging device, and the light that has passed through the filter is imaged. In a three-plate type imaging device, imaging light is separated into three primary colors of red, blue, and green by a color separation prism, and each monochrome image light is individually imaged by an imaging device. Accordingly, the three-plate type imaging device is provided with three imaging elements.

  In the LPF 5, the image signal of each single color image light obtained by these methods is handled as a vector quantity signal indicating the hues of the three primary colors of red, green, and blue (in this case, u = 3 in the claims), and each single color image is processed. The optical image signal is individually subjected to low pass filter processing. Alternatively, a luminance signal and two kinds of color difference signals are generated from the color image signal (in this case, u = 3), and the luminance signal and the color difference signal are individually subjected to low-pass filter processing. Thereby, also in a color imaging device, the light quantity difference of an output image can be corrected.

  The data subjected to the low-pass filter processing as described above is recorded on the recording medium 10 after being interpolated by the signal processing unit 9. The data recorded on the recording medium may be individual signals processed individually by the LPF 5 or may be a signal obtained by synthesizing them.

  A specific example of a single plate type color imaging apparatus according to the fourth embodiment of the present invention will be described below. Since the structure of the components of the imaging apparatus of the present embodiment is substantially the same as that of the imaging apparatus of the first embodiment, the same components are denoted by the same reference numerals and description thereof is omitted.

  FIG. 12 is a diagram showing a basic arrangement pattern 17 of the color arrangement of the transmissive area of the color filter. The basic array pattern 17 includes four light-transmitting regions L arranged in 2 rows and 2 columns. The pattern 17 includes only two green light-transmitting regions L and one red and blue light-transmitting regions. In the basic arrangement pattern 17, the light-transmitting regions L (1, 1) and L (2, 2) are green light-transmitting regions L. The translucent area L (1, 2) is a red translucent area L. The translucent area L (2, 1) is the blue translucent area L.

  The image sensor 2 captures image light imaged through the color filter and outputs an original image signal. The arrangement of the pixels of the original image signal and the correspondence between the pixels and the light reception data are equivalent to the arrangement of the light-transmitting regions L of the color filter in FIG. 12, and each corresponding pixel is a single color light reception data. Have

  The image light imaging operation of the imaging device described above is similar to the image light imaging operation of the first embodiment, and the behavior of the optical system 1, the image sensor 2, the A / D converter 3, and the image memory 4 is as follows. It is equal to the first embodiment. At this time, the image shift moves the imaging position of the image light to the first and second imaging positions A4 and B4 that are separated by a distance PH in the horizontal direction H as shown in FIG. In the fourth embodiment, the horizontal resolution is improved by shifting the image in the horizontal direction.

  The signal processing unit 9 first generates a composite image signal from the first and second original image signals. This composite image is composed of M × N corresponding pixels arranged in M rows and N columns. The arrangement periods of the horizontal and vertical pixels H and V of the composite image are periods PH and PV, respectively. Each corresponding pixel has light reception data of two different types of color light.

  FIG. 14 is a diagram illustrating a basic array pattern 18 of an array of equivalent pixels D of the composite image represented by the above-described composite image signal. This basic array pattern 18 is composed of four pixels arranged in 2 rows and 2 columns. Pixels D (1,1) and D (1,2) are corresponding pixels of green and red. Pixels D (2,1) and D (2,2) are corresponding pixels of green and blue. In FIG. 14, the symbol “Fa” is written for the corresponding pixel of green and red, and the symbol “Fb” is marked for the corresponding pixel of green and blue. From this, it can be seen that in the composite image, the pixel arrangement is equal to the pixel arrangement of the original image, and all the pixels are green corresponding pixels. The odd-numbered pixels are red corresponding pixels, and the even-numbered pixels are virtual pixels having no red corresponding pixels. Even-numbered pixels are blue corresponding pixels, and odd-numbered rows are virtual pixels having no blue corresponding pixels. The arrangement of individual composite images for each red, blue, and green is shown in FIGS. FIG. 15 shows a red array. FIG. 16 shows a blue array. FIG. 17 shows a green array. The pixels of the first original image are represented with right-down hatching, and the pixels of the second original image are represented with right-up hatching.

Next, the LPF 5 which is a characteristic part of the present invention will be described. The LPF 5 individually performs low-pass filter processing on each monochromatic image light image signal. FIG. 18 is a diagram showing the characteristics of the baseband and LPF 5 regarding the red or blue pixels of the composite image on the spatial frequency coordinates. The base band 19 is a rectangular area (hatching area in the figure) having the following four points as vertices.
(FH / 2, fV / 4)
(-FH / 2, fV / 4)
(-FH / 2, -fV / 4)
(FH / 2, -fV / 4)
The frequency component of the horizontal stripe pattern generated when there is a light amount difference between the two original images has a cycle that is twice the shift amount of the image shift in the image shift direction. Therefore, the boundary line connecting the point (fH / 2, fV / 4) and the point (fH / 2, −fV / 4) and the point (−fH / 2, fV / 4) and the point (−fH / 2, −fV). / 4) exists on the boundary line. For this reason, the LPF 5 desirably has a zero gain on this boundary line, and preferably has the base band 15 as a pass band. Further, it is desirable that the LPF 5 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted. A low-pass filter that satisfies such requirements tends to be very large in order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter r13 is raised.

  The low-pass filter r13 is a two-dimensional low-pass filter that is represented as a 7 × 4 matrix. The element in the p-th row and the q-th column is expressed as r13 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r13 is calculated by the following equation.

  Here, s (n, m) represents a red or blue pixel in the nth row and mth column of the composite image before the light amount difference correction. As shown in FIGS. 15 and 16, the virtual pixel has no pixel data. In the calculation of Expression (4), the pixel data of the virtual pixel is assumed to be zero. Further, t (n, m) represents a red or blue pixel in the nth row and mth column of the composite image after the light amount difference correction. When s (n, m), which is a virtual pixel, is calculated by Expression (28), the pixel value of the pixel t (n, m) after light amount difference correction is zero. Accordingly, the calculation of virtual pixels can be omitted with respect to equation (28).

  A low-pass filter r14 that is more simplified than the low-pass filter r13 is shown below.

  The low-pass filter r14 is a two-dimensional low-pass filter expressed as a 5 × 3 matrix. The element in the p-th row and the q-th column is expressed as r14 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r14 is calculated by the following equation.

  The low-pass filter r14 has an advantage that the calculation amount is smaller than that of the low-pass filter r13. However, the low-pass filter r14 has a strong tendency to become smaller than 1 as the gain of the passband 19 approaches the boundary, and as a result, the passband characteristics are not as good as those of the low-pass filter r13, and after the light amount difference correction. There is a drawback that the composite image is blurred.

  A low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r15 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r15 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r15 is calculated by the following equation.

  The low-pass filter r15 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The frequency component of the vertical stripe pattern generated when there is a difference in the amount of light between the two original images is equal to the spatial frequency Fc4 whose cycle is twice the directed line segment A4B4 representing the image shift amount shown in FIG. . The direction of the spatial frequency Fc4 is equal to the horizontal direction H, and is a spatial frequency having a magnitude of fH in this direction. Unlike the above-described two-dimensional low-pass filter, the low-pass filter r15 is limited to cut off the broken line 20 including the spatial frequency Fc4. Even such a low-pass filter r15 with respect to the horizontal direction H can sufficiently correct the light amount difference.

  Furthermore, a low-pass filter with a reduced calculation amount is shown below.

  The low-pass filter r16 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r16 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r16 is calculated by the following equation.

  The low-pass filter r16 performs low-pass filtering on the composite image in the image shift direction shown in FIG. The low-pass filter r16 has an advantage that the calculation amount is smaller than that of the low-pass filter r15. However, the low-pass filter r16 tends to be smaller than 1 as the gain of the passband 19 approaches the broken line 20, and as a result, the passband characteristics are not as good as the low-pass filter r15, and after the light amount difference correction. There is a drawback that the composite image is blurred.

Next, the green pixel of the composite image is described. FIG. 19 is a spatial frequency diagram showing the base band of a green pixel. The LPF 5 uses a rectangular area (hatching area in the figure) having the following four points as vertices as a passband 21.
(FH / 2, fV / 2)
(-FH / 2, fV / 2)
(-FH / 2, -fV / 2)
(FH / 2, -fV / 2)
A checkered frequency component generated when there is a light amount difference between two original images has a period twice as large as the image shift amount in the image shift direction. Therefore, the boundary line connecting the point (fH / 2, fV / 2) and the point (fH / 2, −fV / 2) and the point (−fH / 2, fV / 2) and the point (−fH / 2, −fV). / 2) on the boundary line connecting. For this reason, it is desirable for the LPF 5 to have zero gain on this boundary line. Further, it is desirable that the LPF 5 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted. A low-pass filter that satisfies such requirements tends to be very large in order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter r17 is raised.

  The low-pass filter r17 is a two-dimensional low-pass filter represented as a 4 × 4 matrix. The element in the p-th row and the q-th column is expressed as r17 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r17 is calculated by the following equation.

  Here, G (n, m) represents a pixel in the green nth row and mth column of the composite image before the light amount difference correction. Further, t (n, m) represents a green pixel in the nth row and mth column of the composite image after the light amount difference correction.

  A low-pass filter r18 that is more simplified than the low-pass filter r17 is shown below.

  The low-pass filter r18 is a two-dimensional low-pass filter expressed as a 3 × 3 matrix. The element in the p-th row and the q-th column is expressed as r18 (p, q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r18 is calculated by the following equation.

  The low-pass filter r18 has the advantage of requiring less calculation amount than the low-pass filter r17. However, the low-pass filter r18 tends to be smaller than 1 as the gain of the passband 21 approaches the boundary, and as a result, the passband characteristics are not as good as those of the low-pass filter r17, and after the light amount difference correction. There is a drawback that the composite image is blurred.

  A low-pass filter with a further reduced calculation amount is shown below.

  The low-pass filter r19 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r19 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r19 is calculated by the following equation.

  The low-pass filter r19 performs low-pass filtering of the composite image in the image shift direction shown in FIG. A checkered frequency component generated when there is a light quantity difference between two original images is equal to a spatial frequency Fc4 having a cycle of twice the directed line segment A4B4 representing the image shift amount shown in FIG. The direction of the spatial frequency Fc4 is equal to the horizontal direction H, and is a spatial frequency having a magnitude of fH in this direction.

  Unlike the above-described two-dimensional low-pass filter, the low-pass filter r19 is limited to cut off the broken line 22 including the spatial frequency Fc4. Even such a low-pass filter r19 with respect to the horizontal direction H can sufficiently correct the light amount difference.

  A low-pass filter with a further reduced amount of calculation is shown below.

  The low-pass filter r20 is a one-dimensional low-pass filter expressed as a row vector. The q-th element is represented as r20 (q). The calculation for correcting the light amount difference of the composite image by the low-pass filter r8 is calculated by the following equation.

  The low-pass filter r20 performs low-pass filtering of the composite image in the image shift direction shown in FIG. The low-pass filter r20 has an advantage that the calculation amount is smaller than that of the low-pass filter r19. However, the low pass filter r20 has a strong tendency to become smaller than 1 as the gain of the pass band 11 approaches the broken line 22. As a result, the pass band characteristics are not better than the low pass filter r19, and after the light amount difference correction is performed. There is a drawback that the composite image is blurred.

  As described above, the spatial frequency Fc4 generated when there is a light amount difference between the two original images in the fourth embodiment is a horizontal spatial frequency. However, as can be seen from the green pixel array in FIG. 17, the green pixels that occur when there is a difference in the amount of light have a checkered pattern. The low-pass filter that removes the checkerboard pattern may be a vertical one-dimensional low-pass filter. Since the fourth embodiment performs image shift for the purpose of improving the resolution in the horizontal direction, applying a low-pass filter in the horizontal direction is contrary to this purpose. Therefore, the one-dimensional low-pass filter in the vertical direction is more desirable for the fourth embodiment.

  Further, since the low-pass filter is applied for each pixel, the light amount difference correction can be performed in any part of the image and even when the light amount difference is different for each part.

  In the second to fourth embodiments, as in the first embodiment, (1) after two original image signals are combined to generate a combined image, (2) low-pass filtering is performed on the combined image. After that, (3) The signal processing unit 9 obtains the virtual pixel image data by interpolation, but the processing order is not limited to this, and after obtaining the virtual pixel image data (after the interpolation process) Low-pass filtering may be performed (see the fifth embodiment), or low-pass filtering and interpolation processing may be performed simultaneously.

  (Fifth Embodiment) FIG. 20 shows the arrangement of an image pickup apparatus according to this embodiment. As shown in the figure, the imaging apparatus according to the present embodiment includes an optical system 51, a solid-state imaging device 52, an A / D converter 53, an image memory 54, an interpolation processing unit 55, a driving unit 56, and a synchronization signal generating unit 57. A memory control unit 58, an LPF 59, and a recording medium 60.

  Image light from the subject is condensed by the condenser lens of the optical system 51 and then imaged on the imaging surface of the image sensor 52. A plurality of light receiving regions PD are arranged in a matrix on the imaging surface of the image sensor 52.

  The optical system 51 includes an image shift mechanism in addition to a condensing lens that condenses image light. The image shift mechanism is controlled by the drive unit 56 and shifts the imaging position of the image light on the imaging surface to the first and second imaging positions, respectively, at predetermined time intervals.

  The image sensor 52 captures image light by causing each light receiving region PD to receive image light from the optical system 52 for a predetermined exposure time. When the exposure time elapses, the image sensor 52 derives the light reception data from each light reception region PD as the first or second original image signal to the A / D converter 53 at predetermined time intervals.

  The A / D converter 53 converts the first and second original image signals, which are analog signals from the image sensor 52, into digital signals and stores them in the image memory 54.

  The synchronization signal generation unit 57 generates a synchronization signal corresponding to the imaging operation of the two original image signals, and supplies it to the drive unit 56 and the memory control unit 58. The drive unit 56 performs an image shift operation using an image shift mechanism in the optical system 1. As a result, in the image sensor 52, the image light received by each light receiving area PD is shifted from the image light before movement in the subject image. The memory control unit 58 stores the received light data in the image memory 54 in association with two original image signals having different imaging positions.

  The interpolation processing unit 55 receives the original image signal from the image memory 54 and generates image data that cannot be obtained from the two original image signals by interpolation. That is, image data of pixels that do not have data when the two original image signals are combined is generated by interpolation. Then, the interpolated image signal is stored in the image memory 54 again. Note that the original image signal stored in the image memory 4 is subjected to image quality correction by an LPF 59, which will be described later, and is overwritten and erased by the corrected signal.

  The LPF 59 is a low-pass filter for removing a light amount difference between two original image signals and image quality degradation caused by interpolation processing, and receives the original image signal after interpolation stored in the original image signal image memory 54. Then, image quality correction (light quantity difference correction and interpolation correction) is performed. The signal after the image quality correction is stored in the recording medium 60 as an output image signal.

  Next, a specific example of the imaging device according to the present embodiment will be described. Here, the structure shown in FIG. 2 in the pixel array on the imaging surface of the image sensor 52 is periodically repeated in the horizontal and vertical directions H and V, and the first and second connections of the image light. Assume that the positional relationship between the image positions is as shown in FIG. Here, with reference to the first image formation position A1, the second image formation position B1 is vertical from the first image formation position A1 by a length that is half the arrangement period PH in the horizontal direction H of the light receiving region PD. This is the position moved by half the length of the arrangement period PV in the direction V.

  When image light is imaged at the first and second imaging positions A1 and B1, the image sensor 52 captures the image light and obtains first and second original image signals. FIG. 4 is a diagram showing these synthesized images. The pixels of the first original image are represented by right-down hatching, and the pixels of the second original image are represented by right-up hatching. The operation here is the same as that of the imaging apparatus of the first embodiment shown in FIG.

  The interpolation processing unit 55 interpolates pixel data of pixels (virtual pixels) having no data in the synthesized image from pixel data of surrounding real pixels, and substitutes the pixel data of the virtual pixels for generation.

  In this interpolation processing, for example, linear interpolation or cubic convolution interpolation is used. The linear interpolation method is an interpolation method that interpolates pixel data of a virtual pixel as an average value of pixel data of four real pixels around the virtual pixel in a synthesized image, for example. The cubic convolution interpolation method is an interpolation method in which pixel data of virtual pixels is interpolated using pixel data of 16 real pixels around the virtual pixels.

  The arrangement period of the interpolated image is halved in both horizontal and vertical directions compared to the pixel arrangement of the first and second original images. Therefore, if the resolution of the first and second original images is the standard resolution, the resolution of the output image is higher than the standard resolution in each direction H and V. Further, the arrangement period of the pixels in the oblique direction U of the output image is half of the arrangement period of the composite image. However, the output image signal is obtained by interpolating the composite image signal. At this time, even if the pixel arrangement period of the output image becomes smaller than that of the synthesized image due to the pixels increased by the interpolation processing, the resolution is not different from that of the synthesized image. Therefore, the resolution in the oblique direction U of the output image is lower than the resolution that should be obtained from the pixel arrangement period.

  Next, the LPF 59 that performs light amount difference correction, which is a characteristic part of the present embodiment, will be described. The LPF 59 performs low-pass filter processing on the output image after the interpolation in order to remove the checkered pattern caused by the light amount difference between the two original images and the checkered pattern generated by the interpolation process.

FIG. 5 is a diagram showing the low-pass characteristics of the LPF 59 and the base band of the composite image on the spatial frequency coordinates. The base band 11 is a rectangular area (hatching area in the figure) having the following four points as vertices.
(FH, 0)
(0, fV)
(-FH, 0)
(0, -fV)
Here, for example, (fH, 0) represents a spatial frequency which is a spatial frequency fH in the horizontal direction and a spatial frequency 0 in the vertical direction.

  A checkered frequency component generated when there is a light quantity difference between two original images has a period twice as large as the shift amount by the image shift, and the direction coincides with the image shift direction. This will be described with reference to FIG. In FIG. 4, considering the image shift direction passing through the pixel s (1,1), s (2,2), s (3,3), s (4,4), s (5,5), s ( 6, 6) in a diagonal pixel row. Among these, the pixel s of the composite image corresponding to the pixel of the first original image is s (1,1), s (3,3), s (5,5). Also, the pixel s of the composite image corresponding to the pixel of the second original image is s (2,2), s (4,4), s (6,6). That is, the pixels of the first original image and the pixels of the second original image are alternately arranged. When a light amount difference occurs between two original images, a spatial frequency of a two-pixel cycle is generated in the diagonal pixel row. The spatial frequency of this two-pixel cycle is equal to the first oblique direction U1, and is the spatial frequency of the following magnitude in this direction.

  As described above, when considered in a one-dimensional space with respect to the first oblique direction U1, the spatial frequency generated when there is a light amount difference between the two original images is equal to Fd in FIG. Fd is expressed as (fH / 2, fV / 2) and (−fH / 2, −fV / 2) when expressed as a two-dimensional spatial frequency. The frequency component Fd exists on the boundary line connecting the point (fH, 0) and the point (0, fV) and the boundary line connecting the point (−fH, 0) and the point (0, −fV) in FIG. .

  Also, the checkered frequency component generated when there is a difference in the amount of light between the two original images is equal to Fc in FIG. 5 when considered in the horizontal and vertical directions. That is, the spatial frequencies are (fH, 0), (0, fV), (−fH, 0), and (0, −fV). This will be described with reference to FIGS. FIG. 21 is a diagram illustrating a composite image showing a spatial frequency of (fH, 0) or (−fH, 0). The spatial frequency of (fH, 0) or (−fH, 0) is a vertical stripe with a period PH in the horizontal direction. In FIG. 21, black pixels and white pixels are alternately displayed as vertical stripes. FIG. 22 is a diagram illustrating a composite image indicating a spatial frequency of (0, fV) or (0, −fV). The spatial frequency of (0, fV) or (0, −fV) is a horizontal stripe with a period PV in the vertical direction. In FIG. 22, it is represented as horizontal stripes in which rows of black pixels and rows of white pixels are alternately arranged. In both FIG. 21 and FIG. 22, black pixels include all pixels of the first original image, and white pixels include all pixels of the second original image. Therefore, the spatial frequency generated when there is a light amount difference between two original images includes Fc. Further, the frequency component Fc is on the boundary line connecting the point (fH, 0) and the point (0, fV) and the boundary line connecting the point (−fH, 0) and the point (0, −fV) in FIG. 5. Exists.

  The LPF 59 cuts the spatial frequency as described above. When configured as a one-dimensional filter, the one-dimensional filter direction of the LPF 59 is equal to the first diagonal direction U1 and the gain on the frequency Fd is zero. It is desirable that the base band 11 be a pass band. In this case, the LPF 59 cuts off only on the broken line 12 including the spatial frequency Fd. Further, in order not to attenuate the frequency component originally held in the original image (in order not to blur the image), it is desirable that the gain of the LPF 59 is 1 in the region within the rectangular region excluding the boundary line. Further, it is desirable that the LPF 59 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted.

  When the LPF 59 is configured as a two-dimensional filter, the gain on the boundary line 12 is preferably zero, and the base band 11 is preferably a pass band. Further, in order not to attenuate the frequency component originally held in the original image (in order not to blur the image), it is desirable that the gain of the LPF 59 is 1 in the region within the rectangular region excluding the boundary line. Further, it is desirable that the LPF 59 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted.

  As such a one-dimensional filter or two-dimensional filter, the low-pass filters (r1 to r4) described in the first embodiment can be used. Here, the following low-pass filters will be described.

  The most prominent spatial frequency component causing the stripe pattern is Fc which is the maximum absolute value frequency component in the horizontal and vertical directions in the base band 11. Therefore, when the filter is configured as a two-dimensional filter, the stripe pattern becomes inconspicuous if a low-pass filter that makes only the gain on the frequency component Fc zero is used. The low-pass characteristics of such a low-pass filter will be described with reference to FIG. In the low-pass filter, it is desirable that the gain on the broken lines 63 and 64 in FIG. 23 is zero, and a region 65 (in the drawing, a hatching region that falls to the right, including the baseband 11) is surrounded by the broken lines 63 and 64. A pass band is desirable. Further, in order not to attenuate the frequency component originally held in the original image (in order not to blur the image), it is desirable that the gain of the LPF 59 is 1 in the region within the rectangular region excluding the boundary line. Further, it is desirable that the LPF 59 has a linear phase characteristic so that the image after the low pass, that is, the composite image after the light amount difference correction is not distorted.

  A low-pass filter that satisfies the above requirements tends to have a very large order. A high-order low-pass filter is not practical because it takes time to process and the circuit scale increases. Accordingly, an appropriate order is set in consideration of the conditions to be satisfied by the above-described low-pass filter, processing time, and circuit scale. For example, the following low-pass filter m1 is appropriate.

  The low-pass filter m1 is a two-dimensional low-pass filter expressed as a 4 × 4 matrix. When the element in the p-th row and the q-th column is expressed as m1 (p, q), the calculation for correcting the light amount difference of the composite image by the low-pass filter m1 is calculated by the following equation.

  Here, s (n, m) represents a pixel in the nth row and mth column of the composite image before the light amount difference correction. Further, t (n, m) represents a pixel in the nth row and mth column of the composite image after the light amount difference correction. As shown in FIG. 4, the virtual pixel has no pixel data. Therefore, before the calculation of the low-pass filter m1, the interpolation processing unit 55 obtains the pixel of the virtual pixel by interpolation.

  A low-pass filter m2 simplified from the low-pass filter m1 is shown below.

  The low-pass filter m2 is a two-dimensional low-pass filter represented as a 3 × 3 matrix. When the element in the p-th row and the q-th column is expressed as m2 (p, q), the calculation for correcting the light amount difference of the composite image by the low-pass filter m2 is calculated by the following equation.

  As shown in FIG. 4, the virtual pixel has no pixel data. Therefore, before calculating the low-pass filter m2, the interpolation processing unit 55 obtains the pixel of the virtual pixel by interpolation.

  The low-pass filter m2 has an advantage that the calculation amount is smaller than that of the low-pass filter m1. However, the low-pass filter m2 has a strong tendency to become smaller than 1 as the gain of the pass band 15 approaches the boundary line. As a result, the pass band characteristics are not better than the low-pass filter m1, and after the light amount difference correction is performed. There is a drawback that the composite image is blurred.

  The two-dimensional low-pass filter m1 can be decomposed into a horizontal one-dimensional low-pass filter (hereinafter referred to as horizontal low-pass filter) mh1 and a vertical one-dimensional low-pass filter (hereinafter referred to as vertical low-pass filter) mv1. mh1 is a one-dimensional low-pass filter represented as a row vector. Mv1 is a one-dimensional low-pass filter expressed as a column vector.

  The calculation for correcting the light amount difference of the composite image by the horizontal low-pass filter mh1 and the vertical low-pass filter mv1 is calculated by the following equation.

  Thus, since the two-dimensional low-pass filter m1 can be configured by cascading the horizontal low-pass filter mh1 and the vertical low-pass filter mv1, the circuit configuration can be simplified.

  FIG. 24 is a block diagram showing this configuration example, in which a vertical low-pass filter mv1 and a horizontal LPF mh1 are connected in cascade. By inputting non-interlaced pixel data of the composite image to the cascade connection circuit, pixel data subjected to two-dimensional low-pass filter processing can be output.

  FIG. 25 illustrates a circuit configuration example of the vertical low-pass filter mv1, and FIG. 26 illustrates a circuit configuration example of the horizontal low-pass filter mh1. Both circuits can be calculated with three adders and one 5x multiplier. Note that a 1/8 divider can be calculated by 3-bit shift. The 5-times multiplier can also be constituted by a bit shift and an adder (see FIG. 27).

  On the other hand, FIG. 28 shows a block diagram in which the two-dimensional low-pass filter m1 is configured as a circuit as it is. In this figure, D represents a one-pixel delay element, and a broken-line block represents one horizontal delay line. Therefore, the number of delay pixels in the delay line in the broken line block is 3 pixels less than the number of horizontal pixels in the output image. In the case of this circuit, it can be calculated by 15 adders and 2 5 times multipliers. It can be seen that a considerable number of adders are required compared to the cascade connection type.

  The two-dimensional low-pass filter m2 can be decomposed into a horizontal low-pass filter mh2 and a vertical low-pass filter mv2. mh2 is a one-dimensional low-pass filter expressed as a row vector. Mv2 is a one-dimensional low-pass filter represented as a column vector.

  The calculation for correcting the light amount difference of the composite image by the horizontal low-pass filter mh2 and the vertical low-pass filter mv2 is calculated by the following equation.

  Thus, since the two-dimensional low-pass filter m2 can be configured by connecting the horizontal low-pass filter mh2 and the vertical low-pass filter mv2 in cascade, the circuit configuration can be simplified.

  Further, according to the fifth embodiment, the horizontal edge or the vertical edge included in the image can be accurately corrected. FIG. 29 shows an example in which an edge in the vertical direction is imaged. This is the case where there is a vertical edge between q = 3 and 4, where q ≦ 3 is the black level and q ≧ 4 is the white level in the pixel s (p, q). In the first original image, it is assumed that the black level is 10 and the white level is 200. That is, s (p, 1) = 10, s (p, 3) = 10, and s (p, 5) = 200. In the second original image, it is assumed that the black level is 0 and the white level is 190 because a light amount difference is generated between the two original images. That is, it is assumed that s (p, 2) = 0, s (p, 4) = 190, and s (p, 6) = 190. FIG. 30 shows the pixel value of each pixel and the value obtained by calculating the pixel data of the virtual pixel by the linear interpolation method. What was originally an edge on a straight line in the vertical direction is a zigzag edge. When the light amount difference correction process based on the equation (49) is performed on this image, the result is as shown in FIG. A zigzag edge can be corrected to a vertical edge. Similarly, the horizontal edge can be corrected accurately.

  In addition, according to the present embodiment, since the low-pass filter process is performed after the interpolation process, it is possible to simultaneously correct the image quality degradation caused by the interpolation process, that is, the occurrence of the image unevenness of the specific frequency component. This is because the frequency component of image unevenness generated by the interpolation processing usually matches the frequency component based on the light amount difference at the time of image shift.

  By the way, in a filter such as a low-pass filter, it is difficult to create a filter having ideal characteristics in which the gain at a frequency component to be cut is 0 and the gain of other frequency components is 1. Therefore, normally, if the gain of the frequency component to be cut is set to 0, the gain in the vicinity of the frequency component will be lower than 1. For this reason, when cutting (attenuating) the frequency component corresponding to the stripe pattern generated due to the light amount difference as in the present invention, it is necessary to attenuate a part of the frequency component in the baseband. In the embodiment, the frequency component to be cut by the filter is set to the horizontal and vertical vertices of the baseband expressed in the spatial frequency coordinates (the frequency component having the maximum absolute value on the horizontal axis and the vertical axis). The frequency components that need to be attenuated can be reduced as compared with the case of cutting the baseband boundary as in the fourth embodiment. Therefore, it is possible to suppress deterioration in image quality.

  In the fifth embodiment, (1) after synthesizing two original image signals to generate a synthesized image, (2) the interpolation processing unit 55 interpolates the virtual pixel image data for the synthesized image. (3) The LPF 59 performs low-pass filtering on the interpolated image. However, the processing order is not limited to this, and after performing low-pass filtering on the synthesized image, the interpolation processing is performed. May be. Therefore, the LPF 59 (m1, m2) of this embodiment may be used as the LPF 5 having the configuration of FIG. 1 described in the first embodiment. In this case, the LPF 59 sets the pixel data of the virtual pixel to zero. What is necessary is to perform low-pass filtering. Further, low-pass filtering and interpolation processing may be performed simultaneously.

  In the above-described fifth embodiment, the imaging apparatus that captures a monochrome image has been described. However, the light amount difference correction can be performed similarly for a color imaging apparatus that captures a color image.

  For example, color imaging devices include single-plate and three-plate imaging devices. In a single-plate imaging device, a color filter that restricts the wavelength of light received by each light receiving region is provided on the light incident side of the imaging device, and the light that has passed through the filter is imaged. In a three-plate type imaging device, imaging light is separated into three primary colors of red, blue, and green by a color separation prism, and each monochrome image light is individually imaged by an imaging device. Accordingly, the three-plate type imaging device is provided with three imaging elements.

  In the LPF 59, the image signal of each single color image light obtained by these methods is handled as a vector amount signal indicating the color tone of each of the three primary colors of red, blue, and green (in this case, u = 3 in the claims), and each single color The image light image signal is individually subjected to low pass filter processing. Alternatively, a luminance signal and two kinds of color difference signals are generated from the color image signal (in this case, u = 3), and the luminance signal and the color difference signal are individually subjected to low-pass filter processing. Thereby, also in a color imaging device, the light quantity difference of an output image can be corrected.

  Data subjected to the low-pass filter processing as described above is recorded on the recording medium 60. The data recorded on the recording medium may be individual signals individually processed by the LPF 59, or may be a signal obtained by synthesizing them.

  In the first to fifth embodiments described above, the shift amount does not exceed one pixel in the horizontal direction or the vertical direction due to the image shift. However, when the shift amount exceeds one pixel, The frequency component of the pattern that appears due to the difference in the amount of light has a cycle that is twice the component of the image shift amount less than one pixel. That is, when the image shift amount is X, 2 × (X−INT (X)) is used as a cycle (where INT (y) is the maximum integer not exceeding y). For this reason, it is necessary to select a low-pass filter that can cut the frequency component.

  In the first to fifth embodiments, one composite image is generated from two original images. However, the present invention can also be applied to a case where one composite image is generated from more original images. In that case, it is necessary to select a low-pass filter capable of canceling the light amount difference in consideration of the arrangement of the original images.

  Furthermore, a band pass filter can be used instead of the low pass filter in the first to fifth embodiments.

  INDUSTRIAL APPLICABILITY The present invention can be used for a light amount difference correction method capable of obtaining a high-resolution image. In particular, a plurality of images are picked up by moving image light from a subject relative to an image pickup means, and the images are combined. Thus, it can be used in a light quantity difference correction method for obtaining one high-resolution image.

It is a schematic block diagram which shows the example of 1 structure of the imaging device which concerns on 1st Embodiment. It is a top view which shows the specific pixel arrangement | sequence of the image formation surface of an image pick-up element. It is a figure which shows the positional relationship of the 1st and 2nd image formation position of the image light of 1st Embodiment. It is a figure which shows the synthesized image comprised from the 1st and 2nd original image of 1st Embodiment. It is a spatial frequency figure which shows the base band of the synthesized image of 1st Embodiment. It is a figure which shows the positional relationship of the 1st and 2nd image formation position of 2nd Embodiment. It is a figure which shows the synthesized image comprised from the 1st and 2nd original image of 2nd Embodiment. It is a spatial frequency diagram which shows the base band of the synthesized image of 2nd Embodiment. It is a figure which shows the positional relationship of the 1st and 2nd image formation position of 3rd Embodiment. It is a figure which shows the synthesized image comprised from the 1st and 2nd original image of 3rd Embodiment. It is a spatial frequency diagram which shows the base band of the synthesized image of 3rd Embodiment. It is a figure which shows the basic sequence pattern of the color sequence of the permeation | transmission area | region of the color filter of 4th Embodiment. It is a figure which shows the positional relationship of the 1st and 2nd image formation position of 4th Embodiment. It is a figure which shows the basic arrangement pattern 18 of the arrangement | sequence of the equivalent pixel D of the synthesized image which the synthesized image signal of 4th Embodiment represents. It is a figure which shows the arrangement | sequence of the red pixel of the synthesized image of 4th Embodiment. It is a figure which shows the arrangement | sequence of the blue pixel of the synthesized image of 4th Embodiment. It is a figure which shows the arrangement | sequence of the green pixel of the synthesized image of 4th Embodiment. It is a spatial frequency diagram which shows the baseband in the case of the arrangement | sequence of red or blue of 4th Embodiment. It is a spatial frequency figure which shows the baseband in the case of the green arrangement | sequence of 4th Embodiment. It is a schematic block diagram which shows the example of 1 structure of 5th Embodiment. It is a spatial frequency diagram explaining the vertical stripe produced by the light quantity difference. It is a spatial frequency diagram explaining the horizontal stripe produced by the light quantity difference. It is a spatial frequency diagram showing the pass band of LPF59. It is a schematic block diagram which shows LPF59 comprised by the cascade connection of 5th Embodiment. It is a schematic block diagram which shows the vertical low-pass filter of 5th Embodiment. It is a schematic block diagram which shows the horizontal low-pass filter of 5th Embodiment. It is a block diagram which shows the 5 times multiplier comprised by the bit shift and adder of 5th Embodiment. It is a block diagram of the two-dimensional low-pass filter of 5th Embodiment. It is a figure which shows the synthesized image at the time of imaging the edge of a perpendicular direction. It is a figure which shows the image after the interpolation at the time of imaging the edge of a perpendicular direction. It is a figure which shows the image after light quantity difference correction | amendment at the time of imaging the edge of a perpendicular direction. It is a figure explaining the mechanism of an image shift. It is a figure explaining generation | occurrence | production of a light quantity difference. It is the histogram of the 1st original image in the conventional light quantity difference correction method. It is the histogram of the 2nd original image in the conventional light quantity difference correction method. It is a figure explaining conversion of the pixel data in the conventional light quantity difference correction method.

Explanation of symbols

1,51 Optical system 2,52 Image sensor 3,53 A / C converter 4,54 Image memory 5,59 LPF
6, 56 Drive unit 7, 57 Synchronization signal generation unit 8, 58 Memory control unit 9 Signal processing unit 10, 60 Recording medium 55 Interpolation processing unit

Claims (12)

Capture image light from the subject, generate monochrome image data by the imaging means,
Moving the imaging means to a plurality of relative positions with respect to the image light;
Generating image data in the imaging means at each relative position;
Combining a plurality of image data generated by the imaging means;
A light amount difference correction method, wherein a spatial frequency component generated based on a light amount difference between the plurality of image data is removed. Imaging colored image light from a subject and generating u types of image data per image light by an imaging means;
The image pickup means is moved to v (v ≧ 2) relative positions with respect to the image light, and image data is generated in the image pickup means at each relative position,
The u × v image data generated by the imaging unit is synthesized for each type of image data,
A light amount difference correction method characterized by removing a spatial frequency component generated based on a light amount difference between image data at the v relative positions. In the light quantity difference correction method according to claim 2,
The colored image light from the subject is imaged through a color filter,
U is 3,
V is 2;
A light quantity difference correction method, wherein a basic array pattern of a transmission region of the color filter is a Bayer array. Pixel data that cannot have data by combining the image data is obtained by interpolation and added,
4. The light quantity difference correction method according to claim 1, wherein after the generation of the image data, the spatial frequency component is removed from the image data obtained by the interpolation before being added. 5. . Pixel data that cannot have data by combining the image data is obtained by interpolation and added,
The light quantity difference correction method according to any one of claims 1 to 3, wherein the spatial frequency component is removed from the image data after the pixel data is obtained by interpolation and added.   Relative to the first position and the second position moved from the first position by a half of the horizontal pixel interval in the horizontal direction and by a half of the vertical pixel interval in the vertical direction with respect to the image light The light quantity difference correction method according to claim 1, wherein the light quantity difference is corrected. In the light quantity difference correction method according to claim 3,
The light quantity difference correction characterized by moving the imaging means relative to the image light to a first position and a second position moved horizontally from the first position by a horizontal pixel interval. Method. In the light quantity difference correction method according to claim 3,
The light quantity difference correction characterized in that the image pickup means is moved relative to the image light to a first position and a second position moved vertically from the first position by a vertical pixel interval. Method.   9. The spatial frequency component is removed by using a two-dimensional filter whose pass band is a base band of a pixel array of image data obtained by synthesizing the plurality or u types of image data. The light quantity difference correction method according to claim 1.   When the amount of movement for moving the imaging means to a plurality of relative positions is set to X pixels (X> 0), the spatial frequency having a cycle of 2 × (X−INT (X)) in the moving direction is cut off. 9. The light quantity difference correction method according to claim 1, wherein the spatial frequency component is removed by using a one-dimensional filter having a frequency and a lower spatial frequency as a pass band.   Using a filter that at least attenuates the highest spatial frequency component in the horizontal direction and the highest spatial frequency component in the vertical direction in the base band of the pixel array of the image data obtained by synthesizing the plurality or u types of image data. 9. The light quantity difference correction method according to claim 1, wherein the spatial frequency component is removed.   A horizontal one-dimensional low-pass filter whose cutoff frequency is the highest spatial frequency in the horizontal direction of the baseband, and a vertical one-dimensional low-pass filter whose cutoff frequency is the highest spatial frequency in the vertical direction of the baseband The light quantity difference correction method according to claim 11, wherein the spatial frequency component is removed using a filter formed by cascade connection.

Images