JP2001169186A - Image pickup system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup system, capable of obtaining the image of resolution higher than that of an image pickup element. SOLUTION: After passing through an optical filter 110, an inputted image is converted into an electrical signal by an image sensor 120 and stored in a storage device 130 as a digital signal. The signal, temporarily stored in the device 130, is shifted to a signal processor 200. The characteristic of the optical filter is changed, and image pickup is executed plural number of times to realize extraction of aliasing components. In order to extract the aliasing components, digital signal processing is used. In the signal processing, the correction of the characteristic of the optical filter, Fourier transformation for transforming a photographed image to a signal in a frequency domain and inverse Fourier transformation for performing the reverse operation are executed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup device for an image, and more particularly to an image pickup device for improving the resolution of an image pickup device such as a CCD by image processing.

[0002]

TECHNICAL BACKGROUND CCDs (Charge Coupled Devices) have become commercially successful and their pixel count is now on the order of megapixels. With the recent trend toward multimedia, an imaging device with higher resolution has been demanded. Up to now, high resolution of the image sensor has been realized mainly by miniaturization of the pixel size. However, when an image is captured by an image sensor such as a CCD, the number of accumulated charges decreases due to a decrease in the pixel size, and a problem of deterioration in image quality occurs. Resolution and image quality have a trade-off relationship. Further, as the pixel size becomes smaller, the optical system requires higher accuracy. As described above, there is a limit to increasing the resolution of an image sensor by reducing the pixel size. Therefore, unlike the conventional high resolution,
There is an urgent need to develop entirely new methods.

[0003]

SUMMARY OF THE INVENTION An object of the present invention is to provide an imaging system capable of obtaining an image having a higher resolution than the resolution of an imaging device.

[0004]

To achieve the above object, an imaging system according to the present invention comprises a plurality of optical filters,
An imaging device using an imaging element that obtains an image for each of the plurality of optical filters, and an image processing device that processes a plurality of images from the imaging device, the image processing device is an image for each of the plurality of optical filters , Into a frequency domain, a synthesizing unit for synthesizing the converted image, and an inverse transforming unit for inversely converting the synthesized image. With this configuration, a spatial frequency equal to or higher than the Nyquist frequency determined by the image sensor can be restored, and an image having a higher resolution than the resolution of the image sensor can be obtained. This conversion means performs a Fourier transform,
The inverse transform means performs Fourier inverse transform. In addition, the imaging device may be one that can simultaneously capture a plurality of images for each optical filter. The plurality of optical filters may include a low-pass filter for obtaining an image at or below the Nyquist frequency determined by at least the imaging device, and a band-pass filter for obtaining an image at or above the Nyquist frequency. The filter can be a plurality of types of low-pass filters. In the case of a low-pass filter, it is possible to include an image without a filter, and the synthesizing unit processes and synthesizes image data for each of a plurality of types of low-pass filters.
The image processing apparatus further includes a position correcting unit that corrects a positional deviation of a plurality of images, so that an image closer to the original image can be obtained.

[0005]

Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 shows the outline of the imaging system of the present invention. This system includes an imaging device 100 and a signal processing device 200. The imaging device 100 includes an optical filter 110, an image sensor 120, and a storage device 130, and the signal processing device 200 includes a digital signal processing circuit 21.
Consists of zero. In FIG. 1, an input image passes through an optical filter 110, is converted into an electrical signal by an image sensor 120, and is converted into a digital signal as a digital signal.
0 is stored. The image temporarily stored in the storage device 130 is transferred to the signal processing device 200 by serial communication, a memory card, or the like, where a high-resolution image is generated by digital signal processing described below.

In the imaging system shown in FIG. 1, imaging is performed a plurality of times by changing the characteristics of the optical filter. This number is increased when the resolution of the image is greatly increased. In each of the images captured in this way, as an aliasing, a signal component having a Nyquist frequency (ie, の of the sampling frequency) or more determined by a cell pitch of the image sensor (ie, a sampling frequency) is a component having a Nyquist frequency or less. It is folded back. After the aliasing, the aliasing component cannot be distinguished from the frequency component originally existing below the Nyquist frequency. In the present invention, this aliasing component can be extracted by capturing an image after the optical filter. Digital signal processing is used to extract the aliasing component. In the signal processing, characteristics of an optical filter are corrected, Fourier transform for changing a captured image into a frequency domain signal, and Fourier inverse transform for performing the inverse are performed. Although the device for taking an image and the device for performing digital signal processing can be integrated, it is advantageous to separate them as shown in FIG. By using another device as described above, image capturing can be reduced in size and power consumption, and is suitable for mobile use. Especially, the resolution of the captured image is low unless signal processing is performed,
This image can be used for preview. This eliminates the need to convert the image to a lower resolution for preview. The image stored in the memory device is transferred to a terminal for digital signal processing, and after the signal processing, a high-resolution image is obtained. Hereinafter, image processing performed by digital signal processing will be described in detail.

<Principle for increasing the resolution of a one-dimensional image>
FIG. 2 shows the principle of high-frequency component restoration of a two-dimensional image. FIG.
The original image shown in (a), includes a large spatial frequency components than the Nyquist frequency (f _n) is 1/2 of the sampling frequency determined by the photodiode cell pitch of the image sensor (f _s). First, the input image is two optical filters (LPF: low-pass filter, BPF: band pass filter), the spatial frequency components of the following bandwidth Nyquist frequency (0~f _n, f _n ~
f _s ) (FIG. 2B).
These divided images are independently sampled at spatial sampling frequency f _s of the image sensor. After sampling, the frequency components lower than the Nyquist frequency shown in the lower part of FIG. 2B do not change as shown in the lower part of FIG. 2C. On the other hand, the signal in the upper part of FIG. 2B having a spatial frequency component higher than the Nyquist frequency is folded back to the spatial frequency lower than the Nyquist frequency as shown in the upper part of FIG.
At this time, a low frequency component (0 to f _n ) is generated by the BPF.
Has been removed, so that signal overlap does not occur even if high frequency components are turned back. The two sampled images are converted into frequency components by FFT (Fast Fourier Transform). The frequency of the signal converted into the frequency component can be changed by moving its position. Based on this principle, the signal folded back to the lower frequency in the upper part of FIG. 2C is moved to the original frequency higher than the Nyquist frequency, and is combined with the frequency component lower than the Nyquist frequency in the lower part of FIG. FIG. 2 (d)). Finally, these restored frequency components are transformed into spatial domain signals by IFFT (Inverse Fast Fourier Transform).

<Principle for increasing the resolution of a two-dimensional image> The principle described above is extended to apply it to a two-dimensional image. Equation (1) shows the relationship between the original signal and the aliasing.

(Equation 1) C _mn is a two-dimensional image before sampling, and C ′ _mn
Indicates the Fourier coefficients when the two-dimensional image after sampling is developed into Fourier series. P is the number of horizontal pixels and Q is the number of vertical pixels. m and n are integers, each horizontal,
Corresponds to the vertical frequency. Here, C _mn is m <−P
/ 2, P / 2 ≦ m , n < case of 0 -Q / 2, Q / 2 ≦ n, C mn = C'mn is satisfied. C _mn is m <−P / 2, P
/ 2 ≦ m, n <−Q / 2, Q / 2 ≦ n and not 0,
The components are all C ' _mn (provided that -P / 2 ≦ m <P /
2, an area satisfying −Q / 2 ≦ n <Q / 2).
From equation (1), the relationship between the original signal and the aliasing component is shown in FIG. FIG. 3 shows, for each frequency region, how an image having a frequency component twice as high as the Nyquist frequency is used as an original image, and when a component higher than the Nyquist frequency is sampled, the image is folded back into the Nyquist frequency. I have.

FIG. 3 shows that when an image is sampled after being band-limited by a BPF or LPF, FIG.
It can be seen that the areas B _x , C _x , and D _{x in} (c) to (e) are folded back to A _x independently and without overlap.
In this way, frequency components higher than the Nyquist frequency may be removed from the region A _x is a lower frequency as aliasing. An image sampled for each frequency domain using a filter is combined and reconstructed with the frequency components shown in FIG. Finally, an image converted into an image by IFFT and having a frequency component equal to or higher than the Nyquist of the image sensor can be restored without aliasing noise.

<Processing of two-dimensional image using low-pass filter> In addition to the above-described method using BPF and LPF,
Higher resolution can be achieved by using only the PF. This processing will be described below. The occurrence of aliasing A 'is shown in equation (2) as shown in FIG.

(Equation 2) Here, A, B, C, and D are frequency components of the image signal. Aliasing filtering the signal after A'are as the following formula (3), the filter coefficient H ₁ to H ₄ are represented by suffering form in each of the image signal components to D.

(Equation 3) Here, when sampling after passing through four types of low-pass filters, the following relational expression is established.

(Equation 4) Here, the image signal A _{1'~A 4} occurring aliasing
'Filter coefficients H ₁₁ to H ₄₄ are known. Obtaining the image signal components A to D before aliasing is equivalent to solving the relational expression of Expression (4) for the image signal components A to D. As described above, by performing processing for solving equation (4) using, for example, four types of low-pass filters (LPFs), the image signal before aliasing is generated from the image signal having aliasing. Can be requested. At this time, an image that does not use the LPF can be included in the processing target. Note that the filter coefficients for the image signal when no filter is used are all 1 in the coefficients of one row.
It is.

<Image Position Correction Processing> In the system used in the present invention, it is important how to obtain the relative position accuracy of a plurality of captured images. Therefore, a correction process for maintaining the position accuracy will be described below. First, after capturing a plurality of images, the images are converted into frequency components by FFT. F
Due to the characteristics of the FT, the position information is reflected on the phase information of the frequency component. Therefore, a plurality of images after imaging are shifted by signal processing so that the shift of the phase information is minimized. The shift processing for the position correction is performed in units smaller than the pixel. In this way, by using the phase information of the frequency component, the positional accuracy can be improved, and the influence of noise, filters, and the like mixed into the image itself can be suppressed.

<Implementation> The processing program was developed and executed on a personal computer. All programs were written in C ++ language. Original image is 512x
An 8-bit gray scale was used for a digital image composed of 512 pixels. Circular zone plates (CZP) were used for the images. This CZP has a higher spatial frequency according to the distance from the center of 512 × 512 pixels, and becomes equal to the spatial sampling frequency of the image sensor at the end of the image. The imaging resolution of the image sensor is 256 ×
The number of pixels was 256. Since the number of pixels of the image sensor is 1/4 of the original image, the original image is an image having a spatial frequency up to twice the Nyquist frequency in each of the horizontal and vertical directions of the image sensor. The sampling of the input image by the image sensor is performed in this 512 × 512.
This is realized by skipping one pixel in each of the horizontal and vertical directions in the original image of pixels. In this example, a filter process in a frequency domain that realizes an optical filter by simulation is used as the optical filter.
In this process, the attenuation in the passband is zero and does not pass through the transition band,
The ideal filter characteristic in which the amount of attenuation in the cutoff region is infinite is used. By incorporating a function for correcting the filter characteristics into the signal processing mechanism, not only such an ideal filter but also a filter having a gentle roll-off can be used.

FIG. 4 shows an original image and an image generated by each process. FIG. 4A shows an original image. This image is shown in FIG.
As in (b), it is divided into four from the region A _x shown in FIGS. 3 (a) using one of the LPF and three BPF to D _x. At this time, for the above-described reason, the bandwidth of the divided area was set to the same value as the bandwidth corresponding to the Nyquist frequency of the image sensor. Subsequently, the filtered image (FIG. 4B) is sampled at the same pitch (256 × 256 pixels) as the sampling rate of the image sensor (FIG. 4C). Furthermore, this image is FF
It is converted into a frequency component by T (FIG. 4D). FF
After T, only components within the Nyquist frequency of the image sensor are present. Therefore, it can be seen from the sampling of FIG. 4C that the frequency component higher than the Nyquist frequency is turned back into the Nyquist frequency of the image sensor. That is, these images shown in FIG. 4D are nothing but the high-frequency components shown in FIG. 4B which appeared as aliasing. The signal converted to the frequency component is shown in FIG.
The frequency can be changed from (d) by changing the position as shown in FIG. 4 (e). Using this principle, the signal folded back to the lower frequency in FIG.
Moving to the original frequency component shown in (e), the four signals are combined as shown in FIG. 4 (f). Finally, this frequency component is converted into an image by IFFT (inverse FFT) (FIG. 4 (g)).

FIG. 5A shows details of FIG. 4G, and FIG.
(B) shows the CZP band-limited to the spatial frequency that can be imaged by the image sensor. FIG. 5A shows an original image and a reconstructed image. In these figures, 1/4 of the image
5A, and an enlarged view thereof is shown on the left side in FIG. When the original image and the reproduced image are compared in the enlarged view of FIG. 5A, it is difficult to find a difference. Also, with the resolution of the image sensor set in the simulation, FIG.
As shown in (b), an image is lost in a 3/4 region. Comparing FIGS. 5A and 5B, it is clear that the frequency component missing in the image sensor can be reproduced by using the processing of the present invention. As a result, it has been clarified that high resolution can be achieved with a two-dimensional image. According to the present invention, a post-imaging processing process has been established.

FIG. 6 is a diagram showing that a reproduced image is obtained by using a plurality of LPFs in a CZP image as in FIG. FIG. 6 shows a quarter image as in FIG. FIG. 6A shows a result obtained by performing processing using a plurality of images obtained by using a low-pass filter as shown in FIG. FIG. 6 (a)
As can be seen from the reproduced image shown in FIG. 6, it is also possible to reproduce an image in an area not obtained by the image using the low-pass filter (see FIG. 6B). This processing is performed by solving an equation as shown in equation (4). Further, by further performing signal processing for correcting the relative position between images, it is possible to reproduce a high-resolution image with less noise.

<Configuration of Imaging Apparatus> FIG. 7 shows an example of the configuration of an imaging apparatus in an imaging system used in the present invention. The imaging device shown in FIG. 7 includes three half mirrors 202,
204 and 206 and four types of optical filters 212 and 2
22, 232 and 242, 4 image sensors 2
14, 224, 234 and 244 are shown. In FIG. 7, the input image is divided into two equivalent images when passing through the half mirror 202. These images are further divided into two images by the half mirrors 204 and 206, respectively, for a total of four images. These images are processed by optical filters 212, 222, 232 and 242, respectively, and then processed by image sensors 214, 224, 234 and 2 respectively.
An image is taken at 44. With such a configuration, four different image signals can be obtained simultaneously. As for the configuration of the optical filter, an image can be obtained using the imaging apparatus even with the configuration of, for example, no filter and three types of low-pass filters as described above. An optical Fourier system can be used as the optical filter.

According to the present invention, as described above, since the resolution of a two-dimensional image sensor can be increased, a video camera, a digital still camera, an FA monitoring apparatus, and a scientific imaging apparatus using the image sensor are provided. It can be used in a wide range of fields. The image processing of the present invention may be applied not only to a stand-alone computer system, but also to, for example, a client-server system composed of a plurality of systems. The functions of the present invention can be realized by reading and executing the program from a storage medium storing a program related to image processing according to the present invention by the system. The recording medium includes a floppy disk, a CD-ROM, a magnetic tape, a ROM cassette, and the like.

[0018]

As described above, according to the present invention, the resolution of a two-dimensional image sensor can be increased by using digital signal processing and an optical filter without changing the image sensor.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an imaging system according to the present invention.

FIG. 2 is a diagram showing processing in one dimension.

FIG. 3 is a diagram showing processing in two dimensions.

FIG. 4 is a diagram showing processing on a two-dimensional image.

FIG. 5 is a diagram comparing an original image with a reproduced image according to the present invention.

FIG. 6 is a diagram comparing an original image when an LPF is used and a reproduced image according to the present invention.

FIG. 7 is a diagram illustrating a configuration example up to an imaging device according to the present invention.

[Explanation of symbols]

REFERENCE SIGNS LIST 100 imaging device 110 optical filter 120 image sensor 130 storage device 200 signal processing device 202, 204, 206 half mirror 210 digital signal processing circuit 212, 222, 232, 242 optical filter 214, 224, 234, 244 image sensor

Claims (7)

[Claims] An imaging system, comprising: a plurality of optical filters; an imaging device that uses an imaging element that obtains an image for each of the plurality of optical filters; and an image processing device that processes a plurality of images from the imaging device. The image processing apparatus further comprises: a conversion unit that converts each of the images for each of the plurality of optical filters into a frequency domain; a synthesis unit that synthesizes the converted image; and an inverse conversion unit that performs an inverse conversion of the synthesized image. An imaging system comprising: 2. The imaging system according to claim 1, wherein said conversion means performs a Fourier transform, and said inverse transform means performs an inverse Fourier transform. 3. The imaging system according to claim 1, wherein the imaging device can simultaneously capture a plurality of images for each optical filter. 4. The imaging system according to claim 1, wherein the plurality of optical filters include: a low-pass filter for obtaining an image having a Nyquist frequency or less determined by at least the imaging element; An imaging system, comprising: a band-pass filter for obtaining an image. 5. The imaging system according to claim 1, wherein the plurality of optical filters are a plurality of types of low-pass filters, and the combining unit includes a plurality of types of low-pass filters. An imaging system characterized in that image data is processed and synthesized. 6. The imaging system according to claim 5, wherein the imaging device further obtains an image without a filter, and the synthesizing unit synthesizes the image using the image without a filter. . 7. The imaging system according to claim 1, wherein the image processing apparatus further includes a position correction unit that corrects a positional deviation of a plurality of images.