JP5418258B2 - Imaging apparatus and image processing method - Google Patents

Description

  The present invention relates to an imaging apparatus that captures an image and an image processing method that performs image processing.

  In fields such as surveillance cameras, measurement, and medical use, there are many requests for focusing on every corner of the object scene. In order to extend the depth of field (EDOF: Extension of Depth of Field), the F value (focal length / lens aperture) may be increased, and the lens aperture may be simply reduced. However, when the lens diameter is reduced, the spatial resolution is lowered, the amount of light passing through the lens is reduced, and the image becomes dark.

  On the other hand, in recent years, a technique called WFC (Wavefront Coding) has attracted attention. A specific example of WFC is a combination of a light wavefront modulation element (phase modulation element) that modulates the phase of transmitted light and an image processing technique for digitized image data.

  WFC is a technology that expands the depth of field without sacrificing spatial resolution or the amount of transmitted light, and can increase the depth of field without reducing the lens aperture. In addition, an imaging optical system that is not easily affected by chromatic aberration and the like can be realized, and a so-called optical zoom lens mechanism can be eliminated, so that the number of lenses can be reduced, and the mechanism and assembly adjustment can be simplified.

  FIG. 7 is a diagram for explaining the outline of the WFC. The optical system 10 includes lenses 11 to 13, and an optical wavefront modulation element 10 a is inserted between the lenses 12 and 13. By inserting the light wavefront modulation element 10a, the imaging characteristics of the original optical system including only the lenses 11 to 13 are changed, and an analog image formed in the state of the optical system 10 including the light wavefront modulation element 10a is changed. An intermediate image is generated.

  Since an ordinary optical system is optically designed so that aberrations are sufficiently corrected, if an extra phase element such as a phase plate is inserted, the imaging performance is lowered, and the intermediate image becomes a degraded image. At this time, if a phase element that does not change the degree of degradation of the image even if the position of the subject deviates from the focal position to some extent, an intermediate image in which the same degraded image is arranged along the optical axis can be obtained.

  The light wavefront modulation element 10a is a phase element having a phase distribution with little change in the degree of degradation of the image even if the position of the subject is shifted to some extent, and the phase distribution is a cubic expressed by the following equation (1). This is an optical wavefront modulation element (cubic type optical wavefront modulation element). α is a constant, and x and y are position coordinates in the optical axis direction. It is known that the case of having a third order coefficient increases the effect of EDOF.

φ (x, y) = α (x ³ + y ³ ) (1)
On the other hand, the intermediate image deteriorated by the light wavefront modulation element 10a is picked up by the subsequent image pickup element 20 and converted into a digital image. The image processing unit 30 performs digital image processing using a deconvolution filter (inverse filter), removes the deteriorated component from the deteriorated intermediate image, and generates a high-quality final image.

  FIG. 8 is a diagram showing a phase distribution function. Since the expression (1) of the above phase distribution function can be considered as the thickness of the shape of the light wavefront modulation element 10a, if the thickness is Z, the following expression (1a) is obtained. Therefore, FIG. 8 can also be seen as a schematic view of the surface shape of the light wavefront modulation element 10a.

Z = α (X ³ + Y ³ ) (1a)
The optical wavefront modulation element 10a has a thickness with a shape change with respect to the X axis and the Y axis as shown in the formula (1a), and emits the incident light after phase modulation. By using the light wavefront modulation element 10a having such characteristics, even if the position of the subject is shifted to some extent, the change in the degree of deterioration of the intermediate image is reduced, and the degree of deterioration becomes known. For this reason, by performing appropriate image processing in the image processing unit 30, it is possible to obtain a clear image restored to the diffraction limit with respect to the focal position regardless of the amount of deviation of the subject.

  As a conventional technique, a technique for taking an image by expanding the depth of field by WFC has been proposed (for example, Patent Document 1 and Non-Patent Document 1). In addition, a technique for suppressing image quality deterioration by performing data correction based on an impulse response of an inverse filter has been proposed (for example, Patent Document 2).

JP 2003-235794 A JP 2004-24659 A

E. R. Dowski and W.T.Cathey, “Extended Depth of Field Through Wavefront Coding”, Applied Optics, vol. 34, no 11, pp. 1859-1866, April, 1995

  Image degradation such as image blur is often expressed by a PSF (Point Spread Function). The PSF represents the response of the optical system to the point light source, and is a function representing how much the point image spreads when an ideal point image passes through the optical system.

  For example, when one point of the sample is projected on the screen through the optical system, the spot does not become the same size as the original point due to the diffraction of the lens, but a blurred and spread spot is projected. This spot is an image generated by the PSF, and the blurred image displayed at this time appears as an image obtained by multiplying the original image information by the PSF (convolution).

In order to return the image spread according to the PSF to the original image, an inverse operation, that is, division by PSF (inverse matrix calculation of PSF) may be performed on the obtained image.
Specifically, an inverse filter process is performed by dividing the PSF by an OTF (Optical Transfer Function) representing the spatial frequency transfer characteristic of the imaging optical system obtained by Fourier transform.

  However, when the OTF has a very small spatial frequency or the like, noise is amplified, and therefore an appropriate constraint condition or inverse function is selected and the inverse filter process is executed.

  FIG. 9 is a diagram showing an outline of general inverse filter processing. In normal inverse filter processing, an image matrix obtained by arranging pixels of an image obtained on an image sensor vertically and horizontally is processed by a matrix inverse filter called a kernel.

  For simplicity, the kernel is a 3 × 3 matrix. Further, each element (kernel data) of the kernel is assumed to be a to i, and pixel values of the image area where the kernel acts are assumed to be A to I. First, the pixel values A to I of the image area on which the kernel acts are read from the image matrix, and a product-sum operation (convolution operation) between the read pixel value and kernel data is performed to perform image processing.

In the case of the figure, A × a + B × b + C × c + D × d + E × e + F × f + G × g + H × h + I × i = E ₁ , and the original pixel value E is converted to a new pixel value E ₁ . By preparing an appropriate kernel (inverse filter) and performing such processing on other pixel values, the blurred image can be made sharper, or the image from which noise has been removed can be generated. .

  FIG. 10 is a diagram showing the filter strength of the inverse filter. The inverse filter of the image obtained by WFC is generated by taking the inverse of the OTF obtained by Fourier transforming the PSF after passing through the light wavefront modulation element. Arise. The Z-axis represents the filter strength of the inverse filter, and as shown in the figure, the filter strength is shifted to the second quadrant side.

  The deviation of the filter strength of the inverse filter indicates that there is a deviation in the number of vertical and horizontal elements (number of dots) of the inverse filter, that is, the effective calculation area. As described above, the inverse filter obtained from the light wavefront modulation element has a deviation in the filter strength (a deviation occurs in the effective calculation area). Conventionally, however, such deviation is not taken into consideration. Inverse filter processing was performed.

  That is, in the conventional inverse filter processing on the image obtained by WFC, the first to the second quadrant of the filter strength is applied to the image after A / D conversion by the image sensor regardless of the strength of the filter strength of the inverse filter. The image was restored by performing a deconvolution operation over the entire area of the four quadrants. For this reason, there is a problem that the image processing efficiency is inefficient, and that the influence is more remarkable as the number of pixels of the image sensor is increased.

The present invention has been made in view of these points, and an object of the present invention is to provide an imaging apparatus with improved image processing efficiency.
Another object of the present invention is to provide an image processing method with improved image processing efficiency.

  In order to solve the above problems, an imaging apparatus is provided. The imaging apparatus includes an optical system including a light wavefront modulation element, an imaging element that captures a subject image that has passed through the optical system, and an image processing unit that performs image processing on the subject image using an inverse filter. The image processing unit performs the image processing by limiting a calculation region to a region where the filter strength of the inverse filter is higher than a threshold value.

  It becomes possible to improve the image processing efficiency.

It is a figure which shows the structural example of an imaging device. It is a figure which shows the structural example of an imaging device. It is a figure which shows a flare area | region. It is a figure which shows an inverse filter process. It is a figure which shows the frequency characteristic of an inverse filter. It is a flowchart which shows operation | movement of an imaging device. It is a figure for demonstrating the outline | summary of WFC. It is a figure which shows a phase distribution function. It is a figure which shows the outline | summary of a general reverse filter process. It is a figure which shows the filter strength of an inverse filter.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus. The imaging device 1 includes an optical system 10, an imaging element 20, and an image processing unit 30.
The optical system 10 includes a plurality of lenses (not shown) and forms a subject image. An optical wavefront modulation element 10a is inserted between the lenses. The optical wavefront modulation element 10a is an element that reduces changes in OTF in accordance with the focal length.

  The image pickup device 20 can use a CCD (Charged Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or the like, picks up a subject image that has passed through the optical system 10, and converts the analog image into a digital image.

  The image processing unit 30 performs image processing on the subject image using an inverse filter. In this case, image processing with the calculation area limited is performed using an area where the filter strength of the inverse filter is higher than the set threshold as the calculation area.

  Next, the configuration and operation of the imaging apparatus will be described in detail. FIG. 2 is a diagram illustrating a configuration example of the imaging apparatus. The imaging apparatus 1a includes an optical system 10, an imaging element 20, an image processing unit 30, and a monitor unit 40. The image processing unit 30 includes a processor unit 31, registers 32 a and 32 b, a filter strength processing unit 33, a deconvolution unit 34, and an image correction unit 35.

  The processor unit 31 has a memory (not shown), and information for selecting an optimum inverse filter is registered in this memory in accordance with the exposure state. The processor unit 31 recognizes the optimum inverse filter corresponding to the current exposure state from the registration information, and transmits a selection signal for selecting the inverse filter (kernel) to be used for the calculation to the register 32a.

  The register 32a stores a plurality of inverse filters, receives the selection signal transmitted from the processor unit 31, and determines and outputs the inverse filter used for the calculation. The filter strength processing unit 33 receives the inverse filter from the register 32a. Then, a threshold is set for the inverse filter strength, and elements other than the elements (calculation areas) having the strength equal to or higher than the threshold are deleted.

  The filter strength of the inverse filter obtained by WFC, that is, the filter strength of the inverse filter generated by taking the reciprocal of the OTF obtained by Fourier transforming the PSF after passing through the optical wavefront modulation element 10a is as described above. It appears concentrated on the second quadrant side of the intensity distribution.

  Therefore, the filter strength processing unit 33 deletes the other elements while leaving the second quadrant element and the second quadrant flare area element. FIG. 3 shows the flare area. The flare area is an area in the vicinity of the second quadrant, and in the filter strength distribution, the area in the vicinity of the boundary between the second quadrant and the first quadrant and the area in the vicinity of the boundary between the second quadrant and the third quadrant. Refers to an area.

  The register 32b stores the inverse filter (the inverse filter in which the calculation area is limited to the second quadrant area and the flare area of the second quadrant) processed by the filter intensity processing unit 33 and having a filter strength equal to or higher than the threshold value.

The deconvolution unit 34 performs a deconvolution operation between the image signal after A / D conversion output from the image sensor 20 and an inverse filter stored in the register 32b.
The image correction unit 35 performs a predetermined image correction process on the image signal after the deconvolution operation and outputs the processed image signal. The monitor unit 40 performs monitor display of the image signal output from the image processing unit 30.

  The image correction processing in the image correction unit 35 includes, for example, color interpolation (a function for mixing a tone between adjacent pixels to generate a smooth color change), white balance (correction so that white appears white). Function), compression, filing, and the like.

  Here, the subject passes through the optical system 10 including the light wavefront modulation element 10 a to become a subject image, enters the image sensor 20, is converted from an analog signal to a digital signal image, and is output to the image processing unit 30. .

  In the image processing unit 30, the processor unit 31 selects an optimal inverse filter from a plurality of inverse filters stored in the register 32a in accordance with exposure information (aperture information) in exposure control of the imaging device 1a.

  This is because, when shooting with the exposure reduced, a part of the light wavefront modulation element 10a is covered by the stop and the phase changes, so that it may be difficult to restore an appropriate image. Because.

  Therefore, a plurality of inverse filters capable of forming an image suitable for the exposure state are stored in the register 32a, and the optimum inverse filter is selected from these. In this way, a fine image can be restored by adaptively selecting an inverse filter based on the exposure state (aperture state).

  On the other hand, the filter strength of the inverse filter obtained from the light wavefront modulation element 10a tends to concentrate on the second quadrant side as described above. Therefore, in the imaging device 1a, the inverse filter stored in the register 32b limits the calculation area so as to leave an area with a high distribution intensity with respect to the center of the conventional kernel distribution. At this time, the center of the limited region is characterized by being shifted in the diagonal direction on the second quadrant side from the conventional kernel center.

  That is, the inverse filter stored in the register 32a has a normal kernel size, and the inverse filter stored in the register 32b is closer to the second quadrant than the conventional kernel center (center of all areas). The center is shifted in the diagonal direction, resulting in a kernel size with a limited calculation area.

  The kernel stored in the register 32b is the distribution of the second quadrant and the flare area of the second quadrant. The reason for including the flare area instead of the distribution only in the second quadrant is that the flare area in the second quadrant also includes a kernel necessary for image restoration.

  Therefore, the filter strength processing unit 33 deletes other elements from the kernel distribution stored in the register 32a while leaving the second quadrant element and the second quadrant flare area element. Set an appropriate threshold. The kernel that does not exceed the threshold is deleted and the kernel size is adjusted.

  FIG. 4 is a diagram illustrating inverse filter processing. A deconvolution operation is performed on the digital intermediate image output from the image sensor 20 by the inverse filter f1, and a reproduced image is output. The inverse filter f1 has a filter strength distribution in which the second quadrant and the second quadrant flare region are used as calculation regions and the other regions are not used as calculation regions. FIG. 5 shows the frequency characteristics of the inverse filter. The vertical axis represents amplitude, and the horizontal axis represents frequency.

FIG. 6 is a flowchart showing the operation of the imaging apparatus 1a.
[S1] An object image is formed by the optical system 10 including the light wavefront modulation element 10a to generate an intermediate image.

[S2] The image sensor 20 captures an analog intermediate image, performs A / D conversion, and outputs a digital intermediate image.
[S3] The processor unit 31 transmits a selection signal for selecting an inverse filter corresponding to the exposure state to the register 32a.

[S4] The register 32a outputs an inverse filter suitable for the exposure state based on the selection signal.
[S5] The filter strength processing unit 33 receives the inverse filter from the register 32a and determines whether or not the calculation area is a predetermined calculation area, that is, whether or not the calculation area is a flare area in the second quadrant and the second quadrant. Determine whether. If it is not the predetermined calculation area, the process goes to step S6, and if it is the predetermined calculation area, the process goes to step S7.

[S6] The filter strength processing unit 33 deletes kernels in areas other than the second quadrant and the flare area in the second quadrant. Return to step S5.
[S7] The register 32b is an inverse filter generated by the filter strength processing unit 33, that is, an inverse filter having a filter strength equal to or higher than a set threshold value, and has a calculation area in the second quadrant and the second quadrant flare area. Store and output restricted inverse filter.

  [S8] The deconvolution unit 34 performs a deconvolution operation using the A / D converted image signal input from the image sensor 20 and the inverse filter stored in the register 32b.

[S9] The image correction unit 35 performs a predetermined image correction process on the image signal after deconvolution, and outputs the corrected image signal.
[S10] The monitor unit 40 performs monitor display of the image signal transmitted from the image processing unit 30.

  As described above, in the imaging apparatus, the center of the inverse filter calculation region obtained based on the PSF of the light wavefront modulation element 10a is shifted in the diagonal direction on the second quadrant side from the conventional kernel center, A region including the flare region in the second quadrant and the second quadrant was defined as a calculation region. As a result, deconvolution is performed using a region with a high filter strength as a computation region, so that it is possible to improve image processing efficiency.

  In addition to the phase plate, the optical wavefront modulation element can be used to modulate the phase distribution of light, as well as an optical element that changes the refractive index (refractive index distribution type wavefront modulation lens). A liquid crystal element (liquid crystal spatial phase modulation element) or the like can also be used.

As mentioned above, although embodiment was illustrated, the structure of each part shown by embodiment can be substituted by the other thing which has the same function. Moreover, other arbitrary structures and processes may be added.
(Supplementary note 1) an optical system including a light wavefront modulation element;
An image sensor that images a subject image that has passed through the optical system;
An image processing unit that performs image processing on the subject image using an inverse filter;
With
The image processing unit performs the image processing by limiting a calculation region to a region where the filter strength of the inverse filter is higher than a threshold value.
An imaging apparatus characterized by that.

  (Additional remark 2) The said image processing part sets the said threshold value which makes the said 2nd quadrant area | region of the said filter strength distribution and the flare area | region of the said 2nd quadrant the said calculation area, The imaging of Additional remark 1 characterized by the above-mentioned apparatus.

(Supplementary note 3) The imaging apparatus according to supplementary note 2, wherein a center of the calculation area limited by the image processing unit is shifted in a diagonal direction on the second quadrant side.
(Supplementary note 4) The imaging apparatus according to supplementary note 1, wherein the image processing unit prepares a plurality of the inverse filters and selects the inverse filter according to an exposure state.

(Supplementary Note 5) In the image processing method,
Take a subject image with an optical system that includes a light wavefront modulation element,
Image processing of the subject image by an inverse filter;
At the time of the image processing, the image processing is performed by limiting a calculation region to a region where the filter strength of the inverse filter is higher than a threshold value.
An image processing method.

DESCRIPTION OF SYMBOLS 1 Imaging device 10 Optical system 10a Light wavefront modulation element 20 Imaging element 30 Image processing part

Claims (4)

An optical system including an optical wavefront modulation element;
An image sensor that images a subject image that has passed through the optical system;
An image processing unit that performs image processing on the subject image using an inverse filter;
With
The image processing unit performs the image processing by limiting a calculation region to a region where the filter strength of the inverse filter is higher than a threshold value.
An imaging apparatus characterized by that.   The imaging apparatus according to claim 1, wherein the image processing unit sets the threshold value using the second quadrant area and the second quadrant flare area of the filter intensity distribution as the calculation area.   The imaging apparatus according to claim 1, wherein the image processing unit prepares a plurality of the inverse filters and selects the inverse filter according to an exposure state. In the image processing method,
Take a subject image with an optical system that includes a light wavefront modulation element,
Image processing of the subject image by an inverse filter;
At the time of the image processing, the image processing is performed by limiting a calculation region to a region where the filter strength of the inverse filter is higher than a threshold value.
An image processing method.

Images