JP2008211678A - Imaging apparatus and method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus that can have an optical system simplified to lower the cost, and perform suitable restoration processing on a subject to be photographed, and to provide a method thereof. <P>SOLUTION: The imaging apparatus includes: the optical system 110 designed to have a nearly constant PSF before and behind a focusing position in spite of being not in focus at the focusing position; an imaging device 120; a blur processing unit 160 which performs blur restoration processing; a blur amount detecting circuit 190 which detects a blur amount; and a control unit 200 which detects a blur amount of an image after performing filtering processing, using an arbitrary blur restoring filter, on image data obtained by the imaging device and which selects a filter according to the blur amount from a blur restoring filter group to perform filtering processing. If the blur amount of the image is smaller when the arbitrary restoring filter is used than when the selected blur restoring filter is used, the control unit 200 performs control to perform the filtering processing using the blur restoring filter suitable to the focus position of the optical system. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera, a camera mounted on a mobile phone, a camera mounted on a portable information terminal, an image inspection apparatus, and an industrial camera for automatic control using an imaging element and an optical system.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state image sensor, is used in most cases instead of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

FIG. 25 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 25, the best focus surface is matched with the imaging element surface.
FIGS. 26A to 26C show spot images on the light receiving surface of the imaging element 3 of the imaging lens device 1.

In addition, an imaging apparatus has been proposed in which a luminous flux is regularly dispersed by a phase plate (Wavefront Coding optical element) or the like, and is restored by digital processing so that an image can be taken with a deep depth of field (for example, non-patent) References 1, 2 and Patent References 1 to 6).
Further, an automatic exposure control system for a digital camera that performs filter processing using a transfer function has been proposed (see, for example, Patent Document 7).

  Further, Patent Document 8 includes variable focus condensing means in which a transparent liquid is sealed in an optical path of an optical system that forms a subject image and a transparent elastic film is sealed inside to form a lens shape. A camera that captures images that are focused at all positions where afterimage phenomenon occurs visually, by applying a pressure change to the transparent liquid from the outside in a predetermined cycle, changing the curvature of the curved surface of the transparent elastic film, and changing the focal length. An apparatus has been proposed (see, for example, Patent Document 7).

"Wavefront Coding; jointly optimized optical and digital imaging systems", Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. "Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems", Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A JP 2000-98301 A JP 2004-153497 A Japanese Patent Laid-Open No. 9-230252

In the imaging devices proposed in the above-mentioned documents, all of them are based on the assumption that the PSF (Point-Spread-Function) when the above-described phase plate is inserted into a normal optical system is constant, When the PSF changes, it is extremely difficult to realize an image with a deep depth of field by convolution using a subsequent kernel.
Therefore, apart from a single-focus lens, a zoom system, an AF system, or the like has a great problem in adopting due to the high accuracy of the optical design and the associated cost increase.
In other words, in the conventional imaging apparatus, proper convolution calculation cannot be performed, and astigmatism and coma that cause a shift of a spot (SPOT) image at the time of wide or tele (Tele). Therefore, an optical design that eliminates various aberrations such as zoom chromatic aberration is required.
However, the optical design that eliminates these aberrations increases the difficulty of optical design, causing problems such as an increase in design man-hours, an increase in cost, and an increase in the size of the lens.

In addition, in the depth extension system disclosed in each of the above-mentioned documents, basically (one time) restoration processing is performed. In this case, an optical system for extending the depth is combined, but there is a considerable image difference between the lens set object distance and other object distances. Due to this minute image difference, even if the restoration process is performed, the restoration is not complete.
Therefore, in a general depth extension system, there is a disadvantage that an optimal restoration process cannot be performed on a subject to be photographed.

  An object of the present invention is to provide an imaging apparatus and method that can simplify an optical system, reduce costs, and perform an optimal restoration process on a subject to be photographed.

  An imaging apparatus according to a first aspect of the present invention includes an optical system including a light wavefront modulation element, an imaging element that converts an image obtained by the imaging optical system into an electrical signal, and blurring of an image obtained by the imaging element. A computing unit having a function of detecting the amount, a blur restoration filter group calculated from a point spread function corresponding to a defocus amount with reference to a focal position of the optical system, and filtering processing of the image data by the blur restoration filter The image data obtained by the filter means and the image sensor are subjected to filtering processing by an arbitrary blur restoration filter, and then the blur amount of the image is detected, and it is estimated that the blur amount is minimized as a result of filtering. A control unit that controls to perform the filtering process by selecting a filter corresponding to the blur amount from the blur restoration filter group. The control unit, when the blur amount of the image is smaller than the selected blur restoration filter, the filtering by the blur restoration filter suitable for the focal position of the optical system. Control to perform processing.

  Preferably, the blur amount of the image is detected based on one or both of an amplitude correlation value of image data and an edge differential value.

  Preferably, the control unit estimates a defocus amount from the blur amount of the image, and selects a filter that is optimal for the estimated defocus amount.

  Preferably, the focal position of the optical system is set at the closest end.

  Preferably, a plurality of shooting modes corresponding to shooting distances are provided, and when a shooting mode suitable for short-distance shooting is selected from the plurality of shooting modes, the focal position of the optical system is set to the nearest end. Set.

  Preferably, the arbitrary blur restoration filter is set so as to be optimal for a state where there is no defocus at a position where the imaging optical system is approximately the center of the focal position between the closest end and the infinity. .

  Preferably, either or both of the amplitude correlation value and the edge differential value of the image data are detected in a predetermined area in the screen.

  Preferably, the predetermined area in the screen can be arbitrarily set.

  The imaging method of the second aspect of the present invention is to capture an image of a subject that has passed through an optical system including a light wavefront modulation element with an imaging element, and apply an arbitrary blur restoration filter to image data obtained by the imaging element. Select and perform filtering processing, detect the blur amount of the image after receiving the filtering processing, and filter the filter according to the blur amount estimated to be the minimum blur amount as a result of filtering the focal position of the optical system The filtering process is performed by selecting from the blur restoration filter group calculated from the point spread function corresponding to the defocus amount as a reference, and the arbitrary blur restoration filter is used instead of the selected blur restoration filter. When the amount of blur is small, the filtering process is performed by the blur restoration filter suitable for the focal position of the optical system.

  According to the present invention, the optical system can be simplified, the cost can be reduced, and an optimal restoration process can be performed on the subject to be photographed.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging device 120, an analog / digital converter (A / D) 130, a signal processing unit 140, a first memory 150, a blur restoration processing unit 160, a second memory 170, and restoration. A filter unit 180, a blur amount detection circuit 190, a control unit 200, a digital-analog converter (D / A) 210, and a monitor device 220 are included.

  The imaging apparatus 100 according to the present embodiment basically includes a wavefront aberration control optical system that restores an image regularly dispersed by a light wavefront modulation element to a focused image without moving the optical system 110 by digital processing. A system or a depth expansion optical system (DEOS: Depth Expansion Optical system) is employed, and this restoration processing is performed by the blur restoration processing unit 160 under the control of the control unit 200.

  The imaging apparatus 100 according to the present embodiment adopting DEOS is configured as an imaging system capable of creating a better image by performing an optimal processing routine in an image processing system that increases the imaging depth. Yes.

Image restoration processing in a normal depth extension system performs uniform (one time) restoration processing. In this case, an optical system for extending the depth is combined, but there is a considerable image difference between the lens set object distance and other object distances. Due to this minute image difference, even if the restoration process is performed, the restoration is not complete.
Therefore, in the present embodiment, after performing the restoration process once so that the optimum restoration process can be performed on the subject to be photographed, the optimum image restoration amount is calculated according to the situation of the subject image. Then, it restores again by the optimum restoration process.

  FIG. 2 is a flowchart for explaining the basic concept of the restoration processing according to the present embodiment.

As shown in FIG. 2, the subject is photographed (ST1), and once subjected to a prescribed filtering process, a restoration process is performed (ST2).
Either or both of the amplitude correlation value and the edge differential value of the image data of the subject obtained by the image sensor after receiving the filtering process are detected (ST3), for example, the object distance (restoration processing coefficient) of the subject is determined. (ST4).
Then, a filter corresponding to this determination is selected from the blur restoration filter group, and restoration processing by filtering processing is performed again with the new restoration coefficient (ST5).
Then, it is compared whether or not the image by the new restoration process has a larger amount of blur than the image by the prescribed restoration process (ST6), and if it is larger, the restoration process by the near-end filter is performed (ST7).
As a result, it is possible to perform an optimal restoration process on the subject to be photographed.

  The lens that performs this image restoration processing is fixed because there is no lens driving unit for distance measurement. In the current system, the lens is placed at a close position, and is installed on the close subject side for snapping to ensure basic performance.

The optical system 110 supplies an image obtained by photographing the subject object OBJ to the image sensor 120.
As will be described later, the optical system 110 of the present embodiment includes a light wavefront modulation element, and is formed so that the amount of blurring of the focal point is substantially constant at the in-focus position and the distance before and after the in-focus position.

The image sensor 120 forms an image captured by the optical system 110 and outputs the primary image information of the image formation as a primary image signal (FIM) of an electrical signal to the signal processing unit 140 via the analog-digital converter unit 130. It consists of a CCD or CMOS sensor.
In the present embodiment, the image pickup surface of the image pickup device 120 is disposed at a lens close position of the optical system 110.

The analog-digital converter 130 converts the analog image signal FIM obtained by the image sensor 120 into a digital signal and outputs the digital signal to the signal processing unit 140.
The signal processing unit 140 performs image processing of processing of the digital image signal and stores it in the first memory 150.

The blur restoration processing unit 160 inputs a digital signal of the captured image that is made available in the first memory 150, performs a two-dimensional convolution process, and stores it in the second memory 170.
The blur restoration processing unit 160 uses a digital filter corresponding to the selection result of the blur restoration digital filter based on the exposure information of the control unit 200 and the amplitude correlation value (edge differential value) of the subject image data as the restoration filter 180, and performs optical A filter process is performed on the image data imaged via the dynamic transfer function (OTF), a blur restoration process is performed, and an image after the blur restoration process is stored in the second memory 170.
The blur restoration processing unit 160 generates a restored image signal by correcting the focal blur of the image from the image sensor 120. More specifically, the blur restoration processing unit 160 has a function of generating an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  In the restoration filter unit 180, a plurality of digital filters used for blur restoration processing are prepared in the blur restoration processing unit 160, and a digital filter according to an instruction from the control unit 200 is used in the blur restoration processing unit 160.

  The blur amount detection circuit 190 reads the blur restoration processing image stored in the second memory 170, and based on one or both of the amplitude correlation value and the edge differential value of the image data under the control of the control unit 210. The loss result is output to the control unit 200.

The control unit 200 performs exposure control and has operation inputs of the operation unit and the like, determines the operation of the entire system according to those inputs, and controls arbitration control of the entire system.
The control unit 200 performs predetermined image processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing on the image subjected to the blur restoration processing supplied from the amplitude correlation value detection circuit 190. Storage in a memory that is not used, image display control on the monitor device 220, and the like.
The control unit 200 uses a plurality of digital filters, performs blur restoration processing with each digital filter, and selects an optimum filter used for the two restoration processing according to the detection result of the blur amount detection circuit 190.
Then, the control unit 200 compares whether or not the image by the new restoration process has a larger amount of blur than the image by the prescribed restoration process, and if so, controls to perform the restoration process by the near-end filter. .

  The digital / analog converter 210 converts the digital data processed by the control unit 200 into analog data and outputs the analog data to the monitor device 220.

  The monitor device 220 is formed of, for example, a liquid crystal display device, and displays an image that has undergone predetermined image processing by the control unit 200, a decoding, and the like.

  Here, with reference to FIGS. 3 to 5, the selection process of the digital filter for blur restoration for obtaining the optimal image in focus in the present embodiment will be described.

  As described above, since there is no lens driving unit for distance measurement, the lens is fixed. In the current system, the lens is placed at a close position, and is installed on the close subject side for snapping to ensure basic performance.

  Since the position of the lens is known as the closest end, the blurred image is mostly the front pin on the far side, and information on the degree of lens blur depending on the distance is held in advance by the system, and after shooting If a certain restoration process is performed, a defocus amount (object image distance) is estimated from the contrast (blur amount) of the image, and a restored image process corresponding to the estimated distance is performed.

The amount of blur can be detected based on one or both of the amplitude correlation value and the edge differential value of the image data.
Normally, the blurred state (focus state) of an image is correlated with the amplitude correlation value (contrast strength), but this is affected by the brightness of the subject. Although the amount of received light is controlled to have a constant brightness by automatic exposure control, it cannot be denied that the absolute value changes due to a change in some condition.
Even in such a case, if the differential value of the edge of the image is used, the absolute value can be detected without being affected by the brightness. Of course, it is also possible to detect the amount of blur using a combination of these.

  The feature of this embodiment is that it is possible to improve the estimation accuracy of the object distance by performing the restoration process once, thereby performing the restoration process according to the photographed object image blur (distance). It is. By performing this optimum processing, an imaging system that does not require a distance measuring drive system can be obtained. In particular, a lens system with a small F-number is more effective.

The amount of blur on the sensor surface between the distance difference on the near side and the distance difference on the far side is larger on the near side. For this reason, if the sensor position is installed on the close side, when performing distance estimation (filter estimation) in general imaging, estimation in the close vicinity can be estimated with higher accuracy than on the far side.
On the other hand, although the estimation on the far side becomes rough, the blur change amount of the estimation error on the far side is small.

  FIG. 3 is a diagram showing the relationship between the shooting distance and the amount of blur on the sensor (imaging device).

As shown in FIG. 3, with respect to the shooting distance difference A = B, the blur amount difference (C, D) at each distance difference is C> D.
However, the subject is not always on the infinity side from the closest end. If the subject is located closer to the near end (super close) than the close end of the lens specifications, the resulting image will be blurred, and whether the subject is on the infinite edge side from the close end will be very close Cannot be determined.
In other words, the subject distance does not necessarily reach the infinity side as the blur amount increases, and if the blur restoration process is performed using a filter that is uniquely selected by estimating the subject distance from the blur amount, the subject filter is not blurred. The image is more blurred than the restored image.

In order to solve this problem, when the blur amount of the image subjected to the blur restoration process using the filter selected based on the estimated subject distance is larger than the blur amount of the image subjected to the blur restoration process using the filter before selection, Then, it is determined that the subject is in the extremely close range, and a filter suitable for the close end distance is selected to perform blur restoration processing.
The configuration and processing for setting the lens at the closest distance are particularly effective when the subject is at a short distance. Therefore, it is used in the macro shooting mode, and the lens position is set to infinity in the normal shooting mode. It is also possible to switch and use depending on the shooting mode, such as setting and executing the process.

  The blur restoration filter that is used first to estimate the defocus amount is designed so that the imaging optical system can detect the blur amount accurately on both the near side and the infinity side. Is designed so as to be optimal for a state where there is no defocus at a position which is approximately the center of.

  FIG. 4 is a diagram illustrating the relationship between the subject distance and the focal length.

<1>: A blur amount of A is detected from an image restored by blurring using an arbitrary filter which is an initial setting. However, it is impossible to determine which of the following states is present.
A 'as a result of the subject being at a position b on the infinity side from the closest end,
-A of the result by being in the position of a very close a.

<2>: The subject distance b is estimated from the blur amount A, and a filter suitable for the distance b is selected to perform the blur restoration process.

<3>: When the amount of blur of the restored image is detected, B> A, which is larger than in the case of the default filter.

<4>: The blur restoration process is performed using a filter that is optimal for the closest distance.

Note that, as shown in FIGS. 5A to 5C, for example, as shown in FIG. 5A to FIG. It is also possible to configure to detect for AR3).
FIG. 5A shows, for example, the initial setting area AR1.
FIG. 5B shows a case where the setting area is expanded from AR1 to AR2 so as to cover the main subject.
FIG. 5C shows a case where the setting area is moved from AR1 to AR3 along with the main subject.

  The basic function of the optical system according to the present embodiment and the basic processing according to the presence / absence of the blurred image restoration will be described below, and then the optical system 110 and the blur restoration processing unit 160 that are features of the present embodiment are further described. A specific configuration and function will be specifically described.

Here, a general imaging optical system in which light rays concentrate at the best focus (focal point) position will be described as a comparative example.
FIG. 6 is a diagram illustrating the relationship between the light beam height and defocus of a general imaging optical system.
FIG. 7 is a diagram showing optical characteristics in the vicinity of image formation of the optical system of the present embodiment.

As shown in FIG. 6, in a general imaging optical system, light rays are most densely concentrated at the best focus position, and when the distance from the best focus position is increased, the blur diameter is approximately proportional to the defocus amount. spread.
On the other hand, as shown in FIG. 7, the optical system 110 according to the present embodiment does not concentrate light rays at the best focus position, unlike the general imaging system optical system. ing. However, the optical system 110 according to the present embodiment is designed so that the change in the blur shape (PSF) is insensitive to the defocus amount in the vicinity of the best focus.
Therefore, if a blur restoration process according to the PSF at the best focus position is performed, a clear image can be obtained by removing the blur not only before and after the best focus point.
This is the principle of depth expansion adopted in this embodiment.

Next, the blurred image restoration process will be described.
FIG. 8 is an MTF (Modulation Transfer Function) characteristic diagram at the best focus position of a general optical system.
FIG. 9 is a diagram showing the MTF characteristics of the optical system of the present embodiment.

Since the picked-up image obtained by the image pickup device 120 after passing through the optical system 110 of the present embodiment is blurred, the MTF decreases from the middle range to the high range. This MTF is raised by calculation. In contrast to the MTF, which is the amplitude characteristic of a single lens, the total amplitude characteristic including image processing is called SFR (Spatial Frequency Response).
Since the frequency characteristic of a PSF that generates a blurred image is MTF, a blur restoration filter is designed to have a gain characteristic that is increased to a desired SFR characteristic. The degree of gain is determined by the balance with noise and false images.

  There are two methods for digitally filtering this blur restoration filter to the original image: Fourier transforming the image and multiplying the filter and frequency in the frequency domain, and convolution (convolution) in the spatial domain. is there. Here, the latter implementation method will be described. The convolution operation is expressed by the following equation.

Here, f indicates a filter kernel (here, the one rotated 180 degrees is used for easy calculation).
A indicates an original image, and B indicates a filtered image (blurred restored image).
As can be seen from this equation, f is superimposed on the image and the result of summing the products of the taps is taken as the value of the center coordinate that has been superimposed.

Next, a 3 * 3 filter will be specifically described with reference to FIGS. 10A to 10C.
The restoration filter of FIG. 10A (already rotated by 180 degrees) is overlaid with the filter center f (0,0) on A (i, j) of the blurred image shown in FIG. The product is taken and the nine total values are set as B (i, j) of the blurred restored image shown in FIG.
When (i, j) is scanned over the entire image, a new B image is generated. This is a digital filter. Here, since the filter is used for blur restoration, blur restoration processing can be performed by performing this processing.

  Hereinafter, the configuration and function of the optical system and the image processing apparatus of the present embodiment will be described more specifically.

  FIG. 11 is a diagram schematically illustrating a configuration example of the optical system 110 according to the present embodiment.

An optical system 110 in FIG. 11 is disposed between an object-side lens 111 disposed on the object-side OBJS, an imaging lens 112 for forming an image on the image sensor 120, and between the object-side lens 111 and the imaging lens 112. An optical wavefront modulation element (wavefront forming optical element: wavefront coding element) made of, for example, a phase plate (Cubic Phase Plate) having a three-dimensional curved surface, which deforms the wavefront of the image formation on the light receiving surface of the image sensor 120 by the imaging lens 112. An optical element group 113; A stop (not shown) is disposed between the object side lens 111 and the imaging lens 112.
For example, in this embodiment, a variable aperture is provided, and the aperture (aperture) of the variable aperture is controlled by exposure control (apparatus).

  In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-described third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change by coding on the lens surface (for example, wavefront modulation) A light wavefront modulation element such as a hybrid lens) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element) may be used.

The phase plate 113a shown in the figure is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, the phase plate 113a forms a deep luminous flux (which plays a central role in image formation) and a flare (blurred portion).
As described above, means for restoring the regularly dispersed image to a focused image by digital processing is called the above-described wavefront aberration control optical system (DEOS: Depth Expansion Optical system), and this processing is This is performed in the blur restoration processing unit 160.

Here, the basic principle of DEOS will be described.
As shown in FIG. 12, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

(Equation 2)
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

(Equation 3)
f = H ⁻¹ * g

Here, the kernel size and calculation coefficient regarding H will be described.
Let the zoom positions be ZPn, ZPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, or a PSF is set as a function of a focus blur amount (object distance), calculated by an object distance, and an arbitrary object can be calculated by calculating the H function. It may be possible to set so as to create an optimum filter with respect to the distance. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In the present embodiment, as shown in FIG. 1, an image from the optical system 110 is received by the image sensor 120 and input to the blur restoration processing unit 160, and a conversion coefficient corresponding to the optical system is acquired and acquired. A non-dispersed image signal is generated from the dispersed image signal from the image sensor 120 with the converted coefficient.

  In the present embodiment, as described above, dispersion refers to forming a non-focused image on the image sensor 120 by inserting the phase plate 113a as described above. It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

  Next, the configuration and processing of the blur restoration processing unit 160 will be described.

  As illustrated in FIG. 13, the blur restoration processing unit 160 includes a raw (RAW) buffer memory 161, a two-dimensional convolution operation unit 162, a kernel data storage ROM 163 as a storage unit, and a convolution control unit 164.

  The convolution control unit 164 controls the convolution process on / off, the screen size, the replacement of kernel data, and the like, and is controlled by the control unit 220, for example.

Further, the kernel data storage ROM 163 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. 14 or FIG. Exposure information determined at times is acquired, and kernel data is selected and controlled through the convolution control unit 164.
The exposure information includes aperture information.

  In the example of FIG. 14, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

  In the example of FIG. 15, the kernel data A corresponds to the F number (2.8) as aperture information, the kernel data B corresponds to the F number (4), and the kernel data C corresponds to the data corresponding to the F number (5.6). It has become.

As in the example of FIG. 15, the filtering process according to the aperture information is performed for the following reason.
When shooting with the aperture stopped, the phase plate 113a forming the light wavefront modulation element is covered by the aperture and the phase changes, making it difficult to restore an appropriate image.
Therefore, in the present embodiment, as in this example, appropriate image restoration is realized by performing filter processing according to aperture information in exposure information.

  The example in which the two-dimensional convolution calculation unit 162 performs the filtering process according to only the exposure information has been described above. For example, the calculation coefficient suitable for combining subject distance information, zoom information, or shooting mode information and exposure information. Can be extracted or calculated.

FIG. 16 is a diagram illustrating a configuration example of an image processing apparatus that combines subject distance information and exposure information.
FIG. 16 shows a configuration example of an image processing apparatus 300 that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  As illustrated in FIG. 16, the image processing apparatus 300 includes a convolution apparatus 301, a kernel / numerical operation coefficient storage register 302, and an image processing operation processor 303.

  In this image processing apparatus 300, the image processing arithmetic processor 303 which has obtained information on the approximate distance of the object distance of the subject and the exposure information calculates the kernel size and the arithmetic coefficient used in the appropriate calculation for the object separation position. The image is restored by performing an appropriate calculation in the convolution device 301 that stores the value in the kernel and numerical calculation coefficient storage register 302 and calculates using the value.

As described above, in the case of an imaging device having a phase plate (Wavefront Coding optical element) as a light wavefront modulation element, an image signal without proper aberrations by image processing within the predetermined focal length range However, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 400 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

The image processing is performed by convolution calculation. To achieve this, for example, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the focal length, The correction coefficient is used to correct the calculation coefficient, and an appropriate convolution calculation can be performed using the corrected calculation coefficient.
In addition to this configuration, the following configuration can be employed.

  The kernel size and the convolution calculation coefficient itself are stored in advance according to the focal length, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient according to the focal length is stored in advance as a function. In addition, it is possible to employ a configuration in which a calculation coefficient is obtained from this function based on the focal length and a convolution calculation is performed using the calculated calculation coefficient.

  Corresponding to the configuration of FIG. 16, the following configuration can be taken.

At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 113a are stored in advance in the register 302 as the conversion coefficient storage means according to the subject distance. The image processing arithmetic processor 303 functions as a coefficient selection unit that selects a conversion coefficient corresponding to the distance from the register 302 to the subject based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit. .
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.

Alternatively, as described above, the image processing calculation processor 303 as the conversion coefficient calculation unit calculates the conversion coefficient based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit, and stores it in the register 302. Store.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 110 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 113a is stored in advance in the register 302 that also functions as a second conversion coefficient storage unit.
Then, based on the distance information generated by the object approximate distance information detection device 400 as the subject distance information generation means, the image processing arithmetic processor 303 as the correction value selection means performs a process from the register 302 as the correction value storage means to the subject. Select a correction value according to the distance.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 17 is a diagram illustrating a configuration example of an image processing apparatus that combines shooting mode information and exposure information.
FIG. 17 shows a configuration example of an image processing apparatus 300A that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  As shown in FIG. 17, the image processing apparatus 300A includes a convolution apparatus 301, a kernel / numerical calculation coefficient storage register 302 as a storage unit, and an image processing calculation processor 303, as shown in FIG.

  In the image processing apparatus 300A, the image processing arithmetic processor 303 that has obtained information and exposure information related to the approximate object distance of the subject read from the approximate object distance information detection apparatus 500 performs an appropriate calculation for the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

Also in this case, as described above, in the case of an imaging apparatus having a phase plate (Wavefront Coding optical element) as an optical wavefront modulation element, an appropriate aberration is obtained by image processing within the predetermined focal length range. Although no image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 500 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

  The above image processing is performed by convolution calculation. To realize this, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the object distance. A configuration in which a correction coefficient is used to correct a calculation coefficient and an appropriate convolution calculation is performed using the corrected calculation coefficient, a calculation coefficient corresponding to the object distance is stored in advance as a function, and the calculation coefficient is calculated from this function according to the focal length. The convolution calculation is performed using the calculated calculation coefficient, the kernel size and the convolution calculation coefficient are stored in advance according to the object distance, and the convolution calculation is performed using the stored kernel size and calculation coefficient. It is possible to adopt a configuration or the like.

  In the present embodiment, as described above, the image processing is changed according to the DSC mode setting (portrait, infinity (landscape), macro).

  When associated with the configuration of FIG. 17, the following configuration can be adopted.

As described above, different conversion coefficients are stored in the register 302 as the conversion coefficient storage means according to each shooting mode set by the shooting mode setting unit 600 of the operation unit through the image processing arithmetic processor 303 as the conversion coefficient calculation means. .
Based on the information generated by the object approximate distance information detecting device 500 as the subject distance information generating unit, the image processing arithmetic processor 303 according to the shooting mode set by the operation switch 601 of the shooting mode setting unit 600 converts the conversion coefficient. A conversion coefficient is extracted from the register 302 serving as a storage unit. At this time, for example, the image processing arithmetic processor 303 functions as conversion coefficient extraction means.
Then, the convolution device 301 serving as a conversion unit performs conversion processing according to the image signal shooting mode using the conversion coefficient stored in the register 302.

  Note that the optical system of FIG. 11 is an example, and the present invention is not necessarily used for the optical system of FIG. Also, the kernel data storage ROM of FIGS. 14 and 15 is not necessarily used for the optical magnification, F number, and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.

As described above, in the case of an imaging apparatus having a phase plate (Wavefront Coding optical element) as an optical wavefront modulation element, there is no appropriate aberration by image processing within the predetermined focal length range. Although an image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of image processing, so that only an object outside the range has an aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

18A to 18C show spot images on the light receiving surface of the image sensor 120. FIG.
18A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 18B shows a case where the focal point is a focal point (Best focus), and FIG. 18C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 18A to 18C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (a central role of image formation) is obtained by the wavefront forming optical element group 113 including the phase plate 113a. And flare (blurred part) are formed.

  As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

FIGS. 19A and 19B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens device according to the present embodiment. FIG. 19A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 19B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 160 including a digital signal processor, for example, as shown in FIGS. 19A and 19B. The MTF of the primary image is essentially a low value.

  As described above, the image processing apparatus 160 receives the primary image FIM from the image sensor 120, performs a so-called predetermined correction process for raising the MTF at the spatial frequency of the primary image, and forms a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing device 160 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristics indicated by the curve B in FIG.
The characteristic indicated by the curve B in FIG. 20 is a characteristic obtained when the wavefront is not deformed without using the wavefront forming optical element as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In this embodiment, as shown in FIG. 20, in order to achieve the MTF characteristic curve B to be finally realized with respect to the MTF characteristic curve A with respect to the optically obtained spatial frequency, each spatial frequency is changed to each spatial frequency. On the other hand, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic shown in FIG. 20, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the blur restoration processing unit 160 that forms the primary image in a high-definition final image. In the optical system, a wavefront forming optical element is newly provided, or an optical element surface such as glass, plastic or the like is formed for wavefront shaping, thereby forming a wavefront for imaging. Is deformed (modulated), and such a wavefront is imaged on the imaging surface (light-receiving surface) of the imaging device 120 made up of a CCD or CMOS sensor. An image forming system obtained.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. Therefore, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 150.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the target wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing device 160 composed of a later stage DSP or the like. Thereby, the MTF value is significantly improved.

  Next, the response of the MTF of this embodiment and the conventional optical system will be considered.

FIG. 22 is a diagram showing MTF responses when the object is at the focal position and when the object is out of the focal position in the case of the conventional optical system.
FIG. 23 is a diagram showing the MTF response when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 24 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, even when the object deviates from the focal position, the change in the MTF response is smaller than the optical diameter in which the light wavefront modulation element is not inserted.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

  As described above, according to the present embodiment, the optical system 110 is designed so that the PSF (Point Spread Function point spread function) is substantially constant before and after the focusing position even though the in-focus position is not achieved. An image sensor 120 that captures an image of the system, a blur processing unit 160 that performs blur restoration processing, a blur amount detection circuit 190 that detects a blur amount of image data obtained by the image sensor, and a focal position of the optical system The blur restoration filter group 180 calculated from the point spread function according to the defocus amount and the image data obtained by the image sensor are subjected to filtering processing by an arbitrary blur restoration filter, and then the blur amount of the image is detected. A control unit 200 that controls to perform filtering processing by selecting a filter corresponding to the blur amount from the blur restoration filter group, and the control unit 200, control is performed so that filtering processing is performed by a blur restoration filter suitable for the focal position of the optical system when an arbitrary blur restoration filter is used rather than the selected blur restoration filter. Therefore, it is possible to perform more faithful image restoration by performing more optimal processing than performing image restoration in a system that expands the depth by image processing.

  The imaging apparatus 100 according to the present embodiment can be used for DEOS of a zoom lens considering the small size, light weight, and cost of consumer devices such as a digital camera and a camcorder.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, since it has a blur restoration processing unit 160 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, it is possible to obtain a high-definition image quality There is an advantage that
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a flowchart for demonstrating the basic concept of the decompression | restoration process which concerns on this embodiment. It is a figure which shows the relationship between an imaging distance and the amount of blurs on a sensor (imaging element). It is a figure which shows the relationship between a to-be-photographed object distance and a focus position. It is a figure which shows the example of a setting of the detection area | region of an amplitude correlation value. It is a figure which shows the relationship between the light ray height of a general imaging optical system, and defocusing. It is a figure which shows the optical characteristic of image formation vicinity of the optical system of this embodiment. It is a MTF (Modulation Transfer Function amplitude characteristic) characteristic figure in the best focus (Best Focus) position of a general optical system. It is a figure which shows the MTF characteristic of the optical system of this embodiment. It is explanatory drawing of the blur decompression | restoration process in this embodiment. It is a figure which shows typically the structural example of the optical system of the imaging lens apparatus which concerns on this embodiment. It is a figure for demonstrating the principle of DEOS. It is a block diagram which shows the structural example of the blur restoration process part which concerns on this embodiment. It is a figure which shows an example (optical magnification) of the storage data of kernel data ROM. It is a figure which shows the other example (F number) of the storage data of kernel data ROM. It is a figure which shows the structural example of the blur restoration process part which combines subject distance information and exposure information. It is a figure which shows the structural example of the image processing apparatus which combines imaging | photography mode information and exposure information. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a focus point ( Best focus), (C) is a diagram showing each spot image when the focal point is deviated by −0.2 mm (Defocus = −0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image pick-up element concerning this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image pick-up element of an image pick-up lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 26A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens apparatus of FIG. 25, where FIG. In the case (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Optical system, 120 ... Imaging element, 130 ... Analog-digital converter (A / D), 140 ... Signal processing part, 150 ... 1st memory, 160 ... Bokeh restoration processing unit, 170 ... second memory, 180 ... restoration filter unit, 190 ... blur amount detection circuit, 200 ... control unit, 210 ... digital analog converter (D / A), 220 ... monitor device, 111 ... object side lens, 112 ... imaging lens, 113 ... optical element for wavefront formation, 113a ... phase plate (light wavefront modulation element), 162: Two-dimensional convolution operation unit, 163: Kernel data ROM, 164 ... Convolution control unit.

Claims (9)

An optical system including an optical wavefront modulation element;
An image sensor for converting an image obtained by the imaging optical system into an electrical signal;
Arithmetic means having a function of detecting a blur amount of an image obtained by the image sensor;
A plurality of blur restoration filters calculated from a point spread function corresponding to a defocus amount with reference to the focal position of the optical system;
Filter means for filtering the image data with any one of the plurality of blur restoration filters;
The image data obtained by the image sensor is subjected to a filtering process using an arbitrary blur restoration filter, and then the blur amount of the image is detected. According to the blur amount estimated to be the minimum blur amount as a result of filtering. A control unit that controls to perform the filtering process by selecting again the filter from the plurality of blur restoration filters,
The controller is
When the blur amount of the image is smaller when the arbitrary blur restoration filter is used than the reselected blur restoration filter, the filtering processing is performed by the blur restoration filter suitable for the focal position of the optical system. The imaging device to control. The imaging apparatus according to claim 1, wherein the blur amount of the image is detected based on one or both of an amplitude correlation value and an edge differential value of image data. The controller is
The imaging apparatus according to claim 1, wherein a defocus amount is estimated from a blur amount of the image, and a filter that is optimal for the estimated defocus amount is selected. The imaging device according to any one of claims 1 to 3, wherein a focal position of the optical system is set at a close end. A plurality of shooting modes corresponding to a shooting distance are provided, and when a shooting mode suitable for short-distance shooting is selected from the plurality of shooting modes, the focal position of the optical system is set to the closest end. The imaging device according to any one of 1 to 3. The arbitrary blur restoration filter is:
The said imaging optical system is set so that it may become optimal with respect to the state without a defocus in the position which becomes the approximate center of a near end and a infinity focus position. Imaging device. The imaging device according to any one of claims 2 to 6, wherein either or both of an amplitude correlation value and an edge differential value of the image data are detected for a predetermined region in the screen. The imaging apparatus according to claim 7, wherein the predetermined area in the screen can be arbitrarily set. The subject image that has passed through the optical system including the light wavefront modulation element is captured by the imaging element,
The image data obtained by the imaging device is subjected to a filtering process by selecting an arbitrary blur restoration filter,
After receiving the filtering process, detect the amount of blur of the image,
From the plurality of blur restoration filters calculated from the point spread function corresponding to the defocus amount with reference to the focal position of the optical system, the filter corresponding to the blur amount estimated to be the minimum blur amount as a result of filtering Select to perform the filtering process,
An imaging method in which the filtering process is performed with the blur restoration filter suitable for the focal position of the optical system when the amount of blur of the image is smaller when the arbitrary blur restoration filter is used than the selected blur restoration filter.

Images