JP2008245266A - Imaging apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which can readily detect the definition performance peak position, while simplifying the optical system and reducing the cost, and can obtain a properly reconstructed image while reducing the impact of noise, and to provide its manufacturing apparatus and method. <P>SOLUTION: The imaging apparatus comprises an optical system 110 equipped with an optical wave front modulating element 114, having a function for deforming the wave front of the image formed on the light receiving surface of an image sensor 120 by an image formation lens where emergence of the optical wave front modulation function depends on the outside, an image sensor 120, and an image processor 140 that generates an image signal, having no dispersion from an object dispersion image signal, wherein the outside dependent optical wave front modulation element 114 has such a performance where the light passing through the optical system under one focus state forms an image at a position different in the direction of optical axis, depending on the distance of the light source when the optical wave front modulation function is controlled to non-emergence state by a modulation function control section 200, and becomes multi-focal point state, when it is controlled to be an emerging state. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus and an image pickup apparatus using an image pickup element, such as a digital still camera equipped with an optical system, a camera mounted on a mobile phone, a camera mounted on a portable information terminal, an image inspection apparatus, and an industrial camera for automatic control.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state image sensor, is used in most cases instead of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

FIG. 27 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 27, the best focus surface is matched with the imaging device surface.
FIGS. 28A to 28C show spot images on the light receiving surface of the image sensor 3 of the imaging lens device 1.

In addition, an imaging apparatus has been proposed in which a light beam is regularly dispersed by a phase front (wavefront coding optical element) and restored by digital processing to enable imaging with a deep depth of field (for example, non-patent literature). 1, 2, and patent documents 1 to 5).
In addition, an automatic exposure control system for a digital camera that performs filter processing using a transfer function has been proposed (see, for example, Patent Document 6).
“Wavefront Coding; jointly optimized optical and digital imaging systems”, Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. “Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems”, Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A JP 2004-153497 A

In the imaging devices proposed in the above-mentioned documents, all of them are based on the assumption that the PSF (Point-Spread-Function) when the above-described phase plate is inserted into a normal optical system is constant, When the PSF changes, it is extremely difficult to realize an image with a deep depth of field by convolution using a subsequent kernel.
Therefore, apart from a single-focus lens, a zoom system, an AF system, or the like has a great problem in adopting due to the high accuracy of the optical design and the associated cost increase.
In other words, in the conventional imaging apparatus, proper convolution calculation cannot be performed, and astigmatism and coma that cause a shift of a spot (SPOT) image at the time of wide or tele (Tele). Therefore, an optical design that eliminates various aberrations such as zoom chromatic aberration is required.
However, the optical design that eliminates these aberrations increases the difficulty of optical design, causing problems such as an increase in design man-hours, an increase in cost, and an increase in the size of the lens.

In addition, in the devices disclosed in the above-described documents, noise is also amplified simultaneously when an image is restored by signal processing, for example, in shooting in a dark place.
Therefore, for example, in an optical system including an optical system and signal processing using the above-described optical wavefront modulation element such as a phase plate and subsequent signal processing, noise is amplified when shooting in a dark place. This has the disadvantage of affecting the restored image.

In addition, depth expansion technology has a small amount of change in image performance within a deep depth of field, so it is difficult to detect the optical performance peak position with respect to defocus, and it is difficult to perform an appropriate evaluation, which imposes a burden on the manufacturing process. And has the disadvantage of being disadvantageous in terms of yield.
In depth-expanded optical systems that have undergone phase modulation, the amount of change in image performance is small as the system name implies, so back focus adjustment using the change in resolution performance as a measure is to see the amount of change. Defocus the wide range and calculate the peak-like part. In this case, it takes time for defocusing a wide range, or a rough search is performed so that the accuracy is lowered, or the manufacturing process is burdened or the performance of the product is deteriorated.
In particular, in a depth-expanded optical system premised on restoration, it is extremely difficult to calculate a resolution peak because a change in performance is extremely suppressed within a guaranteed depth.
Therefore, it becomes an issue to be able to easily detect the resolution performance peak position in the depth extension optical system.

Furthermore, in the depth extension optical system, as the name suggests, the blur due to the sense of depth of the subject is reduced or lost. In general photography, there is also a method for close-up of a subject by intentionally generating a blur in the background, and a depth expansion optical system is not suitable for such applications.
If you want to shoot at different depths depending on the application without having multiple cameras, the optical system with dark F value has been used in the past, but an optical system with a low F value reduces shutter speed and increases noise. This is undesirable.

  An object of the present invention is to provide an imaging apparatus capable of easily detecting a resolution performance peak position, simplifying an optical system, reducing costs, and obtaining a restored image with little influence of noise. It is to provide an imaging method.

  An image pickup apparatus according to a first aspect of the present invention picks up an optical system including an externally dependent light wavefront modulation element whose expression or non-expression of an optical wavefront modulation function depends on the outside, and a subject image that has passed through the optical system. An image sensor, a modulation function controller for controlling the on / off expression of the light wavefront modulation function of the externally dependent light wavefront modulator, and image processing for performing predetermined processing on the image signal of the subject from the image sensor And the optical system is in a one-focus state when the light wavefront modulation element is controlled to the non-expressed state by the modulation function control unit. When the state is controlled, a multi-focus state is obtained.

  According to a second aspect of the present invention, an optical system including an optical wavefront modulation element and an expression or non-expression of an inverse optical wavefront modulation function that is disposed in an optical path of the optical system and cancels the optical wavefront modulation function of the optical wavefront modulation element Controls on / off expression of an externally dependent inverse light wavefront modulation element that depends on the outside, an imaging element that captures a subject image that has passed through the optical system, and a light wavefront modulation function of the externally dependent inverse light wavefront modulation element A modulation function control unit, and an image processing unit that performs predetermined processing on an image signal of a subject from the imaging device, wherein the optical system includes the externally dependent inverse light wavefront modulation device, the modulation function When the inverse light wavefront modulation function is controlled to the expression state by the control unit, the focus state is set to one focus state, and when the control unit is controlled to the non-expression state, the multi-focus state is set.

  Preferably, the image processing unit has a function of performing a filter process on an optical transfer function (OTF) in accordance with predetermined information, and switches between the expression and non-expression of the modulation function of the modulation function control unit. Are linked to the switching of the filter acting on the image processing unit.

  Preferably, the image processing unit has a monitor capable of displaying an image after image processing in color or black and white, and switching between the on / off of the modulation function of the modulation function control unit is a color output of the monitor output And link to black and white output switching.

  Preferably, the image processing unit has a chromatic aberration correction function by a filter process on an optical transfer function (OTF) according to predetermined information, and the modulation function of the modulation function control unit is switched between expression and non-expression. This is linked to switching of the chromatic aberration correction function of the image processing unit.

  Preferably, the image processing unit performs the image restoration process using a plurality of types of restoration filters.

  Preferably, the image processing unit includes memory means for storing a calculation coefficient of the filter.

  Preferably, the memory means stores a calculation coefficient for noise reduction processing according to exposure information.

  Preferably, the memory means stores a calculation coefficient for restoring an optical transfer function (OTF) according to exposure information.

  Preferably, the imaging device includes subject distance information generating means for generating information corresponding to a distance to the subject, and the converting means is based on the information generated by the subject distance information generating means. A non-dispersed image signal is generated from the dispersed image signal.

  Preferably, the imaging apparatus includes subject distance information generating means for generating information corresponding to a distance to the subject, and conversion coefficient calculating means for calculating a conversion coefficient based on the information generated by the subject distance information generating means. The conversion unit converts the image signal using the conversion coefficient obtained from the conversion coefficient calculation unit, and generates an image signal without dispersion.

  Preferably, the imaging apparatus includes a shooting mode setting unit that sets a shooting mode of a subject to be shot, and the conversion unit performs different conversion processing according to the shooting mode set by the shooting mode setting unit. Do.

  According to a third aspect of the present invention, an image of a subject that has passed through an optical system including an externally dependent light wavefront modulation element whose expression or non-expression of the light wavefront modulation function depends on the outside is picked up by the image pickup element. An imaging method for performing predetermined processing on an image signal of a subject, wherein the externally-dependent light wavefront modulation element is optically controlled by controlling the light wavefront modulation function to a non-expressed state and setting the optical system to a single focus state. After the light passing through the system is formed so as to form an image at different positions in the optical axis direction depending on the distance of the light source, a multi-focus state is formed by controlling the expression state.

  According to a fourth aspect of the present invention, there is provided an imaging method in which a subject image that has passed through an optical system including a light wavefront modulation element is captured by an imaging element, and predetermined processing is performed on an image signal of the subject from the imaging element. An externally dependent inverse light wavefront modulation element that is arranged in the optical path of the optical system and has an expression or non-expression of an inverse light wavefront modulation function that cancels out the light wavefront modulation function of the light wavefront modulation element, The externally dependent inverse light wavefront modulation element is controlled so that the inverse light wavefront modulation function is manifested, and the light passing through the optical system is imaged at different positions in the optical axis direction depending on the distance of the light source with the optical system as a single focus state. Then, the optical system is controlled to a non-expressing state to bring the optical system into a multi-focus state.

  According to the present invention, there is an advantage that a resolution performance peak position can be easily detected, the optical system can be simplified, cost can be reduced, and a restored image with less influence of noise can be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging element 120, an analog front end unit (AFE) 130, an image processing device 140, a camera signal processing unit 150, an image display memory 160, an image monitoring device 170, and an operation unit. 180, an exposure control device 190, and a modulation function control unit 200.

The optical system 110 supplies an image obtained by photographing the subject object OBJ to the image sensor 120.
The optical system 110 includes, for example, a first lens 111, a second lens 112, a diaphragm 113, an externally dependent light wavefront modulation element 114, a third lens 115, and a fourth lens 116 that are arranged in order from the object side OBJS.
The third lens 115 and the fourth lens 116 are cemented and function as an imaging lens for forming an image on the image sensor 120.
The externally dependent optical wavefront modulation element 114 depends on the outside for the expression and non-expression of the optical wavefront modulation function having the function of deforming the wavefront of the image formed on the light receiving surface of the image sensor 120 by the imaging lens.
When the light wavefront modulation function is controlled to the non-expressed state by the modulation function control unit 200, the externally dependent light wavefront modulation element 114 is configured such that the light passing through the optical system with the optical system 110 in a single focus state is a light source. The optical system 110 is in a multi-focal state when it has a good imaging performance such that an image is formed at a different position in the optical axis direction depending on the distance.

  2A and 2B are diagrams for explaining a configuration example and functions of the externally dependent optical wavefront modulation element according to the present embodiment.

As shown in FIGS. 2A and 2B, the externally dependent light wavefront modulation element 114 can be constituted by, for example, a liquid crystal element 114a.
The liquid crystal element 114a can change the light collection state by switching the voltage applied to the element.
For example, when a voltage is applied by the modulation function control unit 200, as shown in FIG. 2A, the light wavefront modulation function of the liquid crystal element 114a is controlled to be manifested, and the optical system 110 is in a multi-focus state.
On the other hand, when the voltage application is stopped (or set to a lower level than the manifestation state) by the modulation function control unit 200, as shown in FIG. 2B, the liquid crystal element 114a controls the light wavefront modulation function to the non-expression state. As a result, the optical system 110 is in a single focus state.

Further, as shown in FIG. 3, the optical system 110A, for example, as a second lens 112a, deforms the wavefront of image formation on the light receiving surface of the image sensor 120 by the imaging lens, for example, a phase plate having a three-dimensional curved surface, for example. The light wavefront modulation function of the light wavefront modulation element 112a is formed by a light wavefront modulation element (wavefront forming optical element) made of (Cubic Phase Plate), and instead of the externally dependent light wavefront modulation element It is also possible to arrange so that an externally dependent inverse light wavefront modulation element 117 whose expression or non-expression of the inverse light wavefront modulation function that cancels out the above depends on the outside is arranged.
In this case, the optical system 110A passes through the optical system in a one-focus state when the externally dependent inverse light wavefront modulation element 117 is controlled to the manifestation state of the inverse light wavefront modulation function by the modulation function control unit 200. It has a good imaging performance such that light is imaged at different positions in the optical axis direction depending on the distance of the light source, and when it is controlled to the non-expressed state, it becomes a multi-focus state.

  FIGS. 4A and 4B are diagrams for explaining a configuration example and a function of the externally dependent inverse light wavefront modulation element according to the present embodiment.

As shown in FIGS. 4A and 4B, the externally dependent inverse light wavefront modulation element 117 can also be constituted by, for example, a liquid crystal element 117a.
The liquid crystal element 117a can change the light collection state by switching the voltage applied to the element.
For example, when a voltage is applied by the modulation function control unit 200, as shown in FIG. 4A, the liquid crystal element 117a is controlled to be in a state where the inverse light wavefront modulation function is not developed, and the optical system 110A is in a multi-focus state. .
On the other hand, when the voltage application is stopped (or set to a lower level than the manifestation state) by the modulation function control unit 200, as shown in FIG. 4B, the liquid crystal element 117a controls the inverse light wavefront modulation function to the non-expression state. Thus, the optical system 110A is in a single focus state.

The imaging device 120 forms an image captured by the optical system 110, and outputs the primary image information of the image formation as the primary image signal FIM of the electrical signal to the image processing device 140 via the analog front end unit 130. And a CMOS sensor.
In FIG. 1, the image sensor 120 is described as a CCD as an example.

The analog front end unit 130 includes a timing generator 131 and an analog / digital (A / D) converter 132.
The timing generator 131 generates the drive timing of the CCD of the image sensor 120, and the A / D converter 132 converts an analog signal input from the CCD into a digital signal and outputs it to the image processing device 140.

An image processing apparatus (two-dimensional convolution means) 140 that constitutes a part of the image processing unit inputs a digital signal of a captured image coming from the previous AFE 130, performs two-dimensional convolution processing, and performs subsequent camera signal processing. Part (DSP) 150.
The image processing apparatus 140 performs a filter process on the optical transfer function (OTF), for example, according to the exposure information of the exposure control apparatus 190, and performs an image deterioration restoration process that improves chromatic aberration by correcting chromatic aberration. . Note that aperture information is included as exposure information.
The image processing device 140 has a function of generating a non-dispersed image signal from the subject dispersed image signal from the image sensor 120. The image processing unit has a function of performing noise reduction filtering in the first step.
The processing of the image processing apparatus 140 will be described in detail later.

  A camera signal processing unit (DSP) 150 performs processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing, and stores in the memory 160, displays an image on the image monitoring device 170, and the like. The image monitoring device 170 can perform color output or monochrome output under the control of the camera signal processing unit 150.

  The camera signal processing unit 150 cooperates with the exposure control device 190 and the modulation function control unit 200 to express or not express the light wavefront modulation function (reverse light wavefront modulation function) by the modulation function control unit 200, and filter (kernel). Link control with application switching, link with color output or monochrome output switching, link control with chromatic aberration correction function switching, and the like are performed.

  The exposure control device 190 performs exposure control, has operation inputs such as the operation unit 180, determines the operation of the entire system according to those inputs, controls the AFE 130, the image processing device 140, the DSP 150, etc. It governs mediation control of the entire system.

  The modulation function control unit 200 expresses or does not express the light wavefront modulation function (or reverse light wavefront modulation element) of the externally dependent light wavefront modulation element 114 (or reverse light wavefront modulation element 117) in accordance with an instruction from the camera signal processing unit 150. Is controlled by voltage.

  Hereinafter, the configuration and functions of the optical system and the image processing apparatus according to the present embodiment will be described in detail.

In this embodiment, the case where an externally dependent light wavefront modulation element or inverse light wavefront modulation element is used has been described. However, as the light wavefront modulation element or inverse light wavefront modulation element of the present invention, a wavefront deformation element is used. As long as the optical element changes in thickness (for example, the above-described third-order phase plate), the optical element changes in refractive index (for example, a gradient index wavefront modulation lens), An optical wavefront modulation element such as an optical element whose thickness and refractive index change by coding (for example, a wavefront modulation hybrid lens) and a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element) may be used.
In the example of FIG. 3, the case where a regularly dispersed image is formed using a phase plate that is a light wavefront modulation element has been described. However, a lens used as a normal optical system is similar to the light wavefront modulation element. When an image that can form a regularly dispersed image is selected, it can be realized only by an optical system without using a light wavefront modulation element. In this case, it does not correspond to the dispersion caused by the phase plate described later, but corresponds to the dispersion caused by the optical system.
The phase plate is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, a light beam having a deep depth (which plays a central role in image formation) and a flare (blurred portion) are formed by the light wavefront modulation element.

  Means for restoring the regularly dispersed image to a focused image without moving the optical system 110 or 110A by digital processing is a wavefront aberration control optical system or a depth expansion optical system (DEOS: Depth Expansion System). This is called an “optical system”, and this processing is performed in the image processing apparatus 140.

Here, the basic principle of DEOS will be described.
As shown in FIG. 5, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

(Equation 1)
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

(Equation 2)
f = H ⁻¹ * g

Here, the kernel size and calculation coefficient regarding H will be described.
Let the zoom positions be ZPn, ZPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In this embodiment, as shown in FIGS. 1 and 3, an image from the optical system 110 or 110 </ b> A is received by the image sensor 120 and input to the image processing device 140 to obtain a conversion coefficient corresponding to the optical system. Thus, an image signal having no dispersion is generated from the dispersion image signal from the image sensor 120 with the obtained conversion coefficient.

  In the present embodiment, as described above, dispersion refers to the formation of an image that does not fit anywhere on the image sensor 120 by inserting the light wavefront modulation elements 114 and 112a. This is a phenomenon that forms a deep light beam (which plays a central role in image formation) and flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

By the way, in DEOS (depth extension optical system), since the amount of change in image performance is small within a deep depth of field, it tends to be difficult to detect the optical performance peak position for defocusing. Since it is difficult to perform an appropriate evaluation, the manufacturing process is burdened, and there is a disadvantage in terms of yield.
Further, DEOS is not suitable for shooting with intentionally blurred background because the depth of the subject is lost or reduced due to the characteristics of the system.
Therefore, in the present embodiment, the resolution performance peak position detection method in DEOS (depth extension optical system) as described with reference to FIGS. 1 to 4 and an optical system that can be easily detected are provided. An optical system that loses the depth expansion function according to the application has been realized.
As described above, in the present embodiment, in the DEOS (depth extension optical system), the light wavefront modulation element is a liquid crystal element.
This liquid crystal element is characterized in that light wavefront (phase) modulation can be performed by switching a voltage applied to the element.
In the case of an optical system that does not include a light wavefront (phase) modulation action, or when the phase modulation action is canceled as an optical system, it becomes an optical system similar to a general optical system and can have good optical performance. Become. This also means that performance can be performed in the same way as a normal imaging optical system.
Then, by switching the phase modulation action of the liquid crystal element, it is possible to construct a manufacturing process that can easily adjust the optical performance by causing the depth expansion.

In the optical system 110 shown in FIG. 1, an externally dependent light wave whose appearance or non-expression of a light wavefront modulation function having a function of deforming a wavefront of an image formed on the light receiving surface of the image sensor 120 by an imaging lens depends on the outside. A surface modulation element 114 is provided.
When the light wavefront modulation function is controlled to the non-expressed state by the modulation function control unit 200, the externally dependent light wavefront modulation element 114 is configured such that the light passing through the optical system with the optical system 110 in a single focus state is a light source. The optical system 110 is in a multi-focal state when it has a good imaging performance such that an image is formed at a different position in the optical axis direction depending on the distance.

In this case, the optical system 110 has a good imaging performance such that the optical system that does not include the light wavefront modulation element forms an image at different positions in the optical axis direction depending on the distance of the light source. It is said. This means that performance can be performed in the same manner as a normal imaging optical system. Even if an optical wavefront modulation element having no power is added to the optical system 110, the imaging performance does not change. By employing an optical system with a bright F value, it is possible to take a photograph with a sense of depth.
In this state, the adjustment of the optical performance is in accordance with various existing evaluation methods.
Therefore, in the present embodiment, the voltage to the liquid crystal element 114a is changed to cause a light wavefront (for example) phase modulation effect. At this time, the liquid crystal element 114a is in a multi-focus state. By restoring the multi-focus state image, it is possible to realize a depth expansion effect.

  As described above, the imaging apparatus 100 having the optical system 110 in FIG. 1 changes the presence or absence of the light wavefront modulation function by switching the voltage applied to the liquid crystal element 114a. At this time, control is performed so as to link the light wavefront (phase) modulation and presence / absence of image restoration (switching of application of filter (kernel)).

  6A and 6B are diagrams illustrating an example in which the function of the filter (kernel) is switched simultaneously with the switching of the voltage applied to the liquid crystal element.

As shown in FIG. 6A, when the liquid crystal element 114a exhibits (has) a light wavefront modulation function (action), the filter functions to restore the image. Thereby, a restored image whose depth is extended is obtained.
On the other hand, as shown in FIG. 6B, when the liquid crystal element 114a does not exhibit the phase modulation function (action) (when it does not have), the filter does not function. As a result, an optical system in a good imaging state capable of general photographing is obtained.
With such a configuration, the focus adjustment of the optical system 110 is performed in a state where the liquid crystal element 114a does not exhibit the light wavefront modulation function, and the light passing through the optical system forms an image at different positions in the optical axis direction depending on the distance of the light source. After being formed as described above, it is possible to realize the effect of expanding the depth by maintaining the light wavefront modulation function in the expression state. Accordingly, it is possible to solve the problem that it is difficult to perform focus adjustment during manufacturing when the light wavefront modulation element is disposed.
Although the optical systems 110 and 110A in FIGS. 6 to 9 are partially omitted, they are assumed to have the same configuration as the optical systems 110 and 110A in FIGS.

In addition, the imaging apparatus 100 having the optical system 110 in FIG. 1 changes the presence or absence of the light wavefront modulation function by switching the voltage applied to the liquid crystal element 114a. Changes also occur.
Therefore, a color correction is performed on the monitor after realizing a good chromatic aberration correction state during general photographing, while a monochrome output is performed on the monitor during the DEOS system.

  FIGS. 7A and 7B are diagrams illustrating an example in which the output to the monitor is switched simultaneously with the switching of the voltage applied to the liquid crystal element.

As shown in FIG. 7A, when the liquid crystal element 114a exhibits (has) a light wavefront modulation function (action), monochrome output is performed.
On the other hand, as shown in FIG. 7B, when the liquid crystal element 114a does not exhibit the phase modulation function (action) (when it does not have), chromatic aberration is corrected by the optical system, and color output is output to the monitor. do.

  In addition, the imaging apparatus 100 having the optical system 110 in FIG. 1 changes the presence or absence of the light wavefront modulation function by switching the voltage applied to the liquid crystal element 114a. Changes also occur. Therefore, the switching of the voltage applied to the liquid crystal is linked to the presence or absence of the chromatic aberration correction function of the image processing unit.

  FIGS. 8A and 8B are diagrams illustrating an example in which the chromatic aberration correction function in the image processing unit is switched simultaneously with the switching of the voltage applied to the liquid crystal element.

As shown in FIG. 8A, when the liquid crystal element 114a exhibits (has) a light wavefront modulation function (action), it is affected by chromatic aberration. Therefore, the chromatic aberration correction function in the image processing unit is used and is good. Get a good picture.
On the other hand, as shown in FIG. 8B, when the liquid crystal element 114a does not exhibit the phase modulation function (action) (when it does not have), it is not affected by chromatic aberration, so the chromatic aberration correction function is not activated. .

Further, in the optical system 110A shown in FIG. 3, in the configuration having the optical wavefront modulation element 112a having the optical wavefront modulation function, the optical wavefront modulation function of the optical wavefront modulation element 112a is provided instead of the externally dependent optical wavefront modulation element. An externally dependent inverse light wavefront modulation element 117 in which the expression or non-expression of the inverse light wavefront modulation function to cancel is dependent on the outside is arranged.
In the optical system 110A, when the externally dependent inverse light wavefront modulation element 117 is controlled by the modulation function control unit 200 to exhibit the inverse light wavefront modulation function, the light passing through the optical system in a single focus state is a light source. With a good imaging performance such that an image is formed at a different position in the optical axis direction depending on the distance, a multi-focus state is obtained when the non-expression state is controlled.

In this case, it is assumed that an optical system that does not include the light wavefront modulation element 112a has a phase modulation function. Even if an optical wavefront modulation element having no power is added to this optical system, the imaging state does not change. The depth expansion effect is realized by restoring the image obtained at this time.
Therefore, the voltage to the liquid crystal element 117a is changed to cause reverse light wavefront modulation. Due to the canceling action between the light wavefront modulation element 112a and the liquid crystal element 117a, the optical system 110A has a good imaging performance such that light passing through the optical system forms an image at different positions in the optical axis direction depending on the distance of the light source.
In this state, adjusting the optical performance is in accordance with various existing evaluation methods. In addition, by adopting an optical system with a bright F value, it is possible to take a photograph with a sense of depth.

  As described above, the imaging apparatus 100A having the optical system 110A of FIG. 3 changes the presence or absence of the inverse light wavefront modulation function by switching the voltage applied to the liquid crystal element 117a. At this time, control is performed so as to link the light wavefront (phase) modulation and presence / absence of image restoration (switching of application of filter (kernel)).

  FIGS. 9A and 9B are diagrams illustrating an example in which the function of the filter (kernel) is switched simultaneously with the switching of the voltage applied to the liquid crystal element that exhibits the inverse light wavefront modulation function.

As shown in FIG. 9A, the liquid crystal element 117a does not exhibit the inverse light wavefront modulation function (action) (if it does not have) with respect to the optical system 110A having the light wavefront modulation function (action). ) Causes the filter to function and restore the image. Thereby, a restored image whose depth is extended is obtained.
On the other hand, as shown in FIG. 9B, for the optical system 110A having the light wavefront modulation function (action), the liquid crystal element 117a exhibits the inverse light wavefront modulation function (action), and the optical wavefront of the optical system. When canceling the modulation effect, the filter is not functioned. As a result, an optical system in a good imaging state capable of general photographing is obtained.
With such a configuration, after the liquid crystal element 117a adjusts the focus of the optical system 110A in a state where the inverse light wavefront modulation function is manifested to form a favorable imaging state, the liquid crystal element 117a is maintained in a state where the inverse light wavefront modulation function is not manifested. By doing so, it is possible to realize a depth expansion effect. Accordingly, it is possible to solve the problem that it is difficult to perform focus adjustment at the time of manufacturing when the light wavefront modulation element 112a is disposed. Further, the configuration after the image sensor 120 can be the same as that shown in FIGS.

In this embodiment, a resolution performance peak position detection method in DEOS and a lens frame structure (structure of an optical system holding unit) that can be easily detected are realized.
The following is an example.

As shown in FIG. 10, the lens frame structure unit 200 shown here basically includes a lens holding unit 210 and an image sensor holding unit 220 that are configured separately, and these lens holding unit 210 and image sensor holding unit 220. Is fixed by an intermediate member 230, and the linear expansion coefficients of the lens holding unit 210 and the image sensor holding unit 220 are different.
Further, the influence of this linear expansion coefficient is that the coefficient of the lens holding unit 210 is larger than the coefficient of the image sensor holding unit 220, and by controlling this coefficient, the back focus position shift is alleviated and the usage environment extends from low temperature to high temperature. Even so, it can be configured to ensure sufficient performance. Further, the DEOS (Depth Extension Optical System) is configured so that the temperature change of the depth of field can be reduced.

The lens holding unit 210 is formed in, for example, a cylindrical shape, and in order from the object side, the first holding unit 211 that holds the first lens 111, the second holding unit 212 that holds the second lens 112 (112a), and the liquid crystal element 114a. A third holding portion 213 that holds (117a), a fourth holding portion 214 that holds the third lens 115, and a fourth holding portion 215 that holds the fourth lens 116 are formed.
The object side from the axial center of the outer side of the lens holding part 210 is fixed to one end of the intermediate member 230 by, for example, an adhesive 240.
The lens holding part 210 is made of, for example, resin.

The image sensor holding unit 220 is formed in a cylindrical shape having an outer diameter larger than the outer diameter of the lens holding unit 210, the central part is opened in the axial direction, and the image sensor 120 is fixed to the bottom surface side (first surface side) 221. Has been.
Further, one end 231 of the intermediate member 230 is fixed to the upper surface side (object side surface) 222 of the image sensor holding unit 220 with an adhesive or the like.
The image sensor holding unit 220 is made of, for example, resin.

The intermediate member 2330 is formed in a cylindrical shape having an inner diameter larger than the outer diameter of the lens holding portion 210, and an adhesive is injected into one end portion of the inner wall 231 on the circumference when the lens holding portion 210 is fixed. 240 reservoirs 232 are formed.
Further, a flange portion 233 is formed at the other end portion of the intermediate member 230 so as to extend inward. The outer surface (bottom surface) of the flange portion 233 is in contact with the upper surface side 222 of the image sensor holding portion 220. Is fixed.
The intermediate member 230 is made of a metal material having a small linear expansion coefficient, such as aluminum (Al).

  As described above, in the lens frame structure unit 200, the image sensor holding unit 220 and the lens holding unit 210 are fixed and the optical system has a fixed focal point. The material of the lens holding unit 210 and the material of the image sensor holding unit 220 are as follows. By making the coefficient of linear expansion different from each other, a mechanism can be provided that can mitigate back focus position fluctuations due to temperature changes without having a drive mechanism.

  By making the linear expansion coefficient of the intermediate member 230 smaller than the linear expansion coefficients of the lens holding unit 210 and the image sensor holding unit 220, for example, the back focus position variation of the lens system due to temperature is small, and the back focus is sufficiently long. The relative positional fluctuation amount of each lens of the lens frame structure unit 200 can be suppressed with respect to the system.

  In the present embodiment, when the sum of the powers of the resin lenses (for example, the second lens 112) included in the optical system 110 (110A) is negative, the fourth lens 116, which is the final lens, on the image sensor 120 side. The interval between the surface and the image sensor 120 is configured to be shorter at a temperature higher than normal temperature and longer at a low temperature.

  In this embodiment, when the sum of the powers of the resin lenses included in the optical system 110 is positive, the distance between the imaging element 120 side surface of the fourth lens 116 as the final lens and the imaging element is higher than room temperature. It is configured to become longer and shorter at low temperatures.

  Further, the intermediate member 230 and the lens holding part 210 are fixed on the object side from the central part in the axial direction of the lens holding part 210.

  As described above, in this embodiment, the power of the resin lens is suppressed at the time of lens design to suppress the variation of the resin lens due to temperature, and the lens holding unit 210 and the image sensor holding unit 220 are separate. And it is comprised so that the performance deterioration of the back focus fluctuation | variation by temperature may be suppressed by changing the linear expansion coefficient of both members.

  Next, the configuration and processing of the image processing apparatus 140 will be described.

  As illustrated in FIG. 1, the image processing apparatus 140 includes a raw (RAW) buffer memory 141, a convolution calculator 142, a kernel data storage ROM 143 as a storage unit, and a convolution control unit 144.

  The convolution control unit 144 performs control such as on / off of the convolution process, screen size, and replacement of kernel data, and is controlled by the exposure control device 190.

The kernel data storage ROM 143 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. The exposure information is acquired and the kernel data is selected and controlled through the convolution control unit 144.
The exposure information includes aperture information.

  In the example of FIG. 11, the kernel data A is an F number (2.8) as aperture information, the kernel data B is F number (4), and the kernel data C is data corresponding to the F number (5.6). It has become.

As in the example of FIG. 11, the filtering process corresponding to the aperture information is performed for the following reason.
When shooting with the aperture stopped, the optical wavefront modulation element is covered by the aperture and the phase changes, making it difficult to restore an appropriate image.
Therefore, in the present embodiment, as in this example, appropriate image restoration is realized by performing filter processing according to aperture information in exposure information.

FIG. 12 is a flowchart of the switching process based on the exposure information (including aperture information) of the exposure control apparatus 190.
First, exposure information (RP) is detected and supplied to the convolution control unit 144 (ST101).
In the convolution control unit 144, the kernel size and the numerical performance coefficient are set in the register from the exposure information RP (ST102).
The image data captured by the image sensor 120 and input to the two-dimensional convolution operation unit 142 via the AFE 130 is subjected to a convolution operation based on the data stored in the register, and is calculated and converted. The transferred data is transferred to the camera signal processing unit 150 (ST103).

  The convolution operation is expressed by the following equation.

Here, f indicates a filter kernel (here, the one rotated 180 degrees is used for easy calculation).
A indicates an original image, and B indicates a filtered image (blurred restored image).
As can be seen from this equation, f is superimposed on the image and the result of summing the products of the taps is taken as the value of the center coordinate that has been superimposed.

  As described above, the convolution process is performed by the image processing apparatus 140. Pixel data from the image sensor 120 is subjected to convolution processing according to Equation 4.

First, the image processing apparatus 140 reads a blur coefficient by the phase modulation element by using the blur restoration filter by the phase modulation element, in other words, reads a filter coefficient for blur restoration by the phase modulation element from the kernel data storage ROM 143, and uses the filter coefficient. Perform the restoration process.
Then, the blur caused by the phase modulation element is improved after the processing.

  A more specific example of the signal processing unit and kernel data storage ROM of the image processing apparatus 140 will be described below.

FIG. 13 is a diagram illustrating a first configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 13 is a block diagram when a filter kernel corresponding to the exposure information is prepared in advance.

  Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144. The two-dimensional convolution operation unit 142 performs convolution processing using kernel data.

FIG. 14 is a diagram illustrating a second configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 14 is a block diagram when the signal processing unit has a noise reduction filter processing step at the beginning and noise reduction filter processing ST1 corresponding to the exposure information is prepared in advance as filter kernel data.

Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
In the two-dimensional convolution calculation unit 142, after applying the noise reduction filter ST1, the color space is converted by the color conversion process ST2, and then the convolution process ST3 is performed using the kernel data.
The noise process ST4 is performed again, and the original color space is restored by the color conversion process ST5. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
Note that the second noise processing ST4 can be omitted.

FIG. 15 is a diagram illustrating a third configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 15 is a block diagram when an OTF restoration filter corresponding to exposure information is prepared in advance.

Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
The two-dimensional convolution calculation unit 142 performs the convolution process ST13 using the OTF restoration filter after the noise reduction process ST11 and the color conversion process ST12.
Noise processing ST14 is performed again, and the original color space is restored by color conversion processing ST15. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
Note that only one of the noise reduction processes ST11 and ST14 may be performed.

FIG. 16 is a diagram illustrating a fourth configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 16 is a block diagram in the case where a noise reduction filter processing step is included and a noise reduction filter corresponding to exposure information is prepared in advance as filter kernel data.
Note that the second noise processing ST4 can be omitted.
Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
In the two-dimensional convolution operation unit 142, after performing the noise reduction filter process ST21, the color space is converted by the color conversion process ST22, and then the convolution process ST23 is performed using the kernel data.
The noise process ST24 corresponding to the exposure information is performed again, and the original color space is restored by the color conversion process ST25. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
The noise reduction process ST21 can be omitted.

  The example in which the filtering process is performed in the two-dimensional convolution calculation unit 142 according to only the exposure information has been described above. For example, the calculation coefficient suitable for combining subject distance information, zoom information, or shooting mode information and exposure information is suitable. Can be extracted or calculated.

FIG. 17 is a diagram illustrating a configuration example of an image processing apparatus that combines subject distance information and exposure information.
FIG. 12 shows an example of the configuration of the image processing apparatus 300 that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  As illustrated in FIG. 17, the image processing apparatus 300 includes a convolution apparatus 301, a kernel / numerical arithmetic coefficient storage register 302, and an image processing arithmetic processor 303.

  In this image processing apparatus 300, the image processing arithmetic processor 303 that has obtained the information about the approximate distance of the object distance of the subject read from the object approximate distance information detection apparatus 400 and the exposure information, performs an appropriate calculation on the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

As described above, in the case of an imaging apparatus having an optical wavefront modulation function, an image signal without proper aberration can be generated by image processing within a predetermined focal length range. In the case of outside, there is a limit to the correction of image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 400 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

The above image processing is performed by convolution calculation. To realize this, for example, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the focal length, The correction coefficient is used to correct the calculation coefficient, and an appropriate convolution calculation can be performed using the corrected calculation coefficient.
In addition to this configuration, the following configuration can be employed.

  The kernel size and the convolution calculation coefficient itself are stored in advance according to the focal length, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient according to the focal length is stored in advance as a function. In addition, it is possible to employ a configuration in which a calculation coefficient is obtained from this function based on the focal length and a convolution calculation is performed using the calculated calculation coefficient.

  When associated with the configuration of FIG. 17, the following configuration can be adopted.

At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 113 are stored in advance in the register 302 as the conversion coefficient storage means according to the subject distance. The image processing arithmetic processor 303 functions as a coefficient selection unit that selects a conversion coefficient corresponding to the distance from the register 302 to the subject based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit. .
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.

Alternatively, as described above, the image processing calculation processor 303 as the conversion coefficient calculation unit calculates the conversion coefficient based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit, and stores it in the register 302. Store.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 110 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 113 is stored in advance in the register 302 that also functions as the second conversion coefficient storage unit.
Then, based on the distance information generated by the object approximate distance information detection device 400 as the subject distance information generation means, the image processing arithmetic processor 303 as the correction value selection means performs a process from the register 302 as the correction value storage means to the subject. Select a correction value according to the distance.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 18 shows a configuration example of a filter in the case of using exposure information, object distance information, and shooting mode.
In this example, two-dimensional information is formed by object distance information and shooting mode information, and exposure information forms information such as depth.

FIG. 19 is a diagram illustrating a configuration example of an image processing apparatus that combines shooting mode information and exposure information.
FIG. 19 shows a configuration example of an image processing device 300B that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  Similarly to FIG. 17, the image processing apparatus 300 </ b> B includes a convolution apparatus 301, a kernel / numerical calculation coefficient storage register 302 as a storage unit, and an image processing calculation processor 303.

  In this image processing apparatus 300B, the image processing arithmetic processor 303 that has obtained the information about the approximate distance of the object distance of the subject read from the object approximate distance information detection apparatus 500 and the exposure information, calculates appropriate for the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

Also in this case, as described above, in the case of an imaging apparatus having a light wavefront modulation function, an image signal without proper aberration can be generated by image processing within a predetermined focal length range. When it is outside the focal length range, there is a limit to the correction of the image processing, so that only an object outside the range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection apparatus 500 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

  The above image processing is performed by convolution calculation. To realize this, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the object distance. A configuration in which a correction coefficient is used to correct a calculation coefficient and an appropriate convolution calculation is performed using the corrected calculation coefficient, a calculation coefficient corresponding to the object distance is stored in advance as a function, and the calculation coefficient is calculated from this function according to the focal length. The convolution calculation is performed using the calculated calculation coefficient, the kernel size and the convolution calculation coefficient are stored in advance according to the object distance, and the convolution calculation is performed using the stored kernel size and calculation coefficient. It is possible to adopt a configuration or the like.

  In the present embodiment, as described above, the image processing is changed according to the DSC mode setting (portrait, infinity (landscape), macro).

  Corresponding to the configuration of FIG. 19, the following configuration can be taken.

As described above, different conversion coefficients are stored in the register 302 as the conversion coefficient storage unit according to each shooting mode set by the shooting mode setting unit 600 of the operation unit 180 through the image processing arithmetic processor 303 as the conversion coefficient calculation unit. To do.
Based on the information generated by the object approximate distance information detecting device 500 as the subject distance information generating unit, the image processing arithmetic processor 303 according to the shooting mode set by the operation switch 601 of the shooting mode setting unit 600 converts the conversion coefficient. A conversion coefficient is extracted from the register 302 serving as a storage unit. At this time, for example, the image processing arithmetic processor 303 functions as conversion coefficient extraction means.
Then, the convolution device 301 serving as a conversion unit performs conversion processing according to the image signal shooting mode using the conversion coefficient stored in the register 302.

Also, the kernel data storage ROM of FIG. 1 is not necessarily used for the optical magnification, F number, and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.
As shown in FIG. 18, the storage amount increases by setting to three dimensions or even four dimensions or more, but a more suitable one can be selected in consideration of various conditions. The information may be the above-described exposure information, object distance information, imaging mode information, or the like.

As described above, in the case of an imaging apparatus having a light wavefront modulation element, an image signal without proper aberration can be generated by image processing within a predetermined focal length range. If it is out of the range, there is a limit to the correction of the image processing, so that only an object outside the range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

20A to 20C show spot images on the light receiving surface of the image sensor 120.
20A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 20B shows a case where the focal point is a focal point (Best focus), and FIG. 20C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 20A to 20C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (which plays a central role in image formation) and a flare (blurred portion) by the light wavefront modulation function. ) Is formed.

  As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

FIGS. 21A and 21B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens apparatus according to the present embodiment. FIG. 21A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 21B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 140 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. 21A and 21B. The MTF of the primary image is essentially a low value.

  As described above, the image processing apparatus 140 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like for raising the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing apparatus 140 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristics indicated by the curve B in FIG.
The characteristic indicated by curve B in FIG. 22 is a characteristic obtained when the wavefront is not deformed without exhibiting the optical wavefront modulation function, as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 22, in order to achieve the MTF characteristic curve B to be finally realized with respect to the MTF characteristic curve A with respect to the spatial frequency obtained optically, each spatial frequency is changed to each spatial frequency. On the other hand, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic of FIG. 22, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the image processing apparatus 140 that forms the primary image into a high-definition final image. In the optical system, an optical element capable of switching on / off of the light wavefront modulation function is newly provided, or an inverse light wavefront modulation element capable of canceling the light wavefront modulation function in an optical system having the light wavefront modulation function By deforming (modulating) the wavefront of imaging, such a wavefront is imaged on the imaging surface (light-receiving surface) of the imaging device 120 including a CCD or CMOS sensor, The image forming system obtains a high-definition image through the image processing apparatus 140.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. For this reason, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 140.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the target wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 140 made up of a later stage DSP or the like. Thereby, the MTF value is significantly improved.

  Next, the response of the MTF of this embodiment and the conventional optical system will be considered.

FIG. 24 is a diagram illustrating the MTF response when the object is at the focal position and when the object is out of the focal position in the case of a general optical system.
FIG. 25 is a diagram illustrating the response of the MTF when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 26 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, even when the object deviates from the focal position, the change in the MTF response is smaller than the optical diameter in which the light wavefront modulation element is not inserted.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

As described above, according to the present embodiment, the appearance or non-expression of the light wavefront modulation function having the function of deforming the wavefront of the image formed on the light receiving surface of the image sensor 120 by the imaging lens depends on the outside. An optical system 110 including a dependent light wavefront modulation element 114, an image sensor 120 that captures a subject image that has passed through the optical system 110, and image processing that performs predetermined processing on an image signal of the subject from the image sensor 120 The optical system 110 is in a one-focus state when the externally dependent light wavefront modulation element 114 is controlled to the non-expressed state by the modulation function control unit 200. When the state is controlled, a multi-focus state is obtained.
In addition, according to the present embodiment, the optical system 110A including the light wavefront modulation function, the image sensor 120 that captures the subject image that has passed through the optical system 110A, and predetermined processing on the image signal of the subject from the image sensor 120 An externally dependent inverse light wavefront modulation in which the appearance and non-expression of an inverse light wavefront modulation function that cancels the light wavefront modulation function of the light wavefront modulation element 112a depends on the outside. The element 117 is disposed, and the optical system 110A is in a one-focus state when the externally dependent inverse light wavefront modulation element 117 is controlled to be in an expression state by the modulation function control unit 200, and is in a non-expression state. When it is controlled to be in multiple focus state.
Therefore, the following effects can be obtained.

The resolution performance peak position can be easily detected, and there is an advantage that the depth extension function can be eliminated depending on the application while the optical system can be easily detected.
Further, since the focus adjustment uses the DEOS technology, a restored image in which the entire image is in focus can be provided. In particular, it can be said to be an optimal optical system for an electronic image equipment system such as a surveillance camera.

In addition, since the image processing device 140 performs the filtering process on the optical transfer function (OTF) according to the exposure information from the exposure control device 190, the optical system can be simplified and the cost can be reduced. Moreover, there is an advantage that a restored image that is less affected by noise can be obtained.
In addition, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation are made variable, know by input from the operation unit 180, etc. There is an advantage that the lens can be designed without worrying about the image and that the image can be restored by convolution with high accuracy.
Also, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large-sized optical lens and without driving the lens. There are advantages that can be made.
The imaging apparatus 100 according to the present embodiment can be used for DEOS of a zoom lens considering the small size, light weight, and cost of consumer devices such as a digital camera and a camcorder.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, the image processing apparatus 140 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, so that high-definition image quality can be obtained. There is an advantage of becoming.
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a figure for demonstrating the structural example and function of the external dependence type | mold optical wavefront modulation element which concern on this embodiment. It is a figure for demonstrating the other structural example of the optical system which concerns on this embodiment. It is a figure for demonstrating the structural example and function of the external dependence type | mold reverse light wavefront modulation element which concern on this embodiment. It is a figure for demonstrating the principle of DEOS. It is a figure which shows the example which switches the function of a filter (kernel) simultaneously with switching of the voltage concerning a liquid crystal element. It is a figure which shows the example which switches the output to a monitor simultaneously with the switching of the voltage concerning a liquid crystal element. It is a figure which shows the example which switches the chromatic aberration correction function in an image process part simultaneously with switching of the voltage concerning a liquid crystal element. It is a figure which shows the example which switches the function of a filter (kernel) simultaneously with the switching of the voltage concerning the liquid crystal element which expresses a reverse light wavefront modulation function. It is a figure which shows an example of the lens-frame structure which concerns on this embodiment. It is a figure which shows the other example (F number) of the storage data of kernel data ROM. It is a flowchart which shows the outline | summary of the optical system setting process of an exposure control apparatus. It is a figure which shows the 1st structural example about a signal processing part and a kernel data storage ROM. It is a figure which shows the 2nd structural example about a signal processing part and a kernel data storage ROM. It is a figure which shows the 3rd structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the 4th structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the structural example of the image processing apparatus which combines subject distance information and exposure information. It is a figure which shows the structural example of the filter at the time of using exposure information, object distance information, and zoom information. It is a figure which shows the structural example of the image processing apparatus which combines imaging | photography mode information and exposure information. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a case where it is a focusing point ( (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image pick-up element concerning this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image pick-up element of an image pick-up lens apparatus, (B ) Is a diagram showing an MTF characteristic with respect to a spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of a general optical system, and it remove | deviates from a focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 28A is a diagram showing a spot image on the light receiving surface of the image sensor of the imaging lens device of FIG. 27, where FIG. 27A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), and FIG. In the case (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100,100A ... Imaging device, 110, 110A ... Optical system, 120 ... Imaging element, 130 ... Analog front end part (AFE), 140 ... Image processing apparatus, 150 ... Camera Signal processing unit, 180 ... operation unit, 190 ... exposure control device, 200 ... modulation function control unit, 111 ... first lens, 112 ... second lens, 112a ... light wavefront Modulating element, 113... Aperture, 114... Externally dependent light wavefront modulating element, 114 a... Liquid crystal element capable of exhibiting a light wavefront modulating function, 115. Lens, 117... Externally dependent inverse light wavefront modulation element, 117a... Liquid crystal element capable of developing inverse light wavefront modulation function, 142... Convolution calculator, 143. Convolution control unit.

Claims (14)

An optical system including an externally dependent optical wavefront modulation element in which the expression or non-expression of the optical wavefront modulation function depends on the outside;
An image sensor that images a subject image that has passed through the optical system;
A modulation function control unit for controlling the expression and non-expression of the light wavefront modulation function of the externally dependent light wavefront modulation element;
An image processing unit that performs a predetermined process on the image signal of the subject from the image sensor,
The optical system is
The externally dependent light wavefront modulation element is in a single focus state when the light wavefront modulation function is controlled to the non-expressed state by the modulation function control unit, and is in a multi-focus state when controlled to the manifestation state. An imaging device. An optical system including an optical wavefront modulation element;
An externally dependent inverse light wavefront modulation element that is arranged in the optical path of the optical system and has an expression or non-expression of an inverse light wavefront modulation function that cancels the light wavefront modulation function of the light wavefront modulation element;
An image sensor that images a subject image that has passed through the optical system;
A modulation function controller for controlling the expression and non-expression of the light wavefront modulation function of the externally dependent inverse light wavefront modulation element;
An image processing unit that performs a predetermined process on the image signal of the subject from the image sensor,
The optical system is
The externally dependent inverse light wavefront modulation element is in a single focus state when the inverse light wavefront modulation function is controlled to be expressed by the modulation function control unit, and is in a multi-focus state when controlled to a non-expressed state. An imaging device. The image processing unit
A function of performing a filtering process on an optical transfer function (OTF) according to predetermined information;
The imaging apparatus according to claim 1, wherein switching between the expression and non-expression of the modulation function of the modulation function control unit is linked to switching of a filter that acts on the image processing unit. A monitor capable of displaying the image after the image processing of the image processing unit in color or black and white;
The imaging apparatus according to any one of claims 1 to 3, wherein switching between the expression and non-expression of the modulation function of the modulation function controller is linked to the switching of the color output and the monochrome output of the monitor output. The image processing unit
A function of correcting chromatic aberration by filtering the optical transfer function (OTF) according to predetermined information,
The imaging apparatus according to any one of claims 1 to 4, wherein switching between the on / off of the modulation function of the modulation function control unit is linked to the switching of the chromatic aberration correction function of the image processing unit. The imaging apparatus according to any one of claims 1 to 5, wherein the image processing unit performs the image restoration process using a plurality of types of restoration filters. The imaging apparatus according to claim 1, wherein the image processing unit includes a memory unit that stores a calculation coefficient of a filter. The imaging device according to claim 7, wherein the memory means stores a calculation coefficient for noise reduction processing according to exposure information. The imaging device according to claim 7, wherein the memory means stores a calculation coefficient for restoring an optical transfer function (OTF) according to exposure information. The imaging device
Subject distance information generating means for generating information corresponding to the distance to the subject,
The imaging device according to any one of claims 1 to 9, wherein the conversion unit generates an image signal that is less dispersed than the dispersed image signal based on information generated by the subject distance information generation unit. The imaging device
Subject distance information generating means for generating information corresponding to the distance to the subject;
Conversion coefficient calculation means for calculating a conversion coefficient based on the information generated by the subject distance information generation means,
The imaging device according to any one of claims 1 to 9, wherein the conversion unit converts an image signal by using the conversion coefficient obtained from the conversion coefficient calculation unit to generate an image signal without dispersion. The imaging device
Photographing mode setting means for setting the photographing mode of the subject to be photographed,
The imaging device according to any one of claims 1 to 9, wherein the conversion unit performs different conversion processing according to a shooting mode set by the shooting mode setting unit. A subject image that has passed through an optical system including an externally dependent light wavefront modulation element whose expression or non-expression of the light wavefront modulation function depends on the outside is captured by an image sensor, and a predetermined image signal of the subject from the image sensor is obtained. An imaging method that performs the process of
The externally dependent light wavefront modulation element,
After the light wavefront modulation function is controlled to the non-expressed state and the optical system is in a single focus state, the light passing through the optical system is formed so as to form an image at different positions in the optical axis direction depending on the distance of the light source,
An imaging method in which a multi-focus state is formed by controlling the expression state. An imaging method for imaging a subject image that has passed through an optical system including a light wavefront modulation element with an imaging element, and performing predetermined processing on an image signal of the subject from the imaging element,
An externally-dependent inverse light wavefront modulation element that is arranged in the optical path of the optical system and has an inverse light wavefront modulation function that cancels out the light wavefront modulation function of the light wavefront modulation element, the non-expression of which depends on the outside,
The externally dependent inverse light wavefront modulation element,
After the inverse light wavefront modulation function is controlled to the manifestation state and the optical system is in a single focus state, the light passing through the optical system is formed so as to form an image at different positions in the optical axis direction depending on the distance of the light source,
An imaging method in which the optical system is controlled to a non-expressing state to bring the optical system into a multifocal state.

Images