JP2008011492A - Imaging device and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device and its method for maintaining a high frame rate, and for simplifying an optical system without requiring any expensive blur restoration processing hardware, and for reducing costs, and for obtaining a restored image not significantly affected by noise. <P>SOLUTION: This imaging device is provided with: a first filter 210 to be used for the image restoration processing of a conversion means in a static image photographic mode; a second filter 220 to be used for the image restoration processing of the conversion means in the case of moving image photographing and/or through-image display; and a second switching part 140-2. The first filter or the second filter is selected according to a photographic mode, and image restoration processing corresponding to the photographic mode is performed in an image processor 150. Thus, it is possible to perform image restoration processing in the case of capturing, previewing or moving image photographing in a real time, and to obtain a restored image which is in focus without deteriorating the restored image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera, a camera mounted on a mobile phone, a camera mounted on a portable information terminal, an image inspection apparatus, and an industrial camera for automatic control using an imaging element and an optical system.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state image sensor, is used in most cases instead of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

FIG. 36 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 36, the best focus surface is matched with the imaging element surface.
37A to 37C show spot images on the light receiving surface of the imaging element 3 of the imaging lens device 1.

In addition, an imaging apparatus has been proposed in which a light beam is regularly dispersed by a phase front (wavefront coding optical element) and restored by digital processing to enable imaging with a deep depth of field (for example, non-patent literature). 1, 2, and patent documents 1 to 5).
In addition, an automatic exposure control system for a digital camera that performs filter processing using a transfer function has been proposed (see, for example, Patent Document 6).
"Wavefront Coding; jointly optimized optical and digital imaging systems", Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. "Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems", Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A JP 2004-153497 A

In the imaging devices proposed in the above-mentioned documents, all of them are based on the assumption that the PSF (Point-Spread-Function) when the above-described phase plate is inserted into a normal optical system is constant, When the PSF changes, it is extremely difficult to realize an image with a deep depth of field by convolution using a subsequent kernel.
Therefore, apart from a single-focus lens, a zoom system, an AF system, or the like has a great problem in adopting due to the high accuracy of the optical design and the associated cost increase.
In other words, in the conventional imaging apparatus, proper convolution calculation cannot be performed, and astigmatism and coma that cause a shift of a spot (SPOT) image at the time of wide or tele (Tele). Therefore, an optical design that eliminates various aberrations such as zoom chromatic aberration is required.
However, the optical design that eliminates these aberrations increases the difficulty of optical design, causing problems such as an increase in design man-hours, an increase in cost, and an increase in the size of the lens.

In addition, in the devices disclosed in the above-described documents, noise is also amplified simultaneously when an image is restored by signal processing, for example, in shooting in a dark place.
Therefore, for example, in an optical system including an optical system and signal processing using the above-described optical wavefront modulation element such as a phase plate and subsequent signal processing, noise is amplified when shooting in a dark place. This has the disadvantage of affecting the restored image.

  In addition, the method of reducing the F value (aperture) for deepening the depth of field requires an aperture mechanism, or the lens becomes dark when the aperture is reduced. Is disadvantageous in that it tends to cause camera shake and subject shake.

  In addition, the method of performing out-of-focus (blur) restoration processing by image processing is very heavy, so that the necessary resources are always large and expensive hardware is required to perform out-of-focus restoration even during live view display. I need it. Alternatively, there is a restriction that the frame rate cannot be increased according to the processing capability, and there is a possibility that the shutter timing may be missed when the frame rate is considerably lowered.

For example, in the existing technology, the size of the filter used for the image restoration process is a fixed size regardless of the shooting mode, and the image restoration process at the time of moving image shooting is performed with the filter size of the image restoration process at the time of still image capture. There is a disadvantage that the processing time per frame becomes long and it becomes impossible to process in real time.
On the other hand, if the filter size is set so that real-time processing is possible at the time of live view display (preview), there is a disadvantage that the restored image at the time of still image capture is deteriorated.

  The object of the present invention is to simplify the optical system and reduce the cost without requiring expensive blur restoration processing hardware while maintaining a high frame rate. An object of the present invention is to provide an imaging apparatus and method that can realize processing without deteriorating a restored image and that can obtain a restored image that is less affected by noise.

  An image pickup apparatus according to a first aspect of the present invention includes an optical system formed so that a focal blur amount is substantially constant at an in-focus position and a distance before and after the in-focus position, and a subject image that has passed through the optical system. An image pickup device that picks up an image, a conversion unit that generates a restored image signal by correcting a defocus of an image from the image pickup device, a signal processing unit that performs predetermined image processing on the image signal, and a signal processing unit Image monitoring means for displaying a processed image, a first filter used for image restoration processing of the conversion means in still image shooting mode, and image restoration processing of the conversion means in moving image shooting mode and / or through image display A second filter to be used, and the conversion means selects the first filter or the second filter according to the shooting mode or presence / absence of through image display, and performs the image restoration process.

  An imaging apparatus according to a second aspect of the present invention includes an optical system formed so that a focal blur amount is substantially constant at an in-focus position and a distance before and after the in-focus position, and a subject image that has passed through the optical system. An image pickup device that picks up an image, a conversion unit that generates a restored image signal by correcting blurring of a focal point of an image from the image pickup device, and a first filter used for image restoration processing of the conversion unit in a still image shooting mode; A second filter used for the image restoration process of the conversion unit in the moving image shooting mode, and the conversion unit selects the first filter or the second filter according to the shooting mode, and the image restoration process I do.

  Preferably, the conversion means performs a simple restoration process using the second filter according to the resolution of the image monitor means.

  Preferably, the second filter includes an image restoration filter using a sampled PSF (Point Spread Function).

  Preferably, the second filter has a band enhancement function.

  Preferably, the size of the second filter is smaller than the size of the first filter.

  Preferably, there is setting means capable of setting whether or not the image restoration processing using the second filter is to be performed at the time of moving image shooting and through image display.

  Preferably, the optical system has a stop at or near the optical system, and the control means controls to stop the stop when displaying a through image.

  Preferably, the optical system includes an optical wavefront modulation element, and the control unit controls the diaphragm to a portion where the influence of the modulation surface of the optical wavefront modulation element can be eliminated.

  Preferably, trigger signal generating means for generating a trigger signal, and during live view display, an image signal subjected to image restoration processing by the second filter in the converting means is input to the signal processing means, and the trigger signal generating means And a control means for inputting, to the signal processing means, an image signal subjected to the image restoration process by the first filter in the conversion means.

  Preferably, the image processing apparatus includes recording means for recording an image, and the control means supplies an image signal subjected to image restoration processing by the second filter to the signal processing means for displaying the through image at the time of moving image shooting. When the image is input and the image is recorded in the recording unit, the conversion unit performs an image restoration process using the first filter.

  Preferably, the image processing apparatus includes recording means for recording an image, and the control means inputs an image signal subjected to image restoration processing by the second filter to the signal processing means at the time of moving image shooting, and the recording means To record.

  Preferably, a switching unit that selectively inputs an image signal from the image sensor or an image signal from the conversion unit to the signal processing unit, a trigger signal generation unit that generates a trigger signal, and the switching during a through image display. And a control means for inputting an image signal from the imaging device to the signal processing means through the means, and inputting an image signal from the conversion means to the signal processing means when receiving the trigger signal from the trigger signal generating means.

  Preferably, the image processing apparatus includes a recording unit that records an image, and the control unit performs signal processing on an image signal from the imaging element that has not performed the image restoration processing on the through image display during the moving image shooting. When the image is input to the recording unit and the image is recorded on the recording unit, an image restoration process is performed by the conversion unit.

  Preferably, the image processing apparatus includes a recording unit that records an image, and the control unit causes the signal processing unit to input an image signal from the imaging element that has not performed the image restoration process at the time of moving image shooting. An image that has not been subjected to image restoration processing is recorded in the recording means, and when the image is reproduced from the recording means, the image restoration processing by the conversion means is performed.

  Preferably, the amount of blur of the focus is set with the resolution of the image monitor means as an upper limit.

  Preferably, the resolution of the image monitoring means is not more than twice the pixel pitch of the image monitoring means.

  Preferably, the control means causes the image monitor means to display the image processed by the signal processing means after being restored by the conversion means for the still image captured by receiving the trigger signal.

  Preferably, the control means causes the image pickup device to output an image having a necessary and sufficient resolution as the resolution of the image monitoring means during the through image display.

  Preferably, the optical system includes a light wavefront modulation element and a zoom optical system, and includes zoom information generation means for generating information corresponding to a zoom position or a zoom amount of the zoom optical system, and the conversion means includes: Based on the information generated by the zoom information generating means, an image signal that is less dispersed than the dispersed image signal is generated.

  Preferably, there is subject distance information generating means for generating information corresponding to the distance to the subject, and the converting means is less dispersed than the dispersed image signal based on information generated by the subject distance information generating means. An image signal is generated.

  Preferably, it has subject distance information generating means for generating information corresponding to the distance to the subject, and conversion coefficient calculating means for calculating a conversion coefficient based on the information generated by the subject distance information generating means, The conversion means converts the image signal using the conversion coefficient obtained from the conversion coefficient calculation means and generates an image signal without dispersion.

  Preferably, there is provided a shooting mode setting means for setting a shooting mode of a subject to be shot, and the conversion means performs different conversion processing according to the shooting mode set by the shooting mode setting means.

  Preferably, the imaging apparatus is capable of exchanging a plurality of lenses, and the imaging element is capable of imaging a subject aberration image that has passed through at least one of the plurality of lenses and the light wavefront modulation element, Furthermore, it has a conversion coefficient acquisition means for acquiring a conversion coefficient corresponding to the one lens, and the conversion means converts an image signal using the conversion coefficient obtained from the conversion coefficient acquisition means.

  Preferably, exposure control means for performing exposure control is provided, and the signal processing means performs filter processing on an optical transfer function (OTF) in accordance with exposure information from the exposure control means.

  According to a third aspect of the present invention, an image pickup device picks up a subject image that has passed through an optical system formed so that a focal blur amount is substantially constant at a focus position and a distance before and after the focus position. During image display, predetermined image processing is performed on the image signal from the image sensor, and the focal point of the image from the image sensor is corrected and restored, and the predetermined image processing is performed on the restored image signal. In the still image shooting mode, the image restoration process is performed using the first filter, and in the moving image shooting mode and / or when the through image is displayed, the image restoration process is performed using the second filter having a different size from the first filter. Do.

  According to the present invention, the optical system can be simplified and the cost can be reduced without requiring expensive blur restoration processing hardware while maintaining a high frame rate. Further, there is an advantage that real-time processing according to the shooting mode can be realized without deteriorating the restored image, and that a restored image with less influence of noise can be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging element 120, an analog front end unit (AFE) 130, a first switching unit 140-1, a second switching unit 140-2, and an image processing device as a conversion unit. 150, a camera signal processing unit 160, an image display memory 170, an image monitoring device 180, an operation unit 190, an exposure control device 200 as a control means, and a first filter used for image restoration processing of the image processing device 150 in the still image shooting mode. 210 and a second filter 220 used for image restoration processing of the image processing apparatus 150 at the time of moving image shooting and / or display of a through image.

The optical system 110 supplies an image obtained by photographing the subject object OBJ to the image sensor 120.
As will be described later, the optical system 110 of the present embodiment includes a light wavefront modulation element, and is formed so that the amount of blurring of the focal point is substantially constant at the in-focus position and the distance before and after the in-focus position.
This focal blur amount is set with the resolution of the image monitoring device 180 as the upper limit.
The optical system 110 is provided with a variable stop 110a. The variable aperture 110a is controlled so as to limit, for example, when displaying a through image. More specifically, under the control of the exposure control device 200, when displaying a through image, it is narrowed down to a portion where the influence of the modulation effect of the modulation surface of the light wavefront modulation element included in the optical system 110 can be eliminated. The
In the present embodiment, the variable stop 110a is disposed in the optical system 110, but may be disposed in the vicinity of the subject side of the optical system 110.

The image sensor 120 forms an image captured by the optical system 110, and outputs a primary image information of the formed image as a primary image signal FIM of an electrical signal to the switching unit 140 via the analog front end unit 130. It consists of a CMOS sensor.
In FIG. 1, the image sensor 120 is described as a CCD as an example.

The analog front end unit 130 includes a timing generator 131 and an analog / digital (A / D) converter 132.
The timing generator 131 generates the drive timing of the CCD of the image sensor 120, and the A / D converter 132 converts the analog signal input from the CCD into a digital signal and outputs it to the switching unit 140.

The first switching unit 140-1 selectively inputs the captured image signal of the image sensor 120 via the AFE 130 to the image processing device 150 or the camera signal processing unit 160 in accordance with the switching control signal SCTL 1 from the exposure control device 200.
In the switching unit 140-1, the fixed contact a is connected to the output line of the A / D converter 132 of the AFE 130, the operation contact b is connected to the input line of the image processing device 150, and the operation contact c is connected to the camera signal processing unit 160. Connected to the signal input line.
For example, when the through image is displayed on the image monitoring device 180, the switching unit 140-1 is controlled so that the fixed contact a is connected to the operation contact c while the through image is being displayed. On the other hand, for example, when an image is captured and recorded in the memory 170, the fixed contact a is controlled to be connected to the operating contact b.

The second switching unit 140-2 connects the first filter 210 or the second filter 220 to the image processing device 150 according to the switching control signal SCTL2 from the exposure control device 200, and selectively uses the filter according to the imaging mode. Then, switching control is performed so as to perform two-dimensional convolution processing of the image processing apparatus 150 described later.
In the second switching unit 140-2, the fixed contact a is connected to the input / output line of the image processing apparatus 150, the operating contact b is connected to the first filter 210, and the operating contact c is connected to the second filter 220. .
The second switching unit 140-2 is controlled to connect the fixed contact a to the operation contact b so that the first filter 210 is used for image restoration processing of the image processing device 150 during still image shooting. On the other hand, the fixed contact a is controlled to be connected to the operating contact c so that the second filter 220 is used for image restoration processing of the image processing device 150 during moving image shooting and / or through image display.

In the present embodiment, the second filter 220 has image edge enhancement and band enhancement functions. As a filter having edge enhancement and band enhancement functions, for example, a low-pass filter, a high-pass filter, a convolution filter, and the like can be employed. The convolution filter is formed as an image restoration filter using a sampled PSF (Point Spread Function).
For example, in the image processing apparatus 150, during the live view display, the image processing apparatus 150 performs a so-called simple restoration process using the second filter 220 without executing the convolution operation using the first filter 210. .
In this embodiment, by performing a simple restoration process using a PSF sampled according to the resolution of the image monitoring device 180 as an image monitoring unit, an image quality corresponding to the image monitoring device 180 is obtained even if it is a through image. It is configured to be able to.

  In the present embodiment, the size of the second filter 220 is set to be smaller than the size of the first filter 210.

  Further, the exposure control apparatus 200 according to the present embodiment controls both the first switching unit 140-1 and the second switching unit 140-2 according to the operation content of the operation unit 190 functioning as a setting unit and a trigger signal generation unit. It is configured to perform the operation, or the first switching unit 140-1 is configured to connect the fixed contact a and the operating contact b without performing the switching control, or the second switching unit 140. -2 is configured so that only the first filter or only the second filter is used in the image restoration process in the image processing apparatus 150 without being operated, or both filters are not used, or when moving images are taken, At the time of display, it is possible to select various configurations and functions, such as setting whether to perform image restoration processing using a filter.

An image processing apparatus (two-dimensional convolution means) 150 functioning as a conversion means inputs a digital signal of a captured image coming from the AFE 130 at the previous stage, performs a two-dimensional convolution process, and performs a camera signal processing section (DSP) at the subsequent stage. To 160.
Filter processing is performed on the optical transfer function (OTF) according to the exposure information of the image processing device 150 and the exposure control device 200. Note that aperture information is included as exposure information.
The image processing device 150 generates a restored image signal by correcting the defocus of the image from the image sensor 120. More specifically, the image processing device 150 has a function of generating a non-dispersed image signal from the subject dispersed image signal from the image sensor 120. The image processing apparatus 150 has a function of performing noise reduction filtering in the first step (processing).
The processing of the image processing apparatus 150 will be described in detail later.

  The camera signal processing unit (DSP) 160 performs predetermined image processing such as color interpolation, white balance, YCbCr conversion processing, compression, filing, etc., and performs storage in the memory 170, image display on the image monitoring device 180, and the like.

The image monitoring device 180 is formed by a liquid crystal display device, for example, and displays a through image or a captured image that has undergone predetermined image processing by the camera signal processing unit 160.
The amount of blur of the light wavefront modulation element is set to be not more than twice the pixel pitch of the image monitoring device 180, for example.

  The memory 170 stores a captured image or the like that has undergone predetermined image processing by the camera signal processing unit 160.

The operation unit 190 includes an input switch that instructs the exposure control device 200 to perform predetermined function control. The operation unit 190 includes switches such as a shutter button, a release button, an enlarge button, a move key, a move key, a move key right, and a move key left.
For example, when the user operates the shutter button or the release button, the operation unit 190 generates a trigger signal instructing to capture a desired through image portion and record it in the memory 170 and output the trigger signal to the exposure control device 200. Has a signal generation function.

The exposure control device 200 performs exposure control, has operation inputs such as the operation unit 190, determines the operation of the entire system according to those inputs, controls the AFE 130, the image processing device 150, the DSP 160, etc. It governs mediation control of the entire system.
For example, during live view display, the exposure control apparatus 200 connects the fixed contact a and the operating contact c of the switching unit 140-1 by the switching control signal SCTL1, and the captured image signal of the image sensor 120 that has not been restored. Are directly input to the camera signal processing unit 160.
On the other hand, when the exposure control device 200 receives the trigger signal from the operation unit 190, the exposure control device 200 determines that the mode is a mode for capturing an image and recording the image in the memory 170, and the fixed control point of the switching unit 140-1 by the switching control signal SCTL1. a and the operation contact b are connected and input to the image processing apparatus 150 to perform the restoration process.
That is, the exposure control apparatus 200 receives the trigger signal, and after the image processing apparatus 150 restores the captured image, the camera signal processing section 160 performs predetermined image processing on the image that has been subjected to image monitoring. It is displayed on the device 180.
Further, for example, the exposure control device 200 causes the image sensor 120 to output an image having a necessary and sufficient resolution with the resolution of the image monitoring device 180 while displaying the through image on the image monitoring device 180 through the camera signal processing unit 160. An instruction is issued to the timing generator 131.

  In addition, the exposure control apparatus 200 according to the present embodiment uses the switching control signal SCTL2 to the second switching unit 140-2 in accordance with the operation of the operation unit 190 in the still image shooting mode. The fixed contact a is connected to the operating contact b so that the first filter 210 is used for the image restoration process, and the second filter 220 is used for the image restoration process of the image processing device 150 when shooting a moving image or displaying a live view. Control is performed so that the fixed contact a is connected to the operation contact c.

  In the present embodiment, depending on the operation content of the operation unit 190, the exposure control device 200 may display an image signal that has not been subjected to image restoration processing by the image processing device 150 for displaying a through image during moving image shooting. It is also possible to control the image processing apparatus 150 to perform the image restoration process when the image processing apparatus 150 inputs the signal to the signal processing unit 160 and records the image in the memory 170.

  Alternatively, depending on the operation content of the operation unit 190, the exposure control device 200 causes the camera signal processing unit 160 to input an image signal that has not been subjected to the image restoration process by the image processing device 150 during moving image shooting. It is also possible to record an image that has not been performed in the memory 170 and control the image processing apparatus 150 to perform an image restoration process when reproducing the image from the memory 170.

FIG. 2 is a diagram showing a basic control flow when used for image restoration processing in accordance with the shooting mode.
At the time of capture, image data is sent from the first switching unit 140-1 to the image processing apparatus 150.
In this case, since the photographing mode is capture, the first filter 210 is connected to the image processing apparatus 150 by the second switching unit 140-2, and the convolution calculation is performed using the first filter 210 (ST146, ST144).
As processing at the time of preview, when the user selects no image restoration processing by the operation unit 190, the first switching unit 140-1 passes through the image data from the image sensor and is directly transferred to the camera signal processing unit 160 to perform predetermined processing. (ST145).
When the restoration process is selected, image data is sent from the first switching unit 140-1 to the image processing apparatus 150. Since the shooting mode is preview, the second filter 220 for moving image shooting is selected. Convolution processing is performed using filters of different sizes at the time of capture, preview, and moving image shooting (ST143, ST144).

In addition, the exposure control device 200 controls the aperture 110a to be reduced by the control signal S200 when displaying a through image. More specifically, the exposure control device 200 uses the control signal S200 to move the aperture 110a to a portion where the influence of the modulation action of the modulation surface of the light wavefront modulation element (the influence of the phase plane) can be eliminated during the through image display. Control to squeeze.
In this way, when the live view is displayed, the exposure control device 200 controls the aperture 110a to be stopped by the control signal S200, so that it is not affected by the phase plane when the live view is displayed. Such an image can be displayed.
Note that the screen display becomes dark when the aperture 110a is stopped, but the brightness problem can be solved by increasing the gain of the screen display signal of the image monitoring apparatus 180.

  The basic function of the optical system according to the present embodiment and the basic processing according to the presence / absence of the blurred image restoration will be described below, and then the image processing that functions as the optical system 110 and the conversion unit, which are features of the present embodiment. A more specific configuration and function of the device 150 will be specifically described.

Here, a general imaging optical system in which light rays concentrate at the best focus (focal point) position will be described as a comparative example.
FIG. 3 is a diagram showing the relationship between the beam height and defocus of a general imaging optical system.
FIG. 4 is a diagram showing optical characteristics in the vicinity of image formation of the optical system of the present embodiment.

As shown in FIG. 3, in a general imaging optical system, light rays are most densely concentrated at the best focus position, and when the distance from the best focus position is increased, the blur diameter is approximately proportional to the defocus amount. spread.
On the other hand, as shown in FIG. 4, the optical system 110 of this embodiment differs from a general imaging system optical system in that light rays do not concentrate at the best focus position. ing. However, the optical system 110 according to the present embodiment is designed so that the change in the blur shape (PSF) is insensitive to the defocus amount in the vicinity of the best focus.
Therefore, if a blur restoration process according to the PSF at the best focus position is performed, a clear image can be obtained by removing the blur not only before and after the best focus point.
This is the principle of depth expansion adopted in this embodiment.

Next, the blurred image restoration process will be described.
FIG. 5 is an MTF (Modulation Transfer Function) characteristic diagram at the best focus position of a general optical system.
FIG. 6 is a diagram showing the MTF characteristics of the optical system of the present embodiment.

Since the picked-up image obtained by the image pickup device 120 after passing through the optical system 110 of the present embodiment is blurred, the MTF decreases from the middle range to the high range. This MTF is raised by calculation. In contrast to the MTF, which is the amplitude characteristic of a single lens, the total amplitude characteristic including image processing is called SFR (Spatial Frequency Response).
Since the frequency characteristic of a PSF that generates a blurred image is MTF, a blur restoration filter is designed to have a gain characteristic that is increased to a desired SFR characteristic. The degree of gain is determined by the balance with noise and false images.

  There are two methods for digitally filtering this blur restoration filter to the original image: Fourier transforming the image and multiplying the filter and frequency in the frequency domain, and convolution (convolution) in the spatial domain. is there. Here, the latter implementation method will be described. The convolution operation is expressed by the following equation.

Here, f indicates a filter kernel (here, the one rotated 180 degrees is used for easy calculation).
A indicates an original image, and B indicates a filtered image (blurred restored image).
As can be seen from this equation, f is superimposed on the image and the result of summing the products of the taps is taken as the value of the center coordinate that has been superimposed.

Next, a 3 * 3 filter will be specifically described with reference to FIGS. 7A to 7C.
The restoration filter of FIG. 7A (already rotated 180 degrees) is overlaid on the center A (i, j) of the blurred image shown in FIG. The product is taken and the nine total values are set as B (i, j) of the blurred restored image shown in FIG.
When (i, j) is scanned over the entire image, a new B image is generated. This is a digital filter. Here, since the filter is used for blur restoration, blur restoration processing can be performed by performing this processing.

Theoretically and technically, it is possible to display the image subjected to the blur restoration process even during the live view display by always applying this process.
However, if this convolution operation is processed in real time while maintaining a high rate such as 30 fps, a line buffer corresponding to the filter size and a large number of multipliers are required, resulting in high cost and high power consumption. Occurs and is not very realistic.
Therefore, in the present embodiment, the convolution calculation can be performed only when a capture execution trigger is input without performing the convolution calculation during an unnecessary period such as during live view display.

In this embodiment, an image is photoelectrically converted by the image sensor 120 through the optical system 110 and converted by the AFE 130 to generate digital image data.
During live view display, the image signal from the image sensor 120 is sent to the camera signal processing unit 160 by the switching unit 140 after passing the blurred image restoration processing of the image processing device 150.
For example, in the case of Bayer color, the camera signal processing unit 160 performs demosaic processing, white balance processing, gamma conversion, YUV conversion, and the like, and outputs an image to the monitor in real time.
Next, for example, when a release button of the operation unit 190 is operated and a trigger signal is input to the exposure control device 200, digital data from the image sensor 120 and the AFE 130 is input to the image processing device 150 by the switching unit 140. The camera signal processing unit 160 performs camera signal processing on the output through the above-described blurred image restoration processing. Although the camera signal processing has been performed, the image is recorded in the memory 170 or displayed on the image monitoring device 180.
If the enlarge button is pressed here, the captured image can be enlarged and displayed on the monitor, or the position of the enlarged portion can be changed by operating the movement key in this state.

  In the present embodiment, the blur width of the PSF is set with the resolution of the image monitoring device 180 as the upper limit. Therefore, the above-described blur restoration process is not performed in order to lighten the process during live view display. Even if it does, the monitor display which does not become conspicuous becomes possible.

  Furthermore, in the present embodiment, since the size of the blur width is set to about twice or less the pixel pitch of the image monitoring device 180, the live image during live view display is not substantially blurred.

The reason why the display is substantially unblurred when the image pitch is about twice or less is generally that when performing color display on a monitor device, R, G, and B, which are the three primary colors of light, are assigned to each pixel, and at least This is because color display is performed with these three pixels as one unit.
As shown in FIGS. 8A to 8D, the pixel arrangement includes a plurality of methods such as a stripe arrangement, a diagonal arrangement, a delta arrangement, and a lettering arrangement. A 2 × 2 size is configured as a minimum unit.
Therefore, if the resolution of the monitor display image is less than twice the pixel pitch, it can be said that the level is substantially satisfactory.

  In this embodiment, the blurred image is restored by the image processing device 150, and the blurred image is automatically output to the image monitoring device 180 when the camera signal processing is completed by the camera signal processing unit 160. Therefore, even if the blur restoration process takes a certain amount of time, it can be viewed at the same time as the process is completed, and there is no need for the photographer to perform a special operation for executing the restoration process.

  Further, in the present embodiment, during live view display, camera signal processing is performed without passing through an image processing device 150 serving as an image sensor 120 blur conversion unit for an image having a sufficiently small size with the resolution of the image monitoring device 180. Since the image is output to the image monitoring apparatus 180, the image processing is not burdened, and high-speed processing in real time is possible.

  Hereinafter, the configuration and function of the optical system and the image processing apparatus of the present embodiment will be described more specifically.

FIG. 9 is a diagram schematically illustrating a configuration example of the zoom optical system 110 according to the present embodiment. This figure shows the wide angle side.
FIG. 10 is a diagram schematically illustrating a configuration example of a zoom optical system on the telephoto side of the imaging lens device according to the present embodiment.
FIG. 11 is a diagram showing a spot shape at the center of the image height on the wide angle side of the zoom optical system according to the present embodiment, and FIG. 12 is a diagram of the center of the image height at the telephoto side of the zoom optical system according to the present embodiment. It is a figure which shows a spot shape.

9 and 10 includes an object side lens 111 disposed on the object side OBJS, an image forming lens 112 for forming an image on the image sensor 120, and between the object side lens 111 and the image forming lens 112. An optical wavefront modulation element (wavefront forming optics) made of a phase plate (Cubic Phase Plate) having a three-dimensional curved surface, for example, which deforms the wavefront of the image formation on the light receiving surface of the image sensor 120 by the imaging lens 112. Elements: Wavefront Coding Optical Element) group 113. A stop (not shown) is disposed between the object side lens 111 and the imaging lens 112.
For example, in this embodiment, a variable aperture is provided, and the aperture (aperture) of the variable aperture is controlled by exposure control (apparatus).

  In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-described third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change by coding on the lens surface (for example, wavefront modulation) A light wavefront modulation element such as a hybrid lens) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element) may be used.

The zoom optical system 110 in FIGS. 9 and 10 is an example in which an optical phase plate 113a is inserted into a 3 × zoom used in a digital camera.
The phase plate 113a shown in the figure is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, the phase plate 113a forms a deep luminous flux (which plays a central role in image formation) and a flare (blurred portion).
A means for restoring the regularly dispersed image to a focused image by digital processing is called a wavefront aberration control optical system system or a depth expansion optical system (DEOS), and this processing is an image. This is performed in the processing device 150.

Here, the basic principle of DEOS will be described.
As shown in FIG. 13, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

(Equation 2)
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

(Equation 3)
f = H ⁻¹ * g

Here, the kernel size and calculation coefficient regarding H will be described.
Let the zoom positions be ZPn, ZPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In the present embodiment, as shown in FIG. 1, an image from the optical system 110 is received by the image sensor 120 and input to the image processing device 150, and a conversion coefficient corresponding to the optical system is acquired and acquired. An image signal having no dispersion is generated from the dispersion image signal from the image sensor 120 with a conversion coefficient.

  In the present embodiment, as described above, dispersion refers to forming a non-focused image on the image sensor 120 by inserting the phase plate 113a as described above. It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

  FIG. 14 shows the shape of wavefront aberration expressed by the following equation, where the optical axis of the optical system including the light wavefront modulation element of the present embodiment is the z axis and two axes orthogonal to each other are x and y. is there.

In the range where the wavefront aberration is 0.5λ or less, the phase change is small, and the OTF is the same as that of a normal optical system. Therefore, it is only necessary to control so that the wavefront aberration is reduced to about 0.5λ when displaying a through image.
FIG. 15 shows the shape of the wavefront aberration and the range of 0.5λ or less by a bold line.
However, λ uses, for example, wavelengths in the visible light region and the infrared region.

  The shape shown in FIG. 14 is an example, and the phase of the light wavefront modulation element is represented by the following equation when the optical axis of the optical system is the z axis and the two axes orthogonal to each other are x and y. Anything is applicable.

  As described above, the image processing apparatus 150 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like for raising the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.

  Next, the configuration and processing of the image processing apparatus 150 will be described.

  As illustrated in FIG. 1, the image processing apparatus 150 includes a raw (RAW) buffer memory 151, a convolution calculator 152, a kernel data storage ROM 153 as a storage unit, and a convolution control unit 154.

  The convolution control unit 154 controls the on / off of the convolution process, the screen size, the replacement of kernel data, and the like, and is controlled by the exposure control apparatus 200.

Further, the kernel data storage ROM 153 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. 16 or FIG. Exposure information determined at the time of setting is acquired, and kernel data is selected and controlled through the convolution control unit 154.
The exposure information includes aperture information.

  In the example of FIG. 16, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

  In the example of FIG. 17, the kernel data A is the F number (2.8) as aperture information, the kernel data B is the data corresponding to the F number (4), and the kernel data C is the data corresponding to the F number (5.6). It has become.

As in the example of FIG. 17, the filtering process corresponding to the aperture information is performed for the following reason.
When shooting with the aperture stopped, the phase plate 113a forming the light wavefront modulation element is covered by the aperture and the phase changes, making it difficult to restore an appropriate image.
Therefore, in the present embodiment, as in this example, appropriate image restoration is realized by performing filter processing according to aperture information in exposure information.

FIG. 18 is a flowchart of the switching process based on the exposure information (including aperture removal) of the exposure control apparatus 200.
First, exposure information (RP) is detected and supplied to the convolution control unit 154 (ST101).
In the convolution control unit 154, the kernel size and the numerical operation coefficient are set in the register from the exposure information RP (ST102).
The image data captured by the image sensor 120 and input to the two-dimensional convolution operation unit 152 via the AFE 130 is subjected to convolution operation based on the data stored in the register, and is calculated and converted. The transferred data is transferred to the camera signal processing unit 160 (ST103).

  A more specific example of the signal processing unit and kernel data storage ROM of the image processing apparatus 150 will be described below.

FIG. 19 is a diagram illustrating a first configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 19 is a block diagram when a filter kernel corresponding to exposure information is prepared in advance.

  Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 154. The two-dimensional convolution operation unit 152 performs convolution processing using kernel data.

FIG. 20 is a diagram illustrating a second configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 20 is a block diagram in the case where a noise reduction filter processing step is provided at the beginning of the signal processing unit, and noise reduction filter processing ST1 corresponding to exposure information is prepared in advance as filter kernel data.

Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 154.
In the two-dimensional convolution operation unit 152, after applying the noise reduction filter ST1, the color space is converted by the color conversion process ST2, and then the convolution process ST3 is performed using the kernel data.
The noise process ST4 is performed again, and the original color space is restored by the color conversion process ST5. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
Note that the second noise processing ST4 can be omitted.

FIG. 21 is a diagram illustrating a third configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 21 is a block diagram when an OTF restoration filter corresponding to exposure information is prepared in advance.

Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 154.
The two-dimensional convolution operation unit 152 performs the convolution process ST13 using the OTF restoration filter after the noise reduction process ST11 and the color conversion process ST12.
Noise processing ST14 is performed again, and the original color space is restored by color conversion processing ST15. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
Note that only one of the noise reduction processes ST11 and ST14 may be performed.

FIG. 22 is a diagram illustrating a fourth configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 22 is a block diagram in the case where a noise reduction filter process step is included and a noise reduction filter corresponding to exposure information is prepared in advance as filter kernel data.
It is possible to omit the noise processing ST24 again.
Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 154.
In the two-dimensional convolution calculation unit 152, after performing the noise reduction filter process ST21, the color space is converted by the color conversion process ST22, and then the convolution process ST23 is performed using the kernel data.
The noise process ST24 corresponding to the exposure information is performed again, and the original color space is restored by the color conversion process ST25. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
The noise reduction process ST21 can be omitted.

  The example in which the two-dimensional convolution calculation unit 152 performs the filtering process according to only the exposure information has been described above. For example, the calculation coefficient suitable for combining subject distance information, zoom information, or shooting mode information and exposure information. Can be extracted or calculated.

FIG. 23 is a diagram illustrating a configuration example of an image processing apparatus that combines subject distance information and exposure information.
FIG. 23 shows a configuration example of an image processing apparatus 300 that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  As shown in FIG. 23, the image processing apparatus 300 includes a convolution apparatus 301, a kernel / numerical calculation coefficient storage register 302, and an image processing calculation processor 303.

  In this image processing apparatus 300, the image processing arithmetic processor 303 that has obtained the information about the approximate distance of the object distance of the subject read from the object approximate distance information detection apparatus 400 and the exposure information, performs an appropriate calculation on the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

As described above, in the case of an imaging device having a phase plate (Wavefront Coding optical element) as a light wavefront modulation element, an image signal without proper aberrations by image processing within the predetermined focal length range However, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 400 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

The image processing is performed by convolution calculation. To achieve this, for example, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the focal length, The correction coefficient is used to correct the calculation coefficient, and an appropriate convolution calculation can be performed using the corrected calculation coefficient.
In addition to this configuration, the following configuration can be employed.

  The kernel size and the convolution calculation coefficient itself are stored in advance according to the focal length, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient according to the focal length is stored in advance as a function. In addition, it is possible to employ a configuration in which a calculation coefficient is obtained from this function based on the focal length and a convolution calculation is performed using the calculated calculation coefficient.

  When associated with the configuration of FIG. 23, the following configuration can be adopted.

At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 113a are stored in advance in the register 302 as the conversion coefficient storage means according to the subject distance. The image processing arithmetic processor 303 functions as a coefficient selection unit that selects a conversion coefficient corresponding to the distance from the register 302 to the subject based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit. .
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.

Alternatively, as described above, the image processing calculation processor 303 as the conversion coefficient calculation unit calculates the conversion coefficient based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit, and stores it in the register 302. Store.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 110 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 113a is stored in advance in the register 302 that also functions as a second conversion coefficient storage unit.
Then, based on the distance information generated by the object approximate distance information detection device 400 as the subject distance information generation means, the image processing arithmetic processor 303 as the correction value selection means performs a process from the register 302 as the correction value storage means to the subject. Select a correction value according to the distance.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 24 is a diagram illustrating a configuration example of an image processing apparatus that combines zoom information and exposure information.
FIG. 24 shows a configuration example of an image processing device 300A that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  Similarly to FIG. 23, the image processing device 300A includes a convolution device 301, a kernel / numerical operation coefficient storage register 302, and an image processing operation processor 303 as shown in FIG.

  In this image processing apparatus 300A, the image processing arithmetic processor 303 that has obtained information and exposure information regarding the zoom position or zoom amount read from the zoom information detection apparatus 500 uses the exposure information and the zoom position in an appropriate calculation. Then, the kernel size and its calculation coefficient are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that calculates using the value to restore the image.

As described above, when the phase plate as the light wavefront modulation element is applied to the imaging apparatus having the zoom optical system, the spot image generated differs depending on the zoom position of the zoom optical system. For this reason, when a convolution calculation is performed on a defocus image (spot image) obtained from the phase plate by a DSP or the like at a later stage, different convolution calculations are required depending on the zoom position in order to obtain an appropriate focused image. Become.
Therefore, in the present embodiment, the zoom information detection apparatus 500 is provided, and is configured to perform an appropriate convolution calculation according to the zoom position and obtain an appropriate focused image regardless of the zoom position.

For proper convolution calculation in the image processing apparatus 300 </ b> A, one type of convolution calculation coefficient can be stored in the register 302 in common.
In addition to this configuration, the following configuration can be employed.

  A configuration in which a correction coefficient is stored in advance in the register 302 in accordance with each zoom position, a calculation coefficient is corrected using the correction coefficient, and an appropriate convolution calculation is performed using the corrected calculation coefficient. Then, the kernel 302 and the convolution calculation coefficient itself are stored in the register 302 in advance, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient corresponding to the zoom position is stored in the register 302 as a function in advance. It is possible to adopt a configuration in which a calculation coefficient is obtained from this function based on the zoom position, and a convolution calculation is performed using the calculated calculation coefficient.

  Corresponding to the configuration of FIG. 24, the following configuration can be taken.

At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 113a corresponding to the zoom position or zoom amount of the zoom optical system 110 are stored in advance in the register 302 as the conversion coefficient storage means. A coefficient by which the image processing arithmetic processor 303 selects a conversion coefficient corresponding to the zoom position or the zoom amount of the zoom optical system 110 from the register 302 based on the information generated by the zoom information detecting device 500 as the zoom information generating means. It functions as a selection means.
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.

Alternatively, as described above, the image processing arithmetic processor 303 as the conversion coefficient calculation unit calculates the conversion coefficient based on the information generated by the zoom information detection apparatus 500 as the zoom information generation unit and stores the conversion coefficient in the register 302.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 110 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 113a is stored in advance in the register 302 that also functions as a second conversion coefficient storage unit.
Then, based on the zoom information generated by the zoom information detecting device 500 serving as the zoom information generating unit, the image processing arithmetic processor 303 serving as the correction value selecting unit performs the zoom position of the zoom optical system from the register 302 serving as the correction value storing unit. Alternatively, a correction value corresponding to the zoom amount is selected.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 25 shows a configuration example of a filter when exposure information, object distance information, and zoom information are used.
In this example, two-dimensional information is formed by object distance information and zoom information, and exposure information forms information such as depth.

FIG. 26 is a diagram illustrating a configuration example of an image processing apparatus that combines shooting mode information and exposure information.
FIG. 26 illustrates a configuration example of an image processing device 300B that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 120.

  Like FIG. 23 and FIG. 24, the image processing device 300B includes a convolution device 301, a kernel / numerical operation coefficient storage register 302 as a storage unit, and an image processing operation processor 303, as shown in FIG.

  In this image processing apparatus 300B, the image processing arithmetic processor 303 that has obtained information and exposure information related to the approximate object distance of the subject read from the approximate object distance information detection apparatus 600 performs an appropriate calculation for the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

Also in this case, as described above, in the case of an imaging apparatus having a phase plate (Wavefront Coding optical element) as an optical wavefront modulation element, an appropriate aberration is obtained by image processing within the predetermined focal length range. Although no image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection apparatus 600 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

  The above image processing is performed by convolution calculation. To realize this, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the object distance. A configuration in which a correction coefficient is used to correct a calculation coefficient and an appropriate convolution calculation is performed using the corrected calculation coefficient, a calculation coefficient corresponding to the object distance is stored in advance as a function, and the calculation coefficient is calculated from this function according to the focal length. The convolution calculation is performed using the calculated calculation coefficient, the kernel size and the convolution calculation coefficient are stored in advance according to the object distance, and the convolution calculation is performed using the stored kernel size and calculation coefficient. It is possible to adopt a configuration or the like.

  In the present embodiment, as described above, the image processing is changed according to the DSC mode setting (portrait, infinity (landscape), macro).

  Corresponding to the configuration of FIG. 26, the following configuration can be taken.

As described above, different conversion coefficients are stored in the register 302 as the conversion coefficient storage unit according to each shooting mode set by the shooting mode setting unit 700 of the operation unit 190 through the image processing calculation processor 303 as the conversion coefficient calculation unit. To do.
Based on the information generated by the object approximate distance information detection device 600 as the subject distance information generation unit, the image processing arithmetic processor 303 according to the shooting mode set by the operation switch 701 of the shooting mode setting unit 700 converts the conversion coefficient. A conversion coefficient is extracted from the register 302 serving as a storage unit. At this time, for example, the image processing arithmetic processor 303 functions as conversion coefficient extraction means.
Then, the convolution device 301 serving as a conversion unit performs conversion processing according to the image signal shooting mode using the conversion coefficient stored in the register 302.

Note that the optical systems of FIGS. 9 and 10 are examples, and the present invention is not necessarily used for the optical systems of FIGS. Further, FIG. 11 and FIG. 12 are only examples of the spot shape, and the spot shape of the present embodiment is not necessarily shown in FIG. 11 and FIG.
Also, the kernel data storage ROM of FIGS. 16 and 17 is not necessarily used for the optical magnification, F number, and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.
As shown in FIG. 25, the amount of storage is increased by using three dimensions or even four or more dimensions, but a more suitable one can be selected in consideration of various conditions. The information may be the above-described exposure information, object distance information, zoom information, shooting mode information, and the like.

As described above, in the case of an imaging apparatus having a phase plate (Wavefront Coding optical element) as an optical wavefront modulation element, there is no appropriate aberration by image processing within the predetermined focal length range. Although an image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of image processing, so that only an object outside the range has an aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

FIGS. 27A to 27C show spot images on the light receiving surface of the image sensor 120.
27A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 27B shows a case where the focal point is a focal point (Best focus), and FIG. 27C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 27A to 27C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (a central role of image formation) is obtained by the wavefront forming optical element group 113 including the phase plate 113a. And flare (blurred part) are formed.

  As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

28A and 28B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens device according to the present embodiment. FIG. 28A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 28B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 150 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. 28 (A) and 28 (B). The MTF of the primary image is essentially a low value.

  As described above, the image processing apparatus 150 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like for raising the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing apparatus 150 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristics indicated by the curve B in FIG.
A characteristic indicated by a curve B in FIG. 29 is a characteristic obtained when the wavefront is not deformed without using the wavefront forming optical element as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 29, in order to achieve the MTF characteristic curve B to be finally realized with respect to the MTF characteristic curve A with respect to the spatial frequency obtained optically, each spatial frequency is On the other hand, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic of FIG. 29, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the image processing apparatus 150 that forms the primary image into a high-definition final image. In the optical system, a wavefront shaping optical element is newly provided, or an optical element such as glass, plastic or the like is formed for wavefront shaping, thereby forming an imaging wavefront. The image is deformed (modulated), and such a wavefront is imaged on the imaging surface (light-receiving surface) of the imaging device 120 made up of a CCD or CMOS sensor. An image forming system.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. Therefore, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 150.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the target wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 150 including a DSP or the like at the subsequent stage. Thereby, the MTF value is significantly improved.

  Next, the response of the MTF of this embodiment and the conventional optical system will be considered.

FIG. 31 is a diagram showing the MTF response when the object is at the focal position and when the object is out of the focal position in the case of the conventional optical system.
FIG. 32 is a diagram showing the MTF response when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 33 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, even when the object deviates from the focal position, the change in the MTF response is smaller than the optical diameter in which the light wavefront modulation element is not inserted.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

As described above, according to the present embodiment, the optical system 110 is designed so that the PSF (Point Spread Function point spread function) is substantially constant before and after the focusing position even though the in-focus position is not achieved. An image sensor 120 that captures an image of the system, an image processing device 150 that performs blur restoration processing, a camera signal processing unit 160 for obtaining a final image, and an image output from the camera signal processing unit 160 are displayed. An image monitoring device 180 for displaying a through image for determining a capture timing and a captured image, a first filter 210 used for image restoration processing of the conversion means in a still image shooting mode, A second filter 220 used for image restoration processing of the conversion means at the time of moving image shooting and / or display of a through image, and a second switching unit 140-2 , And the first filter or the second filter is selected according to the shooting mode, and the image restoration processing according to the shooting mode is performed in the image processing apparatus 150. Image restoration processing at the time of photographing can also be performed in real time, and a restored image in focus can be obtained without deterioration of the restored image.
Further, when image restoration processing is not required particularly during preview, for example, when no image restoration processing is selected during preview, power consumption of the optical system can be reduced.
In other words, according to the above configuration, the frame rate can be set at a high speed while outputting a good image with a size sufficient for a monitor display by performing a simple restoration process during a through image that determines the angle of view and the shutter timing. Can do. And since the restoration process is performed for the first time when it is captured, it may take a certain amount of processing time, so that a low-cost restoration process can be configured.

In addition, according to the present embodiment, the optical system 110 that is designed so that the PSF is substantially constant before and after the in-focus position, the image sensor 120 that captures an image of the optical system, and the blur restoration. An image processing apparatus 150 that performs processing, a camera signal processing unit 160 for obtaining a final image, and an image output from the camera signal processing unit 160 are displayed, and a through image is displayed to determine the angle of view and capture timing. The image monitoring device 180 that displays the captured image, the operation unit 190 that can generate a trigger signal for capturing the image, and the image processing device 15 that performs the blur restoration processing during the live view display. The image signal of the image sensor is directly input to the camera signal processing unit 160, and the image captured by receiving the trigger signal is converted to PSF by the image processing device 150. Since it has a control device 200 which controls so as to output to the camera signal processing unit 160 performs Flip blurring restoration processing, it is possible to obtain the following effects.
While displaying a through image that determines the composition, while maintaining a high frame rate, a certain amount of time can be spent on the blur restoration process at the time of capture, so there is no need for expensive hardware. In addition, even if you perform operations such as enlarging the capture image or moving the enlargement location, the full image of the captured image is restored so that it can be checked in detail and there is no unnaturalness in the operation and appearance. .

In addition, since the PSF blur width is set to about twice or less the pixel pitch of the image monitoring device 180, even if it is a through image that remains blurred, it is at the same level as the fineness of the monitor. There is no worrisome blur.
In addition, for an image captured upon receiving a trigger signal, an image is output to the monitor after performing camera processing after completion of blur restoration processing according to the PSF in the image processing apparatus 150 that performs blur restoration processing. I made it.
In addition, since the image sensor 120 outputs an image with a small image size level that is necessary for the monitor during live view display, a high frame rate can be realized during live view display, and there is room for resources. Can be reduced.
In addition, when displaying a through image, the diaphragm 110a is controlled so that it is not affected by the modulation action of the modulation surface of the light wavefront modulation element during the through image display. It can be displayed.

In addition, according to the present embodiment, the image processing apparatus 150 includes the optical system 110 and the image sensor 120 that form a primary image, and the image processing apparatus 150 that forms the primary image into a high-definition final image. Since the optical transfer function (OTF) is filtered according to the exposure information from the exposure control device 200, the optical system can be simplified, the cost can be reduced, and the influence of noise is small. There is an advantage that an image can be obtained.
In addition, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation can be made variable, know by input from the operation unit 190, etc. There is an advantage that the lens can be designed without worrying about the image and that the image can be restored by convolution with high accuracy.
Also, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large-sized optical lens and without driving the lens. There are advantages that can be made.
The imaging apparatus 100 according to the present embodiment can be used for DEOS of a zoom lens considering the small size, light weight, and cost of consumer devices such as a digital camera and a camcorder.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, the image processing apparatus 150 that forms a high-definition final image FNLIM by performing a predetermined correction process for so-called lifting of the MTF at the spatial frequency of the primary image, and so on, can achieve high-definition image quality. There is an advantage of becoming.
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.

Moreover, the optical system of FIG.9 and FIG.10 is an example, and this invention is not necessarily used with respect to the optical system of FIG.9 and FIG.10. Further, FIG. 11 and FIG. 12 are only examples of the spot shape, and the spot shape of the present embodiment is not necessarily shown in FIG. 11 and FIG.
Also, the kernel data storage ROM of FIGS. 16 and 17 is not necessarily used for the optical magnification, F number, and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.

  In the above embodiment, the case where there is one optical system has been described as an example, but the present invention can also be applied to an imaging apparatus having a plurality of optical systems.

  FIG. 34 is a block diagram showing an embodiment of an imaging apparatus having a plurality of optical systems according to the present invention.

  The optical unit 110A is different from the imaging device 100A in FIG. 1 in the imaging device 100A in FIG. 1 in that the optical unit 110A is replaced with a plurality of optical systems 110-1 and 110-2 by the system controller 200A instead of the exposure controller 200. And an optical system switching control unit 201.

The optical unit 110A includes a plurality (2 in the present embodiment) of optical systems 110-1 and 110-2, and sequentially captures images of the subject object OBJ according to the switching process of the optical system switching control unit 201. This is supplied to the element 120.
The optical systems 110-1 and 110-2 have different optical magnifications, and optically capture an image of the imaging target object (subject) OBJ.

  The system control device 200A basically has the same function as the exposure control device, has operation inputs such as the operation unit 190, and determines the operation of the entire system according to those inputs, and an optical system switching control unit 210, the AFE 130, the switching unit 140, the image processing apparatus 150, the DSP 160, and the like, and controls arbitration control of the entire system.

  Other configurations are the same as those in FIG.

FIG. 35 is a flowchart showing an outline of the optical system setting process of the system control apparatus 200A.
First, the optical system is confirmed (ST111), and kernel data is set (ST112).
When an instruction to switch the optical system is given by operating the operation unit 190 (ST113), the output of the optical system of the optical unit 110A is switched by the optical system switching control unit 201, and the process of step ST111 is performed (ST114).

According to the embodiment of FIG. 34, the following effects can be obtained in addition to the effects of the imaging apparatus of FIG.
That is, in the imaging apparatus of FIG. 34, the optical unit 110A and the imaging device 120 including the plurality of optical systems 110-1 and 110-2 having different magnifications for forming the primary image, and the primary image are formed into a high-definition final image. In the image processing apparatus 150, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation are made variable in accordance with the magnification of the optical system, and are known by input from the operation unit 190 or the like. By adapting the appropriate kernel size according to the magnification of the optical system and the above-mentioned coefficients, the lens can be designed without worrying about the magnification and defocus range, and image reconstruction by high-precision convolution. There is an advantage that becomes possible.
Also, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large-sized optical lens and without driving the lens. There are advantages that can be made.
The imaging apparatus 100A according to the present embodiment can be used for DEOS of a zoom lens in consideration of small size, light weight, and cost of consumer equipment such as a digital camera and a camcorder.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a figure which shows the basic control flow in the case of using for an image restoration process according to imaging | photography mode. It is a figure which shows the relationship between the light ray height of a general imaging optical system, and defocusing. It is a figure which shows the optical characteristic of image formation vicinity of the optical system of this embodiment. It is a MTF (Modulation Transfer Function amplitude characteristic) characteristic figure in the best focus (Best Focus) position of a general optical system. It is a figure which shows the MTF characteristic of the optical system of this embodiment. It is explanatory drawing of the blur decompression | restoration process in this embodiment. It is a figure which shows the example of the pixel arrangement | sequence of a monitor. It is a figure which shows typically the structural example of the zoom optical system of the wide angle side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows typically the structural example of the zoom optical system of the telephoto side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows the spot shape of the image height center on the wide angle side. It is a figure which shows the spot shape of the image height center of a telephoto side. It is a figure for demonstrating the principle of DEOS. It is a figure which shows the shape of the wavefront aberration represented by a type | formula, when the optical axis of the optical system containing the optical wavefront modulation element of this embodiment is set to az axis, and two mutually orthogonal axes are set to x and y. It is the figure which represented the shape of the wavefront aberration and the range below 0.5 (lambda) with the thick line. It is a figure which shows an example (optical magnification) of the storage data of kernel data ROM. It is a figure which shows the other example (F number) of the storage data of kernel data ROM. It is a flowchart which shows the outline | summary of the optical system setting process of an exposure control apparatus. It is a figure which shows the 1st structural example about a signal processing part and a kernel data storage ROM. It is a figure which shows the 2nd structural example about a signal processing part and a kernel data storage ROM. It is a figure which shows the 3rd structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the 4th structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the structural example of the image processing apparatus which combines subject distance information and exposure information. It is a figure which shows the structural example of the image processing apparatus which combines zoom information and exposure information. It is a figure which shows the structural example of the filter at the time of using exposure information, object distance information, and zoom information. It is a figure which shows the structural example of the image processing apparatus which combines imaging | photography mode information and exposure information. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a focus point ( Best focus), (C) is a diagram showing each spot image when the focal point is deviated by −0.2 mm (Defocus = −0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image pick-up element concerning this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image pick-up element of an image pick-up lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. 1 is a block configuration diagram showing an embodiment of an imaging apparatus having a plurality of optical systems according to the present invention. It is a flowchart which shows the outline | summary of the optical system setting process of the system control apparatus of FIG. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 37 is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens apparatus of FIG. 36, where (A) shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), and (B) shows a focused point. In the case (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Imaging device, 110 ... Optical system, 120 ... Image sensor, 130 ... Analog front end part (AFE), 140-1 ... Switching part, 140-2 ... 2nd switching part, 150 ... Image processing apparatus, 160 ... Camera Signal processing unit, 190 ... operation unit, 200 ... exposure control device, 200A ... system control device, 210 ... first filter, 220 ... second filter, 111 ... object side lens, 112 ... imaging lens, 113 ... for wavefront formation Optical element 113a, phase plate (light wavefront modulation element), 152, convolution calculator, 153, kernel data ROM, 154, convolution controller.

Claims (26)

An optical system formed so that the amount of defocusing is substantially constant at the in-focus position and the distance before and after the in-focus position;
An image sensor that images a subject image that has passed through the optical system;
Conversion means for generating a restored image signal by correcting the blur of the focus of the image from the image sensor;
Signal processing means for performing predetermined image processing on the image signal;
Image monitoring means for displaying a processed image of the signal processing means;
A first filter used for image restoration processing of the conversion means in the still image shooting mode;
A second filter used for image restoration processing of the conversion means in the moving image shooting mode and / or when displaying a through image,
The conversion unit performs the image restoration processing by selecting a first filter or a second filter in accordance with a shooting mode or presence / absence of a through image display. An optical system formed so that the amount of defocusing is substantially constant at the in-focus position and the distance before and after the in-focus position;
An image sensor that images a subject image that has passed through the optical system;
Conversion means for generating a restored image signal by correcting the blur of the focus of the image from the image sensor;
A first filter used for image restoration processing of the conversion means in the still image shooting mode;
A second filter used for image restoration processing of the conversion means in the moving image shooting mode,
The conversion unit selects the first filter or the second filter according to a shooting mode, and performs the image restoration process. The imaging apparatus according to claim 1, wherein the conversion unit performs a simple restoration process using the second filter according to the resolution of the image monitor unit. The imaging apparatus according to claim 3, wherein the second filter includes an image restoration filter using a sampled PSF (Point Spread Function). The imaging device according to claim 1, wherein the second filter has a band enhancement function. The imaging device according to claim 1, wherein a size of the second filter is smaller than a size of the first filter. The imaging apparatus according to claim 1, further comprising a setting unit that can set whether or not the image restoration process using the second filter is performed when the moving image is captured and a through image is displayed. Having a stop in or near the optical system,
The imaging apparatus according to any one of claims 1 to 7, wherein the control unit performs control so as to reduce the aperture when displaying a through image. Including an optical wavefront modulation element in the optical system;
The imaging apparatus according to claim 8, wherein the control unit controls the diaphragm to a portion where the influence of the modulation surface of the light wavefront modulation element can be eliminated. Trigger signal generating means for generating a trigger signal;
During live view display, the image signal that has undergone image restoration processing by the second filter in the conversion means is input to the signal processing means, and when the trigger signal is received by the trigger signal generation means, the conversion means uses the first filter. The image pickup apparatus according to claim 1, further comprising: a control unit that inputs an image signal subjected to the image restoration process to the signal processing unit. Having a recording means for recording an image;
The control means is configured to input an image signal subjected to image restoration processing by the second filter to the through image display during the moving image shooting, and to record the image in the recording means. The imaging apparatus according to claim 10, wherein the conversion unit performs image restoration processing using the first filter. Having a recording means for recording an image;
The imaging apparatus according to claim 10, wherein the control unit causes the signal processing unit to input an image signal subjected to the image restoration process by the second filter and record the image signal in the recording unit during the moving image shooting. A switching means for selectively inputting an image signal from the image sensor or an image signal from the conversion means to the signal processing means;
Trigger signal generating means for generating a trigger signal;
Control means for inputting an image signal from the imaging element to the signal processing means through the switching means during a through image display, and for inputting an image signal from the conversion means to the signal processing means when receiving a trigger signal from the trigger signal generating means The imaging apparatus according to any one of claims 1 to 9, further comprising: Having a recording means for recording an image;
The control unit is configured to input an image signal from the imaging element that has not performed the image restoration processing to the through image display and record the image on the recording unit during the moving image shooting. The imaging apparatus according to claim 13, wherein image conversion processing by the conversion unit is performed. Having a recording means for recording an image;
The control means causes the signal processing means to input an image signal from the imaging element that has not been subjected to the image restoration processing to the recording means at the time of shooting the moving image, and to the recording means. The imaging apparatus according to claim 13, wherein the image restoration process is performed by the conversion unit when recording and reproducing from the recording unit. The imaging apparatus according to any one of claims 1 to 15, wherein the focal blur amount is set with an upper limit of a resolution of the image monitoring unit. The imaging device according to any one of claims 1 to 15, wherein the focal blur amount is set to be equal to or less than twice a pixel pitch of the image monitor unit. 16. The control unit displays the image processed by the signal processing unit on the image monitoring unit after being restored by the conversion unit for a still image captured by receiving the trigger signal. The imaging device according to any one of the above. The imaging device according to any one of claims 10 to 15, wherein the control unit outputs an image having a necessary and sufficient resolution from the image monitoring unit during the through image display from the imaging element. The optical system includes a light wavefront modulation element and a zoom optical system,
Zoom information generating means for generating information corresponding to the zoom position or zoom amount of the zoom optical system;
The imaging device according to any one of claims 1 to 19, wherein the conversion unit generates a non-dispersed image signal from the dispersed image signal based on information generated by the zoom information generating unit. Subject distance information generating means for generating information corresponding to the distance to the subject;
The imaging device according to any one of claims 1 to 19, wherein the conversion unit generates a non-dispersed image signal from the dispersed image signal based on information generated by the subject distance information generating unit. Subject distance information generating means for generating information corresponding to the distance to the subject;
Conversion coefficient calculation means for calculating a conversion coefficient based on the information generated by the subject distance information generation means,
The imaging device according to any one of claims 1 to 19, wherein the conversion unit converts an image signal and generates a non-dispersed image signal using the conversion coefficient obtained from the conversion coefficient calculation unit. Having shooting mode setting means for setting the shooting mode of the subject to be shot;
The imaging device according to any one of claims 1 to 22, wherein the conversion unit performs different conversion processing according to a shooting mode set by the shooting mode setting unit. The imaging device can exchange a plurality of lenses,
The imaging element is capable of imaging a subject aberration image that has passed through at least one of the plurality of lenses and the light wavefront modulation element, and
Conversion coefficient acquisition means for acquiring a conversion coefficient corresponding to the one lens;
The imaging device according to any one of claims 1 to 23, wherein the conversion unit converts an image signal using the conversion coefficient obtained from the conversion coefficient acquisition unit. Exposure control means for performing exposure control,
The imaging apparatus according to any one of claims 1 to 24, wherein the signal processing unit performs a filtering process on an optical transfer function (OTF) in accordance with exposure information from the exposure control unit. An image sensor captures a subject image that has passed through an optical system formed so that the amount of focal blur is substantially constant at the in-focus position and the distance before and after the in-focus position;
During live view display, predetermined image processing is performed on the image signal from the image sensor,
When correcting and restoring the blur of the focus of the image from the image sensor, and performing the predetermined image processing on the restored image signal,
In still image shooting mode, perform image restoration using the first filter,
An image capturing method for performing image restoration processing using a second filter having a size different from that of the first filter in a moving image shooting mode and / or in a through image display.

Images