JP2008028454A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and imaging method which can be simplified of an optical system, which can be reduced to a cost, and whereby an image resulting from composing a blur image and a focused image can be obtained by one imaging and a restored image less affected by noise can be obtained. <P>SOLUTION: A signal processing section comprising an image processing apparatus 140 or the like applies prescribed processing to a dispersed image signal such as generation of an image signal without dispersion and an image signal with less dispersion from an object dispersed image signal from an imaging element 120, and composes an image (blur image) before the processing of the signal processing section with images (intermediate image and focused image) after the processing into a new image, which is displayed and recorded. In the case of the composition, an operation section 180 can establish a blur image region or a focused image region. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an imaging apparatus such as a digital still camera, a mobile phone camera, a mobile information terminal camera, an image inspection apparatus, and an industrial camera for automatic control using an image sensor and an optical system.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state image sensor, is used in most cases instead of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

  FIG. 38 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device. The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor. In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side. In the imaging lens device 1, as shown in FIG. 38, the best focus surface is matched with the imaging device surface.

39A to 39C show spot images on the light receiving surface of the imaging element 3 of the imaging lens device 1. In addition, an imaging apparatus has been proposed in which a light beam is regularly dispersed by a phase front (wavefront coding optical element) and restored by digital processing to enable imaging with a deep depth of field (for example, non-patent literature). 1, 2, and patent documents 1 to 5).
In addition, in photography with a camera, for example, an imaging method is known in which a portion other than the main subject is intentionally blurred by setting the aperture to the open side and focusing on the subject while reducing the depth of the subject. .
In addition, in order to obtain an image in which only the background is blurred without being constrained by the distance relationship between the subject and the background, an imaging technique is known in which images are captured at a plurality of focus positions and combined.
In addition, an automatic exposure control system for a digital camera that performs filter processing using a transfer function has been proposed (see, for example, Patent Document 6).
"Wavefront Coding; jointly optimized optical and digital imaging systems", Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. "Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems", Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A JP 2004-153497 A

In the imaging devices proposed in the above-mentioned documents, all of them are based on the assumption that the PSF (Point-Spread-Function) when the above-described phase plate is inserted into a normal optical system is constant, When the PSF changes, it is extremely difficult to realize an image with a deep depth of field by convolution using a subsequent kernel.
Therefore, apart from a single-focus lens, a zoom system, an AF system, or the like has a great problem in adopting due to the high accuracy of the optical design and the associated cost increase.
In other words, in the conventional imaging apparatus, proper convolution calculation cannot be performed, and astigmatism and coma that cause a shift of a spot (SPOT) image at the time of wide or tele (Tele). Therefore, an optical design that eliminates various aberrations such as zoom chromatic aberration is required.
However, the optical design that eliminates these aberrations increases the difficulty of optical design, causing problems such as an increase in design man-hours, an increase in cost, and an increase in the size of the lens.

  Further, in order to obtain an image that is blurred only in the background described above, in the imaging method in which images are captured at a plurality of focus positions and combined, imaging is performed a plurality of times by changing the focus position, and thus all imaging is completed. There is a problem that it takes a long time. In addition, with this photographing method, there is a problem that the subject or the object in the background moves and changes during photographing, and the combined image may become unnatural.

In addition, in the devices disclosed in the above-described documents, noise is also amplified simultaneously when an image is restored by signal processing, for example, in shooting in a dark place.
Therefore, for example, in an optical system including an optical system and signal processing using the above-described optical wavefront modulation element such as a phase plate and subsequent signal processing, noise is amplified when shooting in a dark place. This has the disadvantage of affecting the restored image.

  The object of the present invention is to simplify the optical system, to reduce the cost, and to obtain an image in which only a desired region is blurred or in-focus image from one imaging, and to reduce noise. An object of the present invention is to provide an imaging apparatus and an imaging method capable of obtaining a restored image having a small influence.

  An image pickup apparatus according to the present invention includes an image pickup device that picks up a subject dispersion image that has passed through at least an optical system and a light wavefront modulation device, an image signal that is less dispersed than a dispersed image signal from the image pickup device, and an image signal that has less dispersion. A signal processing unit that includes one or a plurality of conversion units that generate a plurality of different predetermined processes on an image signal from the image sensor, a blurred image before processing by the signal processing unit, and a focused image without dispersion after processing A recording unit that records an intermediate image with little dispersion after processing, and / or a new image obtained by synthesizing the blurred image, the focused image, or the intermediate image, and an image recorded in the recording unit, or for recording A display unit that displays the image, an operation unit that sets a range in the display unit and / or selects a blurred image, and an in-focus state within or outside the set range on the display unit by the operation unit An image is generated, and a range other than the focused image generation is generated by the intermediate image, or the intermediate image and the blurred image, and a stepwise blurred image in which the degree of blur gradually increases as the distance from the focused image increases. Generating means for synthesizing the focused image and the stepwise blurred image to generate a new image;

  Preferably, the optical system includes a zoom optical system, and further includes zoom information generation means for generating information corresponding to a zoom position or a zoom amount of the zoom optical system, and the conversion means includes the zoom information generation means. Based on the generated information, a non-dispersed image signal is generated from the dispersed image signal.

  Preferably, a subject distance information generation unit that generates information corresponding to a distance to the subject is provided, and the conversion unit is less dispersed than the dispersed image signal based on information generated by the subject distance information generation unit. An image signal is generated.

  Preferably, comprising: subject distance information generating means for generating information corresponding to the distance to the subject; and conversion coefficient calculating means for calculating a conversion coefficient based on the information generated by the subject distance information generating means, The conversion means converts the image signal using the conversion coefficient obtained from the conversion coefficient calculation means and generates an image signal without dispersion.

  Preferably, a photographing mode setting unit that sets a photographing mode of a subject to be photographed is provided, and the conversion unit performs different conversion processing according to the photographing mode set by the photographing mode setting unit.

  Preferably, the imaging device is capable of exchanging a plurality of lenses, and the imaging element is capable of imaging a subject aberration image that has passed through at least one of the plurality of lenses and the light wavefront modulation element, Furthermore, a conversion coefficient acquisition unit that acquires a conversion coefficient corresponding to the one lens is provided, and the conversion unit converts an image signal using the conversion coefficient obtained from the conversion coefficient acquisition unit.

  Preferably, an exposure control unit that performs exposure control is provided, and the signal processing unit performs a filtering process on an optical transfer function (OTF) in accordance with exposure information from the exposure control unit.

  According to the present invention, the optical system can be simplified, the cost can be reduced, and an image in which only a desired area is blurred by one shooting, or a restored image with a small influence of noise (that is, a focused image). Further, there is an advantage that a composite image of them can be obtained.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.
The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging element 120, an analog front end unit (AFE) 130, an image processing device 140, a camera signal processing unit 150, an image display memory 160, an image monitoring device 170, and an operation unit. 180 and an exposure control device 190.
The optical system 110 supplies an image obtained by photographing the subject object OBJ to the image sensor 120.
The optical system 110 of this embodiment is configured to include a light wavefront modulation element, as will be described in detail later.

The image pickup device 120 forms an image captured by the optical system 110 including a light wavefront modulation device, and performs image processing via the analog front end unit 130 using the formed primary image information as a primary image signal FIM of an electrical signal. It consists of a CCD or CMOS sensor that outputs to the device 140.
In FIG. 1, the image sensor 120 is described as a CCD as an example.

The analog front end unit 130 includes a timing generator 131 and an analog / digital (A / D) converter 132.
The timing generator 131 generates the drive timing of the CCD of the image sensor 120, and the A / D converter 132 converts an analog signal input from the CCD into a digital signal and outputs it to the image processing device 140.

An image processing apparatus (two-dimensional convolution means) 140 constituting a part of the signal processing unit inputs a digital signal of a captured image coming from the previous AFE 130, performs two-dimensional convolution processing, and performs subsequent camera signal processing. Part (DSP) 150.
The camera signal processing unit (DSP) 150 performs a filtering process on the optical transfer function (OTF) according to the exposure information of the image processing device 140 and the exposure control device 190.
The image processing apparatus 140 has a function of generating an image (focused image) signal that is less dispersed than an object dispersed image signal from the image sensor 120 and one or a plurality of image signals (intermediate image) with less dispersion. The signal processing unit has a function of performing noise reduction filtering in the first step.
The processing of the image processing apparatus 140 will be described in detail later.

  A camera signal processing unit (DSP) 150 performs processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing, and stores in the memory 160, displays an image on the image monitoring device 170, and the like.

  The exposure control device 190 performs exposure control, has operation inputs such as the operation unit 180, determines the operation of the entire system according to those inputs, controls the AFE 130, the image processing device 140, the DSP 150, etc. It governs mediation control of the entire system.

The imaging apparatus 100 according to the present embodiment has a plurality of shooting modes, for example, a portrait mode, a macro shooting mode (close-up), and a far-field shooting mode (infinity). It is possible to select and input by.
For example, as shown in FIG. 2, the operation unit 180 includes a MENU button 1801, a zoom button 1802, and a cross key 1803 arranged in the vicinity of a liquid crystal screen 1701 that is an image monitoring device 170 on the back side of the camera (imaging device) 100. Consists of.
The portrait mode is one of the shooting modes set according to the subject during normal shooting, and is a shooting mode suitable for human shooting, focusing on the central person and making the background image blurred. Is. Other modes that can be set include a sport mode, a sunset mode, a night view mode, a black and white mode, and a sepia mode.

  Each mode can be selected and set by the MENU button 1801 and the cross key 1803. In the present embodiment, the portrait mode is configured so that a portrait for horizontal shooting and a portrait for vertical shooting can be selected. Note that the mode switching may be a touch panel type on the liquid crystal screen 1701.

And the imaging device 100 in this embodiment has the following functions in order to make portrait photography easier.
That is, the signal processing unit formed by the image processing device 140, the DSP 150, and the exposure control device 190 generates an image signal that is less dispersed than the subject dispersed image signal from the image sensor 120 and an image signal that has less dispersion. Predetermined signal processing is performed on the dispersed image signal, and a generation function for synthesizing a new image by synthesizing the unprocessed image and the processed image of the signal processing unit is provided.
In this generation function, the background region generates a plurality of images by the blurred image processing, and generates a new image by synthesizing the focused image of the subject region including the main subject after the processing.

Furthermore, a recording function for recording an image before signal processing, a restored image after processing, and a new composite image, for example, in a memory buffer (not shown) or an image display memory 160 is provided.
Since this recording function is provided, the imaging apparatus 100 can obtain the following effects in addition to the effect that portrait photography can be easily performed due to the generation function.
In other words, by recording the image before signal processing and the image after processing, the position and size of the area that you want to make clear (clear) after shooting and recording (in contrast, the area that you want to blur) You can select and create a new image.
Therefore, a portrait photographed image can be created from a recorded image photographed in a mode other than the portrait mode at the time of photographing.

In the signal processing unit of the imaging apparatus 100 having such a function, a focused image of a subject area including a main subject is extracted from an image after the image restoration process, and a background area in contact with the subject area from the image before the image restoration process The out-of-focus image is extracted. The extracted in-focus image of the subject area and the out-of-focus image of the background area are synthesized to generate a new image. Then, the generated image is recorded.
In the present embodiment, the operation unit 180 also functions as a designation unit that allows the user to designate a subject area.

  Hereinafter, an example of a portrait image creation process according to the present embodiment will be described with reference to FIGS.

FIG. 3 is a diagram illustrating an example of creating a portrait image by performing restoration processing only on the shaded portion of the subject portion.
FIG. 4 is a diagram illustrating a central region in the horizontal photographing portrait mode.
FIG. 5 is a diagram illustrating a central region in the portrait shooting portrait mode.
FIG. 6 is a diagram illustrating an example in which the user selects a subject while the preview image is displayed, and the user determines the size and position of the frame indicating the center area using the cross key 1803.
FIG. 7 is a flowchart when the central area of the image is restored, and FIG. 8 is a flowchart when the selected area is restored.

<First example>
The analog signal obtained by the image sensor 120 is digitized by the image processing unit 140, converted into Y, Cb, and Cr signals by the DSP 150 and displayed as a through image on the image monitoring device 170 as a display unit.
When the portrait shooting portrait mode or the landscape shooting portrait mode is selected by the operation unit 180, as shown in FIG. 4 or 5, frames for vertical shooting and horizontal shooting are provided at the center of the captured image. Then, the user photographs a person in the frame.
Then, as shown in FIG. 3, a portrait image can be created by performing image processing only within the frame and restoring only the shaded portion of the subject portion.
Incidentally, whether vertical shooting or horizontal shooting may be automatically detected using an angular velocity sensor.

This processing operation will be described with reference to FIG. 7. When the portrait mode for vertical shooting or the portrait mode for horizontal shooting is set, the imaging device 100 starts the imaging operation by the image sensor 120 and displays it. The preview image is displayed on the image monitoring device 170 as a part (ST1).
When the user presses the shutter key while displaying the preview image (ST2), the image is recorded in the RAM of the buffer (ST3), and only the central area set in advance is restored (ST4), and the recording process is performed. (ST5).

<Second example>
In this case, after the photographing is completed, a portrait image can be created by selecting a subject in the preview image and performing image processing on that portion as shown in FIG.
This processing operation will be described with reference to FIG. 8. When the portrait mode for vertical shooting or the portrait mode for horizontal shooting is set, the imaging apparatus 100 starts the imaging operation by the image sensor 120 and displays it. The preview image is displayed on the image monitoring device 170 as a part (ST11).
Then, when the user presses the shutter key during the preview image display (ST12), the image is recorded in the RAM of the buffer (ST13), the preview image is displayed, and the user selects a subject by the operation unit 180 (ST14). ).
Then, image restoration processing of the selected area portion is performed (ST15), and restored image recording processing is performed (ST16).

Here, a specific example in which a focused image within or outside the setting range on the liquid crystal screen 1701 is generated by the operation unit 180, which is a feature of the present invention, and is combined with the blurred image to generate a new image will be described. .
FIG. 9 is a diagram showing a state in which an image that has been shot and recorded is displayed on the liquid crystal screen 1701. FIG. 9A is a diagram illustrating a state in which an image (blurred image) before signal processing is displayed.
The left side of FIG. 9B is a diagram showing a state in which all areas are designated by shading as an image (focused image) range after signal processing by operation of the operation unit 180, and the right side is the whole area designated by this designation. It is a figure which shows the state which displayed the focused image by.
The present invention is characterized in that the size and position of the range to be a focused image can be arbitrarily changed by the operation unit 180, and the left side of FIG. It is a figure which shows the state designated as an in-focus image range, and the right side is a figure which shows the state which made only the vicinity of a person a focused image by this designation | designated, and displayed the periphery with a blurred image. Note that the shape of the focused image range only needs to be selectable by the operation unit 180, and may be, for example, a trapezoidal shape or a quadrangular shape as illustrated in FIG.
Further, the left side of FIG. 9E is a diagram showing a state in which the lower right part is designated as the blurred image range by the operation unit 180, and the right side is a blurred image only in the lower right part (near the flower) by this designation. It is a figure which shows the state which displayed the other part by the focused image.

  Here, a specific method for specifying the range will be described. For example, the cursor (cross mark) on the liquid crystal screen 1701 is moved with the cross key 1803 to specify the center and radius to determine the circular shape, to specify three points to determine the circular shape, FIG. As shown in A), an elliptical shape may be determined by designating the center and two radii. FIG. 10 is a diagram showing a display state of the liquid crystal screen 1701.

  Alternatively, the polygonal shape may be determined by designating corners of the shape. Further, when the screen is divided into two, it may be determined that two points are specified and divided into two. For example, as shown in FIG. 10B, the trapezoidal shape may be determined by designating four points corresponding to the corners. Here, the arrows shown in FIGS. 10A and 10B indicate selection of whether the designated range is to be a blurred image or an in-focus image with the 10 o'clock key 1803 and / or the designated range. It shows that it is moved.

FIG. 11 shows a procedure by the operation unit 180.
As shown in FIG. 11A, first, the MENU button 1801 is pressed to display a menu in the liquid crystal screen 1701, and a range is selected using the cross key 1803 or the zoom button 1802. The selection of the range may be any of designation by points as shown in FIG. 11 and adjustment of the size by selecting the shape of the range prepared as a template in advance.
When designation by a point is selected, as shown in FIGS. 11B and 11C, an arrow appears on the liquid crystal screen 1701, and this arrow is moved with the cross key 1803 to press the center of the cross key 1803. The angle is determined as a U-shape. By repeating this four times, the state shown in FIG. 11C is obtained. Processing execution is selected by the zoom button 1802, and the center of the cross key 1803 is pressed. Then, although omitted in the drawing, it is selected whether the designated range is a blurred image or a focused image. When the designated range is an in-focus image, an image as shown in FIG. 11D is displayed, and a composite image of the blurred image and the in-focus image is also recorded in the image display memory 160.

  In the embodiments described so far, a case where a range (position / size) of a blurred image or a focused image is designated, and a new image is generated and recorded by combining the blurred image and the focused image is described. However, the present invention generates an intermediate image that is intermediate between the focused image and the blurred image, and the degree of blur gradually increases as the distance from the focused image increases due to the intermediate image and the blurred image, or a plurality of intermediate images. A blurred image is generated and recorded. The stepwise blurred image of the present invention may be processed automatically for the portion outside the range of the in-focus image, and the user may specify the number of steps with the operation unit 180.

  This intermediate image is an image with less dispersion, means an image that is less focused than the focused image, and is less blurred than the blurred image, and is completely the same process that generates the focused image. It can be generated by processing that does not generate a focused image, for example, by processing with a coefficient different from the coefficient for obtaining a focused image. The present invention is characterized in that a new image is synthesized by synthesizing the intermediate image after the signal processing and the focused image.

  With this generation function, it is possible to generate an intermediate image in the background area, generate a focused image in the subject area including the main subject, and synthesize these images to generate a new image. When a blurred image and a focused image are combined, a large difference in image quality may occur in the vicinity of the combined portion, resulting in an unnatural blur, but as described above, instead of a blurred image By using the intermediate image, the difference in image quality in the vicinity of the combined portion is alleviated, and a more natural blur can be obtained. Accordingly, the effect of the present invention can be obtained even when the blurred image of this embodiment is replaced with an intermediate image. Further, as shown in FIG. 12 (a), an intermediate image B is formed around the focused image A, and an intermediate image C or a blurred image C having a greater degree of blur than the intermediate image B is formed around the focused image A. It is possible to generate a stepwise blurred image in which the degree of blur gradually increases as the user moves away from the image, and the portrait captured image can have various variations. FIG. 12B is a diagram showing the degree of blur. The horizontal direction is the X direction in FIG. 12A, the vertical direction is the magnitude indicating the degree of blur, and the higher the level, the smaller the degree of blur. A state is shown in which a stepwise blurred image in which the degree of blurring increases in two steps as the distance from the focused image A increases.

  In the present embodiment, the case where the stepwise blurred image in which the degree of blurring increases in two steps has been described, but a multistage blurred image may be generated as shown in FIG. . Further, by increasing the number of intermediate images having different degrees of blur, a substantially linear blur degree change as shown in FIG. 13B may be obtained. Further, as shown in FIG. 13C, the degree of focus of the focused image can be enhanced by slightly increasing the degree of blurring in the vicinity of the focused image so as to have a curvilinear change. Furthermore, as shown in FIG. 13 (d), by gradually changing the degree of blurring in the vicinity of the focused image so as to have a curvilinear change, an impression of blurring can be given gradually.

  In the present invention, it is necessary to newly form one or a plurality of intermediate images. In this embodiment, the case where there is one signal processing unit 140, 150, 190 has been described. However, when an intermediate image is generated, two or more signal processing units 140, 150, 190, etc. may be provided. . By having two or more, it is possible to classify one as a signal processing unit for generating a focused image and the other as a signal processing unit for generating an intermediate image. The speed can be increased.

The imaging apparatus 100 according to the present embodiment further has a characteristic configuration in the optical system and the image processing apparatus as described below so that a person or the like can be more clearly recognized without being affected by camera shake or the like. have.




Hereinafter, the configuration and function of the optical system and the image processing apparatus according to the present embodiment will be specifically described.

FIG. 14 is a diagram schematically illustrating a configuration example of the zoom optical system 110 according to the present embodiment. This figure shows the wide angle side.
FIG. 15 is a diagram for explaining the principle of the depth extension optical system.
FIG. 16 is a diagram schematically illustrating a configuration example of the zoom optical system 110 according to the present embodiment. FIG. 17 is a diagram showing a spot shape at the center of the image height on the wide angle side, and FIG. 18 is a diagram showing a spot shape at the center of the image height on the telephoto side.

The zoom optical system 110 in FIG. 14 is disposed between an object side lens 111 disposed on the object side OBJS, an image forming lens 112 for forming an image on the image sensor 120, and between the object side lens 111 and the image forming lens 112. An optical wavefront modulation element (wavefront forming optical element: Wavefront) made of, for example, a phase plate (Cubic Phase Plate) having a three-dimensional curved surface, which deforms the wavefront of the image formation on the light receiving surface of the imaging element 120 by the imaging lens 112. Coding Optical Element) group 113 is included. A stop (not shown) is disposed between the object side lens 111 and the imaging lens 112.
In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-described third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change by coding on the lens surface (for example, wavefront modulation) A light wavefront modulation element such as a hybrid lens) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element) may be used.

The zoom optical system 110 in FIG. 14 is an example in which an optical phase plate 113a is inserted into a 3 × zoom used in a digital camera.
The phase plate 113a shown in the figure is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, the phase plate 113a forms a deep luminous flux (which plays a central role in image formation) and a flare (blurred portion).
Means for restoring the regularly dispersed image into a focused image by digital processing is called a depth-of-field expansion optical system, and this processing is performed in the image processing apparatus 140.

Here, the basic principle of this depth-of-field extended optical system will be described.
As shown in FIG. 15, when the subject image f enters the optical system H, an image g is generated.
This is expressed by the following equation.

(Equation 1)
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

(Equation 2)
f = H ^-1 * g
Here, the kernel size and calculation coefficient regarding H will be described.
Let the zoom positions be ZPn, ZPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In the present embodiment, as shown in FIG. 1, an image from the optical system 110 is received by the image sensor 120 and input to the image processing device 140, and a conversion coefficient corresponding to the optical system is acquired and acquired. An image signal having no dispersion is generated from the dispersion image signal from the image sensor 120 with a conversion coefficient.

  In the present embodiment, as described above, dispersion refers to forming a non-focused image on the image sensor 120 by inserting the phase plate 113a as described above. It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

Next, the configuration and processing of the image processing apparatus 140 will be described.
As illustrated in FIG. 1, the image processing apparatus 140 includes a raw (RAW) buffer memory 141, a convolution calculator 142, a kernel data storage ROM 143 as a storage unit, and a convolution control unit 144.

The convolution control unit 144 performs control such as on / off of the convolution process, screen size, and replacement of kernel data, and is controlled by the exposure control device 190.
Further, the kernel data storage ROM 143 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. The exposure information is acquired and the kernel data is selected and controlled through the convolution control unit 144.
In the example of FIG. 19, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

FIG. 20 is a flowchart of the switching process based on the exposure information of the exposure control device 190.
First, exposure information (RP) is detected and supplied to the convolution control unit 144 (ST21).
In the convolution control unit 144, the kernel size and the numerical performance coefficient are set in the register from the exposure information RP (ST22).
The image data captured by the image sensor 120 and input to the two-dimensional convolution operation unit 142 via the AFE 130 is subjected to a convolution operation based on the data stored in the register, and is calculated and converted. The transferred data is transferred to the camera signal processing unit 150 (ST23).

A more specific example of the signal processing unit and kernel data storage ROM of the image processing apparatus 140 will be described below.
FIG. 21 is a diagram illustrating a first configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 21 is a block diagram when a filter kernel corresponding to the exposure information is prepared in advance.

  Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144. The two-dimensional convolution operation unit 142 performs convolution processing using kernel data.

FIG. 22 is a diagram illustrating a second configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 22 is a block diagram when the signal processing unit has a noise reduction filter processing step at the beginning, and noise reduction filter processing ST31 corresponding to exposure information is prepared in advance as filter kernel data.

Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
In the two-dimensional convolution calculation unit 142, after applying the noise reduction filter ST31, the color space is converted by the color conversion process ST32, and then the convolution process ST33 is performed using the kernel data.
Noise processing ST34 is performed again, and the original color space is restored by color conversion processing ST35. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
It should be noted that the second noise processing ST34 can be omitted.

FIG. 23 is a diagram illustrating a third configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 23 is a block diagram in a case where an OTF restoration filter corresponding to exposure information is prepared in advance.
Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
The two-dimensional convolution operation unit 142 performs the convolution process ST43 using the OTF restoration filter after the noise reduction process ST41 and the color conversion process ST42.
Noise processing ST44 is performed again, and the original color space is restored by color conversion processing ST45. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
Note that only one of the noise reduction processes ST41 and ST44 may be performed.

FIG. 24 is a diagram illustrating a fourth configuration example of the signal processing unit and the kernel data storage ROM. For simplification, AFE and the like are omitted.
The example of FIG. 24 is a block diagram when a noise reduction filter processing step is included and a noise reduction filter corresponding to exposure information is prepared in advance as filter kernel data.
It is possible to omit the noise processing ST5 again.
Exposure information determined at the time of exposure setting is acquired, and kernel data is selected and controlled through the convolution control unit 144.
In the two-dimensional convolution operation unit 142, after performing the noise reduction filter process ST51, the color space is converted by the color conversion process ST52, and then the convolution process is performed using the kernel data in ST53.
Again, noise processing ST54 corresponding to the exposure information is performed, and the original color space is restored by color conversion processing ST55. The color conversion process includes, for example, YCbCr conversion, but other conversions may be used.
The noise reduction process ST51 can be omitted.

  The example in which the filtering process is performed in the two-dimensional convolution calculation unit 142 according to only the exposure information has been described above. For example, the calculation coefficient suitable for combining subject distance information, zoom information, or shooting mode information and exposure information is suitable. Can be extracted or calculated.

FIG. 25 is a diagram illustrating a configuration example of an image processing apparatus that combines subject distance information and exposure information.
FIG. 25 shows an example of the configuration of the image processing apparatus 300 that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 220.

As illustrated in FIG. 25, the image processing apparatus 300 includes a convolution apparatus 301, a kernel / numerical operation coefficient storage register 302, and an image processing operation processor 303.
In this image processing apparatus 300, the image processing arithmetic processor 303 that has obtained the information about the approximate distance of the object distance of the subject read from the object approximate distance information detection apparatus 400 and the exposure information, performs an appropriate calculation on the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

As described above, in the case of an imaging device including a phase plate (Wavefront Coding optical element) as an optical wavefront modulation element, an image signal without proper aberrations by image processing within the predetermined focal length range However, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the above range has an image signal with aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 400 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

  The image processing is performed by convolution calculation. To achieve this, for example, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the focal length, The correction coefficient is used to correct the calculation coefficient, and an appropriate convolution calculation can be performed using the corrected calculation coefficient.

In addition to this configuration, the following configuration can be employed.
The kernel size and the convolution calculation coefficient itself are stored in advance according to the focal length, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient according to the focal length is stored in advance as a function. In addition, it is possible to employ a configuration in which a calculation coefficient is obtained from this function based on the focal length and a convolution calculation is performed using the calculated calculation coefficient.

When associated with the configuration of FIG. 25, the following configuration can be adopted.
At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 213a are stored in advance in the register 302 as the conversion coefficient storage means in accordance with the subject distance. The image processing arithmetic processor 303 functions as a coefficient selection unit that selects a conversion coefficient corresponding to the distance from the register 302 to the subject based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit. .
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.
Alternatively, as described above, the image processing calculation processor 303 as the conversion coefficient calculation unit calculates the conversion coefficient based on the information generated by the object approximate distance information detection device 400 as the subject distance information generation unit, and stores it in the register 302. Store.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 210 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 213a is stored in advance in the register 302 that also functions as the second conversion coefficient storage unit.
Then, based on the distance information generated by the object approximate distance information detection device 400 as the subject distance information generation means, the image processing arithmetic processor 303 as the correction value selection means performs a process from the register 302 as the correction value storage means to the subject. Select a correction value according to the distance.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 26 is a diagram illustrating a configuration example of an image processing apparatus that combines zoom information and exposure information.
FIG. 26 shows an example of the configuration of the image processing apparatus 300A that generates a non-dispersed image signal from the subject dispersed image signal from the image sensor 220.
Similarly to FIG. 25, the image processing device 300A includes a convolution device 301, a kernel / numerical value operation coefficient storage register 302, and an image processing operation processor 303 as shown in FIG.

  In this image processing apparatus 300A, the image processing arithmetic processor 303 that has obtained information and exposure information regarding the zoom position or zoom amount read from the zoom information detection apparatus 500 uses the exposure information and the zoom position in an appropriate calculation. Then, the kernel size and its calculation coefficient are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that calculates using the value to restore the image.

  As described above, when a phase plate as a light wavefront modulation element is applied to an imaging apparatus provided in a zoom optical system, the spot image generated differs depending on the zoom position of the zoom optical system. For this reason, when a convolution calculation is performed on a defocus image (spot image) obtained from the phase plate by a DSP or the like at a later stage, different convolution calculations are required depending on the zoom position in order to obtain an appropriate focused image. Become.

Therefore, in the present embodiment, the zoom information detection apparatus 500 is provided, and is configured to perform an appropriate convolution calculation according to the zoom position and obtain an appropriate focused image regardless of the zoom position.
For proper convolution calculation in the image processing apparatus 300 </ b> A, one type of convolution calculation coefficient can be stored in the register 302 in common.

In addition to this configuration, the following configuration can be employed.
A configuration in which a correction coefficient is stored in advance in the register 302 in accordance with each zoom position, a calculation coefficient is corrected using the correction coefficient, and an appropriate convolution calculation is performed using the corrected calculation coefficient. Then, the kernel 302 and the convolution calculation coefficient itself are stored in the register 302 in advance, the convolution calculation is performed using the stored kernel size and calculation coefficient, and the calculation coefficient corresponding to the zoom position is stored in the register 302 as a function in advance. It is possible to adopt a configuration in which a calculation coefficient is obtained from this function based on the zoom position, and a convolution calculation is performed using the calculated calculation coefficient.

Corresponding to the configuration of FIG. 26, the following configuration can be taken.
At least two or more conversion coefficients corresponding to the aberration caused by the phase plate 213a corresponding to the zoom position or zoom amount of the zoom optical system 210 are stored in advance in the register 302 as the conversion coefficient storage means. A coefficient by which the image processing arithmetic processor 303 selects a conversion coefficient corresponding to the zoom position or the zoom amount of the zoom optical system 210 from the register 302 based on the information generated by the zoom information detecting device 400 as the zoom information generating means. It functions as a selection means.
Then, a convolution device 301 as a conversion unit converts an image signal using the conversion coefficient selected by the image processing arithmetic processor 303 as a coefficient selection unit.

Alternatively, as described above, the image processing arithmetic processor 303 as the conversion coefficient calculation means calculates the conversion coefficient based on the information generated by the zoom information detection device 400 as the zoom information generation means and stores it in the register 302.
Then, a convolution device 301 as a conversion unit converts an image signal using a conversion coefficient obtained by an image processing calculation processor 303 as a conversion coefficient calculation unit and stored in the register 302.

Alternatively, at least one correction value corresponding to the zoom position or zoom amount of the zoom optical system 210 is stored in advance in the register 302 serving as a correction value storage unit. This correction value includes the kernel size of the subject aberration image.
A conversion coefficient corresponding to the aberration caused by the phase plate 213a is stored in advance in the register 302 that also functions as the second conversion coefficient storage unit.
Then, based on the zoom information generated by the zoom information detecting device 400 serving as the zoom information generating unit, the image processing arithmetic processor 303 serving as the correction value selecting unit receives the zoom position of the zoom optical system from the register 302 serving as the correction value storing unit. Alternatively, a correction value corresponding to the zoom amount is selected.
The convolution device 301 serving as the conversion unit generates an image based on the conversion coefficient obtained from the register 302 serving as the second conversion coefficient storage unit and the correction value selected by the image processing arithmetic processor 303 serving as the correction value selection unit. Perform signal conversion.

FIG. 27 shows a configuration example of a filter when exposure information, object distance information, and zoom information are used.
In this example, two-dimensional information is formed by object distance information and zoom information, and exposure information forms information such as depth.

FIG. 28 is a diagram illustrating a configuration example of an image processing apparatus that combines shooting mode information and exposure information.
FIG. 28 shows a configuration example of an image processing device 300B that generates an image signal having no dispersion from the subject dispersion image signal from the image sensor 220.

  Similar to FIGS. 22 and 23, the image processing device 300B includes a convolution device 301, a kernel / numerical operation coefficient storage register 302 as a storage unit, and an image processing operation processor 303, as shown in FIG.

  In this image processing apparatus 300B, the image processing arithmetic processor 303 that has obtained information and exposure information related to the approximate object distance of the subject read from the approximate object distance information detection apparatus 600 performs an appropriate calculation for the object separation position. The kernel size and its calculation coefficient used in the above are stored in the kernel and numerical calculation coefficient storage register 302, and an appropriate calculation is performed by the convolution device 301 that uses the value to restore the image.

Also in this case, as described above, in the case of an imaging apparatus including a phase plate (Wavefront Coding optical element) as a light wavefront modulation element, an appropriate aberration is obtained by image processing within the predetermined focal length range. Although no image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of the image processing, so that only an object outside the range has an aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.
In this example, the distance to the main subject is detected by the object approximate distance information detection device 400 including the distance detection sensor, and different image correction processing is performed according to the detected distance.

The image processing is performed by convolution calculation. To achieve this, one type of convolution calculation coefficient is stored in common, and a correction coefficient is stored in advance according to the object distance. A configuration in which a correction coefficient is used to correct a calculation coefficient and an appropriate convolution calculation is performed using the corrected calculation coefficient, a calculation coefficient corresponding to the object distance is stored in advance as a function, and the calculation coefficient is calculated from this function according to the focal length. The convolution calculation is performed using the calculated calculation coefficient, the kernel size and the convolution calculation coefficient are stored in advance according to the object distance, and the convolution calculation is performed using the stored kernel size and calculation coefficient. It is possible to adopt a configuration or the like.
In the present embodiment, as described above, the image processing is changed according to the DSC mode setting (portrait, infinity (landscape), macro).

Corresponding to the configuration of FIG. 28, the following configuration can be taken.
As described above, different conversion coefficients are stored in the register 302 as the conversion coefficient storage unit according to each shooting mode set by the shooting mode setting unit 700 of the operation unit 180 through the image processing arithmetic processor 303 as the conversion coefficient calculation unit. To do.
Based on the information generated by the object approximate distance information detection device 402 as the subject distance information generation unit, the image processing arithmetic processor 303 according to the shooting mode set by the operation switch 701 of the shooting mode setting unit 700 converts the conversion coefficient. A conversion coefficient is extracted from the register 302 serving as a storage unit. At this time, for example, the image processing arithmetic processor 303 functions as conversion coefficient extraction means.
Then, the convolution device 301 serving as a conversion unit performs conversion processing according to the image signal shooting mode using the conversion coefficient stored in the register 302.

Note that the optical systems in FIGS. 14 and 16 are examples, and the present invention is not necessarily used for the optical systems in FIGS. 14 and 16. Also, FIG. 17 and FIG. 18 are only examples of the spot shape, and the spot shape of the present embodiment is not necessarily shown in FIG. 17 and FIG.
Also, the kernel data storage ROM of FIG. 19 is not necessarily used for the optical magnification and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.

  As shown in FIG. 27, the amount of storage is increased by using three dimensions or even four or more dimensions, but a more suitable one can be selected in consideration of various conditions. The information may be the above-described exposure information, object distance information, zoom information, imaging mode information, and the like.

As described above, in the case of an imaging device including a phase plate (Wavefront Coding optical element) as a light wavefront modulation element, there is no appropriate aberration by image processing within the predetermined focal length range. Although an image signal can be generated, if it is outside the predetermined focal length range, there is a limit to the correction of image processing, so that only an object outside the range has an aberration.
On the other hand, by performing image processing in which no aberration occurs within a predetermined narrow range, it is possible to bring out a blur to an image outside the predetermined narrow range.

  In this embodiment, it is possible to obtain a high-definition image quality by adopting a depth expansion optical system, and it is possible to simplify the optical system and reduce costs.

Hereinafter, this feature will be described.
FIGS. 29A to 29C show spot images on the light receiving surface of the image sensor 120.
29A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 26B shows a case where the focal point is a focal point (Best focus), and FIG. 25C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 29A to 29C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (a central role in image formation) is obtained by the wavefront forming optical element group 113 including the phase plate 113a. And flare (blurred part) are formed.
As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

FIGS. 30A and 30B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens device according to the present embodiment. FIG. 30A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 30B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 140 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. The MTF of the primary image is essentially a low value.

As described above, the image processing apparatus 140 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.
The MTF correction processing of the image processing apparatus 140 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristic indicated by the curve B in FIG.

The characteristic indicated by the curve B in FIG. 31 is a characteristic obtained when the wavefront is not deformed without using the wavefront forming optical element as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 31, in order to achieve the MTF characteristic curve B to be finally realized with respect to the optically obtained MTF characteristic curve A with respect to the spatial frequency, each spatial frequency is changed to each spatial frequency. On the other hand, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic of FIG. 31, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.
That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the image processing apparatus 140 that forms the primary image into a high-definition final image. In the optical system, a wavefront shaping optical element is newly provided, or an optical element such as glass, plastic or the like is formed for wavefront shaping, thereby forming an imaging wavefront. The wavefront is deformed (modulated), and an image of such a wavefront is formed on the image pickup surface (light receiving surface) of the image pickup device 120 formed of a CCD or CMOS sensor. An image forming system.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. For this reason, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 140.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the desired wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 140 made up of a later stage DSP or the like. Thereby, the MTF value is significantly improved.

Next, the response of the MTF of this embodiment and the conventional optical system will be considered.
FIG. 33 is a diagram showing the MTF response when the object is at the focal position and when the object is out of the focal position in the case of the conventional optical system.
FIG. 34 is a diagram showing the response of the MTF when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 35 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, even when the object deviates from the focal position, the change in the MTF response is smaller than the optical diameter in which the light wavefront modulation element is not inserted.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

  As described above, according to the present embodiment, the signal processing unit formed by the image processing device 140, the DSP 150, and the exposure control device 190 is an image signal that is less dispersed than the subject dispersed image signal from the image sensor 120, In addition, a predetermined signal processing is performed on the dispersed image signal such as generating an image signal with less dispersion, but the blurred image before processing of this signal processing unit, the intermediate image after processing, and the focused image after processing are synthesized. A generation function for synthesizing a new image. In this generation function, a step-by-step blur image in which the degree of blur gradually increases as the background area moves away from the focused image is generated, and the main subject after the processing is generated. The in-focus image of the subject area is synthesized to generate a new image, and the blurred image before the signal processing, the restored image (the intermediate image and the in-focus image) after the processing, and the synthesis It is possible to obtain the following effects because it has a recording function of recording a new image in the memory buffer and image display memory 160, not shown.

  You can easily shoot portraits, and by recording images before and after signal processing, you want to make them clear after shooting and recording (clearly, you want to blur) The position and size of the (area) can be selected to create a new image, and there is an advantage that a portrait photographed image can be created from a recorded image photographed in a mode other than the portrait mode at the time of photographing. Furthermore, there is an advantage that variations of the portrait mode can be increased.

  In addition, the image processing apparatus 140 includes an optical system 110 and an image sensor 120 that form a primary image, and an image processing device 140 that forms a primary image into a high-definition final image. Since the optical transfer function (OTF) is filtered according to the information, the optical system can be simplified, the cost can be reduced, and a restored image with less noise can be obtained. There is.

  In addition, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation are made variable, and the magnification and the defocus range can be set by making the appropriate kernel size and the above-described coefficient correspond to each other by inputting the operation unit 180 or the like. There is an advantage that the lens can be designed without concern and the image can be restored by convolution with high accuracy.

In addition, it is a so-called natural method that the object to be photographed is focused and the background is gradually blurred without the need for an expensive, large and expensive optical lens and without driving the lens. There is an advantage that an image can be obtained.
The imaging apparatus 100 according to the present embodiment can be used in a zoom lens depth extending optical system system in consideration of small size, light weight, and cost of consumer devices such as digital cameras and camcorders.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, the image processing apparatus 140 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, so that high-definition image quality can be obtained. There is an advantage of becoming.
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.
However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.

  The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.
Moreover, the optical system of FIG.14 and FIG.16 is an example, and this invention is not necessarily used with respect to the optical system of FIG.14 and FIG.16. Also, FIG. 17 and FIG. 18 are only examples of the spot shape, and the spot shape of the present embodiment is not necessarily shown in FIG. 17 and FIG.
Also, the kernel data storage ROM of FIG. 19 is not necessarily used for the optical magnification and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.
In the above embodiment, the case where there is one optical system has been described as an example, but the present invention can also be applied to an imaging apparatus having a plurality of optical systems.

FIG. 36 is a block diagram showing an embodiment of an imaging apparatus having a plurality of optical systems according to the present invention.
The optical unit 110A is different from the imaging device 100A in FIG. 1 in the imaging device 100A in FIG. 1 in that the optical unit 110A is replaced with a plurality of optical systems 110-1 and 110-2 by a system control device instead of the exposure control device 190. 200, and an optical system switching control unit 201 is further provided.

The optical unit 110A includes a plurality (2 in the present embodiment) of optical systems 110-1 and 110-2, and sequentially captures images of the subject object OBJ according to the switching process of the optical system switching control unit 201. This is supplied to the element 120.
The optical systems 110-1 and 110-2 have different optical magnifications, and optically capture an image of the imaging target object (subject) OBJ.

The system control device 200 basically has the same function as the exposure control device, has operation inputs such as the operation unit 180, and determines the operation of the entire system according to those inputs, and an optical system switching control unit 140, the AFE 130, the image processing apparatus 140, the DSP 150, and the like, and controls arbitration of the entire system.
Other configurations are the same as those in FIG.

FIG. 37 is a flowchart showing an outline of the optical system setting process of the system control apparatus 200.
First, the optical system is confirmed (ST61), and kernel data is set (ST62).
When an instruction to switch the optical system is given by operating the operation unit 180 (ST63), the optical system switching control unit 201 switches the output of the optical system of the optical unit 110A, and the process of step ST61 is performed (ST64).

According to the embodiment of FIG. 35, the following effects can be obtained in addition to the effects of the imaging apparatus of FIG.
36, the optical unit 110A and the image sensor 120 including a plurality of optical systems 110-1 and 110-2 having different magnifications for forming a primary image and the primary image are formed into a high-definition final image. In the image processing apparatus 140, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation are made variable according to the magnification of the optical system, and can be known by input from the operation unit 180 or the like. By adapting the appropriate kernel size according to the magnification of the optical system and the above-mentioned coefficients, the lens can be designed without worrying about the magnification and defocus range, and image reconstruction by high-precision convolution. There is an advantage that becomes possible.
Also, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large-sized optical lens and without driving the lens. There are advantages that can be made.
The imaging apparatus 100 according to the present embodiment can be used in a zoom lens depth extending optical system system in consideration of small size, light weight, and cost of consumer devices such as digital cameras and camcorders.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a figure which shows the structural example of the operation part which concerns on this embodiment. It is a figure which shows the example which restores only the shaded part of a to-be-photographed part, and produces a portrait image. It is a figure which shows the center area | region at the time of horizontal imaging | photography portrait mode. It is a figure which shows the center area | region at the time of portrait photography portrait mode. FIG. 10 is a diagram illustrating an example in which a user selects a subject while a preview image is displayed, and the user determines the size and position of a frame indicating a central region by an operation unit (key input unit). It is a flowchart in the case of restoring the central area of an image. It is a flowchart in the case of restoring the selected area. It is a figure which shows the state which designates the range of a focused image or a blurred image, and the display by it. It is a figure which shows the state which designates the range of a focused image or a blurred image, and the display by it. It is a figure which shows the procedure which designates the range of a focused image or a blurred image. It is a figure which shows the procedure until the range of a focused image or a blurred image is designated and displayed. It is a figure which shows the procedure until the range of a focused image or a blurred image is designated and displayed. It is a figure which shows the state which synthesize | combined the focused image and the stepwise blurred image by this invention. It is a figure which shows the various examples of the blurring degree of the stepwise blur image of this invention. It is a figure which shows typically the structural example of the zoom optical system of the wide angle side of the imaging lens apparatus which concerns on this embodiment. It is a figure for demonstrating the principle of a depth expansion optical system system. It is a figure which shows typically the structural example of the zoom optical system of the telephoto side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows the spot shape of the image height center on the wide angle side. It is a figure which shows the spot shape of the image height center of a telephoto side. It is a figure which shows an example of the storage data of kernel data ROM. It is a flowchart which shows the outline | summary of the optical system setting process of an exposure control apparatus. It is a figure which shows the 1st structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the 2nd structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the 3rd structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the 4th structural example about a signal processing part and kernel data storage ROM. It is a figure which shows the structural example of the image processing apparatus which combines subject distance information and exposure information. It is a figure which shows the structural example of the image processing apparatus which combines zoom information and exposure information. It is a figure which shows the structural example of the filter at the time of using exposure information, object distance information, and zoom information. It is a figure which shows the structural example of the image processing apparatus which combines imaging | photography mode information and exposure information. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a focus point ( Best focus), (C) is a diagram showing each spot image when the focal point is deviated by −0.2 mm (Defocus = −0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image pick-up element concerning this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image pick-up element of an image pick-up lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. 1 is a block configuration diagram showing an embodiment of an imaging apparatus having a plurality of optical systems according to the present invention. FIG. 33 is a flowchart showing an outline of an optical system setting process of the system control apparatus of FIG. 32. FIG. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 35 is a diagram showing a spot image on the light receiving surface of the image sensor of the imaging lens device of FIG. 34, where (A) shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), and (B) shows a focused point. In the case (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm).

Explanation of symbols

DESCRIPTION OF SYMBOLS 100,100A ... Imaging device 110 ... Optical system 110A ... Optical unit 120 ... Imaging element 130 ... Analog front end part (AFE)
DESCRIPTION OF SYMBOLS 140 ... Image processing apparatus 150 ... Camera signal processing part 180 ... Operation part 190 ... Exposure control apparatus 200 ... System control apparatus 201 ... Optical system switching control part 111 ... Object side lens 112 ... Imaging lens 113 ... Optical element 113a for wavefront formation ... Phase plates (light wavefront modulation elements)
142 ... Convolution calculator 143 ... Kernel data ROM
144: Convolution control unit.

Claims (7)

An image pickup device that picks up a subject dispersion image that has passed through at least the optical system and the light wavefront modulation device;
A signal processing unit that includes conversion means for generating one or a plurality of image signals that are less dispersed and less dispersed than the dispersed image signals from the image sensor, and that performs a plurality of different predetermined processes on the image signals from the image sensor When,
A blurred image before processing by the signal processing unit, a focused image without dispersion after processing, an intermediate image with little dispersion after processing, and / or a new image obtained by synthesizing the blurred image, focused image, or intermediate image A recording section for recording
A display unit for displaying an image recorded in the recording unit or an image for recording;
An operation unit for setting a range in the display unit and / or selecting a blurred image;
A focused image is generated within or outside the setting range on the display unit by the operation unit, and a range other than the focused image generation is generated by the intermediate image or the intermediate image and the blurred image. An imaging apparatus comprising: a generation unit that generates a stepwise blur image in which a degree of blur gradually increases with distance from an image, and generates a new image by combining the focused image and the stepwise blur image. The optical system includes a zoom optical system,
Zoom information generating means for generating information corresponding to the zoom position or zoom amount of the zoom optical system;
The imaging apparatus according to claim 1, wherein the conversion unit generates an image signal having less dispersion than the dispersed image signal based on information generated by the zoom information generation unit. Subject distance information generating means for generating information corresponding to the distance to the subject,
The imaging apparatus according to claim 1, wherein the conversion unit generates an image signal that is less dispersed than the dispersed image signal based on information generated by the subject distance information generation unit. Subject distance information generating means for generating information corresponding to the distance to the subject;
Conversion coefficient calculation means for calculating a conversion coefficient based on the information generated by the subject distance information generation means,
The imaging apparatus according to claim 1, wherein the conversion unit converts the image signal using the conversion coefficient obtained from the conversion coefficient calculation unit to generate an image signal without dispersion. A shooting mode setting means for setting a shooting mode of a subject to be shot;
The imaging apparatus according to claim 1, wherein the conversion unit performs different conversion processing according to a shooting mode set by the shooting mode setting unit. The imaging device can exchange a plurality of lenses,
The imaging element is capable of imaging a subject aberration image that has passed through at least one of the plurality of lenses and the light wavefront modulation element;
Conversion coefficient acquisition means for acquiring a conversion coefficient corresponding to the one lens;
The imaging apparatus according to claim 1, wherein the conversion unit converts an image signal using the conversion coefficient obtained from the conversion coefficient acquisition unit. With exposure control means for controlling exposure,
The imaging apparatus according to claim 1, wherein the signal processing unit performs a filtering process on an optical transfer function (OTF) in accordance with exposure information from the exposure control unit.

Images