JP2009033607A - Imaging apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of obtaining images, even to an image containing high-frequency components close to the Nyquist frequency, with high resolution, without requiring switching of processings due to high/low-frequency components of a subject, and to provide an image processing method. <P>SOLUTION: An imaging apparatus includes an optical system 110, including a light wave front modulation element; an imaging device 120 for imaging a subject image that has passed through the optical system 110; a drive section 200, capable of varying relative positions of the optical system 110 and the imaging device 120; an image processing device 140 for applying predetermined arithmetic processings and filtering processings to image data of the subject from the image sensor 120; and a controller 190, which varies the relative positions of the optical system 110 and the image sensor 120 at the drive section 200, to obtain image data which are imaged several times, calculates and combines a plurality of image data items obtained by the image processing device 140 to apply filtering processings thereto. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus using an image pickup device and applicable to a digital still camera equipped with an optical system, a camera mounted on a mobile phone, a camera mounted on a portable information terminal, an image inspection device, an industrial camera for automatic control, an information code reader, and the like. And an image processing method.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state imaging device, is used for the imaging surface in place of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

FIG. 29 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 29, the best focus surface is matched with the imaging element surface.
30A to 30C show spot images on the light receiving surface of the image sensor 3 of the imaging lens device 1.

In addition, imaging devices have been proposed in which light beams are regularly dispersed by a phase plate and restored by digital processing to enable imaging with a deep depth of field (for example, Non-Patent Documents 1 and 2 and Patent Documents). 1-5).
“Wavefront Coding; jointly optimized optical and digital imaging systems”, Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. “Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems”, Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A

By the way, in the above technique, restoration by a filter is performed in a state where the image sensor is fixed. In this case, it is difficult to restore a component close to Nyquist of the image sensor well.
The reason is that even if the resolution as a lens is sufficient, the signal disappears depending on the relationship between the spatial frequency of the subject and the pixel pitch of the image sensor.
This is because even if the resolving power of the lens resolves to the Nyquist frequency of the image sensor, if the Nyquist frequency component in the real space actually forms an image, the signal may disappear by straddling the pixels.
A lost signal cannot be recovered by a filter. Further, if the Nyquist component is forcibly restored with a filter, the S / N ratio becomes worse and noise increases.

  An object of the present invention is to provide an imaging apparatus and an image processing method capable of obtaining a high-resolution image including a high-frequency component close to the Nyquist frequency without requiring switching of processing depending on the level of the frequency component of the subject. It is in.

  An imaging apparatus according to a first aspect of the present invention includes an optical system that includes a light wavefront modulation element, an imaging element that captures a subject image that has passed through the optical system, and a relative position of the optical system and the imaging element that varies. A possible drive unit, an image processing unit that performs predetermined arithmetic processing and filtering processing on the image data of the subject from the image sensor, and the drive unit varies the relative position of the optical system and the image sensor. A control unit that obtains image data captured a plurality of times, combines the plurality of image data obtained by the image processing unit, and performs the filtering process.

  Preferably, the control unit causes the drive unit to change the relative position of the optical system and the imaging element in a direction perpendicular to the optical axis.

  Preferably, the control unit causes the drive unit to change the relative position of the optical system and the image sensor in the optical axis direction.

  Preferably, the amount to be changed by the driving unit is less than one pixel pitch as the amount of change of the image on the image sensor.

  Preferably, the filtering includes a function of increasing the contrast.

  Preferably, the control unit is configured so that the drive unit has a relative position between the optical system and the image sensor on a plane perpendicular to the optical axis, the longitudinal direction of the image sensor, a direction perpendicular to the longitudinal direction, and a diagonal of the pixel. The direction is varied in any one or combination.

  Preferably, the image processing unit generates an image signal with less dispersion than the subject dispersion image signal from the image sensor.

  A second aspect of the present invention is an image processing method for capturing a subject image that has passed through an optical system including a light wavefront modulation element, and performing image processing on the captured image data. The method includes a step of obtaining image data captured a plurality of times while changing the relative position, and a step of combining the obtained plurality of image data and performing a filtering process on the composite image data.

  According to the present invention, an image including a high-frequency component close to the Nyquist frequency can be obtained with high resolution without requiring processing switching due to the level of the frequency component of the subject.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging element 120, an analog front end unit (AFE) 130, an image processing device 140, a camera signal processing unit 150, an image display memory 160, an image monitoring device 170, and an operation unit. 180, a control device 190, and a drive unit 200.

As will be described in detail later, the imaging apparatus 100 according to the present embodiment applies a light wavefront modulation element to the optical system 110, regularly disperses the light flux by the light wavefront modulation element, and restores the light by digital processing. Adopting a wavefront aberration control optical system or a depth expansion optical system (DEOS) system that enables imaging with deep depth of field, and accurately captures subject images (for example, information codes such as barcodes). It can be read with high accuracy.
The light wavefront modulation function can be expressed by arranging the light wavefront modulation element in the optical system as described above, but is expressed by the light wavefront modulation surface formed in the optical element forming the optical system. It is also possible to configure as described above.

  The optical system 110 includes an optical wavefront modulation element such as a phase modulation element, which will be described in detail later, and supplies an image obtained by photographing the subject object OBJ to the imaging element 120.

The image sensor 120 functions as a photoelectric conversion element, an image captured by the optical system 110 is formed, and the imaged primary image information is converted into an electrical signal primary image signal FIM through the analog front end unit 130. It consists of a CCD or CMOS sensor output to the processing device 140.
In FIG. 1, the image sensor 120 is described as a CCD as an example.

  The relative position between the optical system 110 and the image sensor 120 is configured to be variable by the drive unit 200.

The analog front end unit 130 includes a timing generator 131 and an analog / digital (A / D) converter 132.
The timing generator 131 generates the drive timing of the CCD of the image sensor 120, and the A / D converter 132 converts an analog signal input from the CCD into a digital signal and outputs it to the image processing device 140.

An image processing apparatus (two-dimensional convolution means) 140 constituting a part of the image signal processing unit inputs a digital signal of a captured image coming from the previous AFE 130, performs two-dimensional convolution processing, and performs a subsequent camera signal. The data is transferred to the processing unit (DSP) 150.
The image processing device 140 has a function of synthesizing a plurality of image data obtained by imaging a plurality of times by changing the relative position of the optical system 110 and the image sensor 120, and performing a filtering process on the synthesized image data. Have
The filtering performed by the image processing apparatus 140 includes a function for increasing the contrast.
The image processing device 140 performs a filtering process on the image data according to various types of information from the control device 190.
The image processing device 140 has a function of generating a non-dispersed image signal from the subject dispersed image signal from the image sensor 120. The image signal processing unit has a function of performing noise reduction filtering in the first step.

  As shown in FIG. 1, the image processing apparatus 140 basically includes a raw (RAW) buffer memory 141, a convolution calculator 142, a kernel data storage ROM 143 as storage means, and a convolution control unit 144.

  The convolution control unit 144 performs control such as on / off of the convolution process, screen size, and kernel data exchange, and is controlled by the control device 190.

  The kernel data storage ROM 143 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. 2 or FIG. 3, for example, and passes through the convolution control unit 144. Select and control kernel data.

  In the example of FIG. 2, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

In the example of FIG. 3, the kernel data A is data corresponding to an object distance information of 100 mm, the kernel data B is data corresponding to an object distance of 500 mm, and the kernel data C is data corresponding to an object distance of 4 m.
The processing of the image processing apparatus 140 will be described in detail later.

  The camera signal processing unit (DSP) 150 performs processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing, and performs storage in the image display memory 160, image display on the image monitoring device 170, and the like.

  The control device 190 performs exposure control, has operation inputs such as the operation unit 180, determines the operation of the entire system in accordance with those inputs, and controls the AFE 130, the image processing device 140, the DSP 150, etc. It is responsible for overall mediation control.

In addition, the control device 190 controls the drive unit 200 with the control signal CTL, changes the relative position between the optical system 110 and the image sensor 120, stores image data captured a plurality of times in a memory, and sets the image processing device 140. A plurality of image data obtained by the control are calculated and combined to perform filtering processing.
For example, the control device 190 causes the drive unit 200 to change the relative position of the optical system 110 and the image sensor 120 in a direction perpendicular to the optical axis.
For example, as illustrated in FIG. 4, the control device 190 varies the optical system 110 in a direction perpendicular to the optical axis, so that the relative position between the imaging lens of the optical system 110 and the image sensor 120 is in a direction perpendicular to the optical axis. Fluctuate.
Alternatively, as illustrated in FIG. 5, the control device 190 varies the imaging element 120 in a direction perpendicular to the optical axis, and changes the relative position between the imaging lens of the optical system 110 and the imaging element 120 in the direction perpendicular to the optical axis. To fluctuate.
Alternatively, the control device 190 integrates FIGS. 4 and 5 to change the optical system 110 and the image sensor 120 in a direction perpendicular to the optical axis, so that the relative position between the imaging lens of the optical system 110 and the image sensor 120 is changed. In the direction perpendicular to the optical axis.

In addition, the control device 190 changes the relative position between the optical system 110 and the image sensor 120 in the optical axis direction instead of changing the relative position between the optical system 110 and the image sensor 120 in the direction perpendicular to the optical axis. Let
For example, as illustrated in FIG. 6, the control device 190 shifts the optical system 110 in the optical axis direction to change the magnification, thereby changing the relative position of the optical system 110 and the image sensor 120 in the optical axis direction.

  Further, the control device 190 causes the drive unit 200 to set the relative position of the optical system 110 and the image sensor 120 on the plane perpendicular to the optical axis, the longitudinal direction of the image sensor 120, the direction perpendicular to the longitudinal direction, the diagonal direction of the pixel, It is also possible to configure to vary either or a combination of these.

  In the present embodiment, the amount that the control device 190 varies by the drive unit 200 is less than one pixel pitch as the amount of change in the image on the image sensor 120.

  The drive unit 200 varies the relative position between the optical system 110 and the image sensor 120 as described above in accordance with the control signal CTL from the control device 190.

  As shown in FIG. 1, the drive unit 200 includes a drive control device 210, an optical system drive mechanism unit 220, and an image sensor drive mechanism unit 230.

  The drive control device 210 outputs the drive signal DRV1 to the optical system drive mechanism unit 220 in order to change the relative position of the optical system 110 and the image sensor 120 in accordance with the control signal CTL from the control device 190, and outputs the drive signal DRV2. The image is output to the image sensor driving mechanism unit 230.

  The optical system drive mechanism unit 220 drives the optical system 110 in a direction perpendicular to the optical axis or in a direction parallel to the optical axis in response to the drive signal DRV1 from the drive control device 210.

  In response to the drive signal DRV2 from the drive control device 210, the image sensor driving mechanism unit 230 drives the image sensor 120 in a direction perpendicular to the optical axis or in a direction parallel to the optical axis.

  FIGS. 7A and 7B are diagrams showing a configuration example of an optical system drive mechanism unit and an image sensor drive mechanism unit that vary the optical system and the image sensor in a direction perpendicular to the optical axis. FIG. 7A shows a configuration example of the optical system driving mechanism unit, and FIG. 7B shows a configuration example of the image sensor driving mechanism unit.

  As shown in FIG. 7A, the optical system driving mechanism unit 220 is disposed at a position orthogonal to each other on the side surface of the cylindrical optical system 110 in the X and Y directions in the orthogonal coordinate system set in the drawing. Actuators 221 and 222 are provided.

  As shown in FIG. 7B, the image sensor driving mechanism unit 230 includes a rectangular movable stage 231 on which an image sensor (hereinafter also referred to as a sensor) 120 is placed, and orthogonal coordinates set in the figure. Actuators 232 and 233 are disposed on the side surfaces of the movable stage 231 at positions orthogonal to each other in the X and Y directions in the system.

  8A and 8B are diagrams showing a configuration example of an optical system drive mechanism unit and an image sensor drive mechanism unit that cause the optical system and the image sensor to change in the direction along the optical axis, and FIG. Shows an example of the configuration of the optical system drive mechanism, and FIG. 8B shows an example of the configuration of the image sensor drive mechanism.

  As shown in FIG. 8A, the optical system drive mechanism section 220A has an actuator 223 arranged on the side surface orthogonal to the optical axis of the optical system 110 in the Z direction in the orthogonal coordinate system set in the figure. ing.

  The image sensor driving mechanism unit 230A has an actuator 234 disposed on a surface of the movable stage 231 that faces the surface on which the image sensor 120 is disposed in the Z direction in the orthogonal coordinate system set in the drawing.

  The drive mechanism is intended for minute fluctuations of less than one pixel pitch as the amount of image change, and therefore an actuator such as a piezo element, a magnetostrictive element, or a voice coil motor is suitable.

As described above, the imaging apparatus 100 according to the present embodiment includes the optical system (lens) 110 and the relative position of the imaging element 120 in addition to the configuration of the optical system corresponding to DEOS and the DEOS image processing described in detail later. The high frequency restoration processing close to the Nyquist component is performed by varying the.
Specifically, the problem is solved by two processing methods. First, image processing is performed by shifting the relative position between the optical system and the image sensor in a direction perpendicular to the optical axis. Second, the image processing is performed by shifting the relative position between the optical system and the image sensor in the optical axis direction.
The two shift amounts are less than one pixel (pixel) as the image change amount.
The Nyquist frequency component lost by straddling pixels is recovered by processing a plurality of images obtained by any of the above methods.

In the present embodiment, in order to improve the resolving power of an image including a high-frequency component, an image is obtained by using a depth extension optical system while taking an image while slightly changing the relative position between the image sensor and the optical system (less than 1 pixel). A filter process such as depth extension is executed by combining the plurality of images.
By using the depth extension optical system, the contrast is lowered in the high-frequency region but the resolution characteristics are not reversed, so that an accurate image restoration can be realized.
That is, a high-frequency component close to Nyquist does not lose the signal as much as possible by mechanical driving, which leads to good restoration. By using this technique for the depth extension system, it is possible to restore an image by the same filtering process, taking advantage of the small change in OTF at any object distance. In restoration by this filter, in order to maintain the S / N, priority is given to restoration of low frequencies rather than restoration of high frequency components.

  In the following, Nyquist restoration across Nyquist components, Nyquist restoration due to optical system (lens) shift, Nyquist restoration due to imaging element shift, and Nyquist restoration due to focus shift will be illustrated and described.

FIG. 9 is a diagram illustrating an example in which an image disappears due to straddling the imaging element.
In this example, it is assumed that there is no deterioration due to the lens, and the image pickup device 120 rides on the image sensor 120 without any deviation and on the image pickup device 120 with a deviation of half a pixel.
If the pixel is shifted by half a pixel, all the pixels become 50% of the signal and disappear.

FIG. 10 is a diagram illustrating an example in a case where pixel shift is performed by lens (optical system) shift.
Here, the objective is to extract a component close to Nyquist by shifting the lens by a sub-pixel shift (a shift smaller than one pixel).
If the shift amount is recorded in advance, restoration processing can be performed by combining a plurality of images (pictures) shifted by the image processing performed by the image processing apparatus 140.
However, the restoration by this lens shift has an effect of restoring the edge to the high frequency component close to Nyquist, but the contrast does not tend to increase.
Therefore, the image processing apparatus 140 simultaneously restores both resolution and contrast by using image restoration using a filter.

FIG. 11 is a diagram illustrating an example in which pixels are shifted by image sensor shift.
The purpose here is to extract a component close to Nyquist by subpixel shifting the image sensor 120.
Similar to the lens shift, if the shift amount is recorded in advance, a restoration process can be performed by combining a plurality of images (pictures) shifted by the image processing.
However, the restoration by the image pickup element shift has an effect of restoring the edge to the high-frequency component close to Nyquist, but the contrast tends not to increase.
Therefore, the image processing apparatus 140 simultaneously restores both resolution and contrast by using image restoration using a filter.

FIG. 12 is a diagram illustrating an example of magnification variation due to focus shift.
The purpose here is to restore the lost signal by changing the magnification. In a general optical system, since the image starts to be blurred when the focus is removed, the restoration becomes difficult. However, in the case of DEOS, the depth is expanded, so that the OTF does not change with a minute image plane shift.
Therefore, the lost resolution is recovered by shifting the focus.

  FIGS. 13A to 13C are diagrams illustrating an example of variation in relative position in a plane perpendicular to the optical axis, and FIG. 13A illustrates an example of variation in the longitudinal direction (horizontal direction) of a pixel. FIG. 13B shows an example of variation in the direction perpendicular to the longitudinal direction of the pixel, and FIG. 13C shows an example of variation in the diagonal direction of the pixel.

The variation method shown in FIGS. 13A and 13B can improve the resolution in the horizontal and vertical directions, respectively.
The horizontal and vertical directions are simultaneously changed by the fluctuation method <3> shown in FIG. 13C, and the horizontal and vertical resolution can be improved by one fluctuation. It does not contribute to the improvement of the resolution in the direction perpendicular to the direction.
By performing synthesis using the method shown in FIGS. 13A to 13C, it is possible to improve the resolution in each of the horizontal, vertical, and diagonal directions.

  As the image composition method before and after the change, it can be realized in a calculation process such as a simple addition average in units of pixels or an addition average after shift correction of the amount of change.

Next, image restoration processing according to the present embodiment will be described.
FIG. 14 is a flowchart of image restoration processing according to the present embodiment.

As shown in FIG. 14, image data captured by the optical system 110 and the image sensor 120 is supplied to the image processing apparatus 140 through the AFE 130. In the image processing apparatus 140, the image data is stored in the RAW buffer memory (ST1, ST2).
Here, under the control of the control device 190, the image pickup device 120 or the optical system 110 is changed (driven) by the drive unit 200 (ST3).
Then, in the image processing device 140, it is determined whether or not the image sensor 120 or the optical system 110 has returned to the initial position (ST4). The element shift or the phase shift generated by the lens (optical system) is restored by synthesizing the shift image. That is, the high frequency component is restored (ST5).
Next, in the image processing apparatus 140, the contrast of the image restored from the high frequency component is further restored by a filter. That is, the low frequency component is restored (ST6).

  According to the present embodiment that performs the above processing, an image including a high frequency component close to the Nyquist frequency can be obtained with high resolution without switching processing depending on the level of the frequency component of the subject.

Here, the fluctuation amount and the size of the memory are considered.
Preferably, a memory (RAW buffer memory) for storing an image has a size that is an inverse number divided into sub-pixels. This is because there is an effect of increasing the number of samplings by the image sensor 120, and the Nyquist of the image sensor 120 can be increased in a pseudo manner.
For example, when considered in one dimension, it is assumed that the imaging device or the lens is driven so as to shift by 0.5 pixels. In that case, it is preferable that the memory has a size twice as large as the image size. For example, in the case of an image sensor with a width of 1280 pixels, there may be 2560 pixels. The same goes for the vertical.

The image restoration process that is a feature of the present embodiment has been described above.
Hereinafter, the configuration and functions of the optical system and the image processing apparatus related to the DEOS of this embodiment will be described in detail.

FIG. 15 is a diagram schematically illustrating a configuration example of the zoom optical system 110 according to the present embodiment. This figure shows the wide angle side.
FIG. 16 is a diagram schematically illustrating a configuration example of a zoom optical system on the telephoto side of the imaging lens device according to the present embodiment.
FIG. 17 is a diagram showing a spot shape at the center of the image height on the wide angle side of the optical system according to the present embodiment, and FIG. 18 is a spot shape at the center of the image height on the telephoto side of the optical system according to the present embodiment. FIG.

The optical system 110 in FIGS. 15 and 16 includes an object-side lens 111 disposed on the object-side OBJS, an imaging lens 112 for forming an image on the image sensor 120, and between the object-side lens 111 and the imaging lens 112. An optical wavefront modulation element (wavefront forming optical element) group 113 made of, for example, a phase plate having a three-dimensional curved surface, which is disposed and deforms a wavefront of an image formed on the light receiving surface of the imaging element 120 by the imaging lens 112. . A stop (not shown) is disposed between the object side lens 111 and the imaging lens 112.
For example, in the present embodiment, a variable aperture 110a (see FIG. 1) is provided, and the aperture degree of the variable aperture is controlled in exposure control (apparatus).

In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-mentioned third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change due to coding on the lens surface (for example, a wavefront) A light wavefront modulation element such as a modulation hybrid lens or a phase plane formed on the lens surface) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element). .
Further, in the present embodiment, the case where a regularly dispersed image is formed using a phase plate that is a light wavefront modulation element has been described. However, the lens used as a normal optical system has the same rule as the light wavefront modulation element. When an image that can form a dispersed image is selected, it can be realized only by an optical system without using a light wavefront modulation element. In this case, it does not correspond to the dispersion caused by the phase plate described later, but corresponds to the dispersion caused by the optical system.

The phase plate 113a shown in FIGS. 15 and 16 is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, the phase plate 113a forms a deep luminous flux (which plays a central role in image formation) and a flare (blurred portion).
Means for restoring the regularly dispersed image to a focused image without moving the optical system 110 by digital processing is a wavefront aberration control optical system system or a depth expansion optical system (DEOS: Depth Expansion Optical system system). This processing is performed in the image processing apparatus 140.

Here, the basic principle of DEOS will be described.
As shown in FIG. 19, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

[Equation 1]
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

[Equation 2]
f = H ^-1 * g

Here, the kernel size and calculation coefficient regarding H will be described.
The optical switching information is KPn, KPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In this embodiment, as shown in FIG. 1, an image from the optical system 110 is received by the image sensor 120 and input to the image processing device 140 when the aperture is opened, and a conversion coefficient corresponding to the optical system is acquired. The image signal having no dispersion is generated from the dispersion image signal from the image sensor 120 with the obtained conversion coefficient.

  In the present embodiment, as described above, dispersion refers to forming a non-focused image on the image sensor 120 by inserting the phase plate 113a as described above. It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

20A to 20C show spot images on the light receiving surface of the image sensor 120.
20A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 20B shows a case where the focal point is a focal point (Best focus), and FIG. 20C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 20A to 20C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (a central role of image formation) is obtained by the wavefront forming optical element group 113 including the phase plate 113a. And flare (blurred part) are formed.

  As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

FIGS. 21A and 21B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens apparatus according to the present embodiment. FIG. 21A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 21B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 140 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. 21A and 21B. The MTF of the primary image is essentially a low value.

  As described above, the image processing apparatus 140 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing apparatus 140 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristic indicated by the curve B in FIG.
A characteristic indicated by a curve B in FIG. 21 is a characteristic obtained when the wavefront is not deformed without using the wavefront forming optical element as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 22, in order to achieve the MTF characteristic curve B to be finally realized with respect to the MTF characteristic curve A with respect to the spatial frequency obtained optically, each spatial frequency is changed to each spatial frequency. On the other hand, as shown in FIG. 23, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic of FIG. 22, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the image processing apparatus 140 that forms the primary image into a high-definition final image. In the optical system, a wavefront shaping optical element is newly provided, or an optical element such as glass, plastic or the like is formed for wavefront shaping, thereby forming an imaging wavefront. The wavefront is deformed (modulated), and an image of such a wavefront is formed on the image pickup surface (light receiving surface) of the image pickup device 120 formed of a CCD or CMOS sensor. An image forming system.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. For this reason, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 140.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the desired wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 140 made up of a later stage DSP or the like. Thereby, the MTF value is significantly improved.

  Next, the MTF response of this embodiment and a general optical system will be considered.

FIG. 24 is a diagram illustrating the MTF response when the object is at the focal position and when the object is out of the focal position in the case of a general optical system.
FIG. 25 is a diagram illustrating the response of the MTF when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 26 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, the change in the response of the MTF is less than that in an optical system in which no light wavefront modulation element is inserted even when the object deviates from the focal position.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

The absolute value (MTF) of the OTF of the optical system having the phase plate shown in FIG. 23 is preferably 0.1 or more at the Nyquist frequency.
This is because, in order to achieve the OTF after restoration shown in FIG. 24, the gain is increased by the restoration filter, but the noise of the sensor is also raised at the same time. For this reason, it is preferable to perform restoration without increasing the gain as much as possible at high frequencies near the Nyquist frequency.
In the case of a normal optical system, resolution is achieved if the MTF at the Nyquist frequency is 0.1 or more.
Therefore, if the MTF before restoration is 0.1 or more, the restoration filter does not need to increase the gain at the Nyquist frequency. If the MTF before restoration is less than 0.1, the restored image becomes an image greatly affected by noise, which is not preferable.

  As described above, according to the present embodiment, the optical system 110 including the light wavefront modulation element, the image sensor 120 that captures the subject image that has passed through the optical system 110, and the relative relationship between the optical system 110 and the image sensor 120. A drive unit 200 that can change its position, an image processing device 140 that performs predetermined arithmetic processing and filtering processing on image data of a subject from the image sensor 120, and a relative relationship between the optical system 110 and the image sensor 120 in the drive unit 200. And a control device 190 that obtains image data captured multiple times by changing the position, calculates and synthesizes the plurality of image data obtained by the image processing device 140, and performs a filtering process. It is possible to obtain high resolution images including high frequency components close to the Nyquist frequency without switching the processing due to the high and low frequency components. That.

Further, according to the present embodiment, the optical system 110 and the imaging device 120 including the light wavefront modulation element that forms the primary image, and the image processing device 140 that forms the primary image into a high-definition final image are included. The imaging element 120 and the phase modulation element (phase plate) 113a as the light wavefront modulation element can be arranged suitable for restoration, and a depth extension optical system with a narrow tolerance can be realized.
As a result, there is an advantage that a blur that improves the restored image can be realized, and that a good restored image can be obtained with an appropriate image quality and little influence of noise.
In addition, there is an advantage that the optical system can be simplified, the cost can be reduced, and a restored image with an appropriate image quality and less influenced by noise can be obtained.
In addition, the kernel size used at the time of convolution calculation and the coefficient used in the numerical calculation are made variable, know by input from the operation unit 180, etc. There is an advantage that the lens can be designed without worrying about the image and that the image can be restored by convolution with high accuracy.
In addition, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large optical lens and without driving the lens. You can also.
The imaging apparatus 100 according to the present embodiment can be used in a DEOS optical system in consideration of the small size, light weight, and cost of consumer devices such as digital cameras and camcorders.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, the image processing apparatus 140 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, and so on, can achieve high-definition image quality. There is an advantage of becoming.
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.

  Note that the imaging apparatus according to the present embodiment having such characteristics can be applied to an information code reading apparatus as shown in FIG. 27, for example.

FIG. 27 is an external view showing an example of an information code reading apparatus to which the imaging apparatus according to the embodiment of the present invention can be applied.
28A to 28C are diagrams illustrating examples of information codes.

As shown in FIG. 27, the information code reading apparatus 300 according to the present embodiment has a main body 310 connected to a processing device such as an electronic register (not shown) via a cable 311 and, for example, a reflectance printed on a reading object 320. This is a device capable of reading information codes 321 such as different symbols and codes.
When the reading start switch 312 formed on the main body 310 is operated, the information code reading device 300 uses the trigger as a trigger to read the information code, for example, 10 times, and if it can be determined by 7 decoding determinations, It has a function of decoding results.
The reason for having this function is that considering the ease of use of the user, it is assumed that the state where the information cannot be read with respect to disturbance or weak information will continue if the decoding determination is enabled once. It is. By the way, it is configured so that the setting of seven times of this retry can be changed.

  As an information code to be read, for example, a one-dimensional bar code 322 such as a JAN code as shown in FIG. 28A, a stack-type CODE 49 as shown in FIGS. 28B and 28C, Alternatively, a two-dimensional bar code 323 such as a matrix type QR code may be used.

In the information code reading device 300 according to the present embodiment, an illumination light source (not shown) and an imaging device 100 as shown in FIG.
The imaging apparatus 100 in which the information code reader 300 is arranged is provided with an interface part (I / F) extending outside, for example.

  According to the information code reading apparatus 300 according to the present embodiment, it is possible to extend the depth as well as improve the contrast in the image restoration process, and to realize highly accurate code reading.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a figure which shows an example (optical magnification) of the storage data of kernel data ROM. It is a figure which shows the other example of the stored data of kernel data ROM. It is a figure which shows the example which fluctuates an optical system in the direction perpendicular | vertical to an optical axis, and fluctuates the relative position of the imaging lens of an optical system, and an image pick-up element in a direction perpendicular | vertical to an optical axis. It is a figure which shows the example which fluctuates an image pick-up element in the direction perpendicular | vertical to an optical axis, and fluctuates the relative position of the imaging lens of an optical system, and an image pick-up element in the direction perpendicular | vertical to an optical axis. It is a figure which shows the example which changes the relative position of an optical system and the image pick-up element 120 to an optical axis direction by shifting an optical system to an optical axis direction, and changing a magnification. It is a figure which shows the structural example of the optical system drive mechanism part which changes an optical system and an image pick-up element in the direction perpendicular | vertical to an optical axis, and an image pick-up element drive mechanism part, Comprising: (A) shows the structural example of an optical system drive mechanism part. FIG. 4B is a diagram illustrating a configuration example of an image sensor driving mechanism unit. FIG. 4 is a diagram illustrating a configuration example of an optical system drive mechanism unit and an image sensor drive mechanism unit that cause the optical system and the image sensor to change in the direction along the optical axis, and (A) illustrates a configuration example of the optical system drive mechanism unit; FIG. 6B is a diagram illustrating a configuration example of an image sensor driving mechanism unit. It is a figure which shows the example when an image lose | disappears by straddling an image pick-up element. It is a figure which shows the example at the time of pixel shifting by the lens (optical system) shift. It is a figure which shows the example at the time of pixel shifting by image pick-up element shift. It is a figure which shows the example of the magnification fluctuation | variation by a focus shift. It is a figure which shows the example of a fluctuation | variation of the relative position in a plane perpendicular | vertical to an optical axis, Comprising: (A) is a figure which shows the example of a fluctuation | variation to the longitudinal direction (horizontal direction) of a pixel, (B) is perpendicular | vertical to the longitudinal direction of a pixel. The figure which shows the example of a change to a direction, (C) is a figure which shows the example of the change to the diagonal direction of a pixel. It is a flowchart of the image restoration process which concerns on this embodiment. It is a figure which shows typically the structural example of the zoom optical system of the wide angle side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows typically the structural example of the zoom optical system of the telephoto side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows the spot shape of the image height center on the wide angle side. It is a figure which shows the spot shape of the image height center of a telephoto side. It is a figure for demonstrating the principle of DEOS. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a case where it is a focusing point ( (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image sensor which concerns on this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image sensor of an imaging lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. 1 is an external view illustrating an example of an information code reading apparatus to which an imaging apparatus according to an embodiment of the present invention can be applied. It is a figure which shows the example of an information code. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 30A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device of FIG. 29, where FIG. 29A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), and FIG. In the case (Best focus), (C) is a diagram showing each spot image when the focus is shifted by −0.2 mm (Defocus = −0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Image pick-up device, 110 ... Optical system, 120 ... Image pick-up element, 130 ... Analog front end part (AFE), 140 ... Image processing apparatus, 150 ... Camera signal processing part, 180 ... operation unit, 190 ... control device, 200 ... drive unit, 111 ... object side lens, 112 ... imaging lens, 113 ... optical element for wavefront formation, 113a ... Phase plate (light wavefront modulation element), 142... Convolution computing unit, 143... Kernel data ROM, 144... Convolution control unit, 300. code.

Claims (8)

An optical system including an optical wavefront modulation element;
An image sensor that images a subject image that has passed through the optical system;
A drive unit capable of changing a relative position between the optical system and the imaging device;
An image processing unit that performs predetermined calculation processing and filtering processing on the image data of the subject from the image sensor;
The drive unit obtains image data captured a plurality of times by changing the relative position of the optical system and the image sensor, and combines the plurality of image data obtained by the image processing unit to perform the filtering process. An imaging device comprising: a control unit. The controller is
The imaging apparatus according to claim 1, wherein the drive unit causes the relative position of the optical system and the imaging element to vary in a direction perpendicular to the optical axis. The controller is
The imaging apparatus according to claim 1, wherein the driving unit causes a relative position of the optical system and the imaging element to vary in an optical axis direction. The imaging apparatus according to claim 2, wherein the amount to be changed by the driving unit is less than one pixel pitch as the amount of change in the image on the imaging element. The imaging device according to any one of claims 1 to 4, wherein the filtering includes a function of increasing a contrast. The controller is
In the drive unit, the relative position of the optical system and the image sensor varies on the surface perpendicular to the optical axis in any one or a combination of the longitudinal direction of the image sensor, the direction perpendicular to the longitudinal direction, and the diagonal direction of the pixels. The imaging apparatus according to claim 2, 4, or 5. The image processing unit
The imaging device according to any one of claims 1 to 6, wherein the imaging device generates an image signal with less dispersion than a subject dispersion image signal from the imaging device. An image processing method of capturing a subject image that has passed through an optical system including a light wavefront modulation element and performing image processing on the captured image data,
Obtaining image data captured multiple times by changing the relative position of the optical system and the imaging element;
And a step of combining the plurality of obtained image data and performing a filtering process on the combined image data.

Images