JP2008245157A - Imaging device and method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging device and a method for manufacturing the same for realizing blurs for making a restored image satisfactory, and for simplifying an optical system, and for reducing cost, and for acquiring proper picture quality, and for reducing the influence due to noise, and for accurately restoring an image in a required direction, and for acquiring a satifactory restored image. <P>SOLUTION: This imaging device is provided with an optical system 210, including a light wave face modulation element rotatable with an optical axis as a center; an imaging element 220 for imaging through the optical system; an image processor 240 for performing blur restoration processing by using a blur restoration filter, acquired from a PSF acquired by compounding each dot image intensity distribution (PSF), corresponding to a plurality of angles of rotation; and a control device 250 for controlling the focal point depth expansion, by making the blur restoration filter act on the composite image, acquired by compounding images photographed at each of the plurality of angles of rotation. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an imaging apparatus using an imaging device and including an optical system, and a method thereof.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state image sensor, is used in most cases instead of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used in video cameras, digital video units, personal computers, mobile phones, personal digital assistants (PDAs), image inspection devices, industrial cameras for automatic control, and the like.

FIG. 38 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 38, the best focus surface is matched with the imaging device surface.
39A to 39C show spot images on the light receiving surface of the imaging element 3 of the imaging lens device 1.

In addition, imaging devices have been proposed in which light beams are regularly dispersed by a phase plate and restored by digital processing to enable imaging with a deep depth of field (for example, Non-Patent Documents 1 and 2 and Patent Documents). 1-5).
In addition, an automatic exposure control system for a digital camera that performs filter processing using a transfer function has been proposed (see, for example, Patent Document 6).
“Wavefront Coding; jointly optimized optical and digital imaging systems”, Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. “Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems”, Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A JP 2004-153497 A

In the imaging devices proposed in the above-mentioned documents, all of them are based on the assumption that the PSF (Point-Spread-Function) when the above-described phase plate is inserted into a normal optical system is constant, When the PSF changes, it is extremely difficult to realize an image with a deep depth of field by convolution using a subsequent kernel.
Therefore, apart from a single-focus lens, a zoom system, an AF system, or the like has a great problem in adopting due to the high accuracy of the optical design and the associated cost increase.
In other words, in the conventional imaging apparatus, proper convolution calculation cannot be performed, and astigmatism and coma that cause a shift of a spot (SPOT) image at the time of wide or tele (Tele). Therefore, an optical design that eliminates various aberrations such as zoom chromatic aberration is required.
However, the optical design that eliminates these aberrations increases the difficulty of optical design, causing problems such as an increase in design man-hours, an increase in cost, and an increase in the size of the lens.

In addition, in the devices disclosed in the above-described documents, noise is also amplified simultaneously when an image is restored by signal processing, for example, in shooting in a dark place.
Therefore, for example, in an optical system including an optical system and signal processing using the above-described optical wavefront modulation element such as a phase plate and subsequent signal processing, noise is amplified when shooting in a dark place. This has the disadvantage of affecting the restored image.

  Furthermore, in the above technique, for example, when shooting a bright subject, when the aperture is stopped, the phase modulation element is covered with the aperture, so that the phase change becomes small, and if the image restoration processing is performed, the restored image is affected. There are disadvantages.

Furthermore, in the above technique, since the optical image is always in a blurred state regardless of the object distance in the image before the image restoration process, the image quality is not perfect unless the restoration process is performed.
Even if the restoration process is performed, it is difficult to completely restore in the process of blurring and restoring. For this reason, it is necessary to blur the restored image.

Further, in an imaging apparatus that regularly disperses a light beam by the above-described phase plate and restores it by digital processing to enable image capturing with a deep depth of field, the PSF generated by the phase plate spreads out of blur. The direction appears strongly in the vertical and horizontal directions of the angle of view, for example, and the blur component is small in the 45 degree direction.
When this is analyzed by frequency analysis, the modulation transfer function (MTF: Modulation Transfer Function) in the vertical direction and the horizontal direction is there, but the 45 degree direction is considerably lowered.
On the other hand, since the restoration filter is basically a reciprocal of the MTF, the coefficient of the frequency component in the 45 degree direction becomes very high.

  In general, a Wiener filter represented by the following formula is often used as a restoration filter as a countermeasure against noise. When applied, the gain filter is applied to the 45-degree direction where the noise level is lowered, as well as the noise suppression function. As a result, the magnification that is originally required in the 45 degree direction cannot be increased. For this reason, the 45 degree direction cannot be completely restored.

  The present invention can realize blurring that improves the restored image, simplifies the optical system, can reduce the cost, has an appropriate image quality, is less influenced by noise, and is restored in the necessary direction. It is an object of the present invention to provide an imaging apparatus and method capable of accurately performing the above and obtaining a good restored image.

  An imaging apparatus according to a first aspect of the present invention is compatible with an optical system including an optical wavefront modulation element that can rotate around an optical axis, an imaging element that captures an image via the optical system, and a plurality of rotation angles. A blur restoration filter obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs), and the blur restoration filter for a synthesized image obtained by synthesizing images picked up at each of the plurality of rotation angles. And a control unit that performs control so that the depth of focus is extended.

  Preferably, the control unit performs control such that the plurality of PSFs are synthesized by performing different weighting on the plurality of PSFs.

  Preferably, the control unit performs control to change the exposure time according to the weighting ratio of the plurality of PSFs and perform imaging at each of a plurality of rotation angles.

  Preferably, the control unit controls the weighting and / or the ratio of the exposure time so as to be heavily distributed to a rotation angle that places importance on restoration performance.

  Preferably, the control unit sets a rotation angle that places importance on the restoration performance as at least one of a vertical direction and a horizontal direction of an imaging field angle.

  An image pickup apparatus according to a second aspect of the present invention includes an optical system including a plurality of light wavefront modulation elements that can rotate around an optical axis, an image pickup element that picks up an image via the optical system, and the plurality of light waves. Depth of focus expansion by applying a blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to each rotation angle of the surface modulation element, and the blur restoration filter to the captured image. And a control unit for performing

  An image pickup apparatus according to a third aspect of the present invention includes an optical system including one optical wavefront modulation element having characteristics equivalent to a plurality of optical wavefront modulation elements combined with rotating optical axes in terms of optical characteristics; An image pickup device for picking up an image via a system, a blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to each rotation angle of the plurality of light wavefront modulation devices, And a control unit that extends the depth of focus by applying the blur restoration filter to the image.

  Preferably, the rotation angle difference is 45 degrees or 135 degrees.

  An imaging method according to a fourth aspect of the present invention includes a step of imaging a plurality of images via an optical system including an optical wavefront modulation element that can rotate around an optical axis, and an image captured at each of a plurality of rotation angles. And a step of extending the depth of focus by applying a blur restoration filter obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs) corresponding to a plurality of rotation angles with respect to the synthesized image, and Have

  The imaging method according to the fifth aspect of the present invention provides an image via an optical system including one light wavefront modulation element having characteristics equivalent to a plurality of light wavefront modulation elements combined by rotating the optical axis in terms of optical characteristics. And a blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to each rotation angle of the plurality of light wavefront modulation elements with respect to the captured image. Performing a depth of focus extension.

  According to a sixth aspect of the present invention, there is provided an imaging method in which an image is captured via an optical system including one optical wavefront modulation element, and optical wavefront modulation that is biased in a blur spreading direction with respect to the captured image. Expanding the depth of focus by applying a blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to a plurality of rotation angles of the element.

According to the present invention, restoration in a necessary direction can be performed accurately, and a good restored image can be obtained.
In addition, there is an advantage that the optical system can be simplified, the cost can be reduced, and a good restored image can be obtained with an appropriate image quality and little influence of noise.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is an external view showing an example of an information code reading apparatus according to an embodiment of the present invention.
2A to 2C are diagrams illustrating examples of information codes.
FIG. 3 is a block diagram illustrating a configuration example of an imaging apparatus applied to the information code reading apparatus of FIG.

As shown in FIG. 1, the information code reader 100 according to the present embodiment has a main body 110 connected to a processing device such as an electronic register (not shown) via a cable 111, for example, a reflectance printed on a reading object 120. It is a device that can read the information code 121 such as a different symbol or code.
As an information code to be read, for example, a one-dimensional bar code 122 such as a JAN code as shown in FIG. 2A, a stack-type CODE 49 as shown in FIGS. 2B and 2C, Alternatively, a two-dimensional barcode 123 such as a matrix type QR code can be used.

In the information code reading apparatus 100 according to the present embodiment, an illumination light source (not shown) and an imaging apparatus 200 as shown in FIG.
As will be described in detail later, the imaging apparatus 200 applies a light wavefront modulation element to the optical system, regularly disperses the light beam by the light wavefront modulation element, and restores the image by digital processing to capture an image with a deep depth of field. The system adopts a wavefront aberration control optical system or a depth expansion optical system (DEOS: Optical System) that enables a one-dimensional barcode such as a JAN code and a two-dimensional barcode such as a QR code. Information codes having different characteristics (types) such as bar codes can be read accurately and accurately according to the characteristics.

  As shown in FIG. 3, the imaging device 200 of the information code reader 100 includes an optical system 210, an imaging device 220, a selection switching unit 230, an image processing device 240, a control device 250 including a function as a determination unit, and an external device. It has an extended interface unit (I / F) 260.

The optical system 210 includes an optical wavefront modulation element such as a phase modulation element, which will be described in detail later, and supplies an image obtained by photographing the information code 121 that is the subject object OBJ to the imaging element 220.
For example, the light wavefront modulation element of the optical system 210 is arranged to be rotatable about the optical axis by the selection switching unit 230.

The image sensor 220 forms an image captured by the optical system 210, and forms image primary image information as a primary image signal FIM of an electrical signal via an analog front end unit (AFE) (not shown). It consists of a CCD or CMOS sensor that outputs to.
In FIG. 3, the imaging element 220 is described as a CCD as an example.

FIG. 4 is a diagram schematically illustrating a configuration example of the optical system 210 according to the present embodiment.
FIG. 5 is a diagram showing a spot shape at the center of the image height on the wide angle side of the optical system according to the present embodiment, and FIG. 6 is a spot shape at the center of the image height on the telephoto side of the optical system according to the present embodiment. FIG.

  The optical system 210 in FIG. 4 is disposed between an object-side lens 211 disposed on the object OBJ side, an imaging lens 212 for forming an image on the image sensor 220, and between the object-side lens 211 and the imaging lens 212. It has a light wavefront modulation element (wavefront forming optical element) group 213 made of, for example, a phase plate having a three-dimensional curved surface, which deforms the wavefront of image formation on the light receiving surface of the image sensor 220 by the imaging lens 212.

In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-mentioned third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change due to coding on the lens surface (for example, a wavefront) A light wavefront modulation element such as a modulation hybrid lens or a phase plane formed on the lens surface) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element). .
Further, in the present embodiment, the case where a regularly dispersed image is formed using a phase plate that is a light wavefront modulation element has been described. However, the lens used as a normal optical system has the same rule as the light wavefront modulation element. When an image that can form a dispersed image is selected, it can be realized only by an optical system without using a light wavefront modulation element. In this case, it does not correspond to the dispersion caused by the phase plate described later, but corresponds to the dispersion caused by the optical system.

The optical system 210 in FIG. 4 is an example in which an optical phase plate 213a is inserted into a 3 × zoom used in a digital camera.
The phase plate 213a shown in the figure is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 220 is realized.
In other words, the phase plate 213a forms a deep light beam (which plays a central role in image formation) and a flare (blurred portion).
As described above, means for restoring the regularly dispersed image to a focused image without moving the optical system 210 by digital processing is a wavefront aberration control optical system system or a depth extension optical system system (DEOS). : Depth Expansion Optical system), and this processing is performed in the image processing apparatus 240.

The relative positions of the optical wavefront modulation element and the image sensor 220 in the optical system 210 are set by the selection switching unit 230 by relatively rotating the optical wavefront modulation element and the image sensor 220 included in the optical system 210. Is done. In the present embodiment, as will be described later, the light wavefront modulation element is rotated around the optical axis of the optical system 210 or held at the same position, and set to the first position or the second position. Is done.
In the present embodiment, the first position means a reference position where the number of pixels to which an output signal of a point image intensity distribution (PSF) that has passed through the optical system 210 is input is the first number N1. In the case of the first position, the light wavefront modulation element is set at the reference position and the rotation angle is 0 degree.
The second position is a case where the light wavefront modulation element is rotated from the first position by a predetermined angle, for example, 45 degrees or 135 degrees around the optical axis.

The selection switching unit 230 rotates the light wavefront modulation element of the optical system 210 according to the control signal CTL1 of the control device 250, or holds the position as it is, and the first position or the second position described above. The rotational position is selectively switched so that
For example, the selection switching unit 230 holds the first position by the first imaging (first imaging) and the second imaging (second imaging) by the control signal CTL1 of the control device 250. It is controlled to hold the position of 2. The first captured image and the second captured image are combined.

  FIG. 7 is a diagram illustrating a configuration example of the selection switching unit of the present embodiment.

  The selection switching unit 230 </ b> A in FIG. 7 arranges an optical wavefront modulation element (for example, a phase plate) 213 a with a rotationally movable component 231 having a rotation center so that the optical axis coincides with the edge of the component 231 and the motor 232. The light wavefront modulation element 213a is rotated by a predetermined angle, for example, 45 degrees or about the optical axis center AX in accordance with the control signal CTL1 generated by the control device 250, or meshed with the gear 233 attached to the rotation shaft, or left as it is. Hold on.

For example, in the initial state, the selection switching unit 230A is held at a position where the first position portion a1 of the edge portion of the light wavefront modulation element 213a matches the position of the component 231 indicated by P1 in FIG.
The selection switching unit 230A receives the control signal CTL1 from the control device 250 and rotates the component 231 counterclockwise in the drawing by a predetermined angle, for example, 45 degrees.
For example, when the imaging at the second position portion a2 is completed, the selection switching unit 230A receives the control signal CTL1 from the control device 250 and rotates the component 231 by 45 degrees clockwise in the drawing (returned to the reference position). ).

The image processing apparatus 240 receives a digital signal of a captured image coming from an AFE (not shown) in the previous stage, performs a two-dimensional convolution process, and outputs the convolution process to the control apparatus 250 in the subsequent stage.
The image processing device 240 applies a blur restoration filter instructed by the control signal CTL2 of the control device 250 to a composite image obtained by combining images picked up at a plurality of rotation angles (0 degrees and 45 degrees). And filtering the input image signal obtained through the optical transfer function (OTF) of the input image signal.
The image processing apparatus 240 has a function (blur restoration function) for generating an image signal having no dispersion from the subject dispersion image signal from the image sensor 220. The image signal processing unit has a function of performing noise reduction filtering in the first step.
Note that the blur restoration filter is obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs) corresponding to a plurality of rotation angles (0 degrees and 45 degrees in the present embodiment).
The processing of the image processing device 240 will be described in further detail later.

The control device 250 controls communication with a higher-level device (for example, an electronic register) via the selection switching unit 230, the image processing device 240, and the interface unit 260, and controls arbitration control of the entire system.
The control device 250 outputs a control signal CTL1 to the selection switching unit 230 to perform imaging in a state where the light wavefront modulation element 213a is held at the first position without rotating (rotating angle 0 degree). The optical wavefront modulation element 213a is rotated by the control signal CTL1 (rotation angle is 45 degrees), and imaging is performed in a state of being held at the second position.
Then, the control device 250 outputs a control signal CTL2 to the image processing device 240 and outputs a plurality of synthesized images obtained by synthesizing images captured at a plurality of rotation angles (0 degrees and 45 degrees). The depth of focus is expanded by applying a blur restoration filter obtained by calculating from a PSF obtained by synthesizing individual point image intensity distributions (PSF) corresponding to rotation angles (0 degrees and 45 degrees in the present embodiment). To control.
Then, the control device 250 decodes the information code read after the rotation control of the light wavefront modulation element and the selection control of the restoration filter in accordance with the characteristics of the information code, and the decoded information is transmitted to the host device via the interface unit 260, for example. Control to transmit to.

The control device 250 performs control such that the plurality of PSFs are synthesized by performing different weighting on the plurality of PSFs.
Further, the control device 250 performs control so as to change the exposure time according to the weighting ratio of the plurality of PSFs and perform imaging at each of the plurality of rotation angles.
In the present embodiment, the weighting and / or the exposure time ratio are heavily allocated to the rotation angle where restoration performance is important.
In the present embodiment, the rotation angle that places importance on restoration performance is at least one of the vertical direction and the horizontal direction of the imaging field angle.

Here, an example of PSF synthesis processing for performing weighting will be described.
FIG. 8 is a diagram showing PSF2, FIG. 9 is a diagram showing an MTF (Modulation Transfer Function) of PSF2 in FIG. 8, and FIG. 10 is a diagram showing a synthesized PSF3 according to the present embodiment. FIG. 11 is a diagram showing the MTF of the synthetic PSF.

A synthesized PSF 3 is generated by synthesizing PSF 1 (point image) with the blurred PSF 2 shown in FIG. The synthesized PSF is as shown in FIG.
In the combining method, for example, PSF1 (point image) is generated from the first photographing with a short exposure time, and PSF2 is generated from the second image.
Then, the combined PSF 3 is generated in consideration of the weight according to the ratio between the first shooting exposure condition and the second shooting exposure condition.
For example, if the exposure amount of the first shooting is 100 and the exposure amount of the second shooting is 200, PSF1 and PSF2 are combined at 1: 2.
As shown in FIG. 11, the MTF of the combined PSF 3 combines the PSF 1 of an image with a short exposure so that no bounce as shown in FIG. 9 occurs. Therefore, the blur restoration filter that performs frequency modulation generated from the MTF in FIG.

In this embodiment, the relative positions of the optical system 210 and the image sensor 220 are attached by rotating the light wavefront modulation element and the image sensor 220 included in the optical system 210 by the selection switching unit 230. The position is set. In the present embodiment, as described above, the light wavefront modulation element is rotated around the optical axis of the optical system 210 or held in the same position in accordance with the control device 250, and the first position or the second position. The position is set.
For example, PSF is symmetric about a center line passing through the top.

FIGS. 12A and 12B are diagrams showing the analog PSF after a reference position (rotation angle 0 degree) and a predetermined angle rotation (45 degrees rotation).
FIGS. 13A and 13B are diagrams showing the digital PSF after A / D conversion of the analog PSF at the reference position (rotation angle 0 degree) in FIG. 12A and its MTF.
FIGS. 14A and 14B are diagrams showing the digital PSF after A / D conversion of the analog PSF after the predetermined angle rotation (45 degrees rotation) in FIG. 12B and the MTF thereof.
FIGS. 15A and 15B are diagrams showing a digital PSF and its MTF after the PSFs of 0 degrees and 45 degrees are combined.

In an imaging apparatus that regularly disperses a light beam by the above-described phase plate and restores it by digital processing and enables image capturing with a deep depth of field, the PSF generated by the phase plate has a direction in which blur is spread. For example, it appears strongly in the vertical and horizontal directions of the angle of view, and there are few blur components in the 45 degree direction.
When this is analyzed by frequency analysis, the MTFs in the vertical direction and the horizontal direction are there, but as shown in FIG.
On the other hand, since the restoration filter is basically a reciprocal of the MTF, the coefficient of the frequency component in the 45 degree direction becomes very high.

  In general, the above-described Wiener type filter is often used as a restoration filter due to noise countermeasures. However, when it is applied, the noise level is reduced to a noise level as shown in FIG. The effect of suppressing the gain works for the 45 degree direction, and as a result, the originally required magnification in the 45 degree direction cannot be increased. For this reason, the 45 degree direction cannot be completely restored.

  On the other hand, as shown in FIG. 15B, by combining PSFs having a plurality of rotation angles (here, 0 degrees and 45 degrees) as in the present embodiment, the vertical and horizontal MTFs are combined. However, although the 45 degree direction has decreased considerably, it can be increased to the originally required magnification in the 45 degree direction.

  FIG. 16 is a diagram illustrating an example of an original image, FIG. 17 is a diagram illustrating a blurred image at the reference position (rotation angle 0 degree), and FIG. 18 is a restored image having a rotation angle 0 degree. 19 is a restored image with a rotation angle of 45 degrees, FIG. 20 is a view showing a composite blurred image with rotation angles of 0 degrees and 45 degrees, and FIG. 21 is a view with rotation angles of 0 degrees and 45 degrees. It is a figure which shows a synthetic | combination restoration image.

As can be seen from these figures, the restored image with a rotation angle of 0 degrees is an image with blur remaining, and the restored image with a rotation angle of 45 has a difference depending on the angle as compared with the case of 0 degree, but it is still sufficient. A restored image cannot be obtained.
On the other hand, as shown in FIG. 21, a composite restored image with rotation angles of 0 degrees and 45 degrees can provide a sufficiently clear restored image.

  In the present embodiment, the case where it is rotated by 45 degrees has been described, but the effect of the present invention can be fully exhibited even if it is rotated other than 90 degrees, 180 degrees, and 270 degrees from the state of FIG. can do.

  Hereinafter, the principle of the DEOS of the present embodiment, the optical system, and the configuration and function of the image processing apparatus will be specifically described.

First, the basic principle of DEOS will be described.
As shown in FIG. 22, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

[Equation 2]
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

[Equation 3]
f = H ^-1 * g

Here, the kernel size and calculation coefficient regarding H will be described.
The optical switching information is KPn, KPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In the present embodiment, as shown in FIG. 3, an image from the optical system 210 is received by the image sensor 220 and input to the image processing device 240, and a conversion coefficient corresponding to the optical system is acquired and acquired. An image signal having no dispersion is generated from the dispersion image signal from the image sensor 220 with a conversion coefficient.

  In the present embodiment, as described above, dispersion means that, by inserting the phase plate 213a, an image that does not fit anywhere on the image sensor 220 is formed, and the phase plate 213a has a deep light flux ( It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

FIGS. 23A to 23C show spot images on the light receiving surface of the image sensor 220.
23A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 23B shows a case where the focal point is a focal point (Best focus), and FIG. 23C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 23A to 23C, in the imaging apparatus 200 according to the present embodiment, a light beam having a deep depth (a central role of image formation) is generated by the wavefront forming optical element group 213 including the phase plate 213a. And flare (blurred part) are formed.

  As described above, the primary image FIM formed in the imaging apparatus 200 of the present embodiment has a light beam condition with a very deep depth.

24A and 24B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens device according to the present embodiment. FIG. 24A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 24B shows the MTF characteristics with respect to the spatial frequency.
In the present embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 240 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. 24A and 24B. The MTF of the primary image is essentially a low value.

  Hereinafter, each component structure and function when the depth extension optical system is adopted will be further described.

  FIG. 25 shows the shape of the wavefront aberration expressed by the following equation, where the optical axis of the optical system including the light wavefront modulation element of this embodiment is the z axis and two axes orthogonal to each other are x and y.

In the range where the wavefront aberration is 0.5λ or less, the phase change is small, and the OTF is the same as that of a normal optical system. Therefore, the mounting position is adjusted by narrowing down until the wavefront aberration is about 0.5λ.
FIG. 26 shows the shape of the wavefront aberration and the range of 0.5λ or less by a bold line.
However, λ uses, for example, wavelengths in the visible light region and the infrared region.

  The shape shown in FIG. 25 is an example, and the phase of the light wavefront modulation element is represented by the following equation when the optical axis of the optical system is the z axis and the two axes orthogonal to each other are x and y. Anything is applicable.

  As described above, the image processing apparatus 240 receives the primary image FIM from the image sensor 220, performs a so-called predetermined correction process for raising the MTF at the spatial frequency of the primary image, and forms a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing device 240 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristics indicated by the curve B in FIG.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 27, in order to achieve the MTF characteristic curve B to be finally realized with respect to the MTF characteristic curve A with respect to the optically obtained spatial frequency, each spatial frequency is changed to each spatial frequency. On the other hand, as shown in FIG. 28, the original image (primary image) is corrected by applying strength such as edge enhancement.
For example, in the case of the MTF characteristic of FIG. 27, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 200 according to the embodiment basically includes the optical system 210 and the imaging element 220 that form a primary image, and the image processing apparatus 240 that forms the primary image into a high-definition final image. In the optical system, a wavefront shaping optical element is newly provided, or an optical element such as glass, plastic or the like is formed for wavefront shaping, thereby forming an imaging wavefront. The image is deformed (modulated), and such a wavefront is imaged on the imaging surface (light-receiving surface) of the imaging device 220 including a CCD or CMOS sensor, and a high-definition image is obtained from the imaged primary image through the image processing device 240. An image forming system.
In the present embodiment, the primary image from the image sensor 220 has a light flux condition with a very deep depth. For this reason, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 240.

Here, the imaging process in the imaging apparatus 200 in the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position, if the wavefront aberration can be strictly calculated numerically through the optical system 210, the MTF can be calculated.

Therefore, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on an arbitrary imaging plane can be changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the desired wavefront formation is performed, the light flux emitted from the exit pupil is from the dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 240 including a DSP or the like at the subsequent stage. Thereby, the MTF value is significantly improved.

  Next, the response of the MTF of this embodiment and the conventional optical system will be considered.

FIG. 29 is a diagram showing the MTF response when the object is at the focal position and when the object is out of the focal position in the case of the conventional optical system.
FIG. 30 is a diagram illustrating the response of the MTF when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 31 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, the change in the response of the MTF is less than that in an optical system in which no light wavefront modulation element is inserted even when the object deviates from the focal position.
The MTF response of the image formed by this optical system is improved by the processing by the convolution filter.

The absolute value (MTF) of the OTF of the optical system having the phase plate shown in FIG. 30 is preferably 0.1 or more at the Nyquist frequency.
This is because, in order to achieve the OTF after restoration shown in FIG. 31, the gain is increased by the restoration filter, but the noise of the sensor is also raised at the same time. For this reason, it is preferable to perform restoration without increasing the gain as much as possible at high frequencies near the Nyquist frequency.
In the case of a normal optical system, resolution is achieved if the MTF at the Nyquist frequency is 0.1 or more.
Therefore, if the MTF before restoration is 0.1 or more, the restoration filter does not need to increase the gain at the Nyquist frequency. If the MTF before restoration is less than 0.1, the restored image becomes an image greatly affected by noise, which is not preferable.

Next, the blurred image restoration process of the image processing apparatus 240 will be described.
FIG. 32 is an MTF (Modulation Transfer Function) characteristic diagram at the best focus position of a general optical system.
FIG. 33 is a diagram showing the MTF characteristics of the optical system of the present embodiment.

Since the captured image obtained by the image sensor 220 after passing through the optical system 210 of the present embodiment is blurred, the MTF decreases from the middle range to the high range. This MTF is raised by calculation. In contrast to the MTF, which is the amplitude characteristic of a single lens, the total amplitude characteristic including image processing is called SFR (Spatial Frequency Response).
Since the frequency characteristic of a PSF that generates a blurred image is MTF, a blur restoration filter is designed to have a gain characteristic that is increased to a desired SFR characteristic. The degree of gain is determined by the balance with noise and false images.

  There are two methods for digitally filtering this blur restoration filter to the original image: Fourier transforming the image and multiplying the filter and frequency in the frequency domain, and convolution (convolution) in the spatial domain. is there. Here, the latter implementation method will be described. The convolution operation is expressed by the following equation.

Here, f indicates a filter kernel (here, the one rotated 180 degrees is used for easy calculation).
A indicates an original image, and B indicates a filtered image (blurred restored image).
As can be seen from this equation, f is superimposed on the image and the result of summing the products of the taps is taken as the value of the center coordinate that has been superimposed.

Next, a 3 * 3 filter will be specifically described with reference to FIGS. 34 (A) to 34 (C).
The restoration filter of FIG. 34 (A) (already rotated by 180 degrees) is overlaid with the filter center f (0,0) on A (i, j) of the blurred image shown in FIG. The product is taken and the nine total values are set as B (i, j) of the blurred restored image shown in FIG.
When (i, j) is scanned over the entire image, a new B image is generated. This is a digital filter. Here, since the filter is used for blur restoration, blur restoration processing can be performed by performing this processing.

Next, the configuration and processing of the image processing apparatus 240 will be described.
FIG. 35 is a block diagram illustrating a configuration example of the image processing apparatus according to the present embodiment.

  As shown in FIG. 35, the image processing apparatus 240 includes a raw (RAW) buffer memory 241, a convolution calculator 242, a kernel data storage ROM 243 serving as storage means, and a convolution control unit 244.

  The convolution control unit 244 controls the convolution process on / off, the screen size, the replacement of kernel data, and the like, and is controlled by the control device 250.

  The kernel data storage ROM 243 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIGS. Alternatively, the object distance information is acquired, and the kernel data is selected and controlled through the convolution control unit 244.

  In the example of FIG. 36, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

  In the example of FIG. 37, the kernel data A is data corresponding to the object distance information of 100 mm, the kernel data B is data corresponding to the object distance of 500 mm, and the kernel data C is data corresponding to the object distance of 4 m.

As described above, according to the present embodiment, the optical system 210 including the light wavefront modulation element that can rotate around the optical axis, the imaging element 220 that captures an image via the optical system, and a plurality of rotations An image processing device 240 that performs a blur restoration process using a blur restoration filter obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs) corresponding to angles, and an image captured at each of a plurality of rotation angles are synthesized. And a control device 250 that controls the depth of focus to be extended by applying a blur restoration filter to the synthesized image obtained in this manner, and can realize a blur that improves the restored image. The optical system can be simplified, the cost can be reduced, the image quality can be reduced, the influence of noise is small, the necessary direction can be accurately restored, and a good restored image can be obtained. There is a point.
In addition, the kernel size used in the convolution calculation and the coefficient used in the numerical calculation are made variable, know by input from the operation unit, etc., and by making the appropriate kernel size and the above-mentioned coefficient correspond, the magnification and defocus range can be set. There is an advantage that the lens can be designed without concern and the image can be restored by convolution with high accuracy.

In this embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 220 by the imaging lens 212 and the primary image FIM by the imaging element 220 are received. In addition, since the image processing apparatus 240 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, the high-definition image quality can be obtained. There is an advantage of becoming.
Further, the configuration of the optical system 210 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, with the trend of the times, high definition image quality is increasingly required, and in order to form a high definition image, the conventional imaging lens apparatus must have a complicated optical system. If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In the present embodiment, the same effect can be obtained even if the wavefront forming optical element of the optical system is arranged on the object side from the stop, on the same side as the stop, or on the imaging lens side from the stop.

  In the above-described embodiment, the control device 250 outputs the control signal CTL1 to the selection switching unit 230 and keeps the light wavefront modulation element 213a in the first position without rotating (rotating angle 0 degree). Then, the optical wavefront modulation element 213a is rotated by the control signal CTL1 (rotation angle 45 degrees), and the image is captured while being held at the second position. It is not limited to this.

For example, although not shown, a plurality of light wavefront modulation elements that can rotate around the optical axis are arranged in the optical system 210. In this case, for example, a first light wavefront modulation element corresponding to the reference position (rotation angle 0 degree) and a second light wavefront modulation element arranged to correspond to the rotation angle 45 are arranged.
The depth of focus is extended by applying a blur restoration filter to the captured image.
In this case, since the selection switching unit is not necessary, a motor or the like is unnecessary, and the apparatus configuration can be simplified and downsized.

Alternatively, one optical wavefront modulation element having characteristics equivalent to a plurality of optical wavefront modulation elements that are combined by rotating the optical axis is arranged in the optical system 210 in terms of optical characteristics.
The depth of focus is extended by applying a blur restoration filter to the captured image.
In this case, since only one light wavefront modulation element is required in the selection switching unit, it is possible to further simplify the apparatus configuration and reduce the size.

The optical system in FIG. 4 is an example, and the present invention is not necessarily used for the optical system in FIG. 5 and 6 are only examples of the spot shape, and the spot shape of the present embodiment is not necessarily shown in FIGS.
Also, the kernel data storage ROMs in FIGS. 36 and 37 are not necessarily used for the optical magnification, F number, and the size and value of each kernel. Also, the number of kernel data to be prepared is not limited to three.

1 is an external view showing an example of an information code reading device as an imaging device according to an embodiment of the present invention. It is a figure which shows an example of an information code. It is a block which shows the structural example of the imaging device applied to the information code reader of FIG. It is a figure which shows typically the structural example of the optical system which concerns on this embodiment. It is a figure which shows the spot shape of the image height center on the wide angle side. It is a figure which shows the spot shape of the image height center of a telephoto side. It is a figure which shows the structural example of the selection switching part which concerns on this embodiment. It is a figure which shows PSF2. It is a figure which shows MTF of PSF2 of FIG. It is a figure which shows synthetic | combination PSF3 which concerns on this embodiment. It is a figure which shows MTF of synthetic | combination PSF. It is a figure which shows analog PSF after a reference position (rotation angle 0 degree) and predetermined angle rotation (45 degree rotation). It is a figure which shows the digital PSF after the A / D conversion of the analog PSF of the reference position (rotation angle 0 degree) of FIG. It is a figure which shows the digital PSF after the A / D conversion of the analog PSF after the predetermined angle rotation (45 degree rotation) of FIG. It is a figure which shows digital PSF after synthesize | combining 0 degree and 45 degree PSF, and its MTF. It is a figure which shows an example of an original image. It is a figure which shows the blurring image in the case of a reference position (rotation angle 0 degree). It is a figure which shows the decompression | restoration image of rotation angle 0 degree. It is a restored image with a rotation angle of 45 degrees. It is a figure which shows the synthetic | combination blur image of rotation angle 0 degree and 45 degree | times. It is a figure which shows the synthetic | combination restoration image of rotation angle 0 degree and 45 degree | times. It is a figure for demonstrating the principle of DEOS. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a focus point ( Best focus), (C) is a diagram showing each spot image when the focal point is deviated by −0.2 mm (Defocus = −0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image pick-up element concerning this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image pick-up element of an image pick-up lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure which shows the shape of the wavefront aberration represented by a type | formula, when the optical axis of the optical system containing the optical wavefront modulation element of this embodiment is set to az axis, and two mutually orthogonal axes are set to x and y. It is the figure which represented the shape of the wavefront aberration and the range below 0.5 (lambda) with the thick line. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. It is a MTF (Modulation Transfer Function amplitude characteristic) characteristic figure in the best focus (Best Focus) position of a general optical system. It is a figure which shows the MTF characteristic of the optical system of this embodiment. It is explanatory drawing of the convolution calculation of a blur decompression | restoration process in this embodiment. It is a block diagram which shows the structural example of the image processing apparatus which concerns on this embodiment. It is a figure which shows an example of the storage data of kernel data ROM. It is a figure which shows the other example of the stored data of kernel data ROM. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element of the imaging lens apparatus of FIG. In the case (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Information code reading device, 121 ... Information code, 200 ... Imaging device, 210 ... Optical system, 220 ... Imaging element, 230 ... Selection switching part, 240 ... Image Processing device 250... Control device 211... Object side lens 212. Image forming lens 213 Wavefront forming optical element 213 a Phase plate (light wavefront modulation element) 242 ... Convolution calculator, 243 ... Kernel data storage ROM, 244 ... Convolution controller.

Claims (11)

An optical system including an optical wavefront modulation element rotatable around an optical axis;
An image sensor that captures an image via the optical system;
A blur restoration filter obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs) corresponding to a plurality of rotation angles;
An image pickup apparatus comprising: a control unit that performs control to perform depth of focus expansion by applying the blur restoration filter to a composite image obtained by combining images picked up at each of the plurality of rotation angles. An optical system including a plurality of optical wavefront modulation elements combined by rotating around an optical axis;
An image sensor that captures an image via the optical system;
A blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to each rotation angle of the plurality of light wavefront modulation elements;
An imaging apparatus comprising: a control unit that performs depth-of-focus expansion by applying the blur restoration filter to the captured image. An optical system including one light wavefront modulation element;
An image sensor that captures an image via the optical system;
A blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to a plurality of rotation angles of an optical wavefront modulation element biased in the blur spreading direction;
An imaging apparatus comprising: a control unit that performs depth-of-focus expansion by applying the blur restoration filter to the captured image. The controller is
The imaging apparatus according to claim 1, wherein different weightings are performed on the plurality of PSFs so as to combine the plurality of PSFs. The controller is
The imaging apparatus according to claim 4, wherein the exposure time is changed in accordance with a weighting ratio of the plurality of PSFs so as to perform imaging at a plurality of rotation angles. The controller is
The imaging device according to claim 4, wherein the weighting and / or exposure time ratio is controlled so as to be heavily distributed to a rotation angle that places importance on restoration performance. The controller is
The imaging apparatus according to claim 6, wherein the rotation angle that places importance on the restoration performance is at least one of a vertical direction and a horizontal direction of an imaging field angle. The imaging apparatus according to claim 1, wherein the rotation angle difference is 45 degrees or 135 degrees. Capturing a plurality of images via an optical system including an optical wavefront modulation element rotatable about an optical axis;
Synthesizing images captured at a plurality of rotation angles;
An imaging method comprising: expanding a depth of focus by applying a blur restoration filter obtained from a PSF obtained by synthesizing individual point image intensity distributions (PSFs) corresponding to the plurality of rotation angles to a synthesized image. Capturing an image through an optical system including a plurality of light wavefront modulation elements combined by rotating around an optical axis;
The depth of focus is extended by applying a blur restoration filter obtained from a PSF obtained by synthesizing a point image intensity distribution (PSF) corresponding to each rotation angle of the plurality of light wavefront modulation elements to the captured image. An imaging method comprising: Capturing an image via an optical system including one light wavefront modulation element;
By applying a blur restoration filter obtained from a PSF obtained by synthesizing point image intensity distributions (PSFs) corresponding to a plurality of rotation angles of a light wavefront modulation element that is biased in a blur spreading direction to a captured image. An imaging method comprising: performing a depth of focus extension.