JP2009159535A - Imaging apparatus, and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus and an image processing method, capable of carrying out an image recovery process only for a necessary part and improving efficiency of the system. <P>SOLUTION: The imaging apparatus has an optical system 110 including an optical wave surface modulating element, an imaging element 120 which images an image of an object to be imaged that passes through the optical system 110, and an image processing device 140 which subjects the image data of the object to be imaged from the imaging element 120 to a predetermined process. The image processing device 140 recovers dispersed images of a selected particular area as an image signal free from dispersion. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image pickup apparatus using an image pickup device and applicable to a digital still camera equipped with an optical system, a camera mounted on a mobile phone, a camera mounted on a portable information terminal, an image inspection device, an industrial camera for automatic control, an information code reader, and the like. And an image processing method.

In response to the digitization of information, which has been rapidly developing in recent years, the response in the video field is also remarkable.
In particular, as symbolized by a digital camera, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, which is a solid-state imaging device, is used for the imaging surface in place of a conventional film.

  As described above, an imaging lens device using a CCD or CMOS sensor as an imaging element is for taking an image of a subject optically by an optical system and extracting it as an electrical signal by the imaging element. In addition to a digital still camera, It is used for a video camera, a digital video unit, a personal computer, a mobile phone, a personal digital assistant (PDA), an image inspection apparatus, an industrial camera for automatic control, and the like.

FIG. 23 is a diagram schematically illustrating a configuration and a light flux state of a general imaging lens device.
The imaging lens device 1 includes an optical system 2 and an imaging element 3 such as a CCD or CMOS sensor.
In the optical system, the object side lenses 21 and 22, the diaphragm 23, and the imaging lens 24 are sequentially arranged from the object side (OBJS) toward the image sensor 3 side.

In the imaging lens device 1, as shown in FIG. 23, the best focus surface is matched with the imaging element surface.
24A to 24C show spot images on the light receiving surface of the imaging element 3 of the imaging lens device 1.

In addition, an imaging apparatus has been proposed in which a light beam is regularly dispersed by a phase plate that is a light wavefront modulation element and restored by digital processing to enable imaging with a deep depth of field (for example, non-patent literature). 1, 2, and patent documents 1 to 5).
“Wavefront Coding; jointly optimized optical and digital imaging systems”, Edward R. Dowski, Jr., Robert H. Cormack, Scott D. Sarama. “Wavefront Coding; A modern method of achieving high performance and / or low cost imaging systems”, Edward R. Dowski, Jr., Gregory E. Johnson. USP 6,021,005 USP 6,642,504 USP 6,525,302 USP 6,069,738 JP 2003-235794 A

By the way, the depth expansion system proposed in the above proposal performs image restoration on the entire screen.
For this reason, in an optical system aimed at recognizing a part of the screen, there is a problem that a system that performs processing on the entire screen is inefficient.

  An object of the present invention is to provide an imaging apparatus and an image processing method capable of performing image restoration processing only on necessary portions and improving the efficiency of the system.

  An image pickup apparatus according to a first aspect of the present invention provides a depth extension optical system including a light wavefront modulation element, an image pickup element that picks up a subject image that has passed through the depth extension optical system, and an image signal from the image pickup element. Selecting means for selecting a predetermined area, and an image signal processing section for performing predetermined image processing on the image signal of the selected area, wherein the image signal processing section distributes the selected area The image is restored as an image signal without dispersion.

  Preferably, the predetermined area is predetermined in the image.

  Preferably, the predetermined area is determined by extracting a characteristic part from the image and selecting the part.

  Preferably, the feature part extraction processing is performed between photoelectric conversion of an image by the imaging device and restoration processing by the predetermined image processing.

  Preferably, prior to the process of extracting the characteristic part, a binarization process is performed on the image signal from the image sensor.

  Preferably, color arrangement information is used in the process of extracting the characteristic part.

  A second aspect of the present invention is an image processing method for capturing a subject image that has passed through a depth expansion optical system including a light wavefront modulation element, and performing image processing on the captured image data. A selection step for selecting a predetermined region with respect to the image, and an image signal processing step for performing predetermined image processing on the image signal of the selected region. In the image signal processing step, the selected region is selected. The dispersed image of the area is restored as an image signal without dispersion.

  According to the present invention, it is possible to perform image restoration processing only on necessary portions, and the efficiency of the system can be improved.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram illustrating an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 100 according to the present embodiment includes an optical system 110, an imaging element 120, an analog front end unit (AFE) 130, an image processing device 140, a camera signal processing unit 150, an image display memory 160, an image monitoring device 170, and an operation unit. 180 and a control device 190.

As will be described in detail later, the imaging apparatus 100 according to the present embodiment applies a light wavefront modulation element to the optical system 110, regularly disperses the light flux by the light wavefront modulation element, and restores the light by digital processing. Adopting a light wavefront modulation optical system or a depth expansion optical system (DEOS: Optical System) that enables imaging with deep depth of field, the subject image (for example, an information code such as a barcode) is accurately obtained. It can be read with high accuracy.
The light wavefront modulation function can be expressed by arranging the light wavefront modulation element in the optical system as described above, but is expressed by the light wavefront modulation surface formed in the optical element forming the optical system. It is also possible to configure as described above.

  The optical system 110 includes an optical wavefront modulation element such as a phase modulation element, which will be described in detail later, and supplies an image obtained by photographing the subject object OBJ to the imaging element 120.

The image sensor 120 functions as a photoelectric conversion element, an image captured by the optical system 110 is formed, and the imaged primary image information is converted into an electrical signal primary image signal FIM through the analog front end unit 130. It consists of a CCD or CMOS sensor output to the processing device 140.
In FIG. 1, the image sensor 120 is described as a CCD as an example.

The analog front end unit 130 includes a timing generator 131 and an analog / digital (A / D) converter 132.
The timing generator 131 generates the drive timing of the CCD of the image sensor 120, and the A / D converter 132 converts an analog signal input from the CCD into a digital signal and outputs it to the image processing device 140.

An image processing apparatus (two-dimensional convolution means) 140 constituting a part of the image signal processing unit inputs a digital signal of a captured image coming from the previous AFE 130, performs two-dimensional convolution processing, and performs a subsequent camera signal. The data is transferred to the processing unit (DSP) 150.
Further, for example, the image processing apparatus 140 has a function as a selection unit that selects a predetermined area for the image signal from the image sensor 120, and restores the dispersed image signal of the selected area as an image signal without dispersion. It has a function.
In the present embodiment, the predetermined region to be selected is determined in advance in the image, for example.
Further, the predetermined region can be configured to be determined by extracting a characteristic part from the image and selecting the part.
In this case, the image processing apparatus 140 performs the feature part extraction process between the photoelectric conversion of the image by the image sensor 120 and the restoration process by predetermined image processing.
In the present embodiment, the binarization process is performed on the image signal from the image sensor 120 prior to the feature part extraction process.
In addition, color arrangement information can also be used in the feature part extraction process.
This will be illustrated later.
The filtering performed by the image processing apparatus 140 includes a function for increasing the contrast.
The image processing device 140 performs a filtering process on the image data according to various types of information from the control device 190.
The image processing apparatus 140 has a function of performing noise reduction filtering in the first step.

As shown in FIG. 1, the image processing apparatus 140 basically includes a raw (RAW) buffer memory 141, a convolution calculator 142, a kernel data storage ROM 143 as storage means, and a convolution control unit 144.
Note that the image processing apparatus 140 in FIG. 1 shows only an image restoration processing system.

  The convolution control unit 144 performs control such as on / off of the convolution process, screen size, and kernel data exchange, and is controlled by the control device 190.

  The kernel data storage ROM 143 stores kernel data for convolution calculated by the PSF of each optical system prepared in advance as shown in FIG. 2 or FIG. 3, for example, and passes through the convolution control unit 144. Select and control kernel data.

  In the example of FIG. 2, the kernel data A is data corresponding to the optical magnification (× 1.5), the kernel data B is data corresponding to the optical magnification (× 5), and the kernel data C is data corresponding to the optical magnification (× 10).

In the example of FIG. 3, the kernel data A is data corresponding to an object distance information of 100 mm, the kernel data B is data corresponding to an object distance of 500 mm, and the kernel data C is data corresponding to an object distance of 4 m.
The processing of the image processing apparatus 140 will be described in detail later.

  The camera signal processing unit (DSP) 150 performs processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing, and performs storage in the image display memory 160, image display on the image monitoring device 170, and the like.

  The control device 190 performs exposure control, has operation inputs such as the operation unit 180, determines the operation of the entire system in accordance with those inputs, and controls the AFE 130, the image processing device 140, the DSP 150, etc. It is responsible for overall mediation control.

  The control device 190 performs selection control (extraction control) of a predetermined area in the image processing device 140, for example.

The principle of the image restoration process that is a feature of the present embodiment has been described above.
Hereinafter, the configuration and functions of the optical system and the image processing apparatus related to the DEOS of this embodiment will be described in detail.

FIG. 4 is a diagram schematically illustrating a configuration example of the zoom optical system 110 according to the present embodiment. This figure shows the wide angle side.
FIG. 5 is a diagram schematically illustrating a configuration example of a zoom optical system on the telephoto side of the imaging lens device according to the present embodiment.
FIG. 6 is a diagram showing a spot shape at the center of the image height on the wide angle side of the optical system according to the present embodiment, and FIG. 7 is a spot shape at the center of the image height on the telephoto side of the optical system according to the present embodiment. FIG.

4 and 5 includes an object-side lens 111 disposed on the object-side OBJS, an imaging lens 112 for forming an image on the image sensor 120, and between the object-side lens 111 and the imaging lens 112. An optical wavefront modulation element (wavefront forming optical element) group 113 made of, for example, a phase plate having a three-dimensional curved surface, which is disposed and deforms a wavefront of an image formed on the light receiving surface of the imaging element 120 by the imaging lens 112. . A stop (not shown) is disposed between the object side lens 111 and the imaging lens 112.
For example, in the present embodiment, a variable aperture 110a (see FIG. 1) is provided, and the aperture degree of the variable aperture is controlled in exposure control (apparatus).

In the present embodiment, the case where the phase plate is used has been described. However, the optical wavefront modulation element of the present invention may be any element that deforms the wavefront, and an optical element whose thickness changes. (For example, the above-mentioned third-order phase plate), an optical element whose refractive index changes (for example, a gradient index wavefront modulation lens), an optical element whose thickness and refractive index change due to coding on the lens surface (for example, a wavefront) A light wavefront modulation element such as a modulation hybrid lens or a phase plane formed on the lens surface) or a liquid crystal element capable of modulating the phase distribution of light (for example, a liquid crystal spatial phase modulation element). .
Further, in the present embodiment, the case where a regularly dispersed image is formed using a phase plate that is a light wavefront modulation element has been described. However, the lens used as a normal optical system has the same rule as the light wavefront modulation element. When an image that can form a dispersed image is selected, it can be realized only by an optical system without using a light wavefront modulation element. In this case, it does not correspond to the dispersion caused by the phase plate described later, but corresponds to the dispersion caused by the optical system.

The phase plate 113a shown in FIGS. 4 and 5 is an optical lens that regularly disperses the light beam converged by the optical system. By inserting this phase plate, an image that does not fit anywhere on the image sensor 120 is realized.
In other words, the phase plate 113a forms a deep luminous flux (which plays a central role in image formation) and a flare (blurred portion).
Means for restoring the regularly dispersed image to a focused image without moving the optical system 110 by digital processing is a wavefront aberration control optical system system or a depth expansion optical system (DEOS: Depth Expansion Optical system system). This processing is performed in the image processing apparatus 140.

Here, the basic principle of DEOS will be described.
As shown in FIG. 8, when the subject image f enters the DEOS optical system H, a g image is generated.
This is expressed by the following equation.

[Equation 1]
g = H * f
However, * represents convolution.

  In order to obtain the subject from the generated image, the following processing is required.

[Equation 2]
f = H ^-1 * g

Here, the kernel size and calculation coefficient regarding H will be described.
The optical switching information is KPn, KPn-1,. In addition, each H function is defined as Hn, Hn-1,.
Since each spot image is different, each H function is as follows.

The difference in the number of rows and / or the number of columns in this matrix is the kernel size, and each number is the operation coefficient.
Here, each H function may be stored in a memory, and the PSF is set as a function of the object distance, and is calculated based on the object distance. By calculating the H function, an optimum object distance is obtained. It may be possible to set so as to create a filter. Alternatively, the H function may be directly obtained from the object distance using the H function as a function of the object distance.

  In this embodiment, as shown in FIG. 1, an image from the optical system 110 is received by the image sensor 120 and input to the image processing device 140 when the aperture is opened, and a conversion coefficient corresponding to the optical system is acquired. The image signal having no dispersion is generated from the dispersion image signal from the image sensor 120 with the obtained conversion coefficient.

  In the present embodiment, as described above, dispersion refers to forming a non-focused image on the image sensor 120 by inserting the phase plate 113a as described above. It plays a central role in image formation) and a phenomenon of forming flare (blurred portion), and includes the same meaning as aberration because of the behavior of the image being dispersed to form a blurred portion. Therefore, in this embodiment, it may be described as aberration.

In the present embodiment, DEOS can be employed to obtain high-definition image quality, and the optical system can be simplified and the cost can be reduced.
Hereinafter, this feature will be described.

FIGS. 9A to 9C show spot images on the light receiving surface of the image sensor 120.
9A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), FIG. 9B shows a case where the focal point is a focal point (Best focus), and FIG. 9C shows a case where the focal point is shifted by −0.2 mm. In this case, each spot image is shown (Defocus = −0.2 mm).
As can be seen from FIGS. 9A to 9C, in the imaging apparatus 100 according to the present embodiment, a light beam having a deep depth (a central role in image formation) is obtained by the wavefront forming optical element group 113 including the phase plate 113a. And flare (blurred part) are formed.

  As described above, the primary image FIM formed in the imaging apparatus 100 of the present embodiment has a light beam condition with a very deep depth.

FIGS. 10A and 10B are diagrams for explaining a modulation transfer function (MTF) of a primary image formed by the imaging lens device according to the present embodiment. FIG. 10A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens device, and FIG. 10B shows the MTF characteristics with respect to the spatial frequency.
In this embodiment, the high-definition final image is left to the correction processing of the image processing apparatus 140 including a digital signal processor (Digital Signal Processor), for example, as shown in FIGS. 10 (A) and 10 (B). The MTF of the primary image is essentially a low value.

  As described above, the image processing apparatus 140 receives the primary image FIM from the image sensor 120 and performs a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image to form a high-definition final image FNLIM. To do.

The MTF correction processing of the image processing apparatus 140 is performed after edge enhancement, chroma enhancement, etc., using the MTF of the primary image, which is essentially a low value, as shown by a curve A in FIG. In the process, correction is performed so as to approach (reach) the characteristic indicated by the curve B in FIG.
A characteristic indicated by a curve B in FIG. 11 is a characteristic obtained when the wavefront is not deformed without using the wavefront forming optical element as in the present embodiment, for example.
It should be noted that all corrections in the present embodiment are based on spatial frequency parameters.

In the present embodiment, as shown in FIG. 11, in order to achieve the MTF characteristic curve B that is finally realized with respect to the optically obtained MTF characteristic curve A with respect to the spatial frequency, each spatial frequency is changed to each spatial frequency. On the other hand, the original image (primary image) is corrected by applying strength such as edge enhancement as shown in FIG.
For example, in the case of the MTF characteristic of FIG. 11, the curve of edge enhancement with respect to the spatial frequency is as shown in FIG.

  That is, a desired MTF characteristic curve B is virtually realized by performing correction by weakening edge enhancement on the low frequency side and high frequency side within a predetermined spatial frequency band and strengthening edge enhancement in the intermediate frequency region. To do.

As described above, the imaging apparatus 100 according to the embodiment basically includes the optical system 110 and the imaging element 120 that form a primary image, and the image processing apparatus 140 that forms the primary image into a high-definition final image. In the optical system, a wavefront shaping optical element is newly provided, or an optical element such as glass, plastic or the like is formed for wavefront shaping, thereby forming an imaging wavefront. The wavefront is deformed (modulated), and an image of such a wavefront is formed on the image pickup surface (light receiving surface) of the image pickup device 120 formed of a CCD or CMOS sensor. An image forming system.
In the present embodiment, the primary image from the image sensor 120 has a light beam condition with a very deep depth. For this reason, the MTF of the primary image is essentially a low value, and the MTF is corrected by the image processing device 140.

Here, the imaging process in the imaging apparatus 100 according to the present embodiment will be considered in terms of wave optics.
A spherical wave diverging from one of the object points becomes a convergent wave after passing through the imaging optical system. At that time, aberration occurs if the imaging optical system is not an ideal optical system. The wavefront is not a spherical surface but a complicated shape. Wavefront optics lies between geometric optics and wave optics, which is convenient when dealing with wavefront phenomena.
When dealing with the wave optical MTF on the imaging plane, the wavefront information at the exit pupil position of the imaging optical system is important.
The MTF is calculated by Fourier transform of the wave optical intensity distribution at the imaging point. The wave optical intensity distribution is obtained by squaring the wave optical amplitude distribution, and the wave optical amplitude distribution is obtained from the Fourier transform of the pupil function in the exit pupil.
Further, since the pupil function is exactly from the wavefront information (wavefront aberration) at the exit pupil position itself, if the wavefront aberration can be strictly calculated numerically through the optical system 110, the MTF can be calculated.

Accordingly, if the wavefront information at the exit pupil position is modified by a predetermined method, the MTF value on the imaging plane can be arbitrarily changed.
In this embodiment, the wavefront shape is mainly changed by the wavefront forming optical element, but the target wavefront is formed by increasing or decreasing the phase (phase, optical path length along the light beam). .
Then, if the desired wavefront is formed, the exiting light flux from the exit pupil is made up of dense and sparse portions of the light, as can be seen from the geometric optical spot images shown in FIGS. It is formed.
The MTF in the luminous flux state shows a low value at a low spatial frequency and a characteristic that the resolving power is managed up to a high spatial frequency.
That is, if this MTF value is low (or such a spot image state in terms of geometrical optics), the phenomenon of aliasing will not occur.
That is, a low-pass filter is not necessary.
Then, the flare-like image that causes the MTF value to be lowered may be removed by the image processing apparatus 140 made up of a later stage DSP or the like. Thereby, the MTF value is significantly improved.

  Next, the MTF response of this embodiment and a general optical system will be considered.

FIG. 13 is a diagram illustrating the MTF response when the object is at the focal position and out of the focal position in the case of a general optical system.
FIG. 14 is a diagram showing the response of the MTF when the object is at the focal position and when the object is out of the focal position in the optical system of the present embodiment having the light wavefront modulation element.
FIG. 15 is a diagram illustrating a response of the MTF after data restoration of the imaging apparatus according to the present embodiment.

As can be seen from the figure, in the case of an optical system having a light wavefront modulation element, the change in the response of the MTF is less than that in an optical system in which no light wavefront modulation element is inserted even when the object deviates from the focal position.
The response of the MTF is improved by processing the image formed by this optical system using a convolution filter.

The absolute value (MTF) of the OTF of the optical system having the phase plate shown in FIG. 14 is preferably 0.1 or more at the Nyquist frequency.
This is because, in order to achieve the OTF after restoration shown in FIG. 15, the gain is increased by the restoration filter, but the noise of the sensor is also raised at the same time. For this reason, it is preferable to perform restoration without increasing the gain as much as possible at high frequencies near the Nyquist frequency.
In the case of a normal optical system, resolution is achieved if the MTF at the Nyquist frequency is 0.1 or more.
Therefore, if the MTF before restoration is 0.1 or more, the restoration filter does not need to increase the gain at the Nyquist frequency. If the MTF before restoration is less than 0.1, the restored image becomes an image greatly affected by noise, which is not preferable.

  Hereinafter, a configuration example in a case where a predetermined region selected in the imaging apparatus 100 having the above configuration is used as a location necessary for authentication will be described.

In the present embodiment, as described above, the image restoration processing for depth extension is performed only on the target region. The target area is fixed or selected by feature point extraction.
As for application to an authentication system or information code reading, it is possible to improve the accuracy of feature point extraction by binarizing image data to make the outline clear.

That is, it is possible to improve the efficiency of the system by performing the image restoration process only on the part necessary for authentication.
Preferably, a part necessary for authentication is registered in advance as a fixed position in the screen, and the restoration process is performed only on that part, so that the efficiency of the system can be improved.
In addition, after acquiring images, the system searches for the locations required for authentication, restores images only for those locations, and then moves to authentication processing. It is possible to make a system with

  In addition, the depth extension optical system including the light wavefront modulation element can widen the depth region that retains the resolving power as compared with the normal optical system. Therefore, when the screen is binarized as a method for searching for a place necessary for authentication, it is possible to perform authentication in a wider depth region than usual by maintaining the resolving power.

  Further, the depth extension optical system including the light wavefront modulation element tends to have a low contrast in the image before the restoration process. However, since there is no influence on the color arrangement, the subject to be authenticated is suitable for identification by the color arrangement. Therefore, the color arrangement information is used as a method for detecting the part necessary for the authentication. It becomes possible.

  FIG. 16 is a diagram for explaining an example in which a predetermined region to be selected is predetermined in an image.

In the example of FIG. 16, the imaging apparatus 100 according to the present embodiment is mounted on a monitoring camera (not shown). For example, a two-dimensional information code 211 such as a QR code attached to a product 210 flowing on the belt conveyor 200 is displayed at the center of the screen. It is a case to read by. The two-dimensional information code 211 may be a color information code.
The inside of the dotted frame 220 in the figure is a position necessary for authentication fixed in the screen. This position information is registered.
As shown in FIG. 16A, authentication is not performed in a state where the authentication dotted color information code is not included in the fixed dotted line frame 220.
On the other hand, as shown in FIG. 16B, when an authentication code is included in the fixed dotted frame 220, authentication is performed.
Further, by performing image restoration only on a position necessary for authentication and extending the depth, authentication with a depth can be performed as shown in FIG.

FIG. 17 is a flowchart in the case where the process of searching for a place necessary for authentication is performed between the image acquisition and the image restoration process.
In other words, FIG. 17 is a flowchart showing that the feature part extraction processing is performed between photoelectric conversion of an image by the image sensor 120 and restoration processing by predetermined image processing.

  Due to the characteristics of this optical system, the resolution of the image is maintained, but the contrast is deteriorated. That is, since the authentication location can be sufficiently searched (ST11 and ST12), authentication is performed after image acquisition (ST13), and only the necessary location is restored (ST14), thereby reducing the amount of image processing.

  FIGS. 18A and 18B are diagrams for explaining a method of binarizing the screen as a method of searching for a place necessary for authentication.

Due to the characteristics of the present optical system, the contour is correctly left by the binarization process.
For example, as shown in FIG. 18A, if the target is a face 231 of the living body 230, the face is recognized if eyes, nose, mouth, etc. are confirmed inside the face outline.
As shown in FIG. 18B, in the license plate 241 of the car 240, the format described in the number is the same in Japan, so that it can be recognized as a number if characters are confirmed in an appropriate format.

  FIG. 19 is a diagram for explaining an example of processing using color arrangement information as a method for detecting a location necessary for authentication.

In this example, the color label in the back label of the file 250 is searched. In the figure, Y indicates a yellow label, G indicates a green label, B indicates a blue label, and R indicates a red label.
For example, a square label of yellow Y is searched, and a restoration location is determined based on the label.
In addition, an arrangement of several kinds of colors is searched, and a restoration location is determined based on the search.

  Next, a specific example of extraction processing using color array information will be described.

(1): Extraction of Shape For example, Hough transform is known as a method for detecting a straight line or a circle from an image. (US patent by PVCHough in 1962) This extracts the shape.

(2): About color extraction It is possible to convert RGB to HSV and use methods such as hue (hue), saturation (saturation) and value (lightness), and methods using the ratio of RGB components. is there.

(3): Arrangement processing Basically, it can be handled by combining the above (1) and (2).
For example, a search is made as to whether the shape matches a predetermined shape by capturing the contour.
Next, the color information in the outline is acquired.

<Method of extracting feature points for authentication>
(11): Barcode extraction For example, a threshold value is acquired from the entire screen by the “Otsu method” and binarized.
Next, a rectangular area is extracted from the binarization result.

(12): Biometric authentication From the biometric data captured as an image, an environmental factor unnecessary for the determination process is removed.
Next, the spatial position, size, temporal change, etc. are normalized, and a comparison with the stored template is performed.

Here, the color arrangement information code will be further described.
In this embodiment, as shown in FIG. 20, an area in which only one of R, G, and B used in the color information code is sensed extremely higher than the other two colors is shown as the color information code. As an area candidate. When these candidate areas are adjacent and the outline of the set matches the assumed color information code outline shape (for example, square), it is possible to adopt a method of extracting the range as the position of the color information code It is.

FIG. 20 is a diagram illustrating an image example of the color arrangement information code.
FIG. 21 is a flowchart showing the color information code processing.

For example, when an image as shown in FIG. 20 is captured, the following processing is performed.
An area in which only R, G, and B (color components used in the color information code) color components appear strongly in the screen is regarded as an area candidate for the color information code (ST21).
(Pixels labeled R, G, and B in FIG. 20 indicate that the color component is detected higher than the other color components.)
If these areas are adjacent to each other (ST22), a group of color information code area candidates is set (ST23).
When the extraction of the area is completed (ST24), it is determined whether or not the outline of the aggregate of the areas applies to an approximately square shape (ST25).
If the outline matches the square shape, the color information code is decoded (ST26).

  Next, feature point extraction related to authentication will be described.

<For biometric authentication>
22 (A) and 22 (B) are diagrams showing an example of a part to be authenticated by hand. Here, the phalanx (region S1), middle finger (region S2), anonymous finger (region S3), It is the schematic which divided the little finger ball (area | region S4) and the palm into area | region S5-S20.

(21): Perform contour extraction of a target shape (for example, a hand).
(22): Apply electronic zoom to a predetermined size for size normalization.
(23): Extraction of a part to be subjected to biometric authentication (for example, each fingertip for a fingerprint and the whole palm for a vein)
It is done by including the procedure like that.

<For barcode authentication>
A two-dimensional barcode (QR code) will be described as an example.

(31): A threshold value (predetermined value, average of in-screen luminance, etc) is determined, and the image is binarized.
(32): Extract a shape having a square filled in the center of a square frame characteristically provided at the three corners of the QR code as a feature point (33): After extracting the three feature points, Cut out the area that has the positional relationship of the upper left corner, upper right corner, lower left corner,
This makes it possible to extract a QR code.

  As described above, in the present embodiment, image restoration processing for depth extension is performed only on a target region. The target area is fixed or selected by feature point extraction.

As described above, according to the present embodiment, the optical system 110 including the light wavefront modulation element, the image sensor 120 that captures the subject image that has passed through the optical system 110, and the image data of the subject from the image sensor 120. An image processing device 140 that performs predetermined processing on the image, and the image processing device 140 restores the dispersed image of the selected specific area as an image signal without dispersion. There is an advantage that the restoration process can be performed and the efficiency of the system can be improved.
As for application to an authentication system or information code reading, it is possible to improve the accuracy of feature point extraction by binarizing image data to make the outline clear.

Further, according to the present embodiment, the optical system 110 and the imaging device 120 including the light wavefront modulation element that forms the primary image, and the image processing device 140 that forms the primary image into a high-definition final image are included. The imaging element 120 and the phase modulation element (phase plate) 113a as the light wavefront modulation element can be arranged suitable for restoration, and a depth extension optical system with a narrow tolerance can be realized.
As a result, there is an advantage that a blur that improves the restored image can be realized, and that a good restored image can be obtained with an appropriate image quality and little influence of noise.
In addition, there is an advantage that the optical system can be simplified, the cost can be reduced, and a restored image with an appropriate image quality and less influenced by noise can be obtained.
In addition, the kernel size used at the time of convolution calculation and the coefficient used in the numerical calculation are made variable, know by input from the operation unit 180, etc. There is an advantage that the lens can be designed without worrying about the image and that the image can be restored by convolution with high accuracy.
In addition, a so-called natural image is obtained in which the object to be photographed is in focus and the background is blurred without requiring a highly difficult, expensive and large optical lens and without driving the lens. You can also.
The imaging apparatus 100 according to the present embodiment can be used in a DEOS optical system in consideration of the small size, light weight, and cost of consumer devices such as digital cameras and camcorders.

In the present embodiment, the imaging lens system having a wavefront forming optical element that deforms the wavefront of the imaging on the light receiving surface of the imaging element 120 by the imaging lens 112 and the primary image FIM by the imaging element 120 are received. In addition, the image processing apparatus 140 that forms a high-definition final image FNLIM by performing a predetermined correction process or the like that raises the MTF at the spatial frequency of the primary image, and so on, can achieve high-definition image quality. There is an advantage of becoming.
In addition, the configuration of the optical system 110 can be simplified, manufacturing becomes easy, and cost reduction can be achieved.

By the way, when a CCD or CMOS sensor is used as an image sensor, there is a resolution limit determined by the pixel pitch, and if the resolution of the optical system exceeds the limit resolution, a phenomenon such as aliasing occurs, which adversely affects the final image. It is a well-known fact that
In order to improve image quality, it is desirable to increase the contrast as much as possible, but this requires a high-performance lens system.

However, as described above, aliasing occurs when a CCD or CMOS sensor is used as an image sensor.
Currently, in order to avoid the occurrence of aliasing, the imaging lens apparatus uses a low-pass filter made of a uniaxial crystal system to avoid the occurrence of aliasing.
The use of a low-pass filter in this way is correct in principle, but the low-pass filter itself is made of crystal, so it is expensive and difficult to manage. Moreover, there is a disadvantage that the use of the optical system makes the optical system more complicated.

As described above, in order to form a high-definition image, the optical system must be complicated in the conventional imaging lens apparatus in spite of the demand for higher-definition image due to the trend of the times. . If it is complicated, manufacturing becomes difficult, and if an expensive low-pass filter is used, the cost increases.
However, according to this embodiment, the occurrence of aliasing can be avoided without using a low-pass filter, and high-definition image quality can be obtained.

  In this embodiment, the example in which the wavefront forming optical element of the optical system is arranged closer to the object side lens than the stop is shown. An effect can be obtained.

1 is a block configuration diagram showing an embodiment of an imaging apparatus according to the present invention. It is a figure which shows an example (optical magnification) of the storage data of kernel data ROM. It is a figure which shows the other example of the stored data of kernel data ROM. It is a figure which shows typically the structural example of the zoom optical system of the wide angle side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows typically the structural example of the zoom optical system of the telephoto side of the imaging lens apparatus which concerns on this embodiment. It is a figure which shows the spot shape of the image height center on the wide angle side. It is a figure which shows the spot shape of the image height center of a telephoto side. It is a figure for demonstrating the principle of DEOS. It is a figure which shows the spot image in the light-receiving surface of the image pick-up element which concerns on this embodiment, Comprising: (A) is a case where a focus shifts 0.2 mm (Defocus = 0.2 mm), (B) is a case where it is a focusing point ( (Best focus), (C) is a diagram showing each spot image when the focal point is shifted by -0.2 mm (Defocus = -0.2 mm). It is a figure for demonstrating MTF of the primary image formed with the image sensor which concerns on this embodiment, Comprising: (A) is a figure which shows the spot image in the light-receiving surface of the image sensor of an imaging lens apparatus, (B ) Shows the MTF characteristics with respect to the spatial frequency. It is a figure for demonstrating the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure for demonstrating concretely the MTF correction process in the image processing apparatus which concerns on this embodiment. It is a figure which shows the response (response) of MTF when an object exists in a focus position in the case of the conventional optical system, and when it remove | deviated from the focus position. It is a figure which shows the response of MTF when an object exists in a focus position in the case of the optical system of this embodiment which has a light wavefront modulation element, and remove | deviates from a focus position. It is a figure which shows the response of MTF after the data restoration of the imaging device which concerns on this embodiment. It is a figure for demonstrating the example in which the predetermined area | region selected is predetermined within the image. It is a flowchart in case the process which searches a location required for authentication is performed between an image acquisition and an image restoration process. It is a figure for demonstrating the method of binarizing a screen as a method of searching a location required for authentication. It is a figure for demonstrating the example processed using color arrangement information as a method of detecting the location required for authentication. It is a figure which shows the example of an image of a color arrangement | sequence information code. It is a flowchart which shows the process of a color information code. It is a figure which shows the example which showed the site | part which the hand authenticates. It is a figure which shows typically the structure and light beam state of a general imaging lens apparatus. FIG. 24A is a diagram showing a spot image on the light receiving surface of the imaging element of the imaging lens apparatus of FIG. 23, where FIG. 23A shows a case where the focal point is shifted by 0.2 mm (Defocus = 0.2 mm), and FIG. In the case (Best focus), (C) is a diagram showing each spot image when the focus is shifted by −0.2 mm (Defocus = −0.2 mm).

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Image pick-up device, 110 ... Optical system, 120 ... Image pick-up element, 130 ... Analog front end part (AFE), 140 ... Image processing apparatus, 150 ... Camera signal processing part, DESCRIPTION OF SYMBOLS 180 ... Operation part, 190 ... Control apparatus, 111 ... Object side lens, 112 ... Imaging lens, 113 ... Optical element for wavefront formation, 113a ... Phase plate (light wavefront modulation) Element), 142... Convolution calculator, 143... Kernel data ROM, 144.

Claims (7)

A depth extension optical system including an optical wavefront modulation element;
An image sensor that captures a subject image that has passed through the depth extension optical system;
Selecting means for selecting a predetermined region for the image signal from the image sensor;
An image signal processing unit that performs predetermined image processing on the image signal of the selected area,
The image signal processor is
An imaging apparatus that restores a dispersed image of the selected region as an image signal without dispersion. The imaging device according to claim 1, wherein the predetermined region is predetermined in an image. The imaging apparatus according to claim 1, wherein the predetermined area is determined by extracting a characteristic part from an image and selecting the part. The imaging apparatus according to claim 3, wherein the feature part extraction processing is performed between photoelectric conversion of an image by the imaging device and restoration processing by the predetermined image processing. The imaging apparatus according to claim 1, 3, or 4, wherein binarization processing is performed on an image signal from the imaging element prior to the processing of extracting the characteristic part. The imaging apparatus according to claim 1, 3, or 4, wherein color arrangement information is used in the feature part extraction process. An image processing method for capturing a subject image that has passed through a depth expansion optical system including a light wavefront modulation element and performing image processing on the captured image data,
A selection step of selecting a predetermined region for the image signal from the image sensor;
An image signal processing step of performing predetermined image processing on the image signal of the selected area,
In the image signal processing step,
An image processing method for restoring a dispersed image of the selected area as an image signal without dispersion.

Images