JP2016005080A - Imaging apparatus, imaging system, and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an imaging system, and an imaging method, capable of improving image restoration performance even at a defocusing position which is not a focus position, using single imaging means.SOLUTION: A lens unit diffuses PSF to input light, so that an image captured by an imaging device 12 has constant blur in a wide depth instead of a blurred state of an image captured by the imaging device 12. A distance judgment unit 142 judges the subject distance of a subject included in the captured image, based on the characteristic of the captured image captured by the imaging device 12. A selection unit 144 selects an inverse conversion filter corresponding to the judged subject distance. A filter processing unit 145 executes inverse conversion processing to the captured image, using the inverse conversion filter selected by the selection unit 144.

Description

  The present invention relates to an imaging apparatus, an imaging system, and an imaging method.

  In recent years, with the development of digitalization of information, the development of digitization has been remarkable also in the field of imaging devices. In particular, in an imaging apparatus represented by a digital camera, a solid-state imaging device is used instead of a conventional film as an imaging component. A CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, or the like is used as a solid-state imaging device (hereinafter simply referred to as an imaging device).

  As described above, an image pickup apparatus using an image pickup device captures light from a subject by an optical system, converts the light into an electrical signal by a solid-state image pickup device, and extracts it. As such an imaging device, in addition to a digital camera, a video camera, a symbol reader (bar code reader, two-dimensional code reader, OCR (Optical Character Reader), etc.), a mobile phone, a personal digital assistant (PDA) and a personal digital assistant Examples include industrial cameras.

  As such an image pickup apparatus having an image pickup element such as a CCD or CMOS, an image including blur caused by aberration of the optical system is restored by filtering, and the so-called EDOF (Extended Depth) is extended. of Field) is already known. However, the blur can be restored by the filtering process within a range in which the shape of a point spread function (PSF) in the vicinity of a predetermined focus position is approximate. Therefore, since the PSF shape is different at the defocus position deviating from the focus position, even if the same filter processing is performed, the image restoration performance deteriorates.

  In order not to deteriorate the image restoration performance at such a defocus position, a plurality of imaging units (imaging units) are provided, and a filter for filter processing is switched based on distance information obtained by these imaging units. Thus, a technique for improving the restoration performance of the image at the defocus position has been proposed (see Patent Document 1).

  The technique described in Patent Document 1 has a problem in that the size of the apparatus increases and the cost increases because it is necessary to include a plurality of imaging units in order to obtain distance information to the subject.

  The present invention has been made in view of the above, and provides an imaging apparatus, an imaging system, and an imaging method capable of improving image restoration performance even at a defocus position that is not a focus position by a single imaging unit. For the purpose.

  In order to solve the above-described problems and achieve the object, the present invention provides an optical system that gives aberration to incident light, and images the subject by converting the light that has passed through the optical system into pixels 1 Two image pickup means, distance determination means for determining a subject distance of the subject based on characteristics of a captured image picked up by the image pickup means, selection means for selecting an inverse transform filter corresponding to the subject distance, Inverse conversion means for executing an inverse conversion process for restoring blur due to the aberration in the captured image by the inverse conversion filter selected by the selection means is provided.

  According to the present invention, it is possible to improve image restoration performance even at a defocus position that is not a focus position by a single imaging unit.

FIG. 1 is a diagram illustrating an example of the overall configuration of an imaging system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the configuration of the imaging apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a lens configuration of the optical system according to the embodiment. FIG. 5 is a diagram illustrating an example of lateral aberration of the optical system according to the embodiment. FIG. 6 is a diagram illustrating an example of longitudinal aberration of the optical system according to the embodiment. FIG. 7 is a diagram illustrating an example of field curvature of the optical system according to the embodiment. FIG. 8 is a diagram illustrating an example of distortion of the optical system according to the embodiment. FIG. 9 shows an example of the MTF when the subject distance is 350 [mm]. FIG. 10 is an example of the MTF when the subject distance is 400 [mm]. FIG. 11 shows an example of the MTF when the subject distance is 470 [mm]. FIG. 12 is a diagram illustrating an example of a captured image of a two-dimensional code at different subject distances. FIG. 13 is a diagram illustrating an example of a block configuration of an image processing unit of the imaging apparatus according to the embodiment. FIG. 14 is a diagram illustrating an example of a captured image captured by the imaging device of the imaging apparatus according to the embodiment. FIG. 15 is a timing chart illustrating an operation in which pixels output from the image sensor are input to the image buffer unit. FIG. 16 is a diagram illustrating an example of the configuration of the inverse transform filter. FIG. 17 is a diagram for describing filtering of an image using an inverse transform filter. FIG. 18 is a diagram illustrating an operation of scanning a target partial image that is a target of filter processing of an inverse transform filter in an image. FIG. 19 is a flowchart illustrating an example of a flow of calculating a frequency characteristic for determining an inverse transform filter of the filter processing unit of the image processing unit according to the embodiment. FIG. 20 is a diagram illustrating an example of a spatial frequency characteristic of a captured image by light that has passed through the optical system. FIG. 21 is a diagram illustrating the spatial frequency characteristics of an image that has been subjected to inverse transformation processing. FIG. 22 is a diagram illustrating an example of an image of a two-dimensional code that has been filtered by an inverse transform filter for the subject at the best focus position. FIG. 23 is a diagram illustrating an example of a configuration of an inverse transform filter applied to a subject whose subject distance is 350 [mm] and 470 [mm]. FIG. 24 is a diagram illustrating an example of a two-dimensional code image that has been filtered by an inverse transform filter corresponding to a subject distance. FIG. 25 is a diagram showing an example of a histogram for the luminance value of an image when the subject distance is 350 [mm]. FIG. 26 is a diagram illustrating an example of a histogram regarding the luminance value of an image when the subject distance is 400 [mm]. FIG. 27 is a diagram illustrating an example of a histogram regarding the luminance value of an image when the subject distance is 470 [mm]. FIG. 28 is a flowchart illustrating an example of the operation of the imaging apparatus according to the embodiment.

  Hereinafter, embodiments of an imaging apparatus, an imaging system, and an imaging method according to the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

(Overall configuration of imaging system)
FIG. 1 is a diagram illustrating an example of the overall configuration of an imaging system according to an embodiment. A configuration of an imaging system 500 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 1, an imaging system 500 of the present embodiment includes an imaging device 1 and a PC (Personal Computer) 2. The imaging device 1 and the PC 2 are communicably connected by a communication cable 3 such as an Ethernet (registered trademark) cable or a USB (Universal Serial Bus) cable.

  The imaging device 1 images the subject 4 by converting light from the subject 4 into an electrical signal, performs image processing on information of the captured image (hereinafter simply referred to as a captured image), and performs an image processing after the image processing. Is recognized, the recognized symbol is decoded, and the decoded information is transmitted to the PC 2 via the communication cable 3. Here, the operation of symbol recognition and decoding of the recognized symbol by the imaging apparatus 1 is simply referred to as “symbol recognition processing”. Examples of the symbol include a barcode, a two-dimensional code, a character string, and the like. The PC 2 executes predetermined processing based on the decoded information received from the imaging device 1.

  For example, the imaging device 1 images a barcode attached to a product flowing through the production line, executes a recognition process for reading the barcode, and transmits product number information as decoded information to the PC 2. The PC 2 reads and analyzes the barcode information from the received image.

  As shown in FIG. 1, the imaging system 500 is a wired communication system in which the imaging apparatus 1 and the PC 2 communicate data via the communication cable 3, but is not limited to this. For example, the image capturing apparatus 1 and the PC 2 may be capable of communicating data with each other by a wireless communication method such as Wi-Fi (registered trademark) (Wireless Fidelity).

  When the imaging device 1 and the PC 2 are used in a production line, the imaging system 500 may be configured such that the PC 2 is connected to a PLC (Programmable Logic Controller) or the like so as to be able to communicate. In this case, as an operation of the imaging system 500, the following operation is given as an example. The imaging device 1 images a barcode attached to a product flowing through a production line, recognizes and decodes the barcode by a recognition process, and transmits product number information as decoded information to the PC 2. The PC 2 determines the product number of the product flowing through the production line from the received product number information. If the determined product number does not match the product number that has been changed in the production line, the PC 2 transmits a signal indicating that the product corresponding to the determined product number is a product of a different product number to the PLC. When receiving a signal indicating that the product is a product of a different product number from the PC 2, the PLC controls the operation of the production line so as to remove the product from the production line.

(Configuration of information processing device)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. The hardware configuration of the PC 2 that is an example of the information processing apparatus will be described with reference to FIG.

  As illustrated in FIG. 2, the PC 2, which is an example of an information processing device, includes a communication unit 21, an operation unit 22, a display unit 23, a storage unit 24, an external storage device 25, and a control unit 26. ing. The above-described units are connected by a bus 27 and can transmit and receive data to and from each other.

  The communication unit 21 is a device that communicates with the imaging device 1 via the communication cable 3. The communication unit 21 is realized by a communication device such as a NIC (Network Interface Card) or a USB interface device, for example. The communication standard of the communication unit 21 is realized by, for example, TCP (Transmission Control Protocol) / IP (Internet Protocol) or UDP (User Datagram Protocol) / IP. Alternatively, the communication standard of the communication unit 21 may be a standard such as USB 1.1 or USB 2.0.

  The operation unit 22 is a device that performs an operation input for causing the control unit 26 to execute a predetermined process by the user. The operation unit 22 is realized by, for example, an operation input function on a mouse, a keyboard, a numeric keypad, a touch pad, or a touch panel.

  The display unit 23 is a device that displays an application image executed by the control unit 26 or an image output from the imaging device 1. The display unit 23 is realized by, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, a plasma display, an organic EL (Electroluminescence) display, or the like.

  The storage unit 24 is a device that stores various programs executed by the PC 2 and data used for various processes performed by the PC 2. The storage unit 24 is realized by a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), for example.

  The external storage device 25 is a storage device that accumulates and stores images, information decoded from symbols by the imaging device 1, programs, font data, and the like. The external storage device 25 is realized by, for example, a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, or a magneto-optical disk (MO: Magneto-Optical Disk).

  The control unit 26 is a device that controls the operation of each unit of the PC 2. The control unit 26 is realized by, for example, a CPU (Central Processing Unit) and an ASIC (Application Specific Integrated Circuit).

(Configuration of imaging device)
FIG. 3 is a diagram illustrating an example of the configuration of the imaging apparatus according to the embodiment. The configuration of the imaging apparatus 1 according to the present embodiment will be described with reference to FIG.

  As illustrated in FIG. 3, the imaging device 1 includes a lens unit 11 (an example of an optical system), an imaging element 12 (imaging unit), an image processing unit 14, a recognition processing unit 15 (recognition processing unit), and communication. Part 16.

  The lens unit 11 is a unit that collects light from the subject 4 (see FIG. 1) and forms an image on the image sensor 12. The lens unit 11 is realized by an optical system composed of one or more lenses. The subject 4 is, for example, a person, a monitored object, or a symbol such as a barcode or a two-dimensional code.

  The imaging element 12 is a solid-state imaging element that captures an image of the subject 4 by converting light from the subject 4 incident on the lens unit 11 into an electrical signal and generates an image. The image sensor 12 outputs pixels constituting an image taken by each detection unit constituting the solid-state image sensor. The image sensor 12 is realized by, for example, a CCD sensor or a CMOS sensor.

  The image processing unit 14 generates an image subjected to filter processing from the captured image output from the imaging element 12.

  The recognition processing unit 15 recognizes a symbol included in the image based on the image on which the filter processing has been performed by the image processing unit 14, and executes a symbol recognition process for decoding the recognized symbol.

  The communication unit 16 is a device that communicates with the PC 2 via the communication cable 3. For example, the communication unit 16 transmits the decoded information output from the recognition processing unit 15 to the PC 2. The communication unit 16 is realized by a communication device such as a NIC or a USB interface device, for example. The communication standard of the communication unit 16 is realized by, for example, TCP / IP or UDP / IP, or the communication standard of the communication unit 16 may be a standard such as USB 1.1 or USB 2.0.

  The recognition processing unit 15 is configured to be included in the imaging device 1, but may be realized by a function of an external device connected to the imaging device 1. For example, the recognition processing unit 15 may be realized in the PC 2 instead of the imaging device 1.

  The image processing unit 14 and the recognition processing unit 15 may be realized by executing a program that is software, or may be realized by a hardware circuit such as an ASIC or an FPGA (Field-Programmable Gate Array). Good. When at least one of the image processing unit 14 and the recognition processing unit 15 is realized by executing a program, the program is provided by being incorporated in advance in a ROM or the like included in the imaging device 1. A program executed by the imaging apparatus 1 according to the present embodiment is a file in an installable or executable format, such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disc), or the like. The information may be provided by being recorded on a recording medium that can be read by the user. Further, the program executed by the imaging apparatus 1 according to the present embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the imaging apparatus 1 of the present embodiment may be configured to be provided or distributed via a network such as the Internet. The program executed by the imaging apparatus 1 according to the present embodiment has a module configuration including at least one of the image processing unit 14 and the recognition processing unit 15 described above, and the CPU includes the above-described ROM as actual hardware. By reading and executing the program, the image processing unit 14 or the recognition processing unit 15 is loaded on the main storage device and generated.

(Configuration of lens unit)
FIG. 4 is a diagram illustrating an example of a lens configuration of the optical system according to the embodiment. A configuration example of the lens unit 11 of the imaging device 1 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 4, the lens unit 11 includes, for example, lenses LZ1 to LZ3, a phase plate LZ4, a stop SB, and lenses LZ5 to LZ7. These are the lenses LZ1 to LZ3, the phase plate LZ4, the stop SB, and the lenses LZ5 to LZ7 in the direction from the left side in FIG. Are arranged in the order of. The lenses LZ1 to LZ3 and LZ5 to LZ7 are arranged so that their optical axes coincide with each other.

  The phase plate LZ4 has an effect of giving aberration to the light incident on the lens unit 11. As a result, the phase plate LZ4 has a function of diffusing a point spread function (PSF) with respect to the light incident on the image sensor 12, and instead of making the image captured by the image sensor 12 blurred. The blur is constant over a wide depth, that is, the shape of the PSF is constant. Here, the diffusion of the PSF means, for example, a diffusion in which the PSF on the sensor surface of the image sensor 12 extends over 3 pixels or more. Aberration is not limited to the phase plate LZ4, but aberration may be given to the light incident on the lens unit 11 by a lens included in the lens unit 11.

  The stop SB is a member that freely adjusts the amount of light incident on the lens unit 11, and is disposed in the vicinity of the phase plate LZ4.

(Lens unit aberration)
FIG. 5 is a diagram illustrating an example of lateral aberration of the optical system according to the embodiment. FIG. 6 is a diagram illustrating an example of longitudinal aberration of the optical system according to the embodiment. FIG. 7 is a diagram illustrating an example of field curvature of the optical system according to the embodiment. FIG. 8 is a diagram illustrating an example of distortion of the optical system according to the embodiment. Examples of aberration given by the lens unit 11 will be described with reference to FIGS.

  First, the lens unit 11 gives lateral aberration as shown in FIG. Here, the lateral aberration is an aberration generated in the direction of the sensor surface of the image sensor 12. FIG. 5A shows the length from the intersection of the optical axis of the lens unit 11 and the sensor surface of the image sensor 12 to the position where the principal ray of the light rays passing through the lens unit 11 reaches the sensor surface, that is, The graph of lateral aberration when the image height for the principal ray is 0 [mm] is shown. The graph on the left side of FIG. 5A is a graph showing the relationship between the lateral aberration ey and the position Py in the direction corresponding to the Y direction of FIG. 14 to be described later, at the entrance pupil based on the stop SB of the lens unit 11. . The graph on the right side of FIG. 5B is a graph showing the relationship between the lateral aberration ex and the position Px in the direction corresponding to the X direction in FIG. 14 at the entrance pupil. FIGS. 5B and 5C similarly show lateral aberration graphs when the image heights of the principal rays are 1.5 [mm] and 3.0 [mm], respectively.

  As shown in FIG. 5, the lateral aberration given by the lens unit 11 does not become 0 on the horizontal axis (direction parallel to the sensor surface), but is an aberration having a magnitude corresponding to the position of the sensor surface. . Note that the graph shown in FIG. 5 is a visible light ray having a wavelength of 656 [nm] (hereinafter referred to as r), a 588 [nm] ray (hereinafter referred to as g), and a 486 [nm] ray. Although shown for each ray (hereinafter referred to as b), they are not substantially separated but overlap each other.

  Further, the lens unit 11 gives a longitudinal aberration as shown in FIG. Here, the longitudinal aberration is an aberration in the optical axis direction (referred to as a longitudinal direction) of the lens unit 11, and is an aberration that occurs when an image is not formed at one point on the sensor surface of the image sensor 12. The vertical axis in FIG. 6 is the height from the optical axis of the incident point of the light beam incident on the lens unit 11 (hereinafter referred to as the incident height), and the horizontal axis indicates the longitudinal aberration. In addition, as shown in the graph R1 for the light ray r, the graph G1 for the light ray g, and the graph B1 for the light ray b in FIG. 6, it can be seen that the longitudinal aberration is slightly different for each light ray. FIG. 6 shows that a large aberration of 0.7 [mm] or more occurs. This is based on the function of diffusing PSF with the phase plate LZ4 so that blur is constant (PSF shape is constant) at a wide depth. However, as a result, the MTF (Modulation Transfer Function) is lowered as shown in the graphs shown in FIGS.

  The lens unit 11 gives an aberration of curvature of field as shown in FIG. Here, the field curvature means that light in each direction incident on the lens unit 11 does not form an image on the sensor surface of the imaging element 12 that is a plane, and the image plane that is a set of image points does not become a plane. The phenomenon of bending. In FIG. 7, the curvature of field in the tangential plane, that is, the plane including the optical axis of the lens unit 11 and the principal ray, and the curvature of field in the sagittal plane, that is, the plane orthogonal to the tangential plane and including the principal ray are obtained. It is shown. In FIG. 7, the vertical axis represents the image height IH, and the horizontal axis represents the position of the imaging point in the optical axis direction with reference to the sensor surface of the image sensor 12. Specifically, the graph R2_T shown in FIG. 7 shows the field curvature in the tangential plane for the ray r, the graph G2_T shows the field curvature in the tangential plane for the ray g, and the graph B2_T is , Shows curvature of field in the tangential plane for ray b. Also, the graph R2_S shown in FIG. 7 shows the field curvature in the sagittal plane for the ray r, the graph G2_S shows the field curvature in the sagittal plane for the ray g, and the graph B2_S shows the curvature for the ray b. The field curvature in the sagittal plane is shown.

  Moreover, the lens unit 11 gives distortion as shown in FIG. Here, the distortion (distortion aberration) is an aberration in which the distortion rate varies depending on the image height. The vertical axis in FIG. 8 represents the image height IH, and the horizontal axis represents the ratio of the difference between the actual image height and the ideal image height with respect to the ideal image height.

  In addition, although the graph shown in FIG. 8 is shown about each light ray of r light, g light, and b light, it is not isolate | separated substantially but it mutually overlaps.

  The aberrations of the kind described above are generated by the lens unit 11, but as described above, the PSF is diffused with respect to the light incident on the image sensor 12 by the phase plate LZ4 included in the lens unit 11, Instead of blurring the captured image, the blur is constant over a wide depth, that is, the shape of the PSF is constant.

(About reduction of MTF due to aberration)
FIG. 9 shows an example of the MTF when the subject distance is 350 [mm]. FIG. 10 is an example of the MTF when the subject distance is 400 [mm]. FIG. 11 shows an example of the MTF when the subject distance is 470 [mm]. FIG. 12 is a diagram illustrating an example of a captured image of a two-dimensional code at different subject distances. The MTF of an image (captured image) in which light from a subject that has passed through the lens unit 11 that gives aberration is captured by the image sensor 12 will be described with reference to FIGS.

  As described above, the lens unit 11 diffuses the PSF by giving aberration to the light from the subject incident on the image sensor 12, and the blur is constant over a wide depth instead of making the captured image blurred. As a result, the MTF is lowered. Here, the MTF is a value obtained by quantifying how faithfully the contrast of a subject can be reproduced, that is, a value indicating a contrast reproduction rate, and an ideal value is “1”.

  The graph shown in FIG. 9 is a graph showing an example of the MTF characteristics with respect to the spatial frequency for the captured image when the subject distance (for example, defined as the distance from the lens unit 11 to the subject) is 350 [mm]. It is. The graph shown in FIG. 10 is a graph showing an example of the MTF characteristics with respect to the spatial frequency for the captured image when the subject distance is 400 [mm]. The graph shown in FIG. 11 is a graph showing an example of the MTF characteristics with respect to the spatial frequency for the captured image when the subject distance is 470 [mm]. In any graph, the MTF is lower than the ideal value “1” due to the influence of the aberration caused by the lens unit 11, and particularly has a tendency to decrease as the spatial frequency increases.

Further, as one of the characteristics of the graphs shown in FIGS. 9 to 11, the MTF on the low frequency side and the MTF on the high frequency side of the spatial frequency differ depending on the subject distance. For example, when the graphs of FIGS. 9 to 11 are compared with each other and the spatial frequency on the low frequency side is, for example, 10 [mm ⁻¹ ], the MTF is the highest when the subject distance shown in FIG. ] Is an MTF graph. On the other hand, when the spatial frequency on the high frequency side is, for example, 50 [mm ⁻¹ ], the MTF having the highest MTF is the graph of the MTF with the subject distance of 400 [mm] shown in FIG.

  Further, the image of the two-dimensional code shown in FIG. 12 is a picked-up image of the two-dimensional code in which light from the two-dimensional code as a subject that has passed through the lens unit 11 that gives aberration is picked up by the image pickup device 12. 12A shows a captured image of a two-dimensional code with a subject distance of 350 [mm], and FIG. 12B shows a captured image of a two-dimensional code with a subject distance of 400 [mm]. (C) shows a captured image of a two-dimensional code having a subject distance of 470 [mm]. FIG. 12D shows an ideal image of a two-dimensional code. Of the captured images of the two-dimensional code shown in FIGS. 12A to 12C, the captured image with the subject distance of 400 mm shown in FIG. 12B is closest to the ideal image shown in FIG. The pattern of the two-dimensional code is also clearer than other captured images. This subject distance 400 [mm] is assumed to be the best focus position. Here, it is assumed that the best focus position is a position where the influence of the aberration of the lens unit 11 is small and the captured image is clearest in the subject distance.

  However, in any case, the captured images shown in FIGS. 12A to 12C are blurred images and need to be corrected so as to obtain a predetermined MTF value. As a method of correcting blur for such a blurred image taken by the lens unit 11, MTF is improved by performing inverse transformation processing of the point spread function (PSF) to correct the image to a high resolution. can do. The inverse transformation process is realized by restoring the blur of the image due to the aberration by applying a filter process using an inverse transform filter to each pixel that forms an image blurred by the optical system. Hereinafter, details of the block configuration of the image processing unit 14 will be described, and an example of a method for extending the depth of field by inverse conversion processing will be described.

(Configuration and operation of image processing unit)
FIG. 13 is a diagram illustrating an example of a block configuration of an image processing unit of the imaging apparatus according to the embodiment. FIG. 14 is a diagram illustrating an example of a captured image captured by the imaging device of the imaging apparatus according to the embodiment. A block configuration of the image processing unit 14 of the imaging device 1 according to the present embodiment will be described with reference to FIG.

  As described above, the imaging element 12 is a solid-state imaging element that captures an image of the subject 4 by converting light from the subject 4 (see FIG. 1) incident on the lens unit 11 into an electrical signal and generates an image. In the following description, it is assumed that the image sensor 12 forms and outputs a VGA (Video Graphics Array) image. Specifically, as illustrated in FIG. 14, the imaging device 12 includes pixels arranged in a 640 × 480 matrix by, for example, 640 detection elements in the X direction and 480 detection elements in the Y direction. A captured image 101 that is an image is captured.

  Although the size of the image captured by the image sensor 12 is a 640 × 480 VGA image, the image is not limited to this and may be an image of a different size.

  As shown in FIG. 13, the image processing unit 14 of the present embodiment includes an image buffer unit 141, a distance determination unit 142 (distance determination unit), a storage unit 143 (storage unit), and a selection unit 144 (selection unit). ) And a filter processing unit 145 (inverse conversion means).

  The image buffer unit 141 includes a line buffer, a register, and the like, and is a functional unit that sequentially inputs pixels output from the image sensor 12 and buffers a captured image. The image buffer unit 141 outputs the buffered captured image to the distance determination unit 142 and the filter processing unit 145.

  The distance determination unit 142 is a functional unit that determines the subject distance in the captured image based on the variance of the histogram for the luminance value of the captured image. The operation of the distance determination unit 142 will be described later.

  The storage unit 143 is a storage device that stores data of an inverse transform filter corresponding to the subject distance determined by the distance determination unit 142 in advance.

  The selection unit 144 is a functional unit that selects an inverse transform filter corresponding to the subject distance determined by the distance determination unit 142 from the inverse transform filters stored in the storage unit 143 and sends the selected inverse transform filter to the filter processing unit 145.

  The filter processing unit 145 performs predetermined filter processing (inverse conversion processing) on the pixels output from the image buffer unit 141 using the inverse conversion filter received from the selection unit 144. In the present embodiment, as a filter used for the filter process, an inverse transform process for correcting (restoring) blur for a blurred captured image given a point spread function (PSF) by the action of the phase plate LZ4. An inverse transform filter for performing the above will be described as an example. The specific operation of the filter processing unit 145 will be described later with reference to FIGS.

  Note that the image buffer unit 141, the distance determination unit 142, the storage unit 143, the selection unit 144, and the filter processing unit 145 conceptually show functions, and are not limited to such a configuration.

<Operation in which pixels output from the image sensor 12 are input to the image buffer unit 141>
FIG. 15 is a timing chart illustrating an operation in which pixels output from the image sensor are input to the image buffer unit. With reference to FIG. 15, an output operation of an image captured by the image sensor 12 will be described.

  The image sensor 12 outputs the pixels included in one horizontal line while scanning the detected pixels one horizontal line in the X direction. Specifically, the imaging device 12 sequentially outputs pixels included in the first horizontal line in the Y direction from the first pixel in the X direction to the 640th pixel. The image sensor 12 outputs the pixels included in the horizontal lines up to the 480th horizontal line in the Y direction, respectively. For each 640 × 480 pixel of the image captured by the image sensor 12, the Xth pixel in the X direction and the Yth pixel in the Y direction are referred to as (X, Y) pixels.

  The above operation will be described based on the timing chart shown in FIG. As shown in FIG. 15, the imaging device 12 outputs pixels for one frame, that is, one image when the effective frame signal is on. In the image sensor 12, the effective line signal L1 indicating the output permission of the pixels in the first horizontal line in the Y direction is turned on through the frame start period A after the effective frame signal is turned on. The imaging device 12 scans the first horizontal line in the Y direction during the effective data period T in which the effective line signal L1 is on, and the 1st to 640th in the X direction included in the horizontal line. Pixels (pixels (1, 1) to (640, 1)) are output in order. After the image sensor 12 outputs the pixels of the first horizontal line in the Y direction, the effective line signal L1 is turned off.

  In the image sensor 12, after the effective line signal L1 is turned off, the effective line signal L2 indicating the output permission of the pixels in the second horizontal line in the Y direction is turned on after the horizontal blanking period B. The imaging device 12 scans the second horizontal line in the Y direction during the effective data period T in which the effective line signal L2 is in the ON state, and the 1st to 640th in the X direction included in the horizontal line. Pixels (pixels (1, 2) to (640, 2)) are output in order. After the image sensor 12 outputs the pixels of the second horizontal line in the Y direction, the effective line signal L2 is turned off.

  With respect to the above operation, the image sensor 12 detects the 1st to 640th pixels in the X direction included in the 480th horizontal line in the Y direction during the effective data period T in which the effective line signal L480 is on. Repeat until output. In the image sensor 12, the effective frame signal is turned off after the frame end period C after the effective line signal L480 is turned off. With the above operation, the output of pixels for one frame by the image sensor 12 is completed. Further, in the image sensor 12, the effective frame signal is turned on again after the vertical blanking period D after the effective frame signal is turned off, and output of pixels for the next one frame is started.

<Operation of Filter Processing Unit 145>
FIG. 16 is a diagram illustrating an example of the configuration of the inverse transform filter. FIG. 17 is a diagram for describing filtering of an image using an inverse transform filter. FIG. 18 is a diagram illustrating an operation of scanning a target partial image that is a target of filter processing of an inverse transform filter in an image. The inverse transformation process of the filter processing unit 145 of the image processing unit 14 will be described with reference to FIGS. Note that the inverse transform filter used by the filter processing unit 145 corresponds to the subject distance determined by the distance determination unit 142 by the distance determination unit 142 and the selection unit 144 determines the subject distance of the subject included in the captured image. It is assumed that the inverse transform filter selected as.

  For example, as shown in FIG. 16, the filter used for the inverse transformation process has a tap number of 5 × 5 configured by filter coefficients a11 to a15, a21 to a25, a31 to a35, a41 to a45, and a51 to a55. It is assumed that the inverse transform filter 121 is a linear filter. It is assumed that the part of the image that is the target of the inverse transform process by the inverse transform filter 121 is the target partial image 131 shown in FIG. The target partial image 131 is a 5 × 5 partial image including pixels A11 to A15, A21 to A25, A31 to A35, A41 to A45, and A51 to A55.

  As shown in FIG. 17, the filter processing unit 145 has the same value as the value obtained by performing the convolution operation on the target partial image 131 by the inverse transform filter 121, that is, the calculated value represented by the equation (1) in FIG. . The calculation value of the convolution calculation is a value obtained by performing an inverse conversion process on the central data that is a pixel located at the center of the target partial image 131. That is, the convolution calculation value is a pixel at a position corresponding to the central data of the image before the inverse transformation process in the image after the inverse transformation process.

  Next, with regard to the inverse conversion process of the filter processing unit 145, an outline of an operation of performing the inverse conversion process while scanning the horizontal line in the X direction in the image 105 will be described with reference to FIG. FIG. 18A shows a state in which the filter processing unit 145 performs an inverse transform process on the (1, 1) pixel in the image 105 by the inverse transform filter 121. As shown in FIG. 18A, in order to perform the convolution operation with the pixel (1, 1) as the central data 135a, the target partial image 131a having the pixel (1, 1) as the central data, and the image 105 The pixel of the part which overlaps with is required. That is, among the target partial image 131a, pixels corresponding to the pixels A33 to A35, A43 to A45, and A53 to A55 of the target partial image 131 shown in FIG. 17 are necessary. In the target partial image 131a, the pixel of the portion that does not overlap with the image 105 is handled as “0”.

  In the above state, the filter processing unit 145 performs the convolution operation on the target partial image 131a by the inverse transform filter 121, similarly to the convolution operation illustrated in FIG. The filter processing unit 145 performs a convolution operation on the (1, 1) pixel that is the central data 135a of the target partial image 131a of the image 105, and the (1, 1) pixel of the image after the inverse transformation process. Output as.

  Next, as shown in FIG. 18B, the filter processing unit 145 shifts the pixel subject to the convolution operation by one in the X direction, and is the central data 135b of the target partial image 131b (2, 1). The inverse conversion process is performed on the pixels. The filter processing unit 145 repeats the convolution operation while shifting the horizontal line in the X direction, and as shown in FIG. 18C, is the last pixel of the horizontal line in the X direction (640, 1). Inverse conversion processing is performed on the pixel. The pixel (640, 1) is the central data 135c of the target partial image 131c, as shown in FIG.

  As described above, the filter processing unit 145 repeats the convolution operation while shifting the horizontal line in the X direction, and when the inverse conversion process for the last pixel in the horizontal line is completed, the filter processing unit 145 applies the next horizontal line in the Y direction. Similarly, reverse conversion processing is performed.

  FIGS. 18D to 18F show a state in which the filter processing unit 145 performs an inverse conversion process on the pixels in the fourth horizontal line in the Y direction in the image 105. FIG. 18D shows a state in which the filter processing unit 145 performs inverse conversion processing on the (1, 4) pixels in the image 105 by the inverse conversion filter 121. As shown in FIG. 18D, in order to perform the convolution operation with the pixel of (1, 4) as the central data 135d, the target partial image 131d having the pixel of (1, 4) as the central data, and the image 105 The pixel of the part which overlaps with is required. In the target partial image 131d, the pixel in the portion that does not overlap with the image 105 is handled as “0” as described above.

  FIG. 18E shows a state in which the filter processing unit 145 performs an inverse transform process on the pixel (5, 4) in the image 105 by the inverse transform filter 121. As shown in FIG. 18E, since the entire target partial image 131e having the pixel (5, 4) as the central data 135e overlaps with the image 105, the filter processing unit 145 adds the target partial image 131e to the target partial image 131e. Inverse conversion processing using all the included pixels is possible.

  Then, the filter processing unit 145 repeats the convolution calculation while shifting the horizontal line in the X direction, and, as shown in FIG. 18 (f), is the last pixel of the horizontal line in the X direction (640, 4). Inverse conversion processing is performed on the pixel. The pixel (640, 4) is the central data 135f of the target partial image 131f as shown in FIG.

  As described above, the filter processing unit 145 performs a convolution operation by the inverse transform filter 121 on each pixel constituting the image 105 and performs an inverse transform process, so that the image blurred by the phase plate LZ4 is restored. The resolution of the image can be improved.

  As described above, in the target partial image that is the target of the convolution operation by the inverse transform filter 121 in the image 105, the pixel of the portion that does not overlap with the image 105 is set to “0”, but is not limited thereto. It is not a thing. For example, the pixel of the portion that does not overlap with the image 105 of the target partial image is the pixel when the pixel of the portion that overlaps with the image 105 of the target partial image is folded with reference to the center data of the target partial image. It may be used.

  Specifically, the target partial image 131a in FIG. 18A will be described as an example. The names of the pixels of the target partial image 131a are assumed to be the same as the names of the pixels of the target partial image 131 shown in FIG. In this case, the pixel of the part which does not overlap with the image 105 of the object partial image 131a is the pixels A11 to A15, A21 to A25, A31, A32, A41, A42, A51 and A52. In addition, the pixels of the portion overlapping the image 105 of the target partial image 131a are pixels A33 to A35, A43 to A45, and A53 to A55.

  At this time, the pixels A31, A32, A41, A42, A51, and A52 fold back the pixels that overlap the image 105 of the target partial image 131a with reference to the central data, and the pixels A35, A34, A45, The values A44, A55 and A54 are used. In addition, the pixels A13 to A15 and A23 to A25 fold back the pixels that overlap the image 105 of the target partial image 131a with reference to the central data, and use the values of the pixels A53 to A55 and A43 to A45, respectively. . Then, the pixels A11, A12, A21, and A22 are pixels in a point target positional relationship among the pixels of the portion overlapping the image 105 of the target partial image 131a with respect to the central data, that is, A55, The values of A54, A45 and A44 are used. Each pixel of the target partial image may be determined by the above method.

  Further, as shown in FIG. 17, the inverse transform filter 121 used by the filter processing unit 145 is a filter having 5 × 5 taps, but is not limited thereto. That is, the number of taps of the filter may be a different number of taps such as 7 × 7, 15 × 15, or 21 × 21. In this case, it is necessary to match the size of the target partial image according to the number of taps of the filter. In addition, the number of taps of the filter needs to be an odd number so that there is central data to be subjected to the inverse transformation process by the filter.

  In addition, the inverse transform filter preferably has a tap number of 15 × 15 or more, for example. As the number of taps increases, the inverse transform filter can increase the width on the optical axis where blur can be corrected with respect to an image in which blur is added by the phase plate LZ4. Therefore, by using an inverse transform filter having a large number of taps, it is possible to increase design variations with respect to the depth of field of the phase plate and the lens.

<Derivation of frequency characteristics of inverse transform filter>
Next, a method for deriving the frequency characteristics of the above-described inverse transform filter (for example, the inverse transform filter 121 shown in FIG. 17) will be described. A method for deriving the frequency characteristics of the inverse transform filter used in the inverse transform processing for restoring the PSF diffused by the lens unit 11 as the optical system so as to be gathered at one point at the focus position (for example, the best focus position described above) will be described. To do. As the filter that realizes the above-described frequency characteristic, a two-dimensional linear filter and an FIR (Finite Impulse Response) filter are suitable.

First, a model of the influence of the optical system on an image captured by the image sensor 12 is expressed by a two-dimensional convolution calculation (convolution calculation) expression shown in the following expression (2).

Here, image _captured is a pixel of a two-dimensional _captured image detected through the optical system, image _ideal is a pixel of an ideal image showing the subject 4 itself, and h is a PSF of the optical system.

Hereinafter, in consideration of the influence of noise applied to the image processing system (the image sensor 12 and the image processing unit 14), the mean square error regarding the error between each pixel of the image after the inverse transformation process and each pixel of the ideal image is calculated. Consider the derivation of the frequency characteristics of the inverse transform filter to be minimized. The mean square error is expressed by the following equation (3).

Here, E [] indicates an expected value (average value), n indicates a position on the image, and image _processed (n) indicates a pixel obtained by _performing an inverse conversion process on image _captured (n). Note that image _captured considers that noise is included.

According to Parseval's theorem that the sum of all the energies of the waveform x (n) for all regions of n and the sum of all the frequency components of the Fourier transform X (ω) of the energy of the waveform x (n) are equal. Expression (3) is expressed by the following expression (4) as a mean square error in the frequency domain.

Here, IMAGE _ideal (ω) represents the frequency characteristic of image _ideal (n), IMAGE _processed (ω) represents the frequency characteristic of image _processed (n), and ω represents the spatial frequency.

Here, if the frequency characteristic of the inverse transform filter is R (ω), the frequency characteristic R (ω) that gives the minimum value of the following equation (5) is the optimum inverse transform filter.

Here, IMAGE _captured (ω) is the frequency characteristic of image _captured (n).

In Equation (5), IMAGE _ideal (ω) = S (ω) and IMAGE _captured (ω) = X (ω), and in order to find the minimum value of Equation (5), Equation (5) is differentiated by R ^* Then, the following formula (6) is obtained.

Here, E [| X (ω) | ² ] is an average value of the power spectrum of the captured image including noise, and E [S (ω) · X (ω) ^* ] is an image including noise. It is the average value of the mutual power spectrum between the image and the ideal image.

In order to obtain the minimum value of the equation (5), when the rightmost side of the equation (6) is 0, the following equation (7) is obtained.

From the equation (7), the following equation (8) is obtained.

  The inverse transform filter based on the frequency characteristic R (ω) shown in the equation (8) is an optimum filter that minimizes the mean square error shown in the equation (3).

Here, when the frequency characteristic of noise is W (ω), the frequency characteristic of h that is the PSF of the optical system is H (ω), and the above equation (2) is shown in the frequency domain, the following equation (9) is obtained. Indicated.

Assuming that the noise frequency characteristic W (ω) and the frequency characteristic S (ω) are uncorrelated, E [S (ω) · W (ω) ^* ] = 0. Substituting equation (9) into the numerator on the right side of the following equation (10) is obtained.

Similarly, if the frequency characteristic W (ω) of noise and the frequency characteristic S (ω) are uncorrelated, E [W (ω) · S (ω) ^* ] = 0 and E [S (ω) ^* · Since W (ω)] = 0, substituting equation (9) into the denominator on the right side of equation (8) above yields equation (11) below.

From the above equations (8), (10) and (11), the frequency characteristic R (ω) shown in the following equation (12) is obtained.

The inverse transform filter based on the frequency characteristic R (ω) shown in the equation (12) is an optimum filter that minimizes the mean square error shown in the above equation (3) when the noise of the image processing system is taken into consideration. . Here, E [| S (ω) | ² ] is the average value of the power spectrum of the ideal image, E [| W (ω) | ² ] is the average value of the power spectrum of noise, and | H (ω ) | ² is the power spectrum of the frequency characteristic of the optical system.

Further, when the frequency characteristic R (ω) shown in the above equation (12) is applied, an integration in the frequency domain is performed for the square error between each pixel of the image after the inverse transform process and each pixel of the ideal image. The following formula (13) is obtained.

  In the calculation of the above equation (13), as described above, the maximum of the equation (13) using the fact that the noise frequency characteristic W (ω) and the frequency characteristic S (ω) are uncorrelated. The first term on the right side indicates the amount of error that could not be restored in the image after the inverse transformation process. The second term indicates the amount of error due to noise.

  By designing the frequency characteristic H (ω) of the optical system so that the integral value of Expression (13) is minimized, the optical system that minimizes the mean square error in the frequency domain shown in Expression (5) above. And a combination of inverse transform filters. Further, according to Parseval's theorem, it is possible to obtain a combination of an optical system and an inverse transform filter that minimize the mean square error in the real space shown in the above equation (3). Therefore, for example, the inverse transform filter 121 used by the filter processing unit 145 of the image processing unit 14 may be derived based on the frequency characteristic R (ω) shown in Expression (12).

  As described above, an optimum inverse transform filter can be obtained for the focus position from the frequency characteristic R (ω) shown in Expression (12). As a result, the blur can be restored to a wider range at the focus position, and the depth of field can be extended.

  FIG. 19 is a flowchart illustrating an example of a flow of calculating a frequency characteristic for determining an inverse transform filter of the filter processing unit of the image processing unit according to the embodiment. A flow for specifically calculating the frequency characteristic R shown in the equation (12) will be described with reference to FIG.

<< Step S1 >>
First, the PSF is derived by ray tracing calculation for the lens unit 11 based on the optical system parameters such as the surface curvature and surface interval in the lens unit 11 and the characteristics of the phase plate LZ4. Here, a ray tracing calculation is performed using optical system parameters at a plurality of defocus positions to derive a PSF. Then, the process proceeds to step S2.

<< Step S2 >>
The frequency characteristic H of the optical system is derived by Fourier transforming the PSF derived in step S1. Then, the process proceeds to step S5.

<< Step S3 >>
Noise characteristics applied to the image processing system (the image sensor 12 and the image processing unit 14) are measured. Then, the noise frequency characteristic W is derived by Fourier transforming the noise characteristic. If it is difficult to measure the noise characteristic, the noise frequency characteristic W may be derived using the value of the S / N ratio of the image sensor 12 as a constant regardless of the spatial frequency. Then, the process proceeds to step S5.

<< Step S4 >>
An image obtained by capturing an image of a natural landscape or a symbol with various sizes and photographing conditions by the imaging device 1 is set as an ideal image. The values of the pixels constituting the ideal image are Fourier transformed, and the average value for the spatial frequency ω is derived as the frequency characteristic S of the subject. The frequency characteristic S of the subject may be a frequency characteristic of a pixel of a captured image based on light that has passed through an optical system that does not give aberration to light emitted from the subject. The frequency characteristic S of the subject may be a constant. Then, the process proceeds to step S5.

<< Step S5 >>
From the frequency characteristic H of the optical system derived in step S2, the frequency characteristic W of noise derived in step S3, and the frequency characteristic S of the subject derived in step S4, the inverse transformation filter is obtained using the above-described equation (12). Is calculated.

<Spatial frequency characteristics of MTF>
FIG. 20 is a diagram illustrating an example of a spatial frequency characteristic of a captured image by light that has passed through the optical system. FIG. 21 is a diagram illustrating the spatial frequency characteristics of an image that has been subjected to inverse transformation processing. FIG. 22 is a diagram illustrating an example of an image of a two-dimensional code that has been filtered by an inverse transform filter for the subject at the best focus position. The spatial frequency characteristics of the MTF of the image will be described with reference to FIGS.

  First, with reference to FIG. 20, the MTF for the spatial frequency ω for an image in which light that has passed through the lens unit 11 (optical system) is captured by the image sensor 12 will be further described. A target spatial frequency characteristic 201 shown in FIG. 20 indicates a characteristic (MTF = 1) regarding the spatial frequency ω of the MTF when the contrast of the captured image of the subject is completely reproduced.

  As described above, the point spread function (PSF) is diffused in the light that has passed through the lens unit 11 by the action of the lens that gives aberration and the phase plate LZ4. A spatial frequency characteristic 202 in FIG. 20 indicates a characteristic with respect to the spatial frequency ω of the MTF of a captured image captured at the focus position by the image sensor 12 based on the light that has passed through the lens unit 11. A spatial frequency characteristic 203 in FIG. 20 indicates a characteristic with respect to the spatial frequency ω of the MTF of a captured image captured at a certain defocus position by the image sensor 12 based on the light that has passed through the lens unit 11. Since PSF is added to the light that has passed through the lens unit 11, the spatial frequency characteristic 202 at the focus position and the spatial frequency characteristic 203 at the defocus position are both from the target spatial frequency characteristic 201 as shown in FIG. Is also a low value. The spatial frequency characteristics 202 and 203 shown in FIG. 20 schematically show examples of the MTF characteristics shown in FIGS. 9 to 11 described above.

  Next, referring to FIG. 21, an inverse conversion filter for the filter processing unit 145 to perform inverse conversion processing on the captured image of the subject at the focus position is obtained by the frequency characteristic R shown in the above equation (12). Consider the case. In this case, the MTF for the spatial frequency ω for an image obtained by performing an inverse transform process on the image captured by the image sensor 12 by the filter processing unit 145 will be described.

  A spatial frequency characteristic 202a in FIG. 21 schematically shows a characteristic regarding the spatial frequency ω of the MTF of an image obtained by performing inverse transform processing on the captured image in which the subject is captured at the focus position by the image sensor 12. It is a thing. The spatial frequency characteristic 203a in FIG. 21 is a characteristic regarding the spatial frequency ω of the MTF of an image obtained by performing inverse transformation processing with the same inverse transformation filter by the filter processing unit 145 on a captured image taken at a certain defocus position by the imaging element 12. Indicates. As described above, since the equation (12) represents the frequency characteristic R of the inverse transform filter that restores the image to which the PSF is added by the lens unit 11 for the subject at the focus position, the spatial frequency characteristic 202a is MTF = 1 and coincides with the target spatial frequency characteristic 201. However, since the inverse transform filter derived from the frequency characteristic R shown in Expression (12) does not correspond to a defocus position different in shape from the PSF at the focus position, the MTF of the spatial frequency characteristic 203a is the spatial frequency. It becomes lower than the MTF of the characteristic 202a.

  As described above, the filter processing unit 145 performs the lens unit 11 in a predetermined position range including the focus position by the inverse transformation process using the inverse transformation filter obtained based on the frequency characteristic R (ω) shown in Expression (12). Thus, an image in which PSF is diffused can be restored. Therefore, the depth of field can be extended to a wider range at the focus position.

  The image shown in FIG. 22 shows the characteristics indicated by the spatial frequency characteristics 202a and 203a as a two-dimensional code image. In the image shown in FIG. 22, the filter processing unit 145 restores the blur of the captured image of the subject (two-dimensional barcode) existing at the focus position (subject distance 400 [mm] which is the best focus position). This is an image obtained by performing an inverse transformation process on a captured image of a subject existing at a focus position and a defocus position using an inverse transformation filter obtained by Expression (12). Among these images, the image shown in FIG. 22B is an image obtained by capturing an image of a two-dimensional code existing at a focus position (best focus position) with a subject distance of 400 [mm] by the filter processing unit 145 as described above. It is an image that has been subjected to inverse transformation processing by an inverse transformation filter. On the other hand, the images shown in FIGS. 22 (a) and 22 (c) are obtained by filtering the images obtained by taking the two-dimensional code existing at the defocus positions with the subject distances of 350 [mm] and 470 [mm] as subjects. This is an image that has been subjected to inverse transformation processing by the processing unit 145 using the same inverse transformation filter as described above. Since the image shown in FIG. 22B has been subjected to inverse transformation processing by an inverse transformation filter for restoring the blur of the captured image of the subject at the focus position, the images shown in FIGS. 22A and 22C are used. The pattern of the two-dimensional code is clearer. On the other hand, the images shown in FIGS. 22A and 22C are reversed by the inverse transform filter for restoring the blur of the captured image of the subject at the focus position with respect to the captured image of the subject at the defocus position. Since the conversion process is performed, the pattern of the two-dimensional code is less clear than the image shown in FIG. This is because the PSF shape changes as the subject distance changes, and the same inverse transform filter cannot cope with it. Here, the image processing unit 14 according to the present embodiment restores the captured image of the subject existing at the defocus position so that the MTF is improved. Details of the operation will be described below.

<Operation of Distance Determination Unit 142 and Selection Unit 144>
FIG. 23 is a diagram illustrating an example of a configuration of an inverse transform filter applied to a subject whose subject distance is 350 [mm] and 470 [mm]. FIG. 24 is a diagram illustrating an example of a two-dimensional code image that has been filtered by an inverse transform filter corresponding to a subject distance. With reference to FIGS. 23 and 24, the inverse conversion process for the captured image of the subject at the defocus position will be described.

  In the above description, in order to derive an inverse transform filter that restores the PSF diffused by the lens unit 11 at the focus position so as to be collected at one point, the frequency characteristic R (ω) shown in Expression (12) is derived. As described above, it is possible to appropriately restore the blur of the captured image of the subject at the focus position if the inverse transform process is executed by the inverse transform filter based on the frequency characteristic R (ω). . However, even if this inverse transform filter is applied to the captured image of the subject at the defocus position, it cannot be restored properly. Therefore, in order to restore the blur of the captured image of the subject at the defocus position, an inverse transform filter corresponding to the PSF diffused at the defocus position is derived using Expression (12). FIG. 23 shows a configuration example of the inverse transform filter derived in this way. FIG. 23A shows an inverse transform filter 122 (the number of taps is 7 × 7) for restoring the blur of a captured image of a subject having a subject distance of 350 [mm] as a defocus position. FIG. 23B shows an inverse transform filter 123 (the number of taps is set to 7 × 7) for restoring blur of a captured image of a subject with a subject distance of 470 [mm], which is another defocus position.

  FIG. 24 shows an image obtained by performing an inverse transform process on captured images of subjects (two-dimensional codes) at different subject distances using an inverse transform filter corresponding to the subject distance. For example, the image shown in FIG. 24A is an image that has been subjected to the inverse transform process by the inverse transform filter 122 corresponding to the subject with the subject distance 350 [mm], which is the defocus position, by the filter processing unit 145. The image shown in FIG. 24B is an image that has been subjected to the inverse transform process by the filter processing unit 145 using the inverse transform filter corresponding to the subject with the subject distance of 400 [mm] that is the best focus position. It is the same image as the image shown in FIG. The image shown in FIG. 24C is an image obtained by performing the inverse transformation process by the inverse transformation filter 123 corresponding to the subject having the subject distance of 450 mm as the defocus position by the filter processing unit 145. In any of the images shown in FIG. 24, the blur of the subject (two-dimensional code) is restored, the code pattern is clear, and the MTF is improved.

  In this way, for example, an inverse transform filter corresponding to the subject distance is derived using Expression (12), the subject distance of the subject included in the captured image is determined, and the inverse transform filter corresponding to the determined subject distance. By selecting and applying the image, it is possible to obtain an image in which the blur is restored not only for the subject at the focus position but also for the subject at the defocus position. Here, the determination operation of the subject distance of the subject included in the captured image by the distance determination unit 142 and the selection operation of the inverse transform filter according to the subject distance by the selection unit 144 will be described below.

  FIG. 25 is a diagram showing an example of a histogram for the luminance value of an image when the subject distance is 350 [mm]. FIG. 26 is a diagram illustrating an example of a histogram regarding the luminance value of an image when the subject distance is 400 [mm]. FIG. 27 is a diagram illustrating an example of a histogram regarding the luminance value of an image when the subject distance is 470 [mm]. The determination operation of the subject distance of the subject by the distance determination unit 142 will be described with reference to FIGS. 25 to 27, and further, the selection operation of the inverse transform filter according to the subject distance by the selection unit 144 will be described.

  As described above, by selecting the inverse transform filter corresponding to the subject distance and executing the inverse transform process, it is possible to obtain an image with the blur restored regardless of the focus position or the defocus position. However, actually, the subject distance of the subject included in the captured image captured by the image sensor 12 is the same as the lens unit 11 and the image sensor as in the configuration of the imaging device 1 according to the present embodiment (see FIG. 3). Since a plurality of sets 12 are not provided, measurement cannot be performed by the principle of triangulation. Therefore, the image processing unit 14 of the imaging apparatus 1 according to the present embodiment determines the subject distance based on the following characteristics regarding captured images of subjects with different subject distances.

  25 to 27 are histograms of luminance values in captured images of subjects with subject distances of 350, 400, and 470 [mm], respectively. As shown in FIGS. 25 to 27, it can be seen that the greater the subject distance, the smaller the variance of the histogram and the smaller the difference between the maximum luminance value and the minimum luminance value. Using this property, for example, the distance determination unit 142 of the image processing unit 14 calculates the variance of the histogram for the luminance value from the captured image as the characteristic of the captured image. When the calculated variance is less than the predetermined first threshold, the distance determination unit 142 determines that the subject of the captured image is at a subject distance farther from the imaging device 1 than the focus position. In addition, when the variance is equal to or greater than the predetermined second threshold, the distance determination unit 142 determines that the subject of the captured image is at a subject distance closer to the imaging device 1 than the focus position. Then, the distance determination unit 142 determines that the subject of the captured image is at a subject distance near the focus position when the variance is greater than or equal to the first threshold and less than the second threshold. The distance determination unit 142 sends the determined subject distance information to the selection unit 144.

  Note that the distance determination unit 142 determines the subject distance using the variance of the histogram with respect to the luminance value of the captured image, but is not limited to this. That is, the distance determination unit 142 may determine the subject distance based on the difference between the maximum luminance value and the minimum luminance value of the captured image described above. Specifically, the distance determination unit 142 first calculates the difference between the maximum luminance value and the minimum luminance value of the captured image as the characteristic of the captured image. If the calculated difference is less than the predetermined third threshold, the distance determination unit 142 determines that the subject of the captured image is at a subject distance farther from the imaging device 1 than the focus position. In addition, when the calculated difference is equal to or greater than a predetermined fourth threshold, the distance determination unit 142 determines that the subject of the captured image is at a subject distance closer to the imaging device 1 than the focus position. If the calculated difference is greater than or equal to the third threshold value and less than the fourth threshold value, the distance determination unit 142 determines that the subject of the captured image is at a subject distance near the focus position.

  The distance determination unit 142 determines the subject distance using both the variance of the histogram of the brightness value of the captured image and the difference between the maximum brightness value and the minimum brightness value of the captured image as the characteristics of the captured image. It is good. Specifically, the distance determination unit 142 first obtains a histogram of luminance values of the captured image, calculates a variance of the histogram, and calculates a difference between the maximum luminance value and the minimum luminance value of the captured image. When the variance is less than the first threshold and the difference is less than the third threshold, the distance determination unit 142 determines that the subject of the captured image is at a subject distance farther from the imaging device 1 than the focus position. In addition, the distance determination unit 142 determines that the subject of the captured image is closer to the subject distance closer to the imaging device 1 than the focus position when the variance is equal to or greater than the second threshold value and the difference is equal to or greater than the fourth threshold value. Then, the distance determination unit 142 determines that the subject of the captured image is at a subject distance near the focus position when the variance is greater than or equal to the first threshold and less than the second threshold or the difference is greater than or equal to the third threshold and less than the fourth threshold. To do.

  In addition, as described above, the determination of the subject distance by the distance determination unit 142 determines which of the three types of distance regions belongs, but is not limited to this, and there are two types or four or more types. It may be determined which one of the distance regions belongs.

Further, the distance determination unit 142 determines the subject distance based on the histogram of the luminance value of the captured image or the difference between the maximum luminance value and the minimum luminance value. However, the distance determination unit 142 is not limited to this. Absent. For example, the distance determination unit 142 determines the subject distance based on the contrast of the captured image as the characteristic of the captured image, that is, the MTF graph with respect to the spatial frequency of the captured image as illustrated in FIGS. It may be a thing. For example, as shown in FIG. 9, in the MTF graph of the subject distance 350 [mm] closer to the imaging device 1 than the focus position, the MTF of the spatial frequency on the low frequency side (for example, 10 [mm ⁻¹ ]) is high, There is a tendency that the MTF of the spatial frequency on the high frequency side (for example, 50 [mm ⁻¹ ]) is low. Also, as shown in FIGS. 10 and 11, in the MTF graph of the subject distance 400 [mm] at the focus position and the MTF graph of the subject distance 470 [mm] far from the imaging device 1 from the focus position, The difference between the MTF of the spatial frequency and the MTF of the spatial frequency on the high frequency side does not change much. However, the MTF graph of the subject distance 470 [mm] farther from the imaging device 1 than the focus position has a lower MTF overall than the MTF graph of the subject distance 400 [mm] at the focus position (for example, the average MTF) Tend to be low). The distance determination unit 142 can also determine the subject distance based on the MTF graph corresponding to each subject distance. Specifically, for example, the distance determination unit 142 first obtains a difference between an MTF having a predetermined spatial frequency on the low frequency side and an MTF having a predetermined spatial frequency on the high frequency side, and the difference is equal to or greater than a predetermined threshold value. If it is, it is determined that the subject of the captured image is at a subject distance closer to the imaging device 1 than the focus position. In addition, when the obtained difference is less than the predetermined threshold, the distance determination unit 142 determines that the subject of the captured image is a subject distance near the focus position or a subject distance farther from the imaging device 1 than the focus position. Further, the distance determination unit 142 determines that the subject of the captured image is at a subject distance near the focus position when the average value of the MTF is equal to or greater than a predetermined threshold value. It is determined that the subject is far from the subject.

  In the above description, the distance determination unit 142 determines the subject distance based on the characteristics of the entire captured image. However, the present invention is not limited to this. For example, the distance determination unit 142 may divide the captured image into a plurality of images and determine the subject distance of the subject included in the partial image for each partial image based on the above-described characteristics of the divided partial images. Good.

  The selection unit 144 receives information on the subject distance determined by the distance determination unit 142, and selects and extracts an inverse transform filter corresponding to the subject distance from the inverse transform filters stored in advance in the storage unit 143. The selection unit 144 sends information on the selected inverse transform filter to the filter processing unit 145. Then, the filter processing unit 145 performs an inverse conversion process on the captured image received from the image buffer unit 141 using the inverse conversion filter received from the selection unit 144.

  As described above, the distance determination unit 142 determines the subject distance of the subject included in the captured image, the selection unit 144 selects the inverse transform filter corresponding to the determined subject distance, and the filter processing unit 145 selects the selection. The inverse transform process is performed on the captured image with the inverse transform filter. This makes it possible to restore the blur of the captured image regardless of the subject distance.

(Operation of recognition processing unit)
The recognition processing unit 15 performs recognition processing of symbols included in the image on which the inverse conversion processing is performed by the filter processing unit 145 on the captured image captured by the image sensor 12. Specifically, the recognition processing unit 15 recognizes a symbol included in the image after the inverse conversion process and decodes the recognized symbol as a symbol recognition process. Then, the recognition processing unit 15 transmits the decoded information to an external device (for example, the PC 2 illustrated in FIG. 1) via the communication unit 16.

  As described above, since the resolution is improved by the inverse conversion process of the filter processing unit 145, that is, the recognition process is performed on the symbol included in the subject whose blur is restored, the accuracy of symbol recognition is improved. Can do.

(Overall flow of operation of imaging device)
FIG. 28 is a flowchart illustrating an example of the operation of the imaging apparatus according to the embodiment. The overall operation flow of the imaging apparatus 1 will be described with reference to FIG.

<Step S11>
The lens unit 11 collects light from the subject and forms an image on the image sensor 12. At this time, the phase plate LZ4 of the lens unit 11 gives aberration to the light incident on the lens unit 11. As a result, the phase plate LZ4 diffuses the point spread function (PSF) with respect to the light incident on the image sensor 12 so that the image picked up by the image sensor 12 is not in a blurred state. The blur is constant, that is, the shape of the PSF is constant. The image pickup device 12 picks up a subject and generates a picked-up image by converting light from the subject incident on the lens unit 11 into an electrical signal. The image sensor 12 outputs the generated captured image to the image buffer unit 141 of the image processing unit 14. Then, the process proceeds to step S12.

<Step S12>
The image buffer unit 141 sends the buffered captured image to the distance determination unit 142. The distance determination unit 142 is included in the captured image on the basis of the characteristics of the received captured image (dispersion of the histogram regarding the brightness value, the difference between the maximum brightness value and the minimum brightness value, or the MTF graph with respect to the spatial frequency). The subject distance of the subject to be determined is determined. The distance determination unit 142 sends information on the determined subject distance to the selection unit 144. Then, the process proceeds to step S13.

<Step S13>
The selection unit 144 receives information on the subject distance determined by the distance determination unit 142, and selects and extracts an inverse transform filter corresponding to the subject distance from the inverse transform filters stored in advance in the storage unit 143. The selection unit 144 sends information on the selected inverse transform filter to the filter processing unit 145. Then, the process proceeds to step S14.

<Step S14>
The filter processing unit 145 receives the captured image from the image buffer unit 141, and performs an inverse conversion process on the captured image by the inverse conversion filter received from the selection unit 144. The filter processing unit 145 sends the image after the inverse transformation process to the recognition processing unit 15. Then, the process proceeds to step S15.

<Step S15>
The recognition processing unit 15 performs symbol recognition processing on the subject on the image on which the inverse conversion processing has been performed by the filter processing unit 145. Then, the process proceeds to step S16.

<Step S16>
When the recognition processing unit 15 executes symbol recognition processing on the image input from the image processing unit 14 to recognize a symbol, that is, when a symbol candidate is found (step S16: Yes). ), Go to step S17. On the other hand, if the symbol is not recognized, that is, if no symbol candidate is found (step S16: No), the operation is terminated.

<Step S17>
As a result of executing the decoding process on the recognized symbol by the recognition processing unit 15, when the decoding is normally completed (step S17: Yes), the process proceeds to step S18, and when the decoding is not normally completed (step S17). : No), the operation is terminated.

<Step S18>
The recognition processing unit 15 performs symbol recognition processing on the image input from the image processing unit 14, thereby recognizing the symbols included in the image, and decoding the recognized symbol information into an external device (for example, It is transmitted (output) to the PC 2) shown in FIG.

  Through the above flow, the subject distance determination, the inverse transform filter selection, the inverse transform process, and the symbol recognition process by the imaging apparatus 1 are executed. If the symbol recognition process is not normally completed by the operations in steps S11 to S18, it may be repeated a predetermined number of times until it is normally completed.

  As described above, in the imaging apparatus 1 according to the present embodiment, the lens unit 11 diffuses the PSF with respect to the input light, and the image picked up by the image pickup device 12 is not in a blurred state, but has a wide depth. So that the blur is constant. The distance determination unit 142 determines the subject distance of the subject included in the captured image based on the characteristics of the captured image captured by the imaging element 12, and the selection unit 144 uses an inverse transform filter corresponding to the determined subject distance. select. And the filter process part 145 shall perform an inverse transformation process with respect to a captured image with the inverse transformation filter selected by the selection part 144. FIG. This makes it possible to restore the blur of the captured image including the subject regardless of the subject distance. In addition, since the subject distance is determined based on the characteristics of the captured image using a single imaging element 12 without using a plurality of imaging means, the increase in the size of the imaging device 1 is suppressed, and the cost is reduced. Increase can be suppressed. In addition, since the blur of the captured image including the subject can be restored regardless of the subject distance, the recognition processing unit 15 can improve the recognition accuracy of symbols such as a two-dimensional code included in the subject. .

  As described above, the distance determination unit 142 divides the captured image into a plurality of images, and determines the subject distance of the subject included in the partial image for each partial image based on the above-described characteristics of the divided partial images. It is desirable to judge. As a result, when a subject at a different subject distance is included in one captured image, an inverse conversion process is executed for each partial image according to the subject distance of the subject included in the partial image. It can be restored.

  In addition, in order to extend the depth of field, an example of the inverse transformation process by the inverse transformation filter based on the frequency characteristic R shown in the above equation (12) is given as an example, but the method of extending the depth of field is limited to this. It is not something. That is, the depth of field may be extended by an inverse transform process using different inverse transform filters, or by another different process.

  Further, the phase plate LZ4 of the lens unit 11 makes the blur constant at a wide depth, and the blur is restored by executing the inverse transform process of the filter processing unit 145, thereby extending the depth of field. Although it is to be realized, it is not limited to this. That is, this is an operation in which the filter processing unit 145 restores the captured image including the blur due to the aberration of the lens unit 11 by performing an inverse transformation process, and does not particularly extend the depth of field. Even in this case, the above-described effects can be achieved by the determination of the subject distance by the above-described distance determination unit 142 and the selection operation of the inverse transform filter by the selection unit 144.

1 Imaging device 2 PC
3 Communication Cable 4 Subject 11 Lens Unit 12 Image Sensor 14 Image Processing Unit 15 Recognition Processing Unit 16 Communication Unit 21 Communication Unit 22 Operation Unit 23 Display Unit 24 Storage Unit 25 External Storage Device 26 Control Unit 27 Bus 101 Captured Image 105 Image 121- 123 Inverse transform filter 131, 131a-131f Target partial image 135a-135f Central data 141 Image buffer unit 142 Distance determination unit 143 Storage unit 144 Selection unit 145 Filter processing unit 201 Target spatial frequency characteristic 202, 202a Spatial frequency characteristic 203, 203a Space Frequency characteristics 500 Imaging system A Frame start period B Horizontal blanking period B1, B2_S, B2_T graph C Frame end period D Vertical blanking period G1, G2_S, G2_T graph LZ1 to LZ3 'S LZ4 phase plate LZ5~LZ7 lens R1, R2_S, R2_T graph SB stop T valid data period

JP 2013-162369 A

Claims (11)

An optical system that gives aberration to incident light;
One image pickup means for picking up an image of a subject by converting the light passing through the optical system into a pixel;
A distance determination unit that determines a subject distance of the subject based on characteristics of a captured image captured by the imaging unit;
Selecting means for selecting an inverse transform filter corresponding to the subject distance;
Inverse conversion means for executing an inverse conversion process for restoring blur caused by the aberration in the captured image by the inverse conversion filter selected by the selection means;
An imaging apparatus comprising: The distance determination unit divides the captured image into a plurality of partial images, determines, for each partial image, a subject distance of a subject included in the partial image based on characteristics of the partial image,
The imaging apparatus according to claim 1, wherein the selection unit selects the inverse transform filter corresponding to the subject distance for each of the partial images determined by the distance determination unit.   The imaging apparatus according to claim 1, wherein the distance determining unit calculates a variance of a histogram with respect to a luminance value of the captured image as a characteristic of the captured image, and determines the subject distance based on the variance. .   The distance determination unit calculates a difference between a maximum luminance value and a minimum luminance value of the captured image as a characteristic of the captured image, and determines the subject distance based on the difference. The imaging device according to claim 1.   The imaging apparatus according to claim 1, wherein the distance determination unit determines the subject distance based on a contrast of the captured image as a characteristic of the captured image.   The imaging apparatus according to claim 1, further comprising a storage unit that stores the inverse transform filter corresponding to the subject distance.   The inverse transform filter minimizes a mean square error between the frequency characteristic of the pixel of the ideal image of the subject according to the subject distance and the frequency characteristic of the pixel of the image subjected to the inverse transform process by the inverse transform unit. The imaging device according to claim 1, wherein the imaging device is a filter. The optical system gives the aberration such that the blur is constant at a predetermined depth;
The said inversion means expands a depth of field by performing the said inversion process by the said inversion filter which restore | restores the said constant blurring by the said aberration, The depth of field as described in any one of Claims 1-7. Imaging device.   The imaging apparatus according to claim 1, further comprising a recognition processing unit that recognizes a symbol with respect to the image on which the inverse conversion process has been performed by the inverse conversion unit. The imaging device according to any one of claims 1 to 8,
Recognition processing means for recognizing a symbol for the image subjected to the inverse transformation processing by the inverse transformation means;
An imaging system comprising: An optical processing step for providing aberration to the incident light by the optical system;
An imaging step of capturing an image of a subject by converting the light that has passed through the optical system into a pixel by a single imaging unit;
A distance determination step for determining a subject distance of the subject based on characteristics of the captured image;
A selection step of selecting an inverse transform filter corresponding to the subject distance;
An inverse transformation step of performing an inverse transformation process for restoring blur due to the aberration in the captured image by the selected inverse transformation filter;
An imaging method comprising: