JP5614256B2 - Imaging apparatus, image processing apparatus, and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, and an imaging method.

  In general, a digital imaging apparatus performs an imaging operation by forming an image of a subject on a solid-state imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor. In recent years, a technique called WFC (Wavefront Coding) has attracted attention as a technique applied to such an imaging apparatus.

  In an imaging apparatus employing the WFC technology, a phase modulation element is disposed on an optical path from a subject with respect to a solid-state imaging element. The phase modulation element regularly disperses the light beam emitted from the subject and deforms the image connected to the light receiving surface of the solid-state imaging element. As a result, an image out of focus is obtained from the solid-state imaging device, and the image obtained at this time is called an “intermediate image”. The imaging apparatus restores an image of a subject by performing digital processing using a point spread function (PSF) on an intermediate image obtained from a solid-state imaging device.

  The degree of blur occurring in the intermediate image obtained using the phase modulation element does not vary greatly even when the distance between the imaging device and the subject varies. Therefore, the depth of field can be expanded by performing digital processing on the intermediate image with a small fluctuation amount of the degree of blur to restore the subject image.

Such WFC technology is also considered to be adopted in an imaging mechanism mounted on an electronic endoscope system in addition to a general camera product such as a digital camera.
In addition, as another imaging apparatus that performs image processing using PSF, a point spread function (PSF) is calculated for each small region in the captured image, and based on the calculated point spread function, There is one that eliminates the blur caused by chromatic aberration.

JP 2003-235794 A JP 2010-147926 A

Edward R. Dowski, Jr. and W. Thomas Cathey, "Extended depth of field through wave-front coding", Applied Optics, vol.34, no.11, April, 1995, pp. 1859-1866

  In the above WFC technology, the fact that the fluctuation amount of the blur degree is small regardless of the distance from the subject means that the fluctuation amount of the PSF is small regardless of the distance from the subject. For this reason, at the time of image restoration, image processing is performed using the same PSF obtained from the design value of the optical system including the phase modulation element regardless of the distance from the subject. However, the PSF used at the time of image restoration may be slightly different from the PSF in the actual optical system due to a manufacturing error of the optical system.

  In the WFC technology, the accuracy of the PSF used at the time of image processing is directly linked to the image restoration accuracy. When there is an error between the PSF used at the time of image restoration and the PSF based on the actual design value of the optical system, there is a problem that the restoration accuracy of the image is deteriorated, for example, a ghost is generated in the restored image.

  The present invention has been made in view of such a problem, and an object thereof is to provide an imaging apparatus, an image processing apparatus, and an imaging method capable of generating an image with an increased depth of field with high accuracy. And

  In order to solve the above problems, an image sensor that receives light from a subject through a phase modulation element, and an image extraction that extracts a partial image including a region in which point images are dispersed from a captured image obtained by the image sensor. And a restoration processing unit that uses the partial image extracted by the image extraction unit as a point spread function and performs a process of restoring a subject image on the captured image obtained by the imaging device. An apparatus is provided.

  In order to solve the above problems, an image processing apparatus and an imaging method are provided in which processing similar to that of the imaging apparatus is executed.

  According to the above imaging device, image processing device, and imaging method, an image with an expanded depth of field can be generated with high accuracy.

It is a figure which shows the structural example of the imaging device which concerns on 1st Embodiment. It is a figure which shows the structural example of the imaging device which concerns on 2nd Embodiment. It is a figure which shows the example of a shape of a phase plate. It is a figure which shows the example of PSF. It is a graph which shows the example of distribution of MTF in the image imaged without using a phase plate. It is a graph which shows the example of distribution of MTF in the image imaged using the phase plate. It is a graph which shows the example of a characteristic of the inverse filter used for restoration | reconstruction of an image. It is a graph which shows the example of distribution of MTF in the image by which the to-be-photographed image was decompress | restored using the inverse filter of FIG. It is a flowchart which shows the imaging process procedure of the imaging device which concerns on 2nd Embodiment. It is a figure which shows the structural example of the imaging device which concerns on 3rd Embodiment. It is a 1st figure which shows the example of an imaging operation in PSF detection mode. It is a 2nd figure which shows the imaging operation example in PSF detection mode. It is a 3rd figure which shows the imaging operation example in PSF detection mode. 12 is a flowchart illustrating an imaging process procedure in a PSF detection mode of an imaging apparatus according to a third embodiment. 12 is a flowchart illustrating an imaging processing procedure in a main imaging mode of an imaging apparatus according to a third embodiment. It is a figure which shows the example of the to-be-photographed object to be imaged in PSF detection mode. 14 is a flowchart illustrating an imaging process procedure in a PSF detection mode of an imaging apparatus according to a fourth embodiment. It is a figure which shows the structural example of the imaging device which concerns on 5th Embodiment. 14 is a flowchart illustrating an imaging processing procedure of an imaging apparatus according to a fifth embodiment. It is a figure which shows the structural example of the imaging device which concerns on 6th Embodiment. It is a figure which shows the example of the to-be-photographed object to be imaged in PSF detection mode. It is a figure which shows the example of a setting of the area | region which extracts a partial image. It is a figure which shows the example of the division area which divided | segmented the intermediate image. 14 is a flowchart illustrating an imaging process procedure in a PSF detection mode of an imaging apparatus according to a sixth embodiment. 19 is a flowchart illustrating an imaging processing procedure in a main imaging mode of an imaging apparatus according to a sixth embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus according to the first embodiment.

  An imaging apparatus 1 illustrated in FIG. 1 is a digital imaging apparatus that captures a subject image with an imaging element 11. In addition, the imaging apparatus 1 can capture an image whose depth of field is expanded by the WFC technology. That is, the image sensor 11 receives light from the subject through the phase modulation element 12. The phase modulation element 12 disperses the incident subject image in a certain direction. In the imaging apparatus 1, a subject image with an expanded depth of field can be restored by performing image processing on a captured image obtained by the imaging element 11 using a point spread function (PSF). It has become.

  Although not shown, an optical member such as a lens is disposed in addition to the phase modulation element 12 on the subject side of the imaging element 11. The lens is disposed, for example, on the subject side of the phase modulation element 12 or on the imaging element 11 side of the phase modulation element 12, or both.

Here, an example of arithmetic processing when restoring the subject image will be described, and then the configuration of FIG. 1 will be described again.
An image picked up using the phase modulation element 12, that is, a picked-up image obtained from the image pickup element 11 of FIG. 1 is called an intermediate image. Here, if the intermediate image is “h”, the PSF is “g”, and the subject image captured when entering the image sensor 11 without passing through the phase modulation element 12 is “f”, “h” “g” The relationship of the following expression (1) is established between “f”.

When the above equation (1) is Fourier transformed, the following equation (2) is obtained.
F ・ G = H (2)
In the above formula (2), “G”, which is the result of Fourier transform of PSF “g”, is an optical transfer function (OTF) and represents the spatial frequency transfer characteristic in the imaging optical system. . The absolute value of OTF is called MTF (Modulation Transfer Function).

The subject image “F” after the Fourier transform can be obtained using the following equation (3). In this equation (3), “H”, which is the result of Fourier transform of the intermediate image “h”, is multiplied by the inverse filter “H _inv ” of OTF “G”, so that the subject image “Fin transformed” F "is required. Accordingly, the subject image “f” can be obtained by performing inverse Fourier transform on “F” in Expression (3).
F = H _inv · H (3)
The inverse filter “H _inv ” shown in Expression (3) is expressed as the following Expression (4).

As another calculation method, an inverse kernel “h _inv ” is obtained by _performing an inverse Fourier transform on the above inverse filter “H _inv ”, and as shown in the following equation (5), the inverse of the intermediate image “h” is obtained. The subject image “f” can also be obtained by performing a convolution operation with the kernel “h _inv ”.

  As shown in the above formulas (1) to (5), in the WFC technology, the subject image restoration calculation is performed using the PSF. In addition, the degree of blur that occurs in the intermediate image captured using the phase modulation element is substantially constant regardless of the distance between the imaging device and the subject. For this reason, in the WFC technology, it is possible to obtain an image with an expanded depth of field by performing a restoration operation using the same PSF obtained from the design value of the optical system including the phase modulation element. It was.

  However, the PSF used at the time of image restoration may be slightly different from the PSF in the actual optical system due to factors such as manufacturing errors of the optical system. When such a PSF error occurs, there is a possibility that the restoration accuracy of the image deteriorates, for example, a ghost occurs in the restored image.

  In order to solve such a problem, the imaging apparatus 1 according to the present embodiment obtains a PSF from an image actually captured by the imaging element 11, and performs a subject image restoration operation using the obtained PSF. As a result, the amount of PSF error caused by manufacturing errors of the optical system is reduced, and the subject image can be restored with higher accuracy.

  As shown in FIG. 1, the imaging apparatus 1 includes an image extraction unit 13 and a restoration processing unit 14 in addition to the imaging element 11 and the phase modulation element 12. The image extraction unit 13 extracts a partial image including a region in which dot-like images (point images) are dispersed from the captured image obtained by the imaging element 11. The restoration processing unit 14 performs a process of restoring the subject image on the captured image obtained by the imaging element 11. The restoration processing unit 14 uses the partial image extracted by the image extraction unit 13 as the PSF during the subject image restoration processing.

  An example of an image captured by the imaging device 1 is shown on the lower side of FIG. The captured image P <b> 1 is an image captured by the image sensor 11 and input from the image sensor 11 to the image extraction unit 13. This captured image P <b> 1 is also called an “intermediate image” and is in a state where the subject image is blurred mainly due to the action of the phase modulation element 12. As described above, the phase modulation element 12 disperses the incident subject image in a certain direction.

  The partial image P2 is an image extracted from the lower right region of the captured image P1 by the image extraction unit 13, for example. The partial image P2 includes a region where a dot-like image (point image) included in the subject image is captured. The point images in the partial image P2 are in a dispersed state mainly due to the action of the phase modulation element 12. Therefore, the signal distribution in the partial image P2 indicates the PSF of the optical system actually captured.

  The partial image P2 can be extracted by pattern matching, for example. In this case, for example, a comparative image that takes the PSF obtained from the design value of the optical system of the imaging apparatus 1 is stored in a memory (not shown) in the imaging apparatus 1. The image extraction unit 13 performs pattern matching with the comparison image stored in the memory on the captured image P1 obtained from the imaging element 11, and extracts an image region similar to the comparison image as the partial image P2.

  The restoration processing unit 14 uses the extracted partial image P2 as a PSF and restores the subject image according to the above formula (3) and formula (5) with respect to the captured image obtained by the imaging device 11. Apply. An image P3 shown in the lower side of FIG. 1 is restored by the restoration processing unit 14 from the same image as the captured image P1 that is the extraction target of the partial image P2 by the image extraction unit 13. In this case, the restoration processing unit 14 restores the image P3 in which the blur that has occurred in the captured image P1 is eliminated.

  As in the example of the image P3, the captured image that is the processing target of the restoration processing unit 14 may be the same as the captured image that is the extraction target of the partial image by the image extracting unit 13. Alternatively, the captured image to be processed by the restoration processing unit 14 may be a captured image obtained again by the imaging element 11 after the partial image is extracted by the image extracting unit 13. In any of these cases, the restoration processing unit 14 performs a subject image restoration process using a PSF obtained from an image actually captured by the image sensor 11, and therefore the PSF obtained from the design value of the optical system is obtained. Compared to the case where the restoration process is used, the subject image can be restored with higher accuracy.

  In addition, as in the example of the image P3 described above, when the captured image that is the processing target of the restoration processing unit 14 is the same as the captured image that is the extraction target of the partial image by the image extracting unit 13, the imaging is performed. The PSF error corresponding to the distance between the apparatus 1 and the subject is also reduced, and the subject image can be restored with higher accuracy. The point that the PSF error corresponding to the distance from the subject is reduced will be described in detail in the second embodiment.

[Second Embodiment]
FIG. 2 is a diagram illustrating a configuration example of an imaging apparatus according to the second embodiment.
The imaging apparatus 100 includes an optical block 110, an imaging element 121, an A / D (Analog / Digital) converter 122, an image restoration processing unit 130, a nonvolatile storage unit 140, a DSP (Digital Signal Processor) 151, a display unit 152, and a memory card. 153, an imaging control unit 154, and an input unit 155.

  The optical block 110 is a block on which an optical system for condensing a light beam from a subject to the image sensor 121 is mounted, and includes lenses 111 and 112, an iris 113, a phase plate 114, and the like. The phase plate 114 is an example of a phase modulation element, and regularly disperses the light beam from the subject. In the example of FIG. 2, the phase plate 114 is disposed immediately after the iris 113, but the position of the phase plate 114 is not limited to such a position.

  The image sensor 121 is realized as, for example, a CCD or a CMOS sensor, and converts light incident through the optical block 110 into an analog image signal. The A / D converter 122 converts the analog image signal output from the image sensor 121 into a digital image signal.

  The image restoration processing unit 130 performs digital image processing based on the WFC technique on the image signal output from the A / D converter 122 to restore an image with an expanded depth of field. The non-volatile storage unit 140 stores various data used during processing by the image restoration processing unit 130.

  The DSP 151 performs predetermined digital image processing on the image signal output from the image restoration processing unit 130. The DSP 151 executes, for example, various image quality correction processing, detection processing for image quality correction, display image signal generation processing, compression encoding processing for generating recording image signals, and the like. Note that the processing function of the image restoration processing unit 130 may be realized as one function of the DSP 151.

  The display unit 152 is realized by a display device such as an LCD (Liquid Crystal Display), and displays an image based on a display image signal output from the DSP 151. The memory card 153 is a portable semiconductor storage device, and stores the recording image signal output from the DSP 151. The memory card 153 is an example of a storage medium that stores a recording image signal. For example, the image signal may be stored in another type of storage medium such as an HDD (Hard Disk Drive) or an optical disk. .

  The input unit 155 includes various input switches such as a shutter button. The imaging control unit 154 controls an imaging operation in the imaging apparatus 100 using, for example, pressing of a shutter button of the input unit 155 as a trigger. For example, when the shutter button is pressed, the imaging control unit 154 causes the image restoration processing unit 130 to execute digital image processing on the captured image signal captured from the A / D converter 122.

  Next, a configuration example of the image restoration processing unit 130 will be described. The image restoration processing unit 130 includes an image memory 131, an image extraction unit 132, a convolution control unit 133, and a convolution calculation unit 134.

  The image memory 131 is a RAM (Random Access Memory) that is used as a work area for processing in the image restoration processing unit 130, and temporarily stores various data including image data. For example, the image memory 131 temporarily stores RAW data of a captured image captured by the image sensor 121 and digitized by the A / D converter 122. Hereinafter, RAW data of a captured image stored in the image memory 131 is referred to as “intermediate image data”, and an image based on the intermediate image data is referred to as “intermediate image”.

  The image extraction unit 132 uses the intermediate image data stored in the image memory 131 to extract a partial area in which point images are dispersed from the intermediate image. In the present embodiment, the image extraction unit 132 extracts a partial region using the design PSF 141 stored in the nonvolatile storage unit 140. The design PSF 141 is a PSF calculated by simulation or the like based on the design value of the optical system in the optical block 110. The image extraction unit 132 extracts a partial region by performing pattern matching using the design PSF 141 on the intermediate image.

  The convolution control unit 133 and the convolution operation unit 134 perform image processing on the intermediate image data stored in the image memory 131 by using the partial image extracted by the image extraction unit 132 as a PSF. It is a processing unit that performs processing for restoring a subject image. In the present embodiment, the convolution calculation unit 134 performs image restoration calculation according to the above-described equation (5).

The convolution control unit 133 uses the partial image extracted by the image extraction unit 132 as a PSF, obtains the inverse kernel “h _inv ” shown in Expression (5) based on this PSF, and inputs it to the convolution calculation unit 134. To do. The inverse kernel “h _inv ” _obtains “G” in Expression (4) by Fourier transforming the PSF based on the partial image, and uses this “G” to calculate the inverse filter “H _inv ” in Expression (4). Can be obtained by _{performing an} inverse Fourier transform on the inverse filter “H _inv ”.

FIG. 3 is a diagram illustrating a shape example of the phase plate.
The phase plate 114 used in the present embodiment gives, for example, a deviation of exp {iψ (x, y)} to the phase of light passing therethrough. Assuming that ψ (x, y) = α (x ³ + y ³ ) / 2 as an example, the phase plate 114 has one surface being a plane and the other surface being z = iα as shown in FIG. The shape of a cubic surface represented by (x ³ + y ³ ) / 2 is formed. FIG. 3 shows the shape of the cubic curved surface when both x and y are in the range from −1 to 1.

FIG. 4 is a diagram illustrating an example of a PSF. In FIG. 4, for the sake of easy understanding, the signal level is black as it is high and the signal level is white as it is low.
The PSF when the phase plate 114 as shown in FIG. 3 is used is as shown in FIG. According to FIG. 4, it can be seen that the phase plate 114 of FIG. 3 has the characteristic of dispersing the passed light beam in two directions, leftward and downward.

  The imaging apparatus 100 according to the present embodiment stores PSF data as illustrated in FIG. 4 in the nonvolatile storage unit 140 in advance as a design PSF 141. Then, the image extraction unit 132 extracts a region having signal distribution characteristics similar to the design PSF 141 as a partial region from the captured image obtained by the image sensor 121 by pattern matching.

  When the design PSF 141 is as shown in FIG. 4, the image extraction unit 132 causes image dispersion in the left and lower two directions centered on the dotted image from the captured image instead of pattern matching. The partial image may be extracted by searching the obtained image.

Here, the characteristics of the captured image will be described. First, FIG. 5 is a graph showing an example of MTF distribution in an image captured without using a phase plate.
A straight line L1 in FIG. 5 shows an example of MTF distribution in a captured image obtained in a state where the light beam from the subject is focused on the image sensor 121 without using the phase plate 114. In an image captured in a focused state like the straight line L1, the MTF is distributed linearly.

  On the other hand, curves L2 and L3 in FIG. 5 show an example of MTF distribution in a captured image obtained when the lens in the optical block 110 is shifted from the in-focus position without using the phase plate 114. That is, the straight line L1 and the curves L2 and L3 indicate the characteristics of an image obtained when images of subjects with different distances from the imaging device 100 are captured.

  As shown by the curves L2 and L3, the MTF distribution is separated from the straight line L1 in the image captured with the focus shifted. In FIG. 5, a curve L3 is an MTF when the lens is shifted more greatly from the in-focus position than the curve L2, and it can be seen that the degree of degradation of the MTF increases as the focus shift increases.

FIG. 6 is a graph showing an example of MTF distribution in an image captured using a phase plate.
A curved line L11 in FIG. 6 indicates a captured image (hereinafter referred to as an intermediate image A) obtained when the phase plate 114 is inserted and imaging is performed in the same lens position state as when the characteristic of the straight line L1 in FIG. 5 is obtained. An example of the distribution of the MTF in FIG. Also, the curves L12 and L13 in FIG. 6 are captured images obtained when the phase plate 114 is inserted and imaged in the same lens position state as when the characteristics of the curves L2 and L3 in FIG. 5 are obtained, respectively. An example of MTF distribution in the following (hereinafter referred to as intermediate images B and C) is shown.

  As can be seen from these curves L11 to L13, when the phase plate 114 is inserted into the optical system, the amount of change in the MTF with respect to the change in the distance from the imaging device 100 to the subject is compared with the case where the phase plate 114 is not inserted. And extremely small. In other words, when the phase plate 114 is inserted into the optical system, even if the distance from the imaging device 100 to the subject changes, the degree of blur appearing in the intermediate image does not change significantly.

FIG. 7 is a graph illustrating an example of characteristics of an inverse filter used for image restoration. The curve shown in the graph of FIG. 7 shows the characteristics of the inverse filter “H _inv ” obtained by Expression (4) using the PSF shown in FIG.

FIG. 8 is a graph showing an example of MTF distribution in an image obtained by restoring the subject image using the inverse filter of FIG.
Curves L21 to L23 in FIG. 8 show an example of MTF distribution in an image obtained by restoring the subject image by performing an operation on the above-described intermediate images A to C using the inverse filter shown in FIG. Since the image after the calculation using the inverse filter is calculated using the same inverse filter for the intermediate images A to C having a small difference in MTF as shown in FIG. 6, as shown in FIG. In addition, the distribution of MTFs in these images is relatively similar. In the WFC technology, using such characteristics, an intermediate image obtained using a phase plate is calculated using the same MTF to obtain an image with an expanded depth of field.

  However, when images of subjects with different distances from the image capturing apparatus 100 are captured, the MTF distribution is not actually the same as shown in FIG. This is because the MTF in the intermediate image captured using the phase plate 114 slightly varies depending on the distance to the subject. Such a difference in MTF appears as a ghost or the like in the image after the restoration process using the PSF, causing image quality deterioration.

  On the other hand, each time imaging is performed, the imaging apparatus 100 according to the present embodiment obtains a PSF from the intermediate image obtained by the imaging, and performs a restoration process on the intermediate image using the obtained PSF. . By such processing, the PSF error due to the difference in distance from the subject does not theoretically occur, and the subject image can be restored with high accuracy.

  In the imaging apparatus 100 of the present embodiment, not only the PSF error due to the difference in distance from the subject, but also the PSF error due to the manufacturing error of the optical system including the phase plate 114 as in the first embodiment. Needless to say, it can be reduced.

FIG. 9 is a flowchart illustrating an imaging processing procedure of the imaging apparatus according to the second embodiment.
[Step S11] The imaging control unit 154 monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154 detects that the shutter button is pressed, the imaging control unit 154 outputs an imaging operation start signal to the image restoration processing unit 130. The image restoration processing unit 130 that has received the start signal executes the processes after step S12.

  [Step S12] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130 that has received the start signal from the imaging control unit 154, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

  [Step S13] The image extraction unit 132 reads the design PSF 141 from the nonvolatile storage unit 140. The image extraction unit 132 extracts a partial image from the intermediate image based on the intermediate image data stored in the image memory 131 by performing pattern matching using the read design PSF 141.

[Step S14] The convolution control unit 133 sets the partial image data extracted by the image extraction unit 132 as a PSF, and calculates an inverse kernel “h _inv ” based on the PSF. For example, the convolution control unit 133 temporarily stores the calculated inverse kernel “h _inv ” in the image memory 131.

[Step S15] The convolution calculation unit 134 uses the inverse kernel “h _inv ” calculated by the convolution control unit 133 in step S14, and performs the operation on the intermediate image data stored in the image memory 131 in step S12. A convolution operation is performed according to equation (5). Thereby, a subject image restoration process is executed.

  The image data calculated by the convolution calculation unit 134 is subjected to predetermined image quality correction processing and compression coding processing by the DSP 151, and then recorded on the memory card 153.

In FIG. 9, the still image capturing operation has been described. For example, when a monitor image is displayed on the display unit 152 or when a moving image is captured and recorded, the above steps S12 to S12 are performed. What is necessary is just to repeat the process of S15. Further, when displaying a monitor image or recording a moving image, extraction of a partial image and calculation of an inverse kernel “h _inv ” may be executed at regular intervals. In this case, until the new inverse kernel “h _inv ” is calculated, the convolution operation unit 134 performs an operation process using the same inverse kernel “h _inv ”. In this way, by executing partial image extraction and inverse kernel “h _inv ” calculation at regular intervals, the processing load and power consumption in the image restoration processing unit 130 can be reduced.

  In the process of FIG. 9, the partial image extraction and the subject image restoration process are executed based on the same intermediate image data stored in the image memory 131. However, as another processing example, the subject image restoration process may be executed based on intermediate image data newly stored in the image memory 131 after the partial image is extracted.

  For example, the intermediate image data to be subjected to subject image restoration processing is processed by the image extraction unit 132 and the convolution control unit 133 after the intermediate image data to be extracted from the partial image is taken into the image memory 131. The image may be taken into the image memory 131 after the required time. By such processing, there is no delay between the image capturing timing and the timing at which the captured image data is output from the image restoration processing unit 130. In particular, when displaying a monitor image, the display delay causes the delay. A sense of discomfort given to the user can be eliminated.

Further, for example, when a still image is captured, the following processing may be performed. First, when the shutter button is pressed halfway, partial image extraction and inverse kernel “h _inv ” are calculated. Thereafter, when the shutter button is fully pressed, the subject image is obtained by using the inverse kernel “h _inv ” calculated in the half-pressed state with respect to the intermediate image data fetched into the image memory 131 at that time. The restoration process is executed.

[Third Embodiment]
FIG. 10 is a diagram illustrating a configuration example of an imaging apparatus according to the third embodiment. In FIG. 10, components corresponding to those in FIG. 2 are denoted by the same reference numerals.

Imaging apparatus 100a according to the third embodiment, as the imaging mode, the PSF detection mode to calculate the extraction of the partial image and reverse the kernel "h _inv" inverse calculated by the PSF detection mode kernel "h _inv" There are two types of main imaging modes in which a subject image restoration process is performed on a captured image using. These imaging modes are selected, for example, when the user operates a mode selection button provided in the input unit 155. The imaging control unit 154a controls the operation of the image restoration processing unit 130a according to the selected imaging mode.

  The image restoration processing unit 130a includes the image memory 131 and the convolution operation unit 134 illustrated in FIG. However, the convolution calculation unit 134 operates only in the main imaging mode out of the two imaging modes. The image restoration processing unit 130a includes an image extraction unit 132a and a convolution control unit 133a. The image extraction unit 132a operates only in the PSF detection mode of the two imaging modes.

  The image extraction unit 132a extracts a predetermined area as a partial image from the intermediate image based on the intermediate image data stored in the image memory 131. In the present embodiment, as an example, the image extraction unit 132a extracts a certain region centered on the center of the intermediate image as a partial image.

In the PSF detection mode, the convolution control unit 133a calculates the inverse kernel “h _inv ” based on the partial image extracted by the image extraction unit 132 as in the convolution control unit 133 illustrated in FIG. However, the convolution control unit 133 a stores the calculated inverse kernel “h _inv ” in the nonvolatile storage unit 140 as kernel data 142. Then, after switching to the main imaging mode, the convolution control unit 133 a reads the kernel data 142 from the nonvolatile storage unit 140 and supplies the kernel data 142 to the convolution calculation unit 134.

  Here, the imaging operation in the PSF detection mode will be described. The PSF detection mode is selected, for example, at a stage before product shipment of the imaging apparatus 100a, at the first power-on of the imaging apparatus 100a after product shipment, or the like. Alternatively, the PSF detection mode may be automatically set every time the imaging apparatus 100a is turned on.

  In the PSF detection mode, the imaging apparatus 100a acquires an intermediate image by imaging a subject including a point image, and detects a PSF from the obtained intermediate image. There are the following methods for imaging a subject including a point image.

FIG. 11 is a first diagram illustrating an example of an imaging operation in the PSF detection mode.
In the PSF detection mode, the imaging device 100a images the detection subject 200 dedicated to the PSF detection mode as shown in FIG. The detection subject 200 includes, for example, a pinhole 201 and a light source 202, and light is emitted from the light source 202 from the back side of the pinhole 201. The imaging surface of the detection subject 200 is desirably a single color, and the light emitted from the pinhole 201 toward the imaging device 100a has a luminance or color different from that of the imaging surface of the detection subject 200. It is said. The imaging apparatus 100a can recognize the pinhole 201 as a point image by imaging the detection subject 200.

FIG. 12 is a second diagram illustrating an example of an imaging operation in the PSF detection mode.
As illustrated in FIG. 12, a point image 213 can be generated on the imaging surface 211 by irradiating light from a light source 212 such as a laser pointer onto an arbitrary imaging surface 211. In this case, the imaging surface 211 is preferably monochromatic, and the point image 213 generated on the imaging surface 211 has a brightness different from that of the imaging surface 211 or a color different from that of the imaging surface 211. In the PSF detection mode, the imaging device 100a detects the PSF by capturing the point image 213 generated on the imaging surface 211.

  The light source 212 may be mounted on the imaging device 100a. However, in this case, when the distance between the imaging surface 211 and the imaging device 100a is a predetermined distance, the point image 213 generated by the light source 212 is arranged at the center of the angle of view from the imaging device 100a. The irradiation angle of the light source 212 is adjusted.

FIG. 13 is a third diagram illustrating an example of an imaging operation in the PSF detection mode.
As illustrated in FIG. 13, the imaging apparatus 100a may capture an image in which the point images 221 are arranged in advance in the PSF detection mode. However, the point image 221 is arranged separately from the other images 222, and the point image 221 and an image component in which the point image 221 is dispersed by the action of the phase plate 114 are arranged in a single color area. It is desirable.

FIG. 14 is a flowchart illustrating an imaging process procedure in the PSF detection mode of the imaging apparatus according to the third embodiment.
[Step S21] The imaging control unit 154a monitors whether the shutter button of the input unit 155 has been pressed. When the imaging control unit 154a detects that the shutter button is pressed, the imaging control unit 154a outputs an imaging operation start signal to the image restoration processing unit 130a. The image restoration processing unit 130a that has received the start signal executes the processes after step S22.

  [Step S <b> 22] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130a that has received the start signal from the imaging control unit 154a, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

  [Step S23] The image extraction unit 132a extracts a predetermined region as a partial image from the intermediate image based on the intermediate image data stored in the image memory 131. Here, when the user presses the shutter button in step S21, the point image is arranged at the center of the angle of view. As a result, the image extraction unit 132a extracts a region having a predetermined size centered on the central portion of the intermediate image, thereby extracting a region including the point image and the component in which the point image is dispersed as a partial image. can do.

  In step S23, as in the image extraction unit 132 of the second embodiment, a partial image may be extracted by pattern matching the intermediate image and the design PSF 141.

[Step S24] The convolution control unit 133a uses the partial image data extracted by the image extraction unit 132a as a PSF, and calculates an inverse kernel “h _inv ” based on the PSF. The convolution control unit 133 a stores the calculated inverse kernel “h _inv ” in the nonvolatile storage unit 140 as kernel data 142.

FIG. 15 is a flowchart illustrating an imaging processing procedure in the main imaging mode of the imaging apparatus according to the third embodiment.
[Step S31] The imaging control unit 154a monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154a detects that the shutter button is pressed, the imaging control unit 154a outputs an imaging operation start signal to the image restoration processing unit 130a. The image restoration processing unit 130a that has received the start signal executes the processing from step S32.

  [Step S <b> 32] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130a that has received the start signal from the imaging control unit 154a, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

  [Step S33] The convolution control unit 133a reads the kernel data 142 from the nonvolatile storage unit 140. Note that the processing order of steps S32 and S33 may be reversed. Or each process of step S32 and S33 may be performed in parallel.

  [Step S34] The convolution operation unit 134 uses the kernel data 142 read from the non-volatile storage unit 140 by the convolution control unit 133a, and applies the intermediate image data stored in the image memory 131 in step S32. The convolution operation is performed according to the equation (5). Thereby, a subject image restoration process is executed.

  The image data calculated by the convolution calculation unit 134 is subjected to predetermined image quality correction processing and compression coding processing by the DSP 151, and then recorded on the memory card 153.

  In the third embodiment described above, the convolution operation unit 134 performs subject image restoration processing using the kernel data 142 obtained from the intermediate image obtained by actual imaging. For this reason, the subject image can be restored with higher accuracy than when restoration processing is performed using kernel data obtained from the design value of the optical system. In addition, the calculation process of the kernel data 142 is performed in the PSF detection mode. In the main imaging mode, the subject image restoration process is executed using the kernel data 142 stored in the nonvolatile storage unit 140. It is possible to reduce the processing load and speed up the restoration process.

[Fourth Embodiment]
In the third embodiment, a point image arranged in a monochrome region is captured in the PSF detection mode. On the other hand, by setting the point image to be imaged to a specific color, the point image and the image component in which the point image is distributed can be obtained regardless of the subject on which the point image exists. The PSF can be calculated separately from the intermediate image.

  Hereinafter, as a fourth embodiment, an imaging apparatus capable of calculating a PSF by imaging a subject including a point image of a specific color will be described. Note that the imaging apparatus according to the fourth embodiment relates to the third embodiment except that the convolution control unit 133a further has a function of separating pixels having a color of a specific color region from the partial image. The configuration is the same as that of the imaging device 100a. For this reason, in the following description of the fourth embodiment, the same reference numerals as those in the third embodiment are used.

FIG. 16 is a diagram illustrating an example of a subject to be imaged in the PSF detection mode.
The subject shown in FIG. 16 includes an arbitrary image 231 and a point image 232 of a specific color arranged inside the image 231. The color of the point image 232 may be at least a color different from the region including the image component in which the point image 232 is dispersed by the action of the phase plate 114 in the peripheral region of the point image 232. Further, the point image 232 may be preliminarily arranged inside the image 231. In this case, the image 231 including the point image 232 may be used also as, for example, a test image for aberration correction. Alternatively, the point image 232 may be generated by irradiating the image 231 with light from the external light source 212 as shown in FIG.

FIG. 17 is a flowchart illustrating an imaging process procedure in the PSF detection mode of the imaging apparatus according to the fourth embodiment.
[Step S41] The imaging control unit 154a monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154a detects that the shutter button is pressed, the imaging control unit 154a outputs an imaging operation start signal to the image restoration processing unit 130a. The image restoration processing unit 130a that has received the start signal executes the processing after step S42.

  [Step S <b> 42] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130a that has received the start signal from the imaging control unit 154a, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

  [Step S43] The image extraction unit 132a extracts a predetermined area as a partial image from the intermediate image based on the intermediate image data stored in the image memory 131. As in the third embodiment, when the user presses the shutter button in step S41, the point image 232 is arranged at the center of the angle of view. Thereby, the image extraction unit 132a can extract a region including the point image 232 and a component in which the point image 232 is dispersed as a partial image by extracting a predetermined region at the center of the intermediate image.

  In step S43, a partial image may be extracted by pattern matching the intermediate image and the design PSF 141 in the same manner as the image extraction unit 132 of the second embodiment.

  [Step S44] The convolution control unit 133a separates the R (Red) component and the G (Green) component from the partial image data extracted by the image extraction unit 132a. The convolution control unit 133a subtracts the value of the G component from the value of the R component for each pixel of the partial image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th1. When the subtraction result is equal to or greater than the threshold th1, the convolution control unit 133a sets the subtraction result as the pixel value. When the subtraction result is less than the threshold th1, the convolution control unit 133a sets the pixel value to 0. By doing so, an image of the (R−G) component is generated. The convolution control unit 133 a temporarily stores the generated image data in the image memory 131.

  [Step S45] The convolution control unit 133a separates the R component and the B (Blue) component from the partial image data extracted by the image extraction unit 132a. The convolution control unit 133a subtracts the value of the B component from the value of the R component for each pixel of the partial image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th2. The convolution control unit 133a sets the subtraction result to the pixel value when the subtraction result is equal to or greater than the threshold value th2, and sets the pixel value to 0 when the subtraction result is less than the threshold value th2. By doing so, an image of the (RB) component is generated. The convolution control unit 133 a temporarily stores the generated image data in the image memory 131.

Note that the processing order of steps S44 and S45 may be reversed. Alternatively, the processes of steps S44 and S45 may be executed in parallel.
[Step S46] The convolution control unit 133a calculates the PSF by calculating the average of the values of the pixels for the images generated in steps S44 and S45, respectively.

[Step S <b> 47] The convolution control unit 133 a calculates the inverse kernel “h _inv ” based on the calculated PSF, and stores the calculated inverse kernel “h _inv ” in the nonvolatile storage unit 140 as the kernel data 142.

Note that the imaging processing procedure in the main imaging mode is the same as the processing procedure shown in FIG.
In the above-described processing of FIG. 17, an example in which a red point image 232 is detected as a specific color has been described, but the color of the point image 232 is not limited to red. For example, instead of subtracting the G component value from the R component value in step S44 of FIG. 17, the R component value is subtracted from the B component value. In step S45, the B component value is subtracted from the R component value. A blue dot image 232 can be detected by subtracting the value of the G component from the value of the B component instead of subtracting.

  In the fourth embodiment described above, as in the third embodiment, the subject image is more accurately compared with the case where the restoration process is performed using kernel data obtained from the design value of the optical system. The effect that it can be restored is obtained. In addition, the processing load in the main imaging mode can be reduced, and the restoration process can be speeded up. Furthermore, in the fourth embodiment, the versatility of a subject to be imaged in the PSF detection mode can be improved as compared with the third embodiment.

[Fifth Embodiment]
An imaging apparatus according to a fifth embodiment below is equipped with a light source that generates a point image on a subject during imaging for PSF detection. And immediately after acquiring PSF by imaging using a light source, this imaging is performed in the state which does not irradiate light from a light source. By such an operation, not only the PSF error due to the manufacturing error of the optical system but also the PSF error due to the difference in distance from the subject can be reduced.

FIG. 18 is a diagram illustrating a configuration example of an imaging apparatus according to the fifth embodiment. In FIG. 18, the same reference numerals are given to the components corresponding to those in FIG.
In the imaging apparatus 100b illustrated in FIG. 18, the image restoration processing unit 130b includes the image memory 131, the convolution control unit 133, and the convolution calculation unit 134 illustrated in FIG. Furthermore, the image restoration processing unit 130b includes an image extraction unit 132b. Similar to the image extraction unit 132 in FIG. 2, the image extraction unit 132b performs a process of extracting a partial image from the intermediate image by pattern matching. However, the image extraction unit 132b generates a specific color image by separating pixels having a color of a specific color region from the intermediate image, and extracts a partial image from the specific color image.

  Furthermore, the imaging apparatus 100b includes a light source 161 that generates a point image of a specific color on the subject. The light source 161 is, for example, a laser light emitting element. Similar to the point image 232 in FIG. 16, the color of the point image generated by the light source 161 on the subject is an image in which the point image is dispersed by the action of the phase plate 114 in at least the peripheral region of the point image in the subject. The color is different from that of the region including the component. Therefore, for example, the light source 161 may be able to switch and emit light of a plurality of colors so that a point image having a color different from the color of the subject can be generated.

FIG. 19 is a flowchart illustrating an imaging process procedure of the imaging apparatus according to the fifth embodiment.
[Step S51] The imaging control unit 154b monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154b detects that the shutter button has been pressed, the processing after step S52 is executed.

  [Step S52] The imaging control unit 154b turns on the light source 161 to irradiate the subject with light from the light source 161. The imaging control unit 154b causes the imaging device 121 to image the subject irradiated with the light from the light source 161.

  [Step S53] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122 and is taken into the image memory 131 as intermediate image data.

  [Step S54] The image extraction unit 132b separates the R component and the G component from the intermediate image data captured in the image memory 131 in step S53. The image extraction unit 132b subtracts the G component value from the R component value for each pixel of the intermediate image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th1. The image extraction unit 132b sets the subtraction result to the pixel value when the subtraction result is equal to or greater than the threshold th1, and sets the pixel value to 0 when the subtraction result is less than the threshold th1. As a result, an image of the (RG) component is generated. The image extraction unit 132 b temporarily stores the generated image data in the image memory 131.

  [Step S55] The image extraction unit 132b separates the R component and the B component from the intermediate image data captured in the image memory 131 in step S53. The image extraction unit 132b subtracts the B component value from the R component value for each pixel of the intermediate image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th2. The image extraction unit 132b sets the subtraction result to the pixel value when the subtraction result is equal to or greater than the threshold th2, and sets the pixel value to 0 when the subtraction result is less than the threshold th2. As a result, an image of the (RB) component is generated. The image extraction unit 132 b temporarily stores the generated image data in the image memory 131.

Note that the processing order of steps S54 and S55 may be reversed. Or the process of step S54, S55 may be performed in parallel.
[Step S56] The image extracting unit 132b generates a specific color image by calculating the average of the values of the pixels of the images generated in steps S54 and S55. The image extraction unit 132b temporarily stores the generated specific color image data in the image memory 131.

  [Step S57] The image extraction unit 132b reads the design PSF 141 from the nonvolatile storage unit 140. The image extraction unit 132b extracts a partial image from the specific color image generated in step S57 by performing pattern matching using the read design PSF 141.

  Note that by using pattern matching to extract a partial image, for example, even when the position within the angle of view of the point image due to light from the light source 161 becomes unstable, such as when the surface of the subject is a curved surface, the partial image Can be extracted with high accuracy. However, as in the fourth embodiment, the partial image may be extracted from a predetermined region without using pattern matching. In this case, the processing in steps S54 to S57 is performed by extracting a partial image first, as in the processing procedure of steps S43 to S46 in FIG. 17, and (RG) component and (RB) from the extracted partial image. The PSF may be changed by generating each image of the components and taking the average of the images. Even when the partial image is extracted from a predetermined region, for example, when the subject is a flat plate, the partial image can be extracted with high accuracy.

[Step S58] The convolution control unit 133 sets the partial image data extracted by the image extraction unit 132b in step S57 as a PSF, and calculates an inverse kernel “h _inv ” based on the PSF. For example, the convolution control unit 133 temporarily stores the calculated inverse kernel “h _inv ” in the image memory 131.

[Step S <b> 59] The imaging control unit 154 b turns off the light source 161 and causes the imaging device 121 to image a subject in which the point image generated by the light source 161 disappears.
[Step S60] A signal of an image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122, and is taken into the image memory 131 as intermediate image data.

[Step S61] The convolution calculation unit 134 uses the inverse kernel “h _inv ” calculated by the convolution control unit 133 in step S58, and performs intermediate image data stored in the image memory 131 in step S60. A convolution operation is performed according to equation (5). Thereby, a subject image restoration process is executed.

  The image data calculated by the convolution calculation unit 134 is subjected to predetermined image quality correction processing and compression coding processing by the DSP 151, and then recorded on the memory card 153.

In the above-described processing of FIG. 19, an example in which a red dot image is detected as a specific color has been shown, but the color of the dot image is not limited to red.
In the above fifth embodiment, as in the second embodiment, every time imaging is performed, a PSF is obtained from the intermediate image obtained by the imaging, and the obtained PSF is used to perform the processing on the intermediate image. A restoration process is performed. By such processing, the PSF error due to the difference in distance from the subject does not theoretically occur, and the subject image can be restored with high accuracy regardless of the distance from the subject. Further, in the fifth embodiment, not only the PSF error due to the difference in distance from the subject but also the manufacture of the optical system including the phase plate 114 as in the first, third, and fourth embodiments. PSF errors due to errors can also be reduced.

[Sixth Embodiment]
In the first to fifth embodiments described above, a partial image extracted from one place in the intermediate image is set as a PSF, and a subject image restoration process is performed using the single PSF. On the other hand, there is a method in which a plurality of PSFs are obtained and an appropriate PSF is selected from the plurality of PSFs at the time of subject image restoration processing. In the following sixth embodiment, as an example of obtaining a plurality of PSFs, an imaging apparatus that obtains PSFs from a plurality of locations in an intermediate image and selectively uses these PSFs during restoration processing will be described.

FIG. 20 is a diagram illustrating a configuration example of an imaging apparatus according to the sixth embodiment. In FIG. 20, components corresponding to those in FIGS. 2 and 10 are denoted by the same reference numerals.
In the imaging device 100c illustrated in FIG. 20, the image restoration processing unit 130c includes the image memory 131 illustrated in FIG. Furthermore, the image restoration processing unit 130c includes an image extraction unit 132c, a convolution control unit 133c, and a convolution calculation unit 134c.

  Similar to the image extraction unit 132a of FIG. 10, the image extraction unit 132c extracts a partial image from a predetermined region of the intermediate image. However, the image extraction unit 132c extracts partial images from a plurality of regions in the intermediate image.

Similar to the convolution control unit 133a in FIG. 10, the convolution control unit 133c sets the partial image extracted by the image extraction unit 132c as the PSF in the PSF detection mode, and sets the inverse kernel “h _inv ” based on the PSF. The calculated inverse kernel “h _inv ” is stored in the nonvolatile storage unit 140 as kernel data 142. However, the convolution control unit 133c calculates kernel data 142 based on each of the plurality of partial images extracted by the image extraction unit 132c, and stores the calculated plurality of kernel data 142 in the nonvolatile storage unit 140.

  Similar to the convolution operation unit 134 in FIG. 10, the convolution operation unit 134 c performs subject image restoration processing using the kernel data 142 on the intermediate image data stored in the image memory 131 in the main imaging mode. Do. However, the convolution operation unit 134c divides the intermediate image into a plurality of regions, and uses the appropriate kernel data 142 corresponding to each divided region among the plurality of kernel data 142 stored in the nonvolatile storage unit 140. Then, subject image restoration processing is performed. As a result, PSF errors that can occur according to the difference in position within the angle of view can be reduced. As a factor causing the PSF error according to the difference in position within the angle of view, for example, there is an aberration of a lens in the optical block 110.

FIG. 21 is a diagram illustrating an example of a subject to be imaged in the PSF detection mode.
Imaging device 100c of the present embodiment captures a plurality of point images 241 to 245 as shown in FIG. 21 in the PSF detection mode. The point images 241 to 245 are arranged apart from each other so that the components in which the point images 241 to 245 are dispersed by the action of the phase plate 114 do not overlap each other. In the present embodiment, as an example, the point images 241 to 245 have the same color. In this case, by setting the color of each point image 241 to 245 to be different from the color of the background image 246, the imaging apparatus 100c can detect each point image 241 to 245 separately from the background image 246. .

  The point images 241 to 245 may be generated by irradiating a subject with light from an external light source, for example, as shown in FIG. Alternatively, the point images 241 to 245 may be preliminarily arranged in a subject shape.

FIG. 22 is a diagram illustrating an example of setting a region from which a partial image is extracted.
An intermediate image 250 illustrated in FIG. 22 is an image captured in the PSF detection mode. The image extraction unit 132c extracts partial images from each of the predetermined partial image regions 251 to 255 in the region of the intermediate image 250. The partial image areas 251 to 255 are associated with the point images 241 to 245 shown in FIG.

  In the PSF detection mode, the user performs imaging while adjusting the angle of view so that the point images 241 to 245 are arranged at substantially the center portions of the partial image regions 251 to 255, respectively. Here, the size of each of the partial image areas 251 to 255 is determined so that each of the point images 241 to 245 includes a component area dispersed by the phase plate 114.

  For example, in the PSF detection mode, a frame indicating each of the partial image areas 251 to 255 is displayed in the area of the display unit 152 that displays the monitor image, so that the user can display the point images 241 to 245 as the partial images. The angle of view can be adjusted so as to be arranged at substantially the center of the regions 251 to 255. As described above, the image extraction unit 132c is set in advance without using pattern matching by performing the imaging in a state where the point images 241 to 245 are arranged at substantially the center portions of the partial image regions 251 to 255, respectively. A plurality of partial images can be extracted by a simple process of extracting an image from the region.

FIG. 23 is a diagram illustrating an example of a divided region obtained by dividing the intermediate image.
The intermediate image 260 in FIG. 23 is a captured image that is subject to calculation by the convolution calculation unit 134c. In the main imaging mode, the convolution calculation unit 134c divides the intermediate image 260 to be calculated into a plurality of areas, and performs subject restoration processing using the kernel data 142 corresponding to each divided area.

  In the example of FIG. 23, the convolution calculation unit 134c divides the intermediate image 260 to be calculated into five divided regions 261 to 265. The divided areas 261 to 265 are set so as to include the partial image areas 251 to 255 shown in FIG. The convolution operation unit 134 c uses the kernel data 142 calculated based on the partial image region 251 when executing the restoration process for the divided region 261. Similarly, the convolution operation unit 134c uses the kernel data 142 calculated based on the partial image areas 252 to 255, respectively, when executing the restoration process for the divided areas 262 to 265. Thereby, the PSF error corresponding to the difference in position within the angle of view is reduced, and the subject image in each divided region can be restored with higher accuracy.

FIG. 24 is a flowchart illustrating an imaging process procedure in the PSF detection mode of the imaging apparatus according to the sixth embodiment.
[Step S71] The imaging control unit 154a monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154a detects that the shutter button has been pressed, the imaging control unit 154a outputs an imaging operation start signal to the image restoration processing unit 130c. The image restoration processing unit 130c that has received the start signal executes the processes after step S72.

  [Step S <b> 72] The signal of the image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130c that has received the start signal from the imaging control unit 154a, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

  [Step S <b> 73] The image extraction unit 132 c selects one of a plurality of predetermined partial image areas 251 to 255, and selects the selected partial image from the intermediate image based on the intermediate image data stored in the image memory 131. Extract the region as a partial image.

  In step S73, as in the image extraction unit 132 of the second embodiment, a partial image may be extracted by pattern matching between the intermediate image and the design PSF 141. When performing pattern matching, for example, in step S73, the image extraction unit 132c selects one of the divided areas 261 to 265 shown in FIG. 23, and the design PSF 141 is selected for the selected divided area on the intermediate image. And pattern matching.

  [Step S74] The convolution control unit 133c separates the R component and the G component from the partial image data extracted by the image extraction unit 132c in step S73. The convolution control unit 133c subtracts the value of the G component from the value of the R component for each pixel of the partial image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th1. The convolution control unit 133c sets the subtraction result to the pixel value when the subtraction result is equal to or greater than the threshold th1, and sets the pixel value to 0 when the subtraction result is less than the threshold th1. By doing so, an image of the (R−G) component is generated. The convolution control unit 133 c temporarily stores the generated image data in the image memory 131.

  [Step S75] The convolution control unit 133c separates the R component and the B component from the partial image data extracted by the image extraction unit 132a in step S73. The convolution control unit 133c subtracts the value of the B component from the value of the R component for each pixel of the partial image, and determines whether the subtraction result is equal to or greater than a predetermined threshold th2. When the subtraction result is greater than or equal to the threshold th2, the convolution control unit 133c sets the subtraction result as the pixel value. When the subtraction result is less than the threshold th2, the convolution control unit 133c sets the pixel value to 0. By doing so, an image of the (RB) component is generated. The convolution control unit 133 c temporarily stores the generated image data in the image memory 131.

Note that the processing order of steps S74 and S75 may be reversed. Or the process of step S74, S75 may be performed in parallel.
[Step S76] The convolution control unit 133c calculates the average of the values of the pixels for the images generated in steps S74 and S75, thereby calculating the PSF.

[Step S77] The convolution control unit 133c calculates the inverse kernel “h _inv ” based on the PSF calculated in step S76, and stores the calculated inverse kernel “h _inv ” as kernel data 142 in the nonvolatile storage unit 140. Remember.

  [Step S78] The image extraction unit 132c determines whether or not the processing in steps S73 to S77 has been executed for all of the predetermined partial image regions. If the image extraction unit 132c determines that there is an area where the above process has not been completed, the image extraction unit 132c executes the process of step S73. In this case, in step S73, the image extraction unit 132c selects a new area that has not been processed, and extracts the selected area from the intermediate image as a partial image. On the other hand, if the image extraction unit 132c determines that the above process has been executed for all regions, the image extraction unit 132c ends the process in the PSF detection mode.

FIG. 25 is a flowchart illustrating an imaging processing procedure in the main imaging mode of the imaging apparatus according to the sixth embodiment.
[Step S81] The imaging control unit 154a monitors whether the shutter button of the input unit 155 is pressed. When the imaging control unit 154a detects that the shutter button has been pressed, the imaging control unit 154a outputs an imaging operation start signal to the image restoration processing unit 130c. The image restoration processing unit 130c that has received the start signal executes the processing after step S82.

  [Step S82] A signal of an image (intermediate image) captured by the image sensor 121 is converted into digital data by the A / D converter 122. In the image restoration processing unit 130c that has received the start signal from the imaging control unit 154a, the image memory 131 takes in the intermediate image data output from the A / D converter 122.

[Step S83] The convolution control unit 133c selects a divided region to be calculated by the convolution calculation unit 134c from among the regions of the intermediate image.
[Step S84] The convolution control unit 133c reads the kernel data 142 corresponding to the divided area selected in Step S83 from the nonvolatile storage unit 140.

  [Step S85] The convolution calculation unit 134c uses the kernel data 142 read from the nonvolatile storage unit 140 by the convolution control unit 133c, and among the intermediate image data stored in the image memory 131 in Step S82. A convolution operation is performed on the data of the divided region selected in step S83 according to equation (5). The convolution calculation unit 134c temporarily stores the calculated image data in the image memory 131, for example.

  [Step S86] The convolution control unit 133c determines whether or not the processing of steps S83 to S85 has been executed for all the divided regions to be calculated by the convolution calculation unit 134c. If the convolution control unit 133c determines that there is a divided region that has not been subjected to the above process, the convolution control unit 133c executes the process of step S83. In this case, the convolution control unit 133c selects a new divided area that has not been processed, and reads the kernel data 142 corresponding to the selected divided area from the nonvolatile storage unit 140. On the other hand, when the convolution control unit 133c determines that the above process has been executed for all the divided regions, the convolution calculation unit 134c causes the process of step S87 to be executed.

[Step S87] The convolution calculation unit 134c integrates the images that have been calculated in step S85, and outputs them to the DSP 151 as an image in which the subject image is restored.
The image data output from the convolution operation unit 134 c is subjected to predetermined image quality correction processing and compression encoding processing by the DSP 151, and then recorded on the memory card 153.

  In the sixth embodiment described above, the PSF is obtained from a plurality of regions on the intermediate image, and subject image restoration processing is performed using the PSF corresponding to each region for each region on the intermediate image to be restored. Executed. By such processing, the PSF error corresponding to the difference in position within the angle of view is reduced, and the subject image in each divided region can be restored with higher accuracy.

  Note that the processing for restoring a subject image using any one of PSFs obtained from a plurality of areas described in the sixth embodiment is also applied to the third and fifth embodiments. It is possible to apply.

  Another example of obtaining a plurality of PSFs based on the intermediate image and selectively using the PSF during the subject image restoration process is to capture a plurality of subjects at different distances from the imaging device in the PSF detection mode. A method is conceivable in which PSF is obtained from each of a plurality of intermediate images obtained by imaging. In this case, the imaging apparatus includes a distance meter that measures the distance to the subject. In this imaging mode, the distance to the subject is measured by the distance meter, and the subject image is restored using the PSF corresponding to the measured distance. I do. As a result, as in the second embodiment, the PSF error due to the difference in distance to the subject is reduced, so that the subject image can be restored with high accuracy regardless of the distance to the subject.

  In the second to sixth embodiments described above, the phase plate whose thickness changes is used as the phase modulation element, but the phase modulation element is not limited to such a phase plate. Other phase modulation elements include, for example, an optical element whose refractive index changes depending on the location, such as a gradient index wavefront modulation lens, or a thickness or the like by coding on the lens surface, such as a wavefront modulation hybrid lens. A liquid crystal element capable of modulating the phase distribution, such as an optical element whose refractive index changes and a liquid crystal spatial phase modulation element, can be used.

  In the second to sixth embodiments, the phase modulation element that disperses the incident light in two directions is used. However, the phase modulation element is not limited to this, and the incident light is regularly distributed. Various phase modulation elements can be used.

Regarding the above embodiments, the following supplementary notes are further disclosed.
(Appendix 1) An image sensor that receives light from a subject through a phase modulation element;
An image extraction unit that extracts a partial image including a region in which point images are dispersed from a captured image obtained by the imaging element;
A restoration processing unit that uses the partial image extracted by the image extraction unit as a point spread function and performs a process of restoring a subject image on a captured image obtained by the imaging element;
An imaging device comprising:

  (Additional remark 2) The said restoration process part performs the process which decompress | restores a to-be-photographed image with respect to the captured image same as the captured image used as the extraction object of the said partial image by the said image extraction part. Imaging device.

  (Additional remark 3) The said image extraction part performs pattern matching with the other point spread function calculated from the design value of the optical system containing the said phase modulation element with respect to the captured image obtained by the said imaging element The imaging apparatus according to appendix 1 or 2, wherein the partial image is extracted.

  (Additional remark 4) The said restoration process part is a point image of the said partial image extracted by the said image extraction part with respect to the captured image newly acquired by the said image pick-up element after the extraction of the said partial image by the said image extraction part. The imaging apparatus according to appendix 1, wherein a processing for restoring the subject image is performed as a distribution function.

(Additional remark 5) It further has the light emission part which produces | generates the said point image on the said subject by irradiating light with respect to the said subject,
The image extraction unit extracts the partial image from a captured image obtained by the imaging element imaging the subject irradiated with light from the light emitting unit,
The restoration processing unit performs a process of restoring the subject image on a captured image obtained by the imaging element capturing an image of the subject after irradiation of light from the light emitting unit is stopped.
The imaging apparatus according to appendix 4, wherein the imaging apparatus is characterized in that

  (Additional remark 6) The said restoration process part is an image obtained by extracting the component of the specific color area | region containing the color of the said point image from the said partial image extracted by the said image extraction part. The imaging apparatus according to any one of supplementary notes 1, 4, and 5, wherein the imaging apparatus is used as:

  (Supplementary note 7) Any one of Supplementary notes 1, 4, 5, and 6, wherein the image extraction unit extracts a predetermined region from the captured image obtained by the imaging element as the partial image. The imaging device described in 1.

(Additional remark 8) The said predetermined area | region is an area | region centering on the center part of the captured image obtained by the said image pick-up element, The imaging device of Additional remark 7 characterized by the above-mentioned.
(Additional remark 9) The said image extraction part produces | generates the image which extracted the component of the specific color area | region containing the color of the said point image from the captured image obtained by the said image pick-up element, The component of the said specific color area | region is produced | generated. Supplementary note 1 wherein the partial image is extracted by performing pattern matching on the extracted image with another point spread function calculated from a design value of an optical system including the phase modulation element. , 4, 5.

(Additional remark 10) The said image extraction part extracts the some partial image containing the area | region where each individual point image was disperse | distributed from the captured image obtained by the said image pick-up element,
The restoration processing unit selects, for each divided region on the captured image obtained by the image sensor, a partial image corresponding to each divided region from the plurality of partial images, and uses the selected partial image as a point spread function. Apply processing to restore the subject image,
The imaging device according to any one of appendices 1 to 7, characterized in that:

(Additional remark 11) The image extraction part which extracts the partial image containing the area | region where the point image was disperse | distributed from the picked-up image obtained by the image pick-up element which receives the light from a to-be-photographed object through a phase modulation element,
A restoration processing unit that uses the partial image extracted by the image extraction unit as a point spread function and performs a process of restoring a subject image on a captured image obtained by the imaging element;
An image processing apparatus comprising:

(Additional remark 12) It is the imaging method in the imaging device provided with the image sensor which receives light from a subject through a phase modulation element,
Extracting a partial image including a region in which point images are dispersed from a captured image obtained by the imaging element,
Using the extracted partial image as a point spread function and restoring a subject image with respect to a captured image obtained by the imaging element;
An imaging method characterized by the above.

  (Additional remark 13) The said imaging device performs the process which decompress | restores the said to-be-photographed image with respect to the captured image same as the captured image used as the extraction object of the said partial image, The imaging method of Additional remark 12 characterized by the above-mentioned.

  (Additional remark 14) The said imaging device performs the process which decompress | restores the said to-be-photographed image using the said partial image as a point spread function with respect to the captured image obtained by the said image pick-up element after extraction of the said partial image. The imaging method according to appendix 12, which is a feature.

(Supplementary Note 15) The imaging device includes:
A point image is generated on the subject by irradiating the subject with light,
Extracting the partial image from a captured image obtained by the imaging device capturing the subject irradiated with light,
Stop irradiating the subject with light,
A process of restoring the subject image is performed on a captured image obtained by the imaging element capturing the subject after the light irradiation is stopped.
The imaging method according to supplementary note 14, characterized in that:

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging element 12 Phase modulation element 13 Image extraction part 14 Restoration process part P1 Captured image P2 Partial image P3 image

Claims (10)

An image sensor that receives light from a subject through a phase modulation element;
An image extraction unit that extracts a partial image similar to a first point spread function calculated from a design value of an optical system including the phase modulation element, from a captured image obtained by the imaging element ;
Using a second point spread function obtained from the partial image extracted by the image extraction unit, a restoration processing unit that performs a process of restoring a subject image on the captured image obtained by the imaging element;
An imaging device comprising: 2. The imaging according to claim 1, wherein the restoration processing unit performs a process of restoring the subject image on the same captured image as the captured image that is the extraction target of the partial image by the image extraction unit. apparatus. The image extraction unit extracts the partial image by performing pattern matching with the first point spread function on a captured image obtained by the imaging element. Or the imaging device of 2. The restoration processing unit is configured to extract the second image obtained from the partial image extracted by the image extraction unit with respect to a captured image newly obtained by the imaging element after the partial image is extracted by the image extraction unit. The imaging apparatus according to claim 1, wherein a processing for restoring the subject image is performed using a point spread function. A light emitting unit for generating a point image on the subject by irradiating the subject with light;
The image extraction unit extracts the partial image from a captured image obtained by the imaging element imaging the subject irradiated with light from the light emitting unit,
The restoration processing unit performs a process of restoring the subject image on a captured image obtained by the imaging element capturing an image of the subject after irradiation of light from the light emitting unit is stopped.
The imaging apparatus according to claim 4. The restoration processing unit extracts a component of a specific color region from the partial image extracted by the image extraction unit, and uses the second point spread function obtained from the extracted image to extract the subject image. The imaging apparatus according to claim 1, wherein a restoration process is performed. The image extraction unit generates an image obtained by extracting a component of a specific color region from a captured image obtained by the image sensor, and the first image is extracted from the image obtained by extracting the component of the specific color region . The image pickup apparatus according to claim 1, wherein the partial image is extracted by performing pattern matching with a point spread function. The image extraction unit extracts a plurality of partial images respectively similar to the first point spread function from a plurality of locations of the captured image obtained by the imaging element;
The restoration processing unit selects, for each divided region on the captured image obtained by the image sensor, a partial image corresponding to each divided region from the plurality of partial images, and each of the second point spread function To perform a process of restoring the subject image,
The imaging apparatus according to claim 1, wherein A partial image similar to the first point spread function calculated from the design value of the optical system including the phase modulation element is extracted from the captured image obtained by the imaging element that receives light from the subject through the phase modulation element. An image extraction unit to
Using a second point spread function obtained from the partial image extracted by the image extraction unit, a restoration processing unit that performs a process of restoring a subject image on the captured image obtained by the imaging element;
An image processing apparatus comprising: An imaging method in an imaging device including an imaging element that receives light from a subject through a phase modulation element,
Extracting a partial image similar to the first point spread function calculated from the design value of the optical system including the phase modulation element from the captured image obtained by the imaging element ;
Using a second point spread function obtained from the extracted partial image, to restore the subject image with respect to the captured image obtained by the imaging element;
An imaging method characterized by the above.

Images