JP6124849B2 - Image processing method, imaging apparatus using the same, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image sharpening process.

  An unsharp mask process for sharpening an image by adding or subtracting the difference between the original image and the image blurred by applying an unsharp mask to the original image is known. The larger the difference between the blurred image and the input image, the sharper the image. Further, Patent Document 1 reduces the influence of a point spread function (PSF) of an optical system by applying an asymmetric one-dimensional filter to pixel signal sequences arranged in the image height direction. Proposed method.

JP 2010-81263 A

  However, the conventional unsharp mask processing uses a rotationally symmetric filter for the unsharp mask, and it is not possible to sharpen a deteriorated image due to the influence of a PSF having a complicated shape such as asymmetric aberration or sagittal halo. Have difficulty. That is, if an attempt is made to correct an aberration in the azimuth direction in which a large amount of aberration occurs, an undershoot occurs in the azimuth direction in which the aberration is small. Conversely, if the undershoot is suppressed, the aberration cannot be corrected sufficiently.

  In addition, the method of Patent Document 1 only considers asymmetry in the image height direction, and the correction filter is also one-dimensional. Therefore, asymmetry in directions other than the image height direction cannot be improved. . The image height direction is the azimuth direction of meridional. Furthermore, with regard to the filter, the asymmetry of the filter is adjusted by the number of minus tap coefficients, and the correction in the image height direction is also different from the method of blurring the PSF of the optical system. I can't.

  An object of the present invention is to provide an image processing device, an imaging device, an image processing method, and an image processing program that are excellent in a sharpening effect.

The image processing apparatus of the present invention acquires a captured image generated by imaging through an optical system, and is generated based on information on a point spread function of the optical system corresponding to the imaging condition of the optical system. And a processing unit that performs an unsharp mask process on the captured image using a filter, and the filter used for the unsharp mask process on the captured image is a filter in which two-dimensional tap data is distributed rotationally asymmetrically. It is characterized by including .

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and an image processing program that are excellent in a sharpening effect.

It is a block diagram of the imaging device of the present invention. (Examples 1, 2, and 3) It is a flowchart which shows the image processing method of this invention. (Examples 1, 2, and 3) It is a schematic diagram of sharpening by an unsharp mask process. It is a schematic diagram of PSF of the photographing optical system in the xy plane. It is a schematic diagram of the sharpening process by a rotationally symmetric unsharp mask. It is a schematic diagram of the sharpening process by a non-rotationally symmetric unsharp mask. It is the schematic diagram and schematic sectional drawing of an unsharp mask. It is a flowchart which shows the image processing method of this invention. Example 1 It is a schematic diagram of a Bayer arrangement. It is a figure for demonstrating the division | segmentation method of an input image. It is a figure for demonstrating the interpolation method of the image height direction of an input image. It is a flowchart which shows the image processing method of this invention. (Example 2) It is a flowchart which shows the image processing method of this invention. (Example 3)

  FIG. 3 is a schematic diagram of sharpening by unsharp mask processing (image sharpening processing) of the present embodiment. The solid line in FIG. 3A is an input image, and the broken line is an image obtained by blurring the input image with an unsharp mask. The dotted line represents the sharpened image, and the solid line in FIG. 3B represents the correction component. In FIG. 3, the horizontal axis represents coordinates, and the vertical axis represents pixel values or luminance values. FIG. 3 corresponds to a cross section in a predetermined direction (for example, the X direction) of FIG. 4 described later.

  When the original image is f (x, y) and the correction component is h (x, y), the sharpened image g (x, y) can be expressed by the following equation.

g (x, y) = f (x, y) + m × h (x, y) (1)
In Equation (1), the correction signal h (x, y) is multiplied by a constant and added to the input image f (x, y). In Equation (1), m is a constant, and the correction amount can be adjusted by changing the value of m. Note that m may be a constant regardless of the position of the input image, or may be adjusted according to the position of the input image by using an adjustment coefficient m (x, y) that varies depending on the position of the input image. The correction amount can also be adjusted. In addition, the constant m and the adjustment coefficient m (x, y) can be varied according to the photographing conditions such as the focal length, aperture value, and subject distance of the optical system. The adjustment factor m (x, y) can be used instead of the constant m in the following description.

  The correction component h (x, y) can be expressed as the following equation when the unsharp mask is USM. USM (x, y) is, for example, a tap value at a certain coordinate (x, y) of USM.

h (x, y) = f (x, y) -f (x, y) * USM (x, y) (2)
The right side of Formula (2) can be transformed and expressed by the following formula.

h (x, y) = f (x, y) * (δ (x, y) −USM (x, y)) (3)
Here, * is convolution (convolution integration, sum of products), and δ is a delta function (ideal point image). The “delta function” is data in which the number of taps is equal to USM (x, y) and the center value is 1 and the rest are filled with 0.

  Since Formula (3) can be expressed by modifying Formula (2), Formula (2) and Formula (3) are equivalent. Therefore, the generation of the correction component will be described below using Equation (2).

  In Equation (2), a difference between an image obtained by blurring the captured image f (x, y) and the captured image f (x, y) with an unsharp mask USM is calculated, and a correction component h (x, y) is based on the difference information. ) Is generated. In general unsharp mask processing, a smoothing filter such as a Gaussian filter, a median filter, or a moving average filter is used for the unsharp mask USM.

  For example, when a Gaussian filter is used as the unsharp mask USM for the captured image f (x, y) shown by the solid line in FIG. 3A, an image obtained by blurring the captured image f (x, y) is shown in FIG. As indicated by the broken line in FIG. Since the correction component h (x, y) is the difference between the captured image f (x, y) and the blurred image as shown in the equation (2), the correction component h (x, y) is changed from the solid line in FIG. By subtracting the broken line, the component represented by the solid line in FIG. By using the correction component calculated in this way and performing the calculation of Expression (1), the captured image f (x, y) shown by the solid line in FIG. 3A is represented by the dotted line in FIG. Can be sharpened.

  Next, a description will be given of a case where an image is sharpened by applying unsharp mask processing to an image deteriorated by a photographing optical system that forms an optical image of a subject. A photographed image f (x, y) obtained through the photographing optical system is an image (image of a subject) before photographing I (x, y), and a PSF which is a function representing a response to the point light source of the photographing optical system. Assuming psf (x, y), it can be expressed as the following equation.

f (x, y) = I (x, y) * psf (x, y) (4)
Here, if the photographing optical system is a rotationally symmetric coaxial optical system, the PSF corresponding to the center of the image is rotationally symmetric. Therefore, it is possible to sharpen the captured image f (x, y) closer to the original image I (x, y) by applying a rotationally symmetric USM to the center of the image. Since the correction amount is a difference value between the photographed image and the photographed image blurred with the unsharp mask, the unsharp mask USM does not use a simple smoothing filter to correct with high accuracy, but more psf (x, It is better to use a mask having a shape close to y). For example, when a captured image deteriorates due to the influence of spherical aberration, the spherical symmetry affects rotational symmetry, but a smoothing filter such as a Gaussian filter has a distribution shape different from that of PSF due to the influence of spherical aberration. For this reason, even when the effect of blurring in rotational symmetry is reduced, correction using the PSF of the photographing optical system can be performed with higher accuracy.

  In this embodiment, PSF is used for USM. Although the captured image f (x, y) illustrated in FIG. 3A has a symmetric shape for simplification, the shape of the image may not be symmetric. Even if the shape of the original image I (x, y) is asymmetric, if the deterioration function applied to the original image I (x, y) corresponding to psf (x, y) is rotationally symmetric, the rotationally symmetric USM Can be used to sharpen.

  On the other hand, at positions other than the center of the image, the PSF usually has an asymmetric shape even if the photographing optical system is a rotationally symmetric coaxial optical system. 4A and 4B are schematic views of the PSF of the photographing optical system on the xy plane, where FIG. 4A shows an on-axis PSF and FIG. 4B shows an off-axis PSF.

  For example, if the original image (subject) is an ideal point image, the photographed image f (x, y) is the PSF of the photographing optical system from Equation (4). If there is an ideal point image at the angle of view corresponding to FIG. 4B and the original image (subject) has deteriorated due to the influence of the PSF of the photographing optical system, the image obtained as the input image is as shown in FIG. The image is blurred like the shape of B). A case where sharpening by unsharp mask processing is performed on such an asymmetrically blurred image will be described.

  5 and 6 are schematic diagrams of unsharp processing for an asymmetrically deteriorated image. FIG. 5 shows a case where a rotationally symmetric unsharp mask is used, and FIG. 6 shows that processing is performed using a rotationally asymmetric unsharp mask. Shows the case. The vertical axis and the horizontal axis are the same as those in FIG.

  The solid lines in FIGS. 5A and 6A represent the cross section in the y-axis direction of FIG. 4B, and the dotted line represents a captured image blurred with an unsharp mask. A Gaussian filter is applied to the rotationally symmetric unsharp mask in FIG. 5, and the PSF of the imaging device is applied to the non-rotationally symmetric unsharp mask in FIG.

  FIG. 5B and FIG. 6B are plots of the difference values between the captured image blurred with each unsharp mask and the original captured image, and represent correction components. For convenience, in FIGS. 5 (A) and 6 (A), the captured image is more blurred and widened by PSF, and the positive side of the Y-axis is assumed.

  In FIG. 5A, the difference value between the plus-side blurred image and the original image with respect to the peak position of the solid line is small, and the difference value between the minus-side blurred image and the original image is large. Therefore, the extreme value of the correction component in FIG. 5B is also smaller on the left side (minus side) than on the right side (plus side) with respect to the center peak position.

  As can be seen by comparing the curves in FIG. 5A and FIG. 5B, the correction amount of the correction component is small on the plus side of the photographed image and the correction amount is large on the minus side with a narrow base, so Asymmetric blur cannot be corrected even if sharpening is performed. For example, let us consider a case where the correction amount is adjusted by changing the constant m in Expression (4) without changing the unsharp mask. If the value of the constant m is increased to sufficiently correct the positive side of the image, the negative side of the image becomes overcorrected (undershoot), and the value of the constant m is adjusted so that the correction amount on the negative side of the image is appropriate. If is set, the plus side of the image is undercorrected.

  As described above, even if unsharp mask processing is performed on an asymmetrically blurred image using a rotationally symmetric unsharp mask, it is difficult to improve the asymmetry and sharpen the image. Such a problem similarly occurs even when a rotationally symmetric filter other than a Gaussian filter is used as a rotationally symmetric unsharp mask.

  On the other hand, in FIG. 6A, the difference value between the image blurred on the plus side and the original image with respect to the peak position of the solid line is large, and the difference value between the image blurred on the minus side and the original image is large. Is the reverse of FIG. Therefore, the extreme value of the correction component in FIG. 6B is also smaller on the right side (plus side) than on the left side (minus side) with respect to the center peak position.

  If such a correction component is applied to the photographed image represented by the solid line in FIG. 6A, the correction amount is larger when the plus side blur is larger than the peak position, and the minus side blur is present. The smaller the value, the smaller the correction amount.

  In the case of such an asymmetric unsharp mask, the tendency of the balance of the blur of the input image and the balance of the correction amount of the correction components coincide, so there is an excess or deficiency of correction that becomes a problem when applying a rotationally symmetric unsharp mask. It becomes difficult to get up. Furthermore, since it becomes difficult to overcorrect as compared with the rotationally symmetric unsharp mask, the value of the constant m in Equation (4) can also be made relatively large, and sharpening while reducing asymmetry. be able to. Further, in order to perform correction with higher accuracy, the balance of the correction amount of the correction component is the difference between the blurred image and the original image. Therefore, the portion more greatly blurred by the PSF of the photographing optical system is caused by the unsharp mask. It needs to be more blurred than the other parts. Thus, in order to correct with higher accuracy, it is ideal to use the PSF of the photographing optical system as an unsharp mask.

  Subsequently, filters and correction signals used in each embodiment described later, and unsharp mask processing in each embodiment will be described.

  In the first embodiment, sharpening is performed using the following expressions derived from Expressions (1) and (2).

g (x, y) = f (x, y) + m × {f (x, y) −f (x, y) * USM (x, y)} (5)
In the second embodiment, sharpening is performed using the following equations derived from Equations (1) and (3).

g (x, y) = f (x, y) + m × f (x, y) * {δ (x, y) −USM (x, y)} (6)
In the third embodiment, sharpening is performed using the following expression obtained by further modifying Expression (6).

g (x, y) = f (x, y) * {δ (x, y) + m × (δ (x, y) −USM (x, y))} (7)
Further, the formula (7) can be modified as the following formula.

g (x, y) = f (x, y) * {(1 + m) × δ (x, y) −m × USM (x, y)} (8)
In the third embodiment, sharpening is performed using Expression (7). However, sharpening can be similarly performed using Expression (8).

  The PSF differs depending on the photographing conditions including the image height of the image formed through the optical system, the focal length of the optical system, the F value, and the subject distance. In the following embodiments, the image height is described as an example of the photographing condition. However, different aberration information is acquired for the focal length, F value, and subject distance of the optical system, and an unsharp mask is generated based on the acquired aberration information. May be.

  FIG. 1 is a block diagram of an imaging apparatus 100 according to the first embodiment. In the imaging apparatus 100, a program for performing a sharpening process (image processing method) of an input image is installed in the storage unit 120, and the sharpening process is executed by the image processing unit 104 (image processing apparatus) of the imaging apparatus 100. The The storage unit 120 includes a ROM, a hard disk drive, and the like, but may also serve as the recording unit 108 described later.

  The imaging apparatus 100 includes a photographic optical system 101 (lens) and an imaging apparatus main body (camera main body). The photographing optical system 101 includes a diaphragm 101a and a focus lens 101b, and is configured integrally with the imaging apparatus main body. However, the present embodiment is not limited to this, and can also be applied to an image pickup apparatus in which the photographing optical system 101 is mounted to the image pickup apparatus body in a replaceable manner.

  The image sensor 102 is a two-dimensional image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The image sensor 102 photoelectrically converts a subject image (imaging light) obtained via the photographing optical system 101 to generate a photographed image. The subject image is photoelectrically converted by the image sensor 102 and converted into an analog signal (electrical signal). The analog signal is converted into a digital signal by the A / D converter 103, and the digital signal is input to the image processing unit 104. Is done.

  The image processing unit 104 is an image processing unit that performs predetermined processing on the digital signal and performs predetermined unsharp mask processing. In this embodiment, the image processing unit of the imaging apparatus performs the sharpening process. However, a personal computer (PC) or a dedicated device may perform the sharpening process as the image processing apparatus.

  The image processing unit 104 acquires imaging condition information of the imaging device 100 from the state detection unit 107. The imaging condition information is information related to an aperture, a shooting distance, a focal length of a zoom lens, and the like. The state detection unit 107 can acquire imaging condition information directly from the system controller 106, but is not limited to this. For example, the imaging condition information regarding the imaging optical system 101 can be acquired from the imaging optical system control unit 105.

  Subsequently, the image processing unit 104 performs an image sharpening process on the input image. The image processing unit 104 includes a point image intensity distribution selection unit (PSF selection unit) 201, a correction signal generation unit 202, and a correction signal application unit 203. However, when the image processing unit 104 is configured as an image processing apparatus, the system controller 106 of the imaging apparatus 100 may store aberration information in association with the captured image. In that case, the image processing apparatus may include the correction signal generation unit 202 and the correction signal application unit 203 and may not include the PSF selection unit 201.

The output image processed by the image processing unit 104 is stored in the recording unit 108 in a predetermined format. The recording unit 108 also functions as a storage unit that stores the relationship between the imaging conditions of the imaging optical system 101 and the PSF of the imaging optical system.
The image display unit 112 can display an image obtained by performing a predetermined display process after the image sharpening process. The image display unit 112 may display an image obtained by performing simple processing for high-speed display.

  The series of processes described above is controlled by the system controller 106. The system controller 106 is configured as a microcomputer and a CPU (processor). The photographing optical system 101 is mechanically driven by the photographing optical system control unit 105 based on an instruction from the system controller 106.

  An optical element such as a low-pass filter or an infrared cut filter may be inserted into the photographing optical system 101. When using an optical element that affects the characteristics of the PSF, such as a low-pass filter, a higher-accuracy image sharpening process can be performed if the influence of this element is taken into consideration when the unsharp mask is created. The infrared cut filter also affects each PSF of the RGB channel (RGB color component) that is an integral value of the PSF of the spectral wavelength, particularly the PSF of the R channel. It is more preferable to consider.

  Next, an image processing method in the present embodiment will be described with reference to FIG. FIG. 2 is a flowchart showing the overall flow of this embodiment, and “S” represents “step (process)”. The flowchart shown in FIG. 2 can be embodied as a program (image processing program) for causing a computer to execute the function of each step. The same applies to other flowcharts. Each step in FIG. 2 is performed by the image processing unit 104 based on an instruction from the system controller 106.

  First, a captured image is acquired as an input image (S11), and then the PSF selection unit 201 acquires the PSF of the imaging optical system corresponding to the imaging condition of the input image from the recording unit 108 (acquisition step S12). The PSF information acquired by the PSF selection unit 201 may be two-dimensional tap data, a plurality of one-dimensional tap data that is a constituent element of the PSF, or a coefficient.

  Next, the correction signal generation unit 202 generates an unsharp mask and a correction signal using the PSF information acquired in S12 (generation step S13).

  Here, the unsharp mask will be described with reference to FIG. The number of taps of the unsharp mask is determined according to the aberration characteristics of the photographing optical system and the required sharpening accuracy. The unsharp mask of FIG. 7A is a two-dimensional mask of 11 × 11 taps as an example. In FIG. 7A, values (coefficients) in each tap are omitted. FIG. 7B shows a cross section of this unsharp mask. In FIG. 7B, the horizontal axis represents the tap, and the vertical axis represents the tap value.

  The distribution of the values (coefficient values) of each tap of the unsharp mask is ideally a distribution of signal values (PSF of the imaging optical system) spread due to aberration. As described above, the unsharp mask may be generated using the PSF information, or the PSF acquired by the PSF selection unit 201 may be used as it is as the unsharp mask. When the correction signal is generated using the unsharp mask in the correction signal generation unit 202, the correction signal generation processing will be described later.

  Next, the correction signal application unit 203 performs a sharpening process on the input image using the correction signal generated in S13 (sharpening step S14). Details of the processing in S14 will also be described later.

  FIG. 8A is a flowchart showing the detailed processing flow of this embodiment. Each step in FIG. 8A is performed by the image processing unit 104 based on an instruction from the system controller 106.

  First, a captured image is acquired as an input image (S111). Here, the color component data as the correction target used as the input image is, for example, image data of the G channel after demosaicing. However, R channel and B channel image data, RGB channel image data, or image data before demosaicing may be used.

  FIG. 9 is a schematic diagram of a Bayer array that is a discrete regular array. For example, the RGB channel data may be extracted as they are and processed as input images for each color, or only a specific channel may be used as the input image. Alternatively, as shown in FIG. 9, the G channel is divided into two, G1 and G2, and may be handled as four channels. By dividing the G channel into two in this way, the image data extracted from each of R, G1, G2, and B have the same resolution, so that processing and data processing are facilitated.

  Next, the PSF selection unit 201 acquires the PSF information of the photographing optical system corresponding to the input image from the recording unit 108 (S112). The PSF information acquired by the PSF selection unit 201 may be two-dimensional tap data, a plurality of one-dimensional tap data that is a constituent element of the PSF, or a coefficient. As a method of decomposing two-dimensional data into a plurality of one-dimensional data, for example, there is a singular value decomposition theorem. The principal component decomposed using such a theorem may be recorded in the recording unit 108, and the plurality of one-dimensional tap data corresponding to the principal component of the PSF may be acquired according to the photographing conditions.

  Since the PSF changes depending on the image height, it is preferable to change the unsharp mask in accordance with the image height in order to increase the correction accuracy, but the data capacity to be recorded in the recording unit 108 increases the cost. Therefore, in this embodiment, in order to change the unsharp mask for each image height, the input image is divided into a plurality of regions, and interpolation processing is performed using PSF information at at least two image heights for each region. Thus, an unsharp mask corresponding to the middle is generated. Details of the interpolation method will be described later in S114.

  Next, division of an input image area will be described. FIG. 10 is a schematic diagram of an input image. For convenience, the long side direction of the input image is the x axis, the short side direction is the y axis, and the center of the image is the origin of coordinates. In this embodiment, as shown in FIG. 10, as an example, the input image is divided into eight regions from region A to region G, and information on the point image intensity distribution function at the periphery and the origin of each region is acquired. .

  Next, the correction signal generation unit 202 performs a filtering process using the PSF acquired in S112 (S113). In this embodiment, PSF is used as an unsharp mask, and an unsharp mask is convolved (convolution integration, product sum) with an input image. Since there are a total of nine PSFs acquired at S112, one at the center of the periphery, nine input images blurred by the corresponding unsharp mask, that is, nine image data obtained by filtering the unsharp mask can be obtained.

  Next, the correction signal generation unit 202 performs interpolation processing in the image height direction using a plurality of image data filtered in the unsharp mask generated in S113, and generates image data filtered with one unsharp mask ( Interpolation step S114).

  The image height direction interpolation processing will be described with reference to FIG. In FIG. 11, assuming that the direction in which the region C is located with respect to the origin is the positive direction of the x axis and the direction in which the region A is present is the positive direction of the y axis, FIG. The area of the first quadrant of the image is shown. Here, P0 is the origin, P1, P2, and P3 are the peripheral image heights of region A, region B, and region C, respectively. In S112, the PSF selection unit 201 acquires the PSFs of the image heights P0, P1, P2, and P3. And

  In FIG. 11, a point Pn represented by a white circle represents an arbitrary point (image height) in the image, and the point Pn in the region B is created in step S113 using the PSF information of the points P0 and P2. The data obtained by filtering the unsharp mask to the input image is used. Similarly, the point Pn in the region A and the region C uses the data obtained by filtering the unsharp mask corresponding to the image heights of the points P0 and P1, and the points P0 and P3 into the input image in S113.

  Next, generation of interpolation data corresponding to data subjected to filtering processing at an arbitrary point Pn from two image heights in the region will be described. As shown in FIG. 11, it is assumed that there is a point Pn in the region B, the distance from the origin P0 is d0, and the distance from the point P2 is d2. In S113, if the input images filtered using the PSFs corresponding to the points P0 and P2 are F0 and F2, respectively, the interpolation data Fn corresponding to an arbitrary point Pn is expressed by the following equation.

Fn = F0 * (1-d0) + F2 * d2 (9)
By performing such interpolation processing, interpolation data at an arbitrary image height in each region can be generated, and one image data is generated from the plurality of image data generated in S113. Since the image data generated in this way can be reduced in data amount compared to the case where the input image is filtered using different PSFs depending on the image height, the processing speed can be improved.

  Although Equation (9) is a calculation formula for the first quadrant region B, the data after the interpolation processing can be created by performing similar calculations for other regions and other quadrants. Further, the calculation formula used for the interpolation of the image height is not limited to the formula (9), and the calculation formula may be weighted by using a quadratic curve or multiplying each filtered input image by a certain constant. Good.

  Next, the correction signal generation unit 202 generates a correction signal using the interpolation data generated in S114 (S115). In the present embodiment, the correction component is represented by Expression (2), and is generated by taking the difference between the input image and the interpolation data generated in S114.

  Next, the correction signal applying unit 203 sharpens the image by applying the correction signal generated in S115 to the input image (S116). In this embodiment, the correction signal application processing corresponds to Equation (1), and the value of the constant m is determined in consideration of image noise, sharpening overcorrection and undercorrection. Using the constant m determined in this way, the correction signal generated in S115, and the input image, sharpening processing is performed.

  Formula (1) is expressed by adding the first term and the second term. This is a case where the constant m is positive, and subtraction is performed when the constant m is negative. As described above, in the sharpening process of the present embodiment, when the correction signal is applied to the input image, the application process is addition or subtraction, which is essentially the same because of the difference in the sign of the constant m. Therefore, the calculation may be performed as long as it is changed according to the sign of the constant m. In this embodiment, since the PSF of the photographic optical system is used as an unsharp mask, the input image is accurate even if the image is deteriorated by the asymmetric PSF of the photographic optical system as seen in the peripheral portion of the input image. Correct and sharpen well.

  In this embodiment, the interpolation in the image height direction is performed after filtering, but may be performed on the correction signal (FIG. 8B) or may be performed on the sharpened image (FIG. 8). (C)).

  In FIG. 8B, point image intensity distribution functions at image heights of at least two points are acquired for each of a plurality of regions set in the input image, and a plurality of correction signals are obtained based on the plurality of point image intensity distribution functions. Is generated. Then, an interpolation process is performed on the plurality of correction signals in the image height direction to generate one correction signal, and the input image is sharpened by applying the one correction signal to the input image. In FIG. 8C, a point image intensity distribution function at an image height of at least two points is acquired for each of a plurality of regions set in the input image, and a plurality of correction signals are obtained based on the plurality of point image intensity distribution functions. Is generated. Then, a plurality of sharpened images are obtained by applying a plurality of correction signals to the input image, respectively, and a plurality of sharpened images are interpolated in the image height direction to obtain a single sharpened image. Generate an image.

  FIG. 8B is a flowchart showing a detailed processing flow when interpolation processing in the image height direction is performed after the correction signal is generated. S121, S122, and S123 correspond to S111, S112, and S113, respectively. Since the processes of S114 and S115 are different from those in FIG. 8A, S124 and S125 corresponding to those processes in FIG. 8B will be described.

  In S124, the correction signal generation unit 202 generates a correction signal from the data obtained by filtering the unsharp mask on the input image in S123. The correction signal is generated by taking the difference between the input image and the filtered data based on Equation (2) using the input image and the filtered data. The correction signal generated by such processing is generated for the data generated in S123.

  Next, in S125, the correction signal generation unit 202 performs an image height direction interpolation process using the plurality of correction signals generated in S124. Although the interpolation process executed in S125 changes the object to be interpolated, the basic process flow is the same as the interpolation process in S114. In S114, the data convolved with the input image using the PSF acquired in S112 as an unsharp mask is interpolated in the image height direction.

  On the other hand, in S125, the difference between the input image and the data obtained by convolving the unsharp mask with the input image is used as a correction signal, and the correction signal is subjected to interpolation processing in the image height direction. In S114, f (x, y) * USM in Equation (2) is interpolated in the image height direction, and h (x, y) is interpolated in the image height direction in S125.

  Therefore, by replacing the data of f (x, y) * USM in S114 with h (x, y), the processing of S125, that is, the interpolation processing of the correction signal in the image height direction can be realized. In step S126, the correction signal application unit 203 applies the correction signal to the input image using the correction signal data after the correction signal is interpolated in this manner. Since the correction signal application processing in S126 is the same as that in S116, detailed description thereof is omitted.

  Next, with reference to FIG. 8C, the flow of processing when performing interpolation in the image height direction on the input image after applying the correction signal will be described. S131, S132, S133, and S134 correspond to S121, S122, S123, and S124, respectively.

  Since processing of S125 and S126 is different from that in FIG. 8B, S135 and S136 corresponding to these processing will be described. In S135, the correction signal application unit 203 performs a sharpening process on the input image using the correction signal generated in S134. The correction signals generated in S134 are generated for the plurality of unsharp masks created using the PSF selected in S132. Therefore, in S135, the sharpening process is performed by applying each of the plurality of correction signals generated in S134 to the input image based on Expression (1).

  Next, in S136, the plurality of sharpened images generated in S135 are subjected to interpolation processing in the image height direction. Although the interpolation process executed in S136 changes the object to be interpolated, the basic process flow is the same as the interpolation process in S114 or S125. Comparing S125 and S136, S125 interpolates h (x, y) in the formula (1) in the image height direction, and S136 interpolates g (x, y) in the image height direction. Therefore, by replacing the data of h (x, y) in S125 with g (x, y), it is possible to realize the processing of S136, that is, the interpolation processing of the sharpened image in the image height direction.

  The imaging apparatus according to the second embodiment has the same configuration as the imaging apparatus according to the first embodiment. The second embodiment is different from the first embodiment in that the image processing method shown in FIG. 12 is used instead of the image processing method shown in FIG. FIG. 12 is a flowchart illustrating the image processing method according to the second embodiment. Each step is performed by the image processing unit 104 based on an instruction from the system controller 106.

  The present embodiment is different from the first embodiment in the generation method of the correction signal. S211 and S212 are the same as S111 and S112. The first embodiment generates a correction signal based on the formula (2), whereas the present embodiment generates a correction signal based on the formula (3). When a correction signal is generated by the correction signal generation unit 202 of the present embodiment, first, a filter is generated by taking the difference between the ideal point image and the PSF selected by the PSF selection unit 201 (S213). Next, the correction signal generation unit 202 generates a correction signal by convolving the input image with the generated filter (S214). Since S215, S216, and S217 are the same as S114, S115, and S116, description thereof is omitted. Thereby, the sharpening process based on Formula (6) can be performed.

  The interpolation in the image height direction may be performed at the correction signal stage as shown in the first embodiment, or the image height direction interpolation is performed at the stage where the correction signal is applied to the input image and sharpened. May be executed.

  The imaging apparatus according to the third embodiment has a configuration similar to that of the imaging apparatus according to the first embodiment. The third embodiment is different from the first embodiment in that the image processing method shown in FIG. 13 is used instead of the image processing method shown in FIG. FIG. 13 is a flowchart illustrating the image processing method according to the third embodiment. Each step is performed by the image processing unit 104 based on an instruction from the system controller 106.

  The present embodiment differs from the first and second embodiments in the method of generating a correction signal and the method of applying the correction signal. In the present embodiment, a filter is generated based on Expression (7) and generated for an input image. Apply. Therefore, in the flowchart of FIG. 13, the processing contents differ from those in the first and second embodiments are the generation of the filter in S313 and the application of the filter in S314. S311 and S312 are the same as S111 and S112.

  In the present embodiment, the PSF selected in S312 is used as an unsharp mask, and a filter corresponding to the curly bracket portion of Expression (7) is generated (S312). In the present embodiment, the correction signal generated by the correction signal generation unit 202 is a filter. Next, the correction signal applying unit 203 convolves the input image with the filter generated in S313 and sharpens it (S314). In the sharpening process of the present embodiment, a filter (correction signal) generated in advance using the PSF of the photographing optical system as an unsharp mask can be sharpened with a single convolution with respect to the photographed image.

  Although the embodiments of the present invention have been described above, the present invention can be variously modified and changed within the scope of the gist thereof.

  The image processing method of the present invention can be applied to an imaging apparatus or a dedicated image processing apparatus.

DESCRIPTION OF SYMBOLS 104 ... Image processing part 201 ... Point image intensity distribution function selection part (acquisition means), 202 ... Correction signal generation part (generation means), 203 ... Correction signal application part (sharpening means)

Claims (18)

A captured image generated by imaging through the optical system is acquired, and the captured image is obtained using a filter generated based on the information of the point spread function of the optical system corresponding to the imaging condition of the optical system. Having processing means for performing unsharp mask processing;
The filter used for unsharp mask processing on the captured image includes a filter in which two-dimensional tap data is distributed rotationally asymmetrically . The filter used for unsharp mask processing on the captured image includes a filter in which the data of the two-dimensional tap is distributed asymmetrically in an image height direction and a direction other than the image height direction. the image processing apparatus according to 1. The processing means includes
Generating means for generating a correction signal by taking a difference between an image obtained by applying the filter to the captured image and the captured image;
The correction signal generated by the generation unit is multiplied by a constant and added to the captured image, or the correction signal generated by the generation unit is adjusted using an adjustment coefficient according to the position of the captured image. Sharpening means for sharpening the captured image by adding to the captured image;
The image processing apparatus according to claim 1 or 2, characterized in that it has a. The processing means includes
Generating means for generating a correction signal by convolving and integrating the filter generated based on the difference information between the point image intensity distribution function and the ideal point image into the captured image;
The correction signal generated by the generation unit is multiplied by a constant and added to the captured image, or the correction signal generated by the generation unit is adjusted using an adjustment coefficient according to the position of the captured image. Sharpening means for sharpening the captured image by adding to the captured image;
The image processing apparatus according to claim 1 or 2, characterized in that it has a. The processing means includes
The difference information between the point image intensity distribution function and the ideal point image is multiplied by a constant and added to the ideal point image, or the difference information is adjusted using an adjustment coefficient corresponding to the position of the captured image. Generating means for generating a filter by adding to the ideal point image;
Sharpening means for sharpening the captured image by convolving and integrating the filter generated by the generating means with the captured image generated by imaging through the optical system;
The image processing apparatus according to claim 1 or 2, characterized in that it has a. The processing means includes
Based on the difference information between the product of the sum of 1 and the constant and the ideal point image, and the product of the point image intensity distribution function and the constant, or the sum of the adjustment coefficient according to the position of the captured image and the ideal of 1 Generating means for generating a filter based on difference information between a product of point images and a product of the point image intensity distribution function and the adjustment coefficient;
Sharpening means for sharpening the captured image by convolving and integrating the filter generated by the generating means with the captured image;
The image processing apparatus according to claim 1 or 2, characterized in that it has a. The photographing conditions, the image height, the focal length, F values, and an image processing apparatus according to any one of claims 1 to 6, characterized in that it comprises at least one of the object distance. The processing means, in any one of claims 1 to 7, characterized in that the unsharp masking for each color component or a specific color component among the plurality of color components constituting the captured image The image processing apparatus described. The photographed image is image data that is discretely regularly arranged for each color component,
The processing means, the image processing apparatus according to any one of claims 1 to 7, characterized in that the unsharp mask processing on the image obtained by interpolating the color components of the correction target. The said processing means has an acquisition means which acquires the information of the said point image intensity distribution function of the said optical system based on the imaging conditions of the said optical system, The any one of Claim 1 thru | or 9 characterized by the above-mentioned. Image processing apparatus. The processing means has acquisition means for acquiring information of the point image intensity distribution function of the optical system based on imaging conditions of the optical system, and the acquisition means is for a plurality of regions set in the captured image. Obtain point spread function at at least two image heights for each,
The processing means performs interpolation processing in the image height direction using a plurality of image data obtained from a point image intensity distribution function corresponding to a plurality of regions of the photographed image, and generates one image data. the image processing apparatus according to claim 3 or 4, characterized in that to generate the correction signal with the image data. The processing means has acquisition means for acquiring information of the point image intensity distribution function of the optical system based on imaging conditions of the optical system, and the acquisition means is for a plurality of regions set in the captured image. Obtain point spread function at at least two image heights for each,
The processing means generates a plurality of correction signals based on a plurality of point image intensity distribution functions, performs an interpolation process on the plurality of correction signals in the image height direction, generates one correction signal, and generates the one correction signal. the image processing apparatus according to signals, in claim 3 or 4, characterized in that applied to the captured image. The processing means has acquisition means for acquiring information of the point image intensity distribution function of the optical system based on imaging conditions of the optical system, and the acquisition means is for a plurality of regions set in the captured image. Obtain point spread function at at least two image heights for each,
The processing means generates a plurality of correction signals based on a plurality of point image intensity distribution functions, obtains a plurality of sharpened images by applying the plurality of correction signals to the captured image, respectively, the image processing apparatus according to claim 3 or 4, characterized in that to generate the image after a single sharpening performs interpolation processing an image after a plurality of sharpening the image height direction. An image sensor that photoelectrically converts an optical image of a subject formed by the optical system;
An image processing unit that processes a captured image obtained from the imaging element;
The image processing unit performs an unsharp mask process on the captured image using a filter generated based on information of a point spread function of the optical system corresponding to the imaging condition of the optical system,
The filter used for unsharp mask processing on the captured image includes a filter in which two-dimensional tap data is distributed rotationally asymmetrically . The imaging apparatus according to claim 14 , further comprising a storage unit that stores a relationship between an imaging condition of the optical system and a point spread function of the optical system. The imaging apparatus according to claim 14 or 15, characterized in that it has the optical system. Obtaining a captured image generated by imaging through an optical system;
Performing an unsharp mask process on the captured image using a filter generated based on information of a point spread function of the optical system corresponding to the imaging condition of the optical system;
Have
The filter used for unsharp mask processing on the captured image includes a filter in which two-dimensional tap data is distributed rotationally asymmetrically . On the computer,
Obtaining a captured image generated by imaging through an optical system;
Performing an unsharp mask process on the captured image using a filter generated based on information of a point spread function of the optical system corresponding to the imaging condition of the optical system;
An image processing program for executing processing including
The image processing program characterized in that the filter used for unsharp mask processing on the captured image includes a filter in which two-dimensional tap data is distributed rotationally asymmetrically .