RU2656009C1 - Image processing device, image survey device, image processing method and continuous automated data medium - Google Patents

Abstract

FIELD: image processing means.

SUBSTANCE: invention relates to image processing for sharpening. Image processor includes a correction signal generator configured to generate a correction signal by calculating a difference between an image and an image obtained by applying a non-sharp mask formed on the basis of the point scatter function (PSF) corresponding to the image pickup parameters of the optical system for capturing an image, and a correction signal applying unit configured to increase the sharpness of the image by multiplying the correction signal generated by the generator, to the constant and adding the multiplied correction signal to the image.

EFFECT: technical result is simplified image processing.

16 cl, 36 dwg

Description

FIELD OF THE INVENTION

The present invention relates to image processing for sharpening.

State of the art

In a known processing based on an unsharp mask, the difference between the original image and the blurred image obtained by applying the unsharp mask to the original image (input image) is added to or subtracted from the image to increase image sharpness. Higher image sharpness is achieved when the difference between the blurry image and the input image is large. Japanese Patent Publication No. 2010-81263 discloses a method of reducing the influence of the point spread function (PSF) of an optical system by applying an asymmetric one-dimensional filter to pixel signals rotated in the image height direction.

However, in standard processing based on an unsharp mask, an axisymmetric filter is used as a unsharp mask, and therefore there are difficulties in increasing the sharpness of the image, which worsened due to the complex form of PSF exposure, for example, asymmetric aberration and sagittal halo. In particular, the correction of aberration in the azimuthal direction, having a large aberration, leads to "under correction" in the azimuthal direction, having a small aberration, while the elimination of the said undercorrection entails insufficient correction of aberration.

In the method disclosed in Japanese Patent Publication No. 2010-81263, asymmetry is taken into account only in the image height direction, and the correction filter is one-dimensional, and therefore cannot reduce the asymmetry in directions other than the image height direction. The image height direction is the azimuthal meridional direction. In addition, the correction in the direction of the image height cannot provide sufficient sharpness when using the standard method, since the asymmetry of the filter is adjusted by changing the number of negative tap coefficients, and the filter causes a blur other than the blur caused by the PSF of the optical system.

As described above, the standard method cannot provide sufficient correction of complex aberration and increase image sharpness.

To provide the processing necessary to sharpen the input image, it is necessary to know the shooting parameters of the input image and the data of the point spread function corresponding to the image height. However, a large amount of data on previously saved various image capture parameters, as well as a large amount of data corresponding to the image height, will require a large amount of memory, which makes such processing almost impossible.

Disclosure of invention

The present invention provides an image processing apparatus, an image pickup apparatus, an image processing method and an image processing program that provide an exceptional sharpening effect.

According to one aspect of the present invention, an image processing apparatus includes an acquisition unit configured to obtain an image formed by capturing an image through an optical system, and a processor configured to provide image processing based on an unsharp mask by using a filter based on information about the scattering function of the point of the optical system in accordance with the image pickup parameter of the optical system. The filter has two-dimensional data.

According to another aspect of the present invention, an image pickup apparatus includes an image sensor configured to photoelectricly convert an optical image of an object formed through an optical system, an image processor configured to process an image received from an image sensor, and a recording unit configured to store the relationship between the image pickup parameter of the optical system and the point spread function of the optical system. The image processor receives information about the scattering function of the point of the optical system corresponding to the image pickup parameter of the optical system from the recording unit and provides image processing with an unsharp mask by using a filter based on information about the scattering function of the point of the optical system. The filter has two-dimensional data.

According to another aspect of the present invention, an image processing method includes acquiring an image formed by capturing an image through an optical system, and providing image processing based on an unsharp mask by using a filter formed based on information about a point spread function of the optical system in accordance with a parameter shooting images of the optical system. The filter has two-dimensional data.

According to another aspect of the present invention, an image processing program that initiates computer processing that includes acquiring an image formed by capturing an image through an optical system and providing image processing based on an unsharp mask by using a filter formed based on scattering function information points of the optical system in accordance with the shooting parameter of the image of the optical system. The filter has two-dimensional data.

According to a further aspect of the present invention, a read-only computer-readable medium stores an image processing program.

In addition, the features and aspects of the present invention will become apparent from the following description of exemplary embodiments of the invention with reference to the accompanying drawings.

Brief Description of the Drawings

FIG. 1 is a block diagram of an image pickup apparatus for each of embodiments 1, 2, and 3 of the present invention;

FIG. 2 is a flowchart of an image processing method according to each of options 1, 2, and 3;

FIG. 3A and 3B are directivity patterns of sharpening by processing based on an unsharp mask;

FIG. 4A and 4B are radiation patterns of a PSF function of an optical system for capturing an image on the xy plane;

FIG. 5A and 5B are radiation patterns of sharpening processing with an axisymmetric unsharp mask;

FIG. 6A and 6B are radiation patterns of sharpening processing with an axisymmetric unsharp mask;

FIG. 7A and 7B are a radiation pattern and a schematic cross section of an unsharp mask;

FIG. 8A to 8C are flowcharts of an image processing method according to the present invention (option 1);

FIG. 9 is a radiation pattern of a Bayer matrix;

FIG. 10 is an explanatory diagram of a method for dividing an input image;

FIG. 11 is an explanatory diagram of a method for interpolating an input image in an image height direction;

FIG. 12 is a flowchart of an image processing method according to the present invention (option 2);

FIG. 13 is a flowchart of an image processing method according to the present invention (embodiment 3);

FIG. 14 is a flowchart of an image processing method according to Embodiment 1 of the present invention;

FIG. 15A - 15E are explanatory diagrams of the arrangement of the OTF and PSF functions according to embodiment 4;

FIG. 16 is a diagram of an image processing system including an image processing device according to Embodiment 4;

FIG. 17 is an explanatory diagram of coefficient data according to Embodiment 4;

FIG. 18 is a detailed explanatory diagram of coefficient data according to Embodiment 4;

FIG. 19 is an explanatory diagram related to the number of taps and the frequency step according to embodiment 4;

FIG. 20 is an explanatory diagram relating to a different number of taps and a frequency step according to embodiment 4;

FIG. 21 is an explanatory diagram of reconfiguring OTF functions according to embodiment 4;

FIG. 22 is a detailed explanatory diagram of reconfiguring OTF functions according to embodiment 4;

FIG. 23 is a block diagram of a modified example of image processing according to embodiment 4;

FIG. 24 is a configuration of an image pickup apparatus according to Embodiment 5 of the present invention;

FIG. 25 is an explanatory diagram of an image processor provided in an image pickup apparatus according to Embodiment 5.

The implementation of the invention

Below with reference to the accompanying drawings, exemplary embodiments of the present invention are described.

Before describing specific options, definitions will be given of terms used in these options and in image processing.

[Input Image]

An input image is a digital image formed from the output of an image sensor that performs photoelectric conversion of an image of an object formed by an optical image pickup system (hereinafter referred to simply as an “optical system”) in an image pickup device. A digital image is an image degraded by an optical transfer (OTF), including the aberration of an optical system that contains optical elements such as a lens and an optical filter. The image sensor includes photovoltaic cells based on CMOS and CCD (CCD) technology. An optical image pickup system may include a curved mirror (reflective surface). The optical system may be removable from an image pickup device (interchangeability). In the image pickup device, the image pickup system consists of an optical system, an image sensor and a signal processing circuit that generates a digital image (input image) from the output of the image sensor.

The color components of the input image contain information, for example, in the color components of the RGB space. The color components used can be selected from other commonly used color spaces, such as the LCH color space, which reflects brightness, hue, and saturation, and the YCbCR color space, which expresses brightness and color contrast. Other usable color spaces include, for example, XYZ, Lab, Yuv, and Jch color spaces, and color temperature can additionally be used.

Input and output images may contain information on the image pickup parameter, for example, focal length, aperture value and distance to the object (hereinafter referred to as information on the shooting parameters of the object) of the optical system in the image pickup device when the input image is formed (fixed). The input and output images may also contain various correction information used to correct the input image. In a configuration in which the image pickup device outputs an input image to an image processing device housed separately, and the image processing device provides image recovery processing to the input image, it is desirable that the input image contains information about shooting parameters and correction information. Instead of sending information about the image pickup parameters and correction information along with the input image, this information can be transmitted directly or indirectly from the image pickup device to the image processing device via a communication means.

[Processing based on unsharp mask (sharpening processing)]

In FIG. 3A and 3B schematically show an increase in image sharpness by processing based on an unsharp mask. In FIG. 3A, the solid line represents the input image to be processed, and the dashed line represents the image blurred by applying an unsharp mask to the input image. The dotted line represents a sharpened image. The solid line in FIG. 3B represents a correction component (correction signal) used to sharpen. In FIG. 3A and 3B, the horizontal axis represents the coordinate, and the vertical axis represents the pixel value or brightness value. Each of FIG. 3A and 3B correspond to a section in a predetermined direction (for example, in the X direction) in FIG. 4A and 4B, which will be described below.

If the input image is denoted as f (x, y), and the correction component is denoted as h (x, y), then the image g (x, y) with increased sharpness can be represented by the expression (1):

In expression (1), the correction signal h (x, y) is multiplied by the constant m and added to f (x, y) representing the input image. The constant m can be changed to adjust the amount of correction. The constant m can be unchanged and not depend on the position in the input image. Alternatively, to adjust the correction amount depending on the position in the input image, a control coefficient m (x, y) can be used, which changes with a change in position in the input image. The constant m and the adjustment coefficient m (x, y) can vary depending on the image capture parameters, such as focal length, aperture value and distance to the subject. In the following description, the constant m can be replaced by a regulation coefficient m (x, y).

If the unsharp mask is designated as USM, then the correction component h (x, y) can be represented as:

Alternatively, the right-hand side of expression (2) may be represented by expression (3). USM (x, y) is, for example, the retraction value in the coordinates (x, y) of the USM mask.

The symbol * represents a convolution (convolution integral, the sum of products), and the symbol δ represents a delta function (perfect point image), whose integral is unity. Here, the delta function is data whose number of taps is equal to the number of taps USM (x, y) and whose values are equal to zero except for the central value equal to one.

Expression (3) differs from expression (2) by the calculation method used in the above processing. However, since expression (3) can be obtained by overwriting expression (2), expression (3) represents processing equivalent to processing according to expression (2). For this reason, expression (2) is used below to describe the formation of the correction component.

Expression (2) is used to calculate the difference between the input image f (x, y) and the image obtained by reducing the sharpness of the input image f (x, y) with an unsharp USM mask and forming the correction component h (x, y). In a typical unsharp mask processing, the USM unsharp mask is a smoothing filter such as a Gaussian filter, a median filter, and a moving average filter. For example, when as an unsharp USM mask for the input image f (x, y) shown by the solid lines in FIG. 3A, a Gaussian filter is applied to obtain an image resulting from a reduction in sharpness of the input image f (x, y), which is shown by the dashed line in FIG. 3A. Thus, the correction component h (x, y) is the difference between the image f (x, y) and the image with reduced sharpness as in expression (2). Thus, subtracting the value of the sharpened image signal shown by the dashed line in FIG. 3A, from the value of the input image signal shown by the solid line, gives the correction component shown by the solid line in FIG. 3B. The correction component thus calculated is used to calculate expression (1) to sharpen the input image f (x, y) shown in FIG. 1A by a solid line, and obtaining an image shown by a line represented by dots.

The following is a description of sharpening an image by processing, based on an unsharp mask, an optical image of an object degraded when passing through an optical imaging system. If the initial image (image of the object) before passing through the optical system is denoted as I (x, y), and the function PSF representing the response of the optical system to a point light source is denoted as psf (x, y), then the input image f (x, y) formed through an optical system is expressed as:

If the optical system is an axisymmetric coaxial optical system, then the PSF corresponding to the central part of the image will be axisymmetric. This allows sharpening processing to be performed, as a result of which the input image f (x, y) becomes closer to the original image f (x, y) in the central part of the image by applying an axisymmetric USM mask, as described above. Since the correction value is the difference value between the above image and the image with reduced sharpness obtained by processing based on an unsharp mask, a more accurate correction will require the use of an unsharp mask, closer in form to the psf (x, y) function, and not a simple smoothing filter . The reason for this is as follows. For example, if the image is degraded due to spherical aberration, which has an axisymmetric effect, a smoothing filter, such as a Gaussian filter, has a distribution shape different from the PSF function affected by spherical aberration. Thus, the use of PSF optical system allows for more accurate correction while reducing axisymmetric sharpening.

For this reason, in each embodiment of the present invention, the PSF function is used in the form of a USM mask. Although the input image f (x, y) in FIG. 3A used to describe processing based on an unsharp mask, for simplicity of description is shown as a symmetrical image, the shape of the input image may be asymmetric. Even in the case when the original image I (x, y) has an asymmetric shape, the axisymmetric sharpening function, which corresponds to the function psf (x, y), and acts on the original image I (x, y), achieves sharpness with an axisymmetric USM mask.

On the other hand, in places not related to the central part of the image, even when the optical system is an axisymmetric coaxial optical system, the PSF function is generally asymmetric. In FIG. 4A and 4B schematically show the PSF function of the optical system in the x, y plane, namely: in FIG. 4A shows the PSF function on the optical axis, and FIG. 4B shows the PSF function outside the optical axis.

For example, if the original image (image of the object) is an ideal point image, expression (4) indicates that the input image f (x, y) is a PSF of the optical system. When the ideal point image is at an angle according to FIG. 4B, and the original image (object image) is degraded due to the PSF of the optical system, the image obtained as the input image is a blurred image, the shape of which is shown in FIG. 4B. The following is a description of sharpening by processing an image asymmetrically blurred in this way based on an unsharp mask.

In FIG. 5A and 5B and FIG. 6A and 6B schematically show asymmetrically degraded image processing based on an unsharp mask. In FIG. 5A and 5B show processing using an axisymmetric unsharp mask, and in FIG. 6A and 6B show processing using an axisymmetric unsharp mask. Here, the vertical axis and horizontal axis are the same as in FIG. 3A and 3B. The solid lines in FIG. 5A and 6A are a section in the y direction of FIG. 4B, and the dashed lines represent images obtained by blurring the input image using appropriate unsharp masks. The unsharp mask in FIG. 5A and 5B, i.e., an axisymmetric unsharp mask, is a Gaussian filter. On the other hand, the unsharp mask in FIG. 6A and 6B, i.e., an axisymmetric unsharp mask, is a function of the PSF of the optical system.

In FIG. 5B and 6B respectively show correction components in the form of differences between unsharp images shown by dotted lines in FIG. 5A and 6A, and the original input image. In FIG. 5A and 6A, the positive direction Y is considered the direction in which the input image blurred by the PSF function looks like an expanding bell. In the case of using the axisymmetric unsharp mask shown in FIG. 5A and 5B, the difference between the unsharp image and the original input image is smaller on the positive side with respect to the peak position shown in FIG. 5A is a solid line, and more on the negative side of the peak. Therefore, the correction component shown in FIG. 5B has a greater extreme value (negative value relative to 0) on the positive side (right side) relative to the position of the central peak than on the negative side (left side).

When using a correction component, the magnitude of which (correction amount) is less on the positive side, where the bell expands, and more on the negative side, where the bell does not expand, as shown in FIG. 5A and 5B, asymmetric blur cannot be corrected by sharpening according to expression (4).

For example, the correction value is adjusted by changing the constant m in expression (4) without changing the unsharp mask. However, when using the large constant m to provide sufficient correction on the positive side of the input image, the negative side of the input image is excessively corrected. In contrast, when the constant m is set so that the negative side of the input image is corrected correctly, the positive side of the input image is not correctly adjusted. Such processing based on an unsharp mask using an axisymmetric unsharp mask on an asymmetrically blurred input image makes it difficult to eliminate the asymmetry and increase the sharpness of the image. Although this case describes the use of a Gaussian filter as an axisymmetric unsharp mask, an asymmetric blurred input image cannot be made sharp enough in the case of using axisymmetric filters of other types.

The following describes the case of using the axisymmetric unsharp poppies shown in FIG. 6A and 6B. In this case, the difference between the unsharp image and the original input image is larger on the positive side relative to the peak position shown by the solid line in FIG. 6A, and less on the negative side, that is, this ratio is the opposite of FIG. 5A. Thus, the correction component shown in FIG. 6B has a greater extreme value on the negative side (left side) relative to the position of the central peak than on the positive side (right side). The correction component used for the input image, shown by the solid line in FIG. 6A has a large value on the positive side with respect to the peak position where there is a large blur, and a small value on the negative side where the blur is small. Therefore, the use of an asymmetric unsharp mask leads to blurring of the input image, and similar distributions of the correction value, which reduces both the over and under correction that occurs when using the axisymmetric unsharp mask. In addition, the use of an asymmetric unsharp mask is unlikely to lead to excessive correction compared to the case of the use of an axisymmetric unsharp mask, and therefore the constant m in expression (4) can be set relatively large, thereby reducing the asymmetry and further increasing the image sharpness.

Since the amount of correction provided by the correction component is distributed in accordance with the difference between the unsharp image and the original image, the area that is highly blurred by the PSF function of the optical system for capturing the image must have a greater blur provided by the unsharp mask compared to other areas to provide more accurate correction. Thus, the PSF function of the optical system is ideally used as an unsharp mask for more accurate correction. The cross section described in this embodiment is one-dimensional, but may be two-dimensional.

The following is a description of the filter and the correction signal used in each embodiment described below, and processing based on an unsharp mask according to each embodiment.

In options 1 and 4, sharpening is achieved using the expression below, derived from expressions (1) and (2).

In option 2, sharpening is achieved using the expression below, obtained from expressions (1) AND (3).

In option 3, sharpening is achieved using the expression below, obtained from expressions (6).

Expression (7) can be converted to the expression below.

In option 3, sharpening is provided using expression (7), but can be achieved using expression (8).

[Saved data]

It is necessary that the PSF function of the optical system is stored in the image pickup apparatus and the image processing apparatus in order to use it as an unsharp mask, as described above. The PSF function stored in these devices can be used directly as a USM mask for sharpening processing. However, the PSF function will be different depending on the parameters (image pickup parameters) when the input image and image height are fixed, and therefore it is necessary to store data corresponding to different PSF functions in order to provide the processing necessary for accurate correction. In addition, these PSF functions are two-dimensional, and a color image requires data for each of the color components, such as RGB.

The stored data may contain OTF function information for generating the PSF function, or only PSF information as necessary. First, a case of storing OTF information is described.

Converting OTF to PSF requires the use of the Fourier transform, which currently does not form a serious load due to the increased processing speed that modern computing devices differ in. However, the OTF data for each image height is data for several taps in the longitudinal direction × number of taps in the transverse direction × 2 (real part and imaginary part) × 3 (RGB). Saving all image capture parameters in these devices, such as image height, F number (aperture value), zoom position (focal length) and distance to the subject, increases the amount of data. As described above, high-precision processing and the amount of data are generally incompatible, but these options provide the ability to obtain images corrected with high accuracy based on the small amount of data. The OTF function has relatively wide frequency characteristics, as will be described in subsequent versions, which allows for high-precision approximation by means of a function such as a polynomial. Thus, the data to be stored are the approximation coefficients of the function, which leads to a significant reduction in the amount of stored data in comparison with the two-dimensional data stored for OTF.

When the image pickup apparatus is an interchangeable lens, since the pixel pitch of the image pickup is different depending on the model of the image pickup apparatus, the PSF data for the same lens will be different. As described in the above options, the pixel pitch and the number of taps of the PSF function to be generated can be easily changed when the function and its coefficient data for restoring the OTF function are stored in the device.

The following describes the storage of PSF information as stored data. The PSF information to be stored, similar to the OTF case, is PSF data corresponding to image pickup parameters and color components. Alternatively, instead of storing two-dimensional PSF data, the PSF function can be approximated by an empirical function, and its coefficients for reproducing the PSF can be stored as stored data. Using a function suitable for approximating a PSF, it is possible to reduce the amount of data for a PSF by approximating to obtain said function. By analogy with the OTF approximation, saving the function for playing PSF and data on its coefficients allows you to reproduce the pixel pitch and the number of taps of the reproduced PSF. Thus, the information about the stored PSF eliminates the need to perform the Fourier transform, which reduces the computational load.

The PSF function is different for each image capture parameter, including the height of the image formed by the optical system and the focal length, F-number and the distance to the object of the optical system. The following options describe the height of the image as an example of a shooting parameter, but aberration information that is different for each value of the focal length, F-number, and distance to the object of the optical system can be obtained so as to form an inaccurate mask based on this information.

[Option 1]

In FIG. 1 is a block diagram of an image pickup apparatus 100 according to Embodiment 1. In the image pickup apparatus 100, a program is installed in the memory 120 that provides sharpening processing (image processing method) for the input image, wherein sharpening processing is performed by the image processor 104 (image processing apparatus ) of the image pickup apparatus 100. The composition of the memory 120 includes read-only memory (ROM) and a hard disk drive, but the recording unit 108 described below may be used as the memory.

The image pickup apparatus 100 includes an optical system 101 (lens) and an image pickup device body (camera body). The optical image pickup system 101 includes an aperture aperture 101a and a lens 101b, which is integrally formed with the body of the image pickup apparatus. However, this option is not limited to this, and is also applicable to an image pickup apparatus including an image pickup apparatus body mounted with an interchangeable image pickup optical system 101.

The image sensor 102 is a two-dimensional image sensor, such as a charge-coupled device (CCD) or a metal oxide semiconductor (CMOS) structure. The image sensor 102 performs photoelectric conversion of the image of an object (light forming an image) obtained by the optical image pickup system 101 and generates an image. The image of the object is converted photoelectricly into an analog signal (electrical signal) by the image sensor 102. This analog signal is converted by an analog-to-digital converter 103 into a digital signal, which is then input to the image processor 104.

The image processor 104 is an image processing unit that provides processing based on a predetermined unsharp mask, as well as a predetermined processing of said digital signal. In this embodiment, sharpening processing is provided by an image processor included in the image pickup device, but can be performed by a personal computer (PC) or a specialized device serving as the image processing device.

The image processor 104 obtains information on image pickup parameters by the image pickup apparatus 100 from the state detector 107. Information about the shooting parameters of the image is information about the aperture, the distance to the subject and the focal length of the zoom lens. The state detector 107 receives information about the image pickup parameters directly from the system controller 106, but this is not a limitation. For example, information on image pickup parameters of the optical image pickup system 101 can be obtained from the controller 105 of the optical image pickup system.

As such, the image processor 104 provides sharpening processing for the input image. The image processor 104 includes a point image intensity distribution selector 201 (PSF function selector), a correction signal generator 202, and a correction signal application unit 203. However, when the image processor 104 serves as the image processing apparatus, the system controller 106 of the image pickup apparatus 100 may store aberration information in conjunction with the image. In this case, the image processing apparatus may include a correction signal generator 202 and a correction signal application unit 203, rather than a PSF function selector 201.

The output image processed in the image processor 104 is stored in a predetermined format in the recording unit 108. The recording unit 108 also serves as a memory in which the relationship between the image pickup parameters of the optical image pickup system 100 and the PSF function of the optical image pickup system are stored.

The image display unit 112 is capable of displaying an image provided by the predetermined display processing after the sharpening image processing. The image display unit 112 may display an image provided by simplified processing so as to achieve fast display.

Said processing operation sequences are controlled by the system controller 106. The system controller 106 is configured as a microcomputer and a processor (CPU). The mechanical drive of the optical image pickup system 101 is controlled by a controller 105 of the optical image pickup systems based on a command from the system controller 106.

The optical image pickup system 101 may include optical elements such as a low pass filter and an infrared notch filter. When using an optical element that affects the characteristics of the PSF, such as a low-pass filter, the calculation of the influence of the optical element during the formation of an unsharp mask allows for more accurate processing to increase image sharpness. When using an infrared notch filter, which affects the PSF of each of the RGB channels (RGB color components), which are the integral value of the PSF over the spectral wavelengths, especially the PSF of the R channel, this effect of the optical element should be taken into account when forming an unsharp mask.

Next, with reference to FIG. 2, an image processing method according to this embodiment is described. In FIG. 2 is a flowchart illustrating the processing as a whole in this embodiment, where the symbol “S” represents “step (process)”. The block diagram shown in FIG. 2, can be implemented as a program (image processing program), which initiates the computer performing the functions of each stage. Similarly, other flowcharts can be implemented. Each step in FIG. 2 is executed by the image processor 104 based on a command from the system controller 106.

First, an image in the form of an input image is obtained (S11), then the PSF function selector 201 receives from the recording unit 108 a PSF function of the optical image pickup system corresponding to the input image pickup parameters (acquisition step S12). The PSF information obtained by the PSF function selector 201 may be two-dimensional tap data, a plurality of one-dimensional tap data, as components of the PSF functions, and coefficients.

Then, the correction signal generator 202 generates an unsharp mask and a correction signal based on the PSF information obtained in step S12 (generation step S13).

Next, with reference to FIG. 7A and 7B, an unsharp mask is described. The unsharp mask has several taps, determined depending on the characteristics of the aberrations of the optical imaging system and the required sharpening accuracy. The unsharp mask in FIG. 7A is an exemplary two-dimensional mask with 11 × 11 taps. In FIG. 7A does not show the value (coefficient) at each tap, but in FIG. 7B shows an unsharp mask section. In FIG. 7B, the horizontal axis represents the bends, and the vertical axis represents the bends.

The distribution of values (coefficient values) of the taps of the unsharp mask ideally represents the distribution of signal values (PSF of the optical image pickup system), which expands due to aberration. An unsharp mask may be generated based on PSF function information, as described above, or may be a PSF function obtained by the PSF function selector 201. The following describes the processing associated with the formation of the correction signal by using an unsharp mask formed in the correction signal generator 202.

Then, the correction signal applying unit 203 provides sharpening processing for the input signal using the correction signal generated in step S13 (sharpening step S14). The processing details in step S14 are described below.

In FIG. 8A is a flowchart illustrating processing details according to this embodiment. Each step in FIG. 8A is performed by the image processor 104 based on a command from the system controller 106.

First, an image in the form of an input image is obtained (S111). In particular, the color component data as the correction target used as the input image are, for example, image data of the channel G after elimination of the mosaic. However, the color component data may be image data of an R channel or a B channel, image data of all RGB channels, or image data until the mosaic is removed.

In FIG. 9 shows the radiation pattern of a Bayer matrix, which is a discrete ordered matrix. For example, processing may be provided for an input image representing data of each channel from RGB or data of a specific channel. Alternatively, as shown in FIG. 9, the G channel can be divided into channels G1 and G2, as a result of which only four channels are provided. When using a configuration where the G channel is divided into two channels, as described above, the image data for each of the channels G, G1, G2 and B have the same resolution, which facilitates image processing and data processing.

Next, the PSF function selector 201 receives, from the recording unit 108, PSF information of the optical image pickup system corresponding to the input image (S112). The PSF information obtained by the PSF function selector 201 may be two-dimensional tap data, a plurality of one-dimensional tap data as components of the PSF, and also coefficients. Two-dimensional data is divided into many one-dimensional data, for example, using the theorem on the decomposition of singular values. The main component, divided according to the aforementioned theorem, can be recorded in the recording unit 108, and the said one-dimensional tap data corresponding to the main component of the PSF function can be obtained from the recording unit 108 depending on the image pickup parameters.

Since the PSF function varies with the height of the image, it is desirable that the unsharp mask changes with the height of the image in order to increase the accuracy of the correction, but recording data in the recording unit 108 leads to an increase in costs. For this reason, in this embodiment, in order to be able to change the unsharp mask with changing the image height, the input image is divided into many regions and interpolation processing is performed based on information about the PSF function for at least two image height values for each region, which ensures the formation of an intermediate unsharp mask. Details of the interpolation method are described below for step S114.

The following describes the division of the input image into areas. In FIG. 10 shows the directivity pattern of the input image, where the directions along the long side and the short side of the input image are considered the X direction and Y direction, respectively, and the image center is selected as the origin. In this embodiment, as an example, shown in FIG. 10, the input image is divided into eight regions A to G, and information about the point spread function is obtained for each of the peripheral parts of the regions and for the center point.

Next, the correction signal generator 202 provides filtering processing using the PSF function obtained in step S112 (S113). In this embodiment, the PSF function is used as an unsharp mask, and convolution processing (convolution integral, sum of products) using the unsharp mask is provided for the input image. Since the number of PSF functions obtained in step S112 is nine, including eight surrounding areas and the origin, nine input images are formed blurred by the corresponding unsharp masks, that is, nine sets of image data filtered by the unsharp mask.

Then, the correction signal generator 202 provides interpolation processing in the image height direction using the plurality of image data filtered by the blur masks that were generated in step S113, and generates the only image data filtered by the blur masks (interpolation step S114).

Next, with reference to FIG. 11, a description of interpolation processing in the image height direction follows. In FIG. 11, the directions of regions C and A with respect to the origin are considered, respectively, the positive direction X and the positive direction Y, and in FIG. 11 shows a first quadrant of the input image, where both X and Y axes are positive. In FIG. 11, P0 denotes the origin, and P1, P2, and P3 respectively denote the peripheral image points of region A, region B, and region C, and the PSF function selector 201 obtains PSF for each of the points P0, P1, P2, and P3 in step S112.

In FIG. 11, the Pn point shown by the white circle is an arbitrary point (image point) in the image, the Pn point in region B uses input image data filtered by a mask that is generated based on the PSF function information for points P0 and P2 in step S113. Similarly, the points Pn in regions A and C respectively use input image data filtered in step S113 with unsharp masks corresponding to the image heights at points P0 and P1 and at points P0 and P3.

The following is a description of the formation of interpolation data corresponding to the filtered data at an arbitrary point Pn based on two values of the image height in the region. As shown in FIG. 11, the point Pn in region B is at a distance d0 from the origin P0 and at a distance d2 from point P2. If the input images filtered in step S113 based on the PSF functions corresponding to points P0 and P2 are denoted by F0 and F2, then the interpolation data Fn corresponding to the optional point Pn can be represented by the following expression.

Said interpolation processing leads to generation of interpolation data for an arbitrary image height in each region and single image data based on a plurality of image data generated in step S113. Thus, the amount of generated image data is less than the amount of input image data filtered using different PSF functions depending on the image height value, which leads to an increase in processing speed.

Although expression (9) is intended for region B in the first quadrant, similar calculations for other regions and other quadrants allow interpolation data to be generated. Interpolation based on the height of the image can be performed not only by means of expression (9), but also by means of a quadratic curve, or each filtered input image can be weighted by multiplying by a constant.

Then, the correction signal generator 202 generates a correction signal based on the interpolation data generated in step S114 (S115). In this embodiment, the correction component is represented by the expression (2) and is formed based on the difference between the input image and the interpolation data generated in step S114.

Next, the correction signal applying unit 203 applies the correction signal generated in step S115 to the input image and increases the sharpness of the image (S116). In this embodiment, the use of the correction signal corresponds to expression (1), in which the constant m is determined on the basis of image noise and excessive or insufficient correction of sharpening. The constant m thus determined, the correction signal generated in step S115, and the input image are used to provide processing for sharpening.

Expression (1) is presented as the sum of the first term and second term for the positive constant m and as their difference for the negative constant m. Thus, when applying the correction signal to the input image during sharpening processing in this embodiment, such an application is expressed in the summation or subtraction, depending on the sign of the constant m. However, since it is understood that the summation and subtraction differ simply by signs and essentially mean the same operation, the above calculation may be a summation or subtraction depending on the sign of the constant m. In this embodiment, since the PSF function of the optical image pickup system is used as an unsharp mask, the input image with degraded peripheral portions due to the asymmetric function of the PSF of the optical image pickup system can be accurately adjusted with increasing sharpness.

In this embodiment, interpolation in the direction of the image height is provided for the filtered image, but can also be provided for the correction signal (Fig. 8B) and sharpened image (Fig. 8C).

In the process shown in FIG. 8B, point scatter functions are obtained for at least two image height values in each of a plurality of areas set in the input image, and a plurality of correction signals are generated based on these point scatter functions. Then, these correction signals are interpolated in the direction of the image height so as to form a single correction signal, which is then used to sharpen the input image. In the process shown in FIG. 8C, point scatter functions are obtained for at least two image height values of each of a plurality of areas set in the input image, and a plurality of correction signals are generated based on these point scatter functions. Each of these correction signals is then applied to the input image so as to obtain a plurality of sharpened images, which are then interpolated in the image height direction so as to form a single sharpened image.

In FIG. 8B is a flowchart illustrating in detail a processing stream that interpolates in the image height direction after generating correction signals. Steps S121, S122, and S123 correspond to steps S111, S112, and S113. The following describes the processing in steps S124 and S125 of FIG. 8B, other than the processing in steps S114 and S115 of FIG. 8A.

In step S124, the correction signal generator 202 generates correction signals from the filtered data obtained by filtering the input images by unsharp masks in step S123. Correction signals are formed based on the difference between the input image and the filtered data by means of expression (2). These correction signals are generated in the amount of data generated in step S123.

Next, in step S125, the correction signal generator 202 provides interpolation processing in the image height direction based on the correction signals generated in step S124. Interpolation processing in step S125 is provided for another interpolation object other than the interpolation processing object in step S114, but the processing flow is generally similar to the interpolation processing flow in step S114. In step S114, the input image data collapsed by the PSF function obtained in step S112 as an unsharp mask is interpolated in the image height direction.

On the other hand, in step S125, the difference between the input image and the input image data collapsed by the unsharp mask is used as the correction signal, and interpolation processing in the image height direction by this correction signal is performed. The processing in step S114 interpolates f (x, y) * USM in expression (2) in the image height direction, while the processing in step S125 interpolates h (x, y) in the image height direction.

Thus, when using the data f (x, y) * USM in step S114 and replacing it with the data h (x, y), in step S125 interpolation processing is also provided for the correction signals in the direction of the image height. The correction signals interpolated in the above manner are applied to the input image by the correction signal applying unit 203 in step S126. The application of these correction signals in step S126 is similar to the processing in step S116, and therefore is not described in detail here.

Next, with reference to FIG. 8C, the interpolation processing flow in the direction of image height in the input image to which the correction signals were applied is described. Steps S131, S132, S133 and S134 correspond to steps S121, S122, S123 and S124.

The following describes the processing in steps S135 and S136, which is different from the processing in steps S125 and S126 in FIG. 8B. In step S135, the correction signal applying unit 203 provides sharpening processing for the input image based on the correction signals generated in step S134. In step S134, correction signals are generated in the volume of a plurality of unsharp masks formed from the PSF functions selected in step S132. The sharpening processing in step S135 ensures that correction signals generated in step S134 based on expression (1) are applied to the input image.

Next, in step S136, a plurality of sharpened images formed in step S135 are interpolated in the image height direction. The interpolation processing in step S136 is provided for another interpolation object other than the interpolation processing object in step S114 or S125, but the processing flow is generally similar to the interpolation processing flow in step S114 or S125. The processing in step S125 interpolates h (x, y) in expression (1) in the image height direction, while the processing in step S136 interpolates g (x, y) in the image height direction. Thus, when using the data h (x, y) in step S125 and replacing it with the data g (x, y), the processing in step S136 interpolates the images with increased sharpness in the direction of the image height.

[Option 2]

The image pickup device according to option 2 has the same configuration as the image pickup device according to option 1. Option 2 differs from option 1 in that instead of the image processing methods shown in FIG. 8A-8C, the image processing method shown in FIG. 12. In FIG. 12 is a flowchart of an image processing method according to embodiment 2, where each step is performed by the image processor 104 based on a command from the system controller 106.

In this embodiment, a correction signal generation method other than option 1 is used. The processing in steps S211 and S212 is similar to the processing in steps S111 and S112. Although in option 1, correction signals are generated based on expression (2), in this embodiment, correction signals are generated based on expression (3). To generate correction signals, the correction signal generator 202 according to this embodiment first calculates the difference between the ideal point image and the PSF functions selected by the PSF function selector 201 and generates filters (S213). Next, the correction signal generator collapses the input image using the generated filters to generate correction signals (S214).

The processing in steps S215, S216 and S217 is similar to the processing in steps S114, S115 and S116, and therefore, a description thereof is omitted here. In this way, sharpening processing can be performed based on expression (6).

Interpolation in the direction of the image height can be provided for the correction signals, as described in option 1, or can be provided for the input image, the sharpness of which is increased by means of correction signals.

[Option 3]

The image pickup device according to option 3 has the same configuration as the image pickup device according to option 1. Option 3 differs from option 1 in that instead of the image processing methods shown in FIG. 8A-8C, the image processing method shown in FIG. 13. In FIG. 13 is a flowchart of an image processing method according to embodiment 3, where each step is performed by the image processor 104 based on a command from the system controller 106.

In this embodiment, a different method of generating correction signals and a different method of applying correction signals other than options 1 and 2 are used, and in this embodiment, the filters are formed based on expression (7), and the generated filters are applied to the input image. Thus, the block diagram of FIG. 13 differs from options 1 and 2 in the formation of filters in step S313 and the application of these filters in step S314. The processing in steps S311 and S312 is similar to the processing in steps S11 and S112.

In this embodiment, the PSF functions selected in step S312 are used as unsharp masks to form filters corresponding to the term in parentheses in expression (7) (S 312). In this embodiment, the correction signals generated by the correction signal generator 202 are filters. Next, the correction signal applying unit 203 collapses the input image using the filters generated in step S313 to sharpen the image (S314). In the sharpening processing according to this embodiment, sharpening is provided by a single convolution of the image using filters (correction signal) formed as unsharp masks based on the PSF functions of the optical image pickup system.

Each of these options provides an image processing device, an image pickup device, an image processing method and an image processing program that are distinguished by strong sharpening effects.

[Option 4]

In FIG. 14 is a flowchart of a procedure implementing an image processing method according to Embodiment 4 of the present invention. This image processing method is performed by a computer (processor), which is mounted in the image pickup device, or is an image processing device according to an image processing program in the form of a computer program. In step S101, the computer receives as a input image a digital image captured and generated by the image pickup device through the optical system. If the computer is mounted in an image pickup device, it receives an input image from the image pickup system. If the computer is another image processing device other than the image pickup device, then it receives an input image from the image pickup device using wired or wireless communication or a permanent computer-readable medium. The data used as the input image, for example, may be image data of the G channel after eliminating mosaicism. However, the R channel or B channel data may be used as the input image, or the data of each channel of the RGB system may be used as the input image. As an alternative, image data can be used as input before eliminating mosaicism.

In FIG. 9 shows a pixel matrix of the input image, which is similar to option 1, and therefore, its description is omitted here.

Next, in step S102, the computer receives information about the image pickup parameters. Image capture options as described above include focal length, aperture value, and distance to the subject. If the input image is generated in the interchangeable lens image pickup device, then the image pickup parameters include a lens identifier (ID) unique to that lens and a camera ID unique to the image pickup device. Image capture parameters can be obtained directly from the image pickup device, or can be obtained as information using the input image.

Next, in step S103, the computer receives coefficient data corresponding to the image pickup parameters. The coefficient data is used to reconfigure the OTF functions, and this data can be obtained selectively from data previously stored in the memory, server, or network depending on the image capture parameters. The coefficient data corresponding to the image pickup parameter among the image pickup parameters for which the coefficient data for the aperture, distance for the subject, and the focal length of the zoom lens can be stored by interpolation processing based on the stored coefficient data. This allows you to reduce the amount of pre-stored coefficient data. The interpolation processing method may be, for example, but not limited to the well-known bipolar interpolation method (linear interpolation) or bicubic interpolation.

Next, in step S104, the computer uses the coefficient data corresponding to the image pickup parameters for reconfiguration (formation), in particular in the direction of the image height, of a plurality of optical transfer functions, each of which changes with the image height from the reference position, which is the center of the input image, or optical axis of the optical system. The reconfiguration of the optical transfer functions based on the selected coefficient data is equivalent to the selection of the optical transfer functions. The processing details at this stage will be described below. In this embodiment, steps S102-S104 are called the step of selecting an optical transfer function.

Next, in step S105, the computer rotates the reconfigured optical transfer functions around the center of the input image or reference position in the form of the optical axis of the optical system. The computer then interpolates the optical transfer functions for the pixel matrix of the input image (or the sensor of the image pickup device that received the input image). In this way, the computer receives optical transfer functions corresponding to a plurality of specific two-dimensional discrete positions in the input image, and disposes in two dimensions optical transfer functions for the input image. These two-dimensional optical transfer functions are hereinafter referred to as rearranged optical transfer functions. In this embodiment, the processing in step S105 is called the rearrangement of the optical transfer function.

Next, in step S106, the computer converts the rearranged optical transfer functions into point spread functions. The conversion of the optical transfer functions to the point spread function is performed by the inverse Fourier transform. In this embodiment, the processing in step S106 is called the step of generating a scattered point function.

The processing in steps S104-S106 is described in detail below with reference to FIG. 15A-15E. In FIG. 15A, circles represent optical transfer functions (referred to herein as OTF) reconfigured in step S104. The reconfigured OTFs correspond to a plurality of image height values in a particular image height direction (in the longitudinal direction of FIG. 15A) with respect to a reference position in this region (image pickup area) of the described circle of the input image. In step S105, the computer expands the OTF functions in a specific direction of the image height relative to the reference position and provides interpolation corresponding to the pixel matrix of the input image in such a way that it is placed in the two-dimensional representation (in the longitudinal direction and in the transverse direction) of the OTF functions in the input image as shown in FIG. . 15B. Interpolation includes radial interpolation and rotational interpolation, which allows you to place OTF functions at arbitrary pixel positions.

Next, in step S106, the computer performs an estimate based on the inverse Fourier transform of the OTF function at each position (pixel) in the input image so as to form a point spread function (hereinafter referred to as PSF) at that position, as shown by the squares in FIG. 15C. The optical system used to obtain the input image is rotationally symmetrical about the optical axis. This rotational symmetry can be used to rotate the PSF formed in one quadrant (quarter region) relative to the reference position, in another quadrant as shown in FIG. 15D, whereby PSF functions are generated in the entire area of the input image. This method has the advantage that the inverse Fourier transform is calculated four times less than a quarter of the total number of positions on which the PSF functions are located. Furthermore, when the OTF functions shown in FIG. 15B and the PSF functions shown in FIG. 15C are rearranged by rotation and interpolation, as shown in FIG. 15E, and are distributed over the entire area of the input image using rotational symmetry, as shown in FIG. 15D, the number of inverse Fourier transform calculations can be further reduced.

The locations (number of functions and intervals between them) of the OTF and PSF shown in FIG. 15A-15E are only examples, that is, such arrangements can be set arbitrarily depending on changes in the OTF functions of the optical system. In FIG. 14, in step S107, the computer generates unsharp masks based on PSF functions generated at positions (pixels) throughout the input image area. The formation of unsharp masks is described in detail below. Then, in step S108, the computer generates correction signals described above. The generation of these correction signals is also described in detail below. In this embodiment, the processing in step S107 and step S108 is called the correction signal generation step.

Next, in step S109, the computer provides sharpening processing for the input image by adding correction signals to the input image, resulting in a corrected image in the form of the resulting image in step S110. The sharpening processing will be described in detail below. In this embodiment, the processing in step S109 is called the adding process.

The unsharp masks formed in step 107 are similar to the masks shown in FIG. 7A and 7B. To generate correction signals based on these unsharp masks in step S108, the computer first minimizes the input image using unsharp masks formed at positions in the input image. Since these unsharp masks are formed at positions at which PSF functions are generated, as shown in FIG. 15D, unsharp masks at positions between positions at which PSF functions are generated are formed by interpolation. This interpolation is provided in such a way that when the target pixel is a pixel at the position where the unsharp mask is formed, many unsharp masks near the target pixel are weighed depending on their distances to the target pixel, and then they are synthesized. This method allows you to create unsharp masks that continuously change from position to position in the input image. Thus, the unsharp mask generated by conversion from the OTF function or by interpolation of the PSF function is applied to the input image at each position in this image.

Another method includes, for example, applying three unsharp masks near the target pixel for a triangular region surrounded by three unsharp masks to form three images. Then, these three images are weighed depending on their distance from the position of the target pixel and synthesized in order to obtain a synthesized image equivalent to an image obtained by continuously changing unsharp masks with changing positions in the input image. The region to be interpolated can be defined, for example, by four positions instead of three positions.

Alternatively, the input image can be divided into many areas, and the same unsharp mask can be used in each area without interpolation. Alternatively, interpolation may be applied to the sharpened image obtained by applying the correction signals to the input image in step S109, described below.

The following describes the generation of correction signals in step S108. In this embodiment, the signals represented by expression (2) are used as correction signals. In particular, correction signals are generated by calculating the difference between the input image and the image obtained by applying unsharp masks to the input image in step S108.

The following is a description of sharpening processing in step S109. Sharpening processing according to this embodiment is performed by adding correction signals multiplied by a constant (m) to the input image by means of expression (1). The constant (m) is determined taking into account the noise in the image and excessive and insufficient sharpening. Expression (1) is presented as the sum of the first term and second term for the positive constant m and in the form of their difference for the negative constant m. As described above, adding correction signals multiplied by a constant during sharpening processing according to this option means adding or subtracting depending on the sign of the constant m.

As described above, in this embodiment, sharpening processing is performed using dot scatter functions as unsharp masks. This allows for accurate correction (sharpening) even when the input image contains distortions that are often found in the peripheral parts of the image due to the asymmetric functions of the PSF of the optical system.

The following is a description of an example image processing system configured to provide an image processing method described above with reference to FIG. 16. The image processing system includes a coefficient calculator 300, a camera (image pickup device) 310, and an image processing device 320.

The coefficient calculator 300 calculates the coefficients for reconfiguring the OTF functions based on the calculated values or the measured values of the optical system. The coefficient calculator 300 converts the OTF functions to coefficients and determines the orders of the coefficients used for the subsequent reconfiguration of the OTF functions depending on the required accuracy. The coefficient calculator 300, based on the spatial scatter sizes of the PSF functions, also determines the number of taps required for the subsequent reconfiguration of the OTF functions for each image height value. The coefficient calculator 300 calculates the coefficients and the number of taps up to the desired order for various combinations of the optical system (interchangeable lens 312 for capturing an image) and the image sensor 311, which is a camera 310, and outputs this data.

In the camera 310, an image of an object formed by the image pickup lens 312 is subjected to photoelectric conversion by the image sensor 311, and an image is generated based on electrical signals from the image sensor 311. Then, the camera 310 adds to the image the identifier (ID) of the lens and information about the image pickup parameters (for example, aperture value, focal length of the zoom lens and the distance to the subject), as well as the camera ID, and displays the image. Camera IDs are used to determine the Nyquist frequency from among the spatial frequencies available in the image sensor 111.

The image processing device 320 stores the data on the coefficients and the number of taps coming from the coefficient calculator 300, as well as the input image (including information on the image pickup parameters) as the image at the output of the camera 310. This data is then used to correct (sharpen) the input image distorted by the lens 312 for capturing an image.

In the image processing apparatus 320, in the correction information storage unit 321, information on various combinations of the image pickup lens 312 and the image sensor 311 calculated by the coefficient calculator 300 is stored. The stored information includes data on the coefficients and the number of taps, the lens ID, information on the image pickup parameters, as well as the Nyquist frequency of the image sensor 311 obtained based on the camera ID.

The OTF reconfiguration unit 322 obtains the Nyquist frequency of the image sensor 311 in the camera 310, the input image, image pickup parameters, and lens ID. The lens ID and image pickup parameters of the image pickup lens 312 used by the user (photographer) when capturing the image to form the input image are then used to search for coefficients and the number of taps stored in the correction information storage unit 321. The OTF reconfiguration unit 322 reconfigures the OTF functions used by the filter 323 based on the information found in the spatial frequency range up to the Nyquist frequency. Hereinafter, the OTF functions reconfigured by the reconfiguration unit 322 are referred to herein as the reconfigured OTF functions.

Filter 323 generates unsharp masks based on reconfigured OTF functions received from OTF reconfiguration block 322, and corrects (increases the sharpness) of the image based on unsharp masks. If the mentioned coefficients and the number of taps calculated by the coefficient calculator 300 are stored in the correction information storage unit 321, the coefficient calculator 300 does not need to be provided to the user. The user can download and use the information necessary for image processing, such as coefficient data, through a network or various types of computer-readable storage media of long-term storage.

The following is a description of a method for calculating coefficients by a coefficient calculator 300. The coefficient calculator 300 approximates the OTF functions of the optical system (image pickup lens 312) by approximating the function to form the coefficients. In this embodiment, the function used in the approximation is, but not only, the Legendre polynomial, and possibly, for example, the Chebyshev polynomial. The Legendre polynomial is given by expression (10).

In this expression, [x] denotes a maximum integer not exceeding x.

Since the OTF function is expressed as z = f (x, y), the coefficient aij of expression (11) is calculated as follows

Expression (11) is an orthogonal function in which the value of aij is determined independently of the order in the approximation. These characteristics of the orthogonal function can be used to truncate the function at a low order, when it is possible to approximate the low-order OTF function with sufficiently high accuracy. This allows you to minimize the amount of data on the coefficients to be stored in the calculator 300 coefficients.

In FIG. 17 shows a specific method for approximating the OTF function using expression (10) and expression (11). In FIG. 17 fum and fvm denote respectively the Nyquist frequencies of the OTF function in the meridional direction and sagittal direction. Nx and Ny denote, respectively, the odd number of OTF taps in the meridional direction and the sagittal direction. The coefficient calculator 300 calculates the coefficients for the real part and imaginary part of the OTF function by approximation.

The real part of the OTF function is symmetrical in the meridional direction and the sagittal direction. The imaginary part of the OTF function is symmetric with opposite signs in the meridional direction and symmetric in the sagittal direction.

Due to the mentioned symmetry, information for a quarter of the entire region is necessary and sufficient as the data of the OTF function as an approximation target. For this reason, in this embodiment, to achieve a high-precision approximation of the OTF function, the approximation is performed on a quarter region cut out from the entire region so that the constant component is contained both in the real part and in the imaginary part of the OTF function.

This embodiment describes an example in which OTF data is data for Nx (row) × Ny (column) of taps, with data for rows 1 to [Nx / 2] +1 and columns 1 to [Ny / 2] + 1 is cut out from OTF data. However, this option is not limited to this.

In FIG. 18 shows the coefficients calculated in the manner described above. In FIG. Figure 18 shows an example of the coefficients of the real part and the imaginary part of the OTF function, which are calculated for each image height (1-10) up to the tenth order in x and y. The coefficient data is presented as a set of coefficients for each image height along with information on the lens ID, aperture, focal length (zoom position) and the distance to the subject. In this embodiment, as an example, the coefficients for ten values of the image height with image capture parameters are shown, such as: lens ID No. 123, aperture value F2.8, WIDE zoom position (wide angle), and close proximity to the subject.

These coefficients for ten image height values are used to reconfigure the OTF function at ten positions (Fig. 15A). The generated coefficients can be used as functions of the image height for each order.

The coefficient calculator 300 generates the mentioned coefficient data for all combinations of the lens ID, aperture, focal length and the distance to the object and displays this data.

The following is a detailed description of a method for determining the number of taps of a reconfigured OTF. The duration of the filtering processing for an image is highly dependent on the number of filter taps used in the filtering processing. Thus, in filtering processing, it is desirable to minimize the number of filter taps, provided that the required correction is achieved without negative effects such as under correction.

The unsharp mask used by the filter 323 of the image processing apparatus is a filter in the actual space. Thus, the number of taps for this filter is determined in the actual space. The unsharp mask according to this embodiment is a filter used to correct image distortion due to PSF. Therefore, it is sufficient to have a region approximately equal to the scattering of the PSF function in real space. In other words, the required number of taps is the number of taps in said area. Since the real space and the frequency space are mutually inverse, the number of taps defined in the real space can be used in the frequency space.

In FIG. 19 shows that the number of taps is so large that a sufficiently large area is covered in comparison with spatial scattering of PSF. In FIG. 20 shows that the number of taps is set so that a region is found that is almost equal to the spatial scattering of the PSF function used in the case shown in FIG. 19. As shown in FIG. 19, the number of taps in the actual space corresponds to the minimum frequency step in the frequency space. Thus, a small number of taps in the actual space, as shown in FIG. 20, means coarse sampling in the frequency space, and therefore a large minimum frequency step. Meanwhile, the Nyquist frequency in the frequency space does not change.

The following is a detailed description of the OTF reconfiguration unit 322, which provides processing corresponding to the processing in step S104 shown in FIG. 14. The OTF reconfiguration unit 322 obtains a lens ID, image pickup parameter information, and Nyquist frequency information of the image sensor 311 from the camera 310 when capturing an image. Then, the OTF reconfiguration unit 322 reads the number of taps, the lens ID, the image pickup parameter information and the Nyquist frequency information of the image from the correction information storage unit 321, and generates a reconfigured OTF based on said information.

Next, with reference to FIG. 21, a detailed description of a method for generating a reconfigured OTF function follows. The Nyquist frequencies in the meridional direction and in the sagittal direction necessary to form a reconfigured OTF are denoted by fuc_rm and fvc_im, respectively, and the number of taps in the meridional direction and in the sagittal direction are indicated by Mx and My. When using the Nyquist frequencies fum and fvm in the meridional direction and in the sagittal direction, the following relationships are supported:

where Mx and My are odd numbers.

The variables x and y in expression (10) and in expression (11) are replaced by u and m, respectively, and then the regions are defined as follows.

respectively sampled using (Mx / 2) +1 and (My / 2) +1 taps. The OTF reconfiguration unit 122 subtracts the coefficients defined above in expression (9) and generates the reconfigured OTF in one quadrant. The above procedure is repeated for the real part (122-1-1) and the imaginary part (122-2-1) of the reconfigured OTF shown in FIG. 21.

The following is a description of a method for generating, based on a reconfigured OTF function, having real or imaginary parts formed in one quadrant, the reconfigured OTF function in regions being defined as follows:

with the number of taps equal to Mx and My.

Next, a method for generating the real part of the reconfigured OTF function is first described. The OTF reconfiguration unit 122, based on the previously generated real part (122-1-1), forms the real part of the reconfigured OTF in the areas as described below:

area of rows 1 through (Mx / 2) +1 and columns 1 through (My / 2); and

area of rows 1 through (Mx / 2) +1 and column (My / 2) +1.

Next, the OTF reconfiguration block 322 copies the numerical data in the area from rows 1 through (Mx / 2) +1 and columns 1 through (My / 2) into the row area from rows from 1 through (Mx / 2) +1 and columns ( My / 2) +2 to My, as shown in the real part (122-1-2). The data is copied in a linearly symmetrical manner with respect to the area of rows 1 through (Mx / 2) +1 and column (My / 2) +1.

Then, the OTF reconfiguration block 322 divides the real part (122-1-2) of the reconfigured OTF function, formed for half of the region, into the region of rows 1 through (Mx / 2) and columns 1 through My and the region from the row (Mx / 2 ) +1 and columns 1 through My, as shown in the real part (122-1-3). Then, the OTF reconfiguration block 322 copies the numerical data in the area from rows 1 to (Mx / 2) and columns 1 to My into the area from rows (Mx / 2) +2 through Mx and columns 1 through My in a linearly symmetrical manner along relative to the area from row (Mx / 2) +1 and columns 1 through My.

The following is a description of the method of forming the imaginary part of the reconfigured OTF function. Although this imaginary part can be formed in the same way as the real part, the imaginary part (122-2-3) must be formed with a switchable sign. The mentioned method can be used in the formation of the real and imaginary parts of the OTF function due to their characteristics.

In FIG. 22, a section of the reconfigured OTF function is shown for a detailed description of the relationship between the Nyquist frequency of the reconfigured OTF function and the number of taps. As described above, the Nyquist frequency is a parameter determined by the spatial resolution of the image sensor 311, and the number of taps is a parameter dependent on the PSF of the image pickup lens 312. These two parameters can be used with the mentioned coefficients in order to form the required reconfigured OTF function.

In FIG. 22 Nyquist frequencies satisfy the inequality f_nyq1> f_nyq2, and the number of taps satisfies the inequality N> M1> M2. As shown in FIG. 22, the Nyquist frequency and the number of taps can be set to the desired values.

Then, the OTF functions corresponding to the combinations of the image sensor 311 and the image pickup lens 312, as well as the image pickup parameters, are stored as coefficient data in the image processing apparatus 320, which allows image processing in accordance with the shooting parameters when capturing an image.

As shown in FIG. 15A-15E, the entire image area can be corrected by using a suitable number of taps with a small amount of coefficient data, which means the possibility of reducing the amount of stored data.

In FIG. 23 shows a modified example for this embodiment. Steps S401-S405 in FIG. 23 are similar to steps S101-S105 in FIG. 105 and therefore their description is omitted here. Processing up to step S405 ensures that the OTF functions are placed in two dimensions in the quadrant of the input image, as shown by way of example in FIG. 15B.

This modified example describes a transfer function that does not have rotational symmetry (hereinafter referred to as rotationally asymmetric transfer function), such as the transfer function of an optical low-pass filter or a pixel window of a certain shape, different from the transfer function of an optical image pickup system.

In step S406, the computer applies a rotationally asymmetric transfer function to each OTF, in the state shown in FIG. 15B. Then, in steps S407-S411, the computer provides the same processing as the processing in steps S106-S110 in FIG. 14 so as to obtain a corrected image obtained by sharpening the input image.

In FIG. 15B, the OTF functions are placed in a two-dimensional embodiment in said image quadrant. However, depending on the type of symmetry of the transfer function, OTF functions can be placed, for example, over the entire image area before applying the rotationally asymmetric transfer function.

[Option 5]

In FIG. 24 shows a configuration of an image pickup apparatus according to Embodiment 5 of the present invention. An image pickup apparatus is installed together with an image processing program that performs image sharpening processing as an input image. This image processing is performed by a computer (processor) included in the image processor 404 (image processing device) provided in the image pickup device following the image processing program.

An optical system 401 for capturing an image projects light from an object (not shown) onto the image sensor 402. The aperture diaphragm 401a in the optical system 401 for capturing an image has an opening diameter controlled by a variable aperture value. The position of the lens 401b is changed by an autofocus (AF) mechanism or a manual focus mechanism (not shown) in order to focus according to the distance to the subject. The optical system 401 for capturing an image may have optical elements, such as a low-pass filter or an integrated infrared notch filter. When forming the point scattering function, it is necessary to take into account the influence of such an element as a low-pass filter on the characteristics of the OTF function of the optical system 401 for image capture. When using a rotationally asymmetric transfer function, as described with reference to FIG. 23, a rotationally asymmetric transfer function is applied after rearrangement of the OTF functions. A rotationally asymmetric transfer function is, for example, a function in the form of an optical low-pass filter and a pixel window of a certain shape.

The image sensor 402 performs photoelectric conversion of the image of the object and provides an analog electrical signal. This analog electrical signal is converted to a digital signal by an analog-to-digital converter 403, and this digital signal is input to an image processor 404.

The image processor 404 provides predetermined processing of the digital signal to form an image, and this image as an input image is subjected to processing associated with sharpening. In particular, the image processor 404 receives from the state detector 407 information about the image pickup parameters, such as the aperture value, the distance to the subject, and the focal length of the zoom lens when taking the image. The state detector 407 may obtain information about the image pickup parameters directly from the system controller 410 or may receive information about the image pickup parameters, for example, about an optical system 401 for capturing an image from the controller 406 of the optical system for capturing an image.

Then, the image processor 404 provides the image processing described by the flowchart of FIG. 14 or FIG. 23, to sharpen the input image. To provide this processing, the image processor 404 includes, as shown in FIG. 25, an optical transfer function selector 4041, an optical transfer function rearrangement unit 4042, a point spread function generator 4043, a correction signal generator 4044, and a signal adder 4045. These components perform the step of selecting the optical transfer function, the stage of rearrangement of the optical transfer function, the step of generating the point spread function, the step of generating the correction signal and the step of adding (FIGS. 14 and 23).

As shown in FIG. 24, the image pickup apparatus includes a recording unit 408 that stores coefficient data for generating reconfigured OTF functions.

The corrected image (output image) generated by the image processor 404 is stored in a predetermined format on the image recording medium 409. The corrected image is displayed on the 405 display.

The processing sequence is controlled by the system controller 410. The control of the mechanical drive of the optical system 401 for capturing images is provided by the controller 406 of the optical system for capturing images by commands of the system controller 410.

In this embodiment, the optical system 401 for capturing an image is part of an image pickup device, and it can be removable (on the camera body with a removable lens), including an image sensor. When using a camera with a removable lens, coefficient data can be transferred from the storage unit in the lens in the camera body and stored in it. The coefficient data can be shared for different Nyquist frequencies of the image sensor and the characteristics of the optical low-pass filter in the camera body, as described above, which reduces the amount of processed coefficient data.

According to options 4 and 5, the function corresponding to the image pickup parameters and the image height is used as an unsharp mask, so that high-precision sharpening processing for the input image can be provided. In addition, according to each embodiment of the present invention, since the optical transfer functions are placed in two dimensions on the input image before they are converted to the point spread function in the actual space, the optical transfer functions, before being placed in two dimensions, can be shared by many image pickup devices having different pixel Steps. This reduces the amount of data for optical transfer functions that must be stored in advance.

Other embodiments of the present invention

Embodiments of the present invention may also be implemented by a system computer or device that reads and executes computer-executable instructions stored on a storage medium (eg, read-only computer-readable medium) to perform the functions of one or more of the above-described embodiments of the present invention. A computer may comprise one or more central processing units (CPUs), a microprocessor unit (MPUs), or other circuits, and may include a network consisting of separate computers or separate computer processors. Computer-readable instructions can be delivered to a computer, for example, from a network or from a storage medium. A machine-readable medium may include, for example, one or more of the following: a hard disk, random-access memory (RAM), read-only memory (ROM), storage for distributed computing systems, an optical disk (such as a compact disc (CD)) , digital versatile disk (DVD) or Bluray (BD) ™ or the like), flash memory device, memory card, etc.

Although the present invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the exemplary embodiments disclosed herein. The scope of the following claims is to be accorded the broadest interpretation, encompassing all the modifications mentioned, equivalent structures and functions.

Claims (51)

1. An image processing method comprising the steps of: obtaining an image formed by capturing an image through an optical system; selecting a plurality of optical transfer functions in the direction of a particular image height, which depend on the image pickup parameter and image height relative to the reference position; rotate the optical transfer functions relative to the reference position and provide interpolation corresponding to the pixel matrix, so as to arrange the optical transfer functions in the image in two dimensions; generating a point scattering function for each position in said image based on optical transfer functions arranged in two dimensions in this way; forming a filter having an axisymmetric coefficient in the plane of the filter based on the point scattering function; generating a correction signal by applying a filter to the image; and provide sharpening processing for the image based on the correction signal and the image. 2. The image processing method according to claim 1, wherein the Fourier transform of the optical transfer functions located in two dimensions is evaluated to form a point scattering function. 3. The image processing method according to claim 1, wherein the dot scatter function is used as a filter to generate a correction signal based on the difference between the signal obtained by convolution of the image by the filter and the image. 4. The image processing method according to claim 1, wherein the correction signal is multiplied by a constant and added to or subtracted from the image during sharpening processing. 5. The image processing method according to claim 1, wherein the correction signal is adjusted using a control coefficient depending on the position in the image, and added to or subtracted from the image during sharpening processing. 6. The image processing method according to claim 1, in which optical transfer functions corresponding to a plurality of specific discrete positions in an image are calculated as optical transfer functions arranged in two dimensions by rotating and interpolating the optical transfer functions in a specific direction of image height, moreover, the point scattering function corresponding to each position in the entire image area is formed by generating point scattering functions corresponding to specific positions based on optical transfer functions located in two dimensions, and by calculating point scattering functions corresponding to a plurality of positions different from said specific positions , by interpolation based on point scatter functions, moreover, sharpening processing is provided for the image based on dot scatter functions. 7. The image processing method according to claim 1, wherein the transfer function axisymmetric with respect to the reference position is applied to each of the optical transfer functions located in two dimensions. 8. The image processing method of claim 7, wherein the axisymmetric transfer function is an optical low-pass filter or a pixel window of a certain shape. 9. The image processing method according to claim 1, wherein the optical transfer functions in a particular direction of the image height are selected by selecting an optical transfer function coefficient based on the image pickup parameters and the image height. 10. The image processing method according to p. 9, in which the optical transfer function has a real part and an imaginary part, in which the coefficient is the coefficient of both the real part and the imaginary part. 11. The image processing method according to p. 9 or 10, wherein the coefficient selected based on the image pickup parameters and the image height is a coefficient in part of the optical transfer function region, and wherein said coefficient is obtained for the entire region by arranging the coefficient thus selected over the region. 12. An image processing device configured to provide processing for an image formed by capturing an image through an optical system, the image processing device comprising a processor configured to, when processing an image: selecting a plurality of optical transfer functions in a particular direction of the image height, which depend on the image capture parameters and image height relative to the reference position, rotation of the optical transfer functions relative to the reference position and ensure interpolation corresponding to the pixel matrix for the location in two dimensions of the optical transfer functions in the image, the formation of the point scattering function for each position in the image based on the optical transfer functions located in two dimensions, forming a filter having an axisymmetric coefficient on the filter plane, based on the point scattering function, generating a correction signal by applying said filter to the image, and providing sharpening processing for the image based on the correction signal and the image. 13. An image pickup apparatus comprising: an image pickup system configured to form an image by capturing an image through an optical system; and the information processing device according to claim 12, configured to receive an image and provide processing for the image. 14. An image processing method comprising the steps of: obtaining an image formed by capturing an image through an optical system; selecting a plurality of optical transfer functions in the direction of a particular image height, which depend on the image pickup parameter and image height relative to the reference position; rotate the optical transfer functions relative to the reference position and provide interpolation corresponding to the pixel matrix, so as to arrange the optical transfer functions in the image in two dimensions; generating a point scattering function for each position in said image based on optical transfer functions arranged in two dimensions in this way; forming a filter having an axisymmetric coefficient in the plane of the filter based on the point scattering function; and provide sharpening processing for the image by applying said filter to the image. 15. An image processing device configured to provide processing for an image formed by capturing an image through an optical system, the image processing device comprising a processor configured to, when processing an image: selecting a plurality of optical transfer functions in a particular direction of the image height, which depend on the image capture parameters and image height relative to the reference position, rotation of the optical transfer functions relative to the reference position and ensure interpolation corresponding to the pixel matrix for the location in two dimensions of the optical transfer functions in the image, the formation of the point scattering function for each position in the image based on the optical transfer functions located in two dimensions, forming a filter having an axisymmetric coefficient on the plane of the filter based on the point scattering function, and providing sharpening processing for the image by applying a filter to the image. 16. An image pickup apparatus comprising: an image pickup system configured to form an image by capturing an image through an optical system; and the information processing device according to claim 15, configured to receive an image and provide processing for the image.

Images