JP2012156716A - Image processing device, imaging device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress an increase in capacity of data required for image restoration processing.SOLUTION: The program of the present invention makes an information processing device execute the following steps: a specific information acquisition step which acquires specific information specifying an optical transfer characteristic to restore images; a characteristic acquisition step which acquires the optical transfer characteristic specified using the specific information by storage means where a first optical transfer characteristic used in common to a first and a second photographic image which are captured under different photography conditions and a second optical transfer characteristic used to a third photographic image are stored; and a restoration step which generates a restoration image using the optical transfer characteristic acquired by the characteristic acquisition step.

Description

  The present invention relates to a program for performing image recovery (restoration) processing, an image processing apparatus, an image processing method, and an imaging apparatus.

  2. Description of the Related Art Conventionally, as an image processing method for an image (degraded image) in which an influence of an aberration of an imaging optical system of an imaging apparatus appears, there is a process of restoring (restoring) an image using an optical transfer function (OTF). This method is called the term image restoration or image restoration. Hereinafter, processing for correcting or reducing image degradation using the optical transfer function (OTF) of the imaging optical system or a system equivalent thereto will be referred to as image restoration processing.

Patent Document 1 discloses an imaging system that performs image restoration processing of a captured image using a plurality of optical systems with a single device.
Patent Document 2 discloses an image processing system that performs image correction using information added with an index of an imaging device.

JP 2006-31472 A Japanese Patent Registration No. 0402262

  However, in the technique disclosed in Patent Document 1 or 2, when a plurality of imaging devices are used, it is necessary to prepare a recovery filter (hereinafter referred to as an optical transfer characteristic filter) corresponding to each imaging device. For this reason, the required data capacity increases in proportion to the increase in the number of imaging devices.

  Therefore, an object of the present invention is to provide a program, an image processing device, an image processing method, and an imaging device that can suppress the data capacity necessary for image restoration processing.

In order to solve the above problems, the program of the present invention
A specific information acquisition step of acquiring specific information for specifying an optical transfer characteristic for recovering an image;
A memory in which a first optical transfer characteristic used in common for the first and second images taken under different shooting conditions and a second optical transfer characteristic used for the third image are stored. From the means, a characteristic acquisition step of acquiring an optical transfer characteristic specified by the specific information;
A program causing an information processing apparatus to execute a recovery step of generating a recovery image using the optical transfer characteristic acquired in the characteristic acquisition step.

  The present invention can suppress the data capacity required for the image restoration processing by paying attention to the similarity of the optical transfer characteristics.

Schematic configuration diagram of an image processing system Schematic configuration diagram of imaging device Schematic configuration diagram of information processing device Explanatory drawing of the image processing system of Embodiment 1 Illustration of optical transfer characteristics Flow chart showing image processing procedure Schematic configuration diagram of an image processing device Illustration of optical transfer characteristic filter (recovery filter) Explanatory drawing of the image processing system of the modification 1 Explanatory drawing of the image processing system of Embodiment 2. Explanatory drawing of the image processing system of Embodiment 3. Illustration of optical transfer characteristic expression space Illustration of symmetry between azimuth direction and PSF

  First, before describing each embodiment, optical transmission characteristics will be described. The optical transfer characteristic is a characteristic (imaging characteristic) related to imaging of the optical system. For example, a point spread function (hereinafter referred to as PSF), an optical transfer function (hereinafter referred to as OTF), an aberration (for example, wavefront aberration), a pupil function, and the like. In this specification, an optical transfer characteristic filter (recovery filter) used for image restoration processing is also defined as one of the optical transfer characteristics reflecting the imaging characteristics. EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated in detail based on attached drawing.

(Embodiment 1)
FIG. 1 is a schematic configuration diagram of an image processing system having an information processing apparatus in which a program according to the present invention is installed and an imaging apparatus connectable (communicable) with the information processing apparatus. The imaging optical systems 10, 12 and 14 form an image of a subject on an imaging element (light receiving element) such as a CCD or CMOS sensor of the imaging devices 11, 13 and 15. The image pickup element may be any element that can convert an optical image based on transfer characteristics of an optical system or a system equivalent thereto into an image. The imaging optical system may be an imaging lens configured integrally with the camera body (imaging apparatus body) or an interchangeable lens that can be attached to and detached from the camera body. 1 (FIG. 1) is described as an interchangeable lens. In the first embodiment, the imaging apparatuses 100, 101, and 102 include the imaging optical system and the imaging apparatus main body. The information processing apparatus 200 can transmit and receive data such as images to and from the imaging apparatuses 100, 101, and 102. The information processing apparatus 200 performs image recovery processing on the received (acquired) image, and stores the recovered image. Generate.

  Next, a schematic configuration of the imaging apparatus will be described with reference to FIG. The arrows in FIG. 2 represent main information transmission paths. The imaging optical system 10 includes a diaphragm 10a and a focus lens 10b, and forms an image of an object (optical image) on an imaging element. The imaging element 202 (photoelectric conversion element, light receiving element) is a CCD or CMOS sensor that converts an image of a subject into an electrical signal. An analog signal image output from the image sensor 202 is converted into a digital signal image by an A / D (analog-digital) converter 203.

  The image processing unit 204 performs image processing such as processing for correcting coloring of the image converted by the A / D converter 203 and processing for reducing blur. The display unit 205 is a liquid crystal display, an organic EL display, or the like, and can display an image processed by the image processing unit 204 or an image recorded in the storage unit 208 or the image recording medium 209. The imaging optical system control unit 206 is an autofocus mechanism, a manual focus mechanism, or the like for performing focus adjustment according to the object distance (subject distance), and controls the aperture 10a and the focus lens 10b of the imaging optical system 10.

  The state detection unit 207 detects shooting conditions (shooting state) such as an F number (aperture diameter), a focal length (zoom position), and an object distance. The system controller 210 controls the entire system of the imaging apparatus, and also performs control to form an image written in the storage unit 208 into a file and record it on the image recording medium 209.

  Next, a schematic configuration of the information processing apparatus 200 illustrated in FIG. 1 will be described with reference to FIG. The CPU 310 of the information processing device 200 controls the entire information processing device and performs image processing in accordance with instructions of a program stored in a ROM or the like. The storage unit 320 stores various data such as images and optical transmission characteristics, and the CPU 310 reads and writes the storage unit 320 as appropriate. The input unit 340 is a keyboard, a mouse, or the like, and is a unit for receiving information from outside and giving information and instructions to a program being executed. The output unit 350 is a monitor, a display, or the like, and appropriately displays an image subjected to image processing. The CPU 310, the storage unit 320, the input unit 340, and the output unit 350 are connected to each other via a bus 360. The program of the present invention provided on a recording medium such as a DVD-R or CD-R is stored in the storage means 320, and the CPU 310 causes the information processing apparatus 200 to execute image processing in accordance with the instruction of the program according to an instruction of the operator. You may comprise as follows. Alternatively, the CPU 310 may perform image processing while a network I / F 370 connected to the bus 360 communicates with an information processing apparatus or the like that exists in a remote place to exchange programs and data.

  So far, each of the imaging apparatus and the information processing apparatus has been described. Next, processing of the entire image processing system shown in FIG. 1 will be described with reference to FIG. The imaging devices 100 and 101 that can communicate with the information processing device 200 are imaging devices having a positive lead type imaging optical system (lens), and the imaging device 102 is an imaging device having a negative lead type imaging optical system.

  First, the information processing apparatus 200 in which the program of the present invention is installed includes an image 110 (first image) captured by the imaging apparatus 100 and an optical transfer characteristic filter (recovery optical transfer characteristic) used for subsequent image restoration processing. Is obtained. The specific information is, for example, EXIF information, and is information indicating imaging conditions (including differences between imaging devices) including the focal length of the imaging optical system, the F number, the identification code of the imaging device, and the like.

  Next, the CPU 310 of the information processing apparatus determines the optical transfer characteristic filter No. specified by the specifying information 120. 1 and the image 110 and the optical transfer characteristic filter No. 1 are obtained. A restored image 110R (not shown) of the image 110 is generated by performing one two-dimensional convolution operation process. Details of this image restoration processing will be described later. Although the recovered image 110R is generated by the above processing, the CPU 310 may perform signal processing on the recovered image 110R and output the signal-processed image to the output unit 350 as an output image. This signal processing includes, for example, demosaicing, white balance adjustment, edge enhancement processing, noise reduction processing, distortion aberration correction for performing geometric aberration correction, magnification chromatic aberration correction, shading correction, and the like.

  Next, the imaging device 101 and the information processing device 200 will be described. Similar to the imaging apparatus 100, the CPU 310 acquires the image 111 (second image) and the specific information 121 captured by the imaging apparatus 101. The CPU 310 of the information processing apparatus then transmits the optical transfer characteristic filter No. specified by the specific information 121. 1 and the image 111 and the optical transfer characteristic filter No. 1 are obtained. A restored image 111R (not shown) of the image 111 is generated by performing one two-dimensional convolution operation processing.

  Conventionally, in the disclosed technique, the storage means stores optical transfer characteristic filters FA and FB corresponding to the respective imaging devices A and B in order to recover the images captured by the imaging devices A and B. Has been. On the other hand, in the first embodiment, the optical transfer characteristic filter no. 1 (first optical transfer characteristic) is stored. This makes it possible to reduce the data capacity necessary for image restoration.

  The present inventor has found that imaging apparatuses having similar structures such as a similar lens type and a similar focus type exhibit similar optical transfer characteristics under the same imaging conditions through optical design. For example, the point image intensity distributions of the positive lead type imaging device 100 and the imaging device 101 are shown in FIGS. 5A and 5B, and they show the same tendency. Even if the shooting conditions are not the same, some optical transmission characteristics may be similar if the zoom type is the same. For example, the same zoom type imaging optical system having the same focal length at the wide-angle end and different zoom ratios. Such an imaging apparatus exhibits similar optical transfer characteristics even when the telephoto end imaging conditions are different.

  That is, the invention of the first embodiment is an invention made by paying attention to the similarity of the optical transfer characteristics described above, and acquires specific information for specifying the optical transfer characteristics for recovering an image. Next, a first optical transmission characteristic used in common for the first and second captured images captured under different imaging conditions, and a second optical transmission characteristic used for the third captured image The optical transfer characteristic specified by the specific information is acquired from the storage means in which is stored. Then, the information processing apparatus is caused to execute a recovery step of generating a recovery image using the optical transfer characteristic acquired in the characteristic acquisition step. This makes it possible to reduce the data capacity while maintaining the quality of the recovered image, rather than storing separate optical transfer characteristic filters corresponding to individual imaging devices.

  More preferably, the storage means 320 may store an optical transmission filter patterned for each type such as a zoom type, a focus type, and an image stabilization type. Then, even when the negative lead type imaging apparatus 102 is connected to the information processing apparatus 200 (becomes communicable), it is not necessary to newly increase the optical transmission filter.

Hereinafter, processing of the information processing apparatus 200 for the image 112 (third image) captured by the negative lead type imaging apparatus 102 will be described.
The CPU 310 acquires the image 112 and specific information 122 captured by the imaging device 102. Next, the CPU 310 of the information processing apparatus determines the optical transfer characteristic filter No. specified by the specifying information 122. 2 (second optical transfer characteristic) is acquired. The image 112 and the optical transfer characteristic filter No. 2, a restored image 112R (not shown) of the image 112 is generated. Here, the optical transfer characteristic filter No. 2 (second optical transfer characteristic) is an optical transfer characteristic filter No. This is a filter (optical transfer characteristic) different from 1 (first optical transfer characteristic).

More preferably, the image restoration can be performed more effectively if the imaging device is designed so that the optical transmission characteristic of the imaging device approaches the optical transmission characteristic stored in advance.
In the present embodiment and the following embodiments, for the sake of brevity, a mode in which one optical transfer characteristic filter is acquired for one image has been described. However, a plurality of optical transfer characteristic filter groups are described. (Set) may be used for one image. When a plurality of optical transfer characteristic filter groups are used for one image, the CPU 310 may acquire an optical transfer characteristic filter group that can be commonly used by the imaging devices 100 and 101, for example.

  When using a plurality of optical transfer characteristic filters for one image, the CPU 310 may acquire the optical transfer characteristic filter for each pixel of the image. When obtaining an optical transfer characteristic filter for each pixel, if one or more specific pixels are extracted and an optical transfer characteristic filter corresponding to the extracted pixel is obtained, the optical transfer characteristic is obtained for every pixel. This is preferable because the processing speed is improved as compared to obtaining a filter.

  Furthermore, the same type of lens has similar aberration tendencies for each pixel position under the same shooting conditions. For example, curvature of field occurs at the wide-angle end of a negative lead zoom lens, and axial chromatic aberration is large at the telephoto end of a positive lead zoom lens used in a high-power zoom lens. As described above, since the same type of lens often has the same deterioration characteristic at each pixel position of one image, the optical transfer characteristic filter (recovery filter) at each pixel position used for each same type. It is also possible to prepare a set (group). As a result, it is not necessary to acquire a filter for each pixel position, so that the processing speed can be increased.

  Further, the storage unit 320 may or may not store the optical transfer characteristic filter before the program is executed. In the latter case, for example, an optical transfer characteristic filter that can be commonly used by a plurality of imaging devices is recorded on a recording medium in which a program is stored. Then, these optical transfer characteristic filters may be stored in an information processing apparatus in which the program is installed or an external storage device of the image processing apparatus.

  In the first embodiment, the optical transfer characteristic filter itself is stored in the storage unit 320. However, a point spread function, an optical transfer function, or the like may be stored. In that case, the CPU 310 may perform Fourier transform or inverse Fourier transform on the point spread function or optical transfer function to generate an optical transfer characteristic filter. In this specification, the recovery filter and the optical transfer characteristic filter are synonymous, and the optical transfer characteristic includes the recovery filter as described.

  Now, a processing flow of a program that the CPU 310 causes the information processing apparatus 200 to execute will be described with reference to FIG. First, in step S10 (specific information acquisition step), the CPU 310 acquires specific information for specifying an image and an optical transfer characteristic filter. Subsequently, in step S20 (optical transfer characteristic acquisition step), the CPU 310 acquires an optical transfer characteristic filter (optical transfer characteristic) specified by the specific information. Subsequently, in step S30 (recovery step), the CPU 310 performs a process of convolving the optical transfer characteristic filter on the image acquired in step S10 in real space. Note that it is not always necessary to acquire the recovery target image in step S10, and the recovery target image may be acquired in step S30 or in the previous step of step S30. It should be noted that the step of executing the signal processing described above may be appropriately executed as necessary, and thus the signal processing step is not included.

  In the first embodiment, the CPU 310 of the information processing apparatus executes each step according to the instructions of the program. However, the effects of the present invention can be obtained even if each step or a part thereof is configured by hardware. . FIG. 7 shows a schematic configuration diagram in the case where the image processing apparatus is configured by hardware as an example. The description of the image pickup apparatus that can communicate with the image processing apparatus 700 overlaps with the description in FIG.

  The image processing apparatus 700 includes an optical transfer characteristic acquisition unit 710, an image restoration unit 720, a signal processing unit 730, and a storage unit 750. First, the image processing apparatus 700 acquires specific information that can specify an image to be recovered and an optical transfer characteristic filter. The optical transfer characteristic acquisition unit 710 acquires the optical transfer characteristic filter specified by the specific information from the storage unit 750 from the storage unit 750. The image restoration unit 720 generates a restored image by convolving the acquired optical transfer characteristic filter with the image. The signal processing unit 730 performs image processing such as demosaicing on the recovered image. A desired restored image can be obtained by the above means.

  As another example of using the image processing system as described above, it is also possible to execute an image restoration process at the time of printing by using an arithmetic processing unit mounted on a printer instead of the CPU 310 of the information processing apparatus 200. is there.

  As described above, the inventor has focused on the similarity (indicating similar imaging characteristics) in the optical transfer characteristics according to the classification of the imaging device. According to the invention made based on this viewpoint, an optical transmission characteristic (representative optical transmission characteristic) representing similar optical transmission characteristics is stored in the storage means in advance, and recovery processing is performed using the optical transmission characteristics. Thus, it is possible to obtain a high-quality restored image while reducing the data capacity.

  Note that the identification information may be information that can identify the optical transfer characteristics used in the recovery process, and may be imaging conditions themselves or a part of imaging conditions. Shooting conditions include focal length, Fno, object distance, image height, information that can identify the imaging device, information that can identify the optical system, zoom position, focus state with respect to the object distance, aperture state, and lens of the vibration-proof lens Examples of the position, the number of lens groups, and information on the lens structure. Furthermore, since the aperture characteristics also vary depending on the pixels of the image sensor of the image pickup apparatus, information regarding this image sensor may be used as the specific information. Alternatively, when a different low-pass filter is used for each imaging device, information regarding this can also be used as specific information for specifying the optical transfer characteristic.

  In the first embodiment, the imaging apparatus outputs specific information for directly specifying the optical transfer characteristic. However, the CPU 310 generates information that can specify the optical transfer characteristic from the information on the imaging apparatus and the information on the imaging condition itself. May be used as specific information. In this case, the specific information may be generated on the imaging device side or may be generated on the image processing device side. These specific information may be added to a part of the image, or the specific information may be acquired via another device or a network.

  Moreover, although Embodiment 1 described about the method in which CPU acquires the optical transfer characteristic filter specified by specific information from the optical transfer characteristic filter memorize | stored in the memory | storage means, it is not restricted to this. For example, the storage unit stores a lookup table (LUT) that associates the specific information with the address of the optical transfer characteristic filter stored in the storage unit. Then, the CPU may obtain an optical transfer characteristic filter corresponding to the specific information with reference to the lookup table. That is, the specific information in the present specification may be information that directly specifies the optical transfer characteristic, or may be information that indirectly specifies the optical transfer characteristic.

In the first embodiment, the case where the imaging optical system and the imaging apparatus main body are separate has been described. However, the lens and the imaging device may be integrated into a compact camera or the like.
In the first embodiment, the CPU 310 of the information processing apparatus executes all the steps. However, even if the CPUs of the imaging apparatus and the information processing apparatus execute some steps, the effect of the present invention is achieved. Obtainable. That is, the effect of the present invention can be obtained as long as the image processing system executes image restoration processing based on specific information and images output from a plurality of imaging devices.

  Here, an image (image data) handled by the present invention will be briefly described. The image handled by the present invention has, for example, RGB color components and a plurality of components expressed by a color space. The component expressed in the color space is, for example, lightness, hue, and saturation expressed in LCH, or luminance and color difference signals expressed in YCbCr.

  The image may be a mosaic image having a signal value of one color component for each pixel, or a demosaic image having a signal value of a plurality of color components for each pixel by performing color interpolation processing (demosaicing processing) on the mosaic image. good. This mosaic image is also called a RAW image as an image before various image processing such as color interpolation processing (demosaicing processing), signal value conversion called gamma conversion, and image compression known by JPEG. In particular, when obtaining a plurality of pieces of color component information with a single-chip image sensor, a mosaic having a color filter with different spectral transmittances arranged in each pixel and a signal value of one color component in each pixel as described above. An image will be acquired. In this case, an image having a plurality of color component signal values in each pixel can be generated by performing the above-described color interpolation processing. On the other hand, when using a multi-plate, for example, three-plate image sensor, color filters having different spectral transmittances are arranged for each image sensor to obtain image signal values of different color components for each image sensor. In this case, since each pixel corresponding to each imaging element has a signal value of each color component, an image having a plurality of color component signal values in each pixel without performing color interpolation processing in particular. Can be generated.

  Further, the image can be attached with photographing conditions such as a focal length of the lens, a diaphragm, and an object distance. When a series of processing from imaging to output is performed with one closed imaging device, the image can be acquired in the device without adding imaging conditions to the image. In that case, specific information can be acquired from the state detection unit 207 (FIG. 2), for example. However, when a RAW image is acquired from an imaging apparatus and desired image processing or signal processing is performed by a separate image processing apparatus or information processing apparatus, it is preferable to add information on imaging conditions to the image.

  Next, an outline of the above-described image restoration process will be described. Image restoration is a process for reducing aberrations appearing in an image, and aberrations include, for example, spherical aberration, coma aberration, field curvature, astigmatism, and the like of an imaging optical system. Due to these aberrations, an image (point image) that should originally be formed at one point is formed on the image sensor (light receiving element) as a widened blurred image. The blur due to this aberration is optically represented by a point spread function (PSF) or a point spread intensity distribution.

When the original image is f (x, y), the captured image (degraded image) is g (x, y), and the point spread function (PSF) is h (x, y), the following equation is established. However, * shows a convolution integral and (x, y) shows the coordinate on an image.
g (x, y) = h (x, y) * f (x, y) (Formula 1)
In the first embodiment, the original image f (x, y) corresponds to the restored image, and the captured image g (x, y) corresponds to the image output from the imaging device. When Formula 1 is Fourier-transformed and converted into a display format in the frequency space, the right side of Formula 1 is represented by the product of the respective frequencies as shown in Formula 2.
G (u, v) = H (u, v) · F (u, v) (Expression 2)
H (u, v) in Expression 2 is a Fourier transform of the point spread function h (x, y), in other words, an optical transfer function (OTF). G (u, v) and F (u, v) are obtained by Fourier transform of g (x, y) and f (x, y), respectively. (U, v) is a coordinate in a two-dimensional frequency space, that is, a frequency. Further, · is a symbol indicating multiplication.
In order to obtain the restored image F (u, v) from the captured image G (u, v), both sides of Equation 2 may be divided by H.
G (u, v) / H (u, v) = F (u, v) (Equation 3)
This F (u, v), that is, G (u, v) / H (u, v) is subjected to inverse Fourier transform and returned to the real space (real surface), whereby a restored image f (x, y) is obtained. .
Here, if the inverse Fourier transform of 1 / H (u, v) is R (x, y), Equation 3 can be rewritten as Equation 4.
g (x, y) * R (x, y) = f (x, y) (Formula 4)
That is, as shown in Expression 4, a recovery image can be obtained by performing convolution integration processing on R (x, y) on a captured image in real space.

  This R (x, y) is a recovery filter (optical transfer characteristic filter). Generally, when the image is two-dimensional, this recovery filter is a two-dimensional filter having taps (cells) corresponding to the respective pixels of the image as shown in FIG. Further, since the recovery accuracy generally increases as the number of taps of the recovery filter increases, the number of taps that can be realized may be set according to the required image quality, image processing capability, aberration characteristics, and the like. Since this recovery filter needs to reflect the characteristics of aberrations, it is different in nature from a conventional edge enhancement filter (high-pass filter) of about 3 × 3 taps.

  FIG. 8B is a cross-sectional view of the recovery filter of FIG. 8A, where the horizontal axis is the tap and the vertical axis is the tap value. The distribution of values (coefficient values) possessed by each tap plays a role of returning the signal value spatially spread by the aberration to the original point. As a method for creating this recovery filter, for example, there is a method in which an optical transfer function (OTF) of an imaging optical system is calculated or measured, and a function based on the inverse function is obtained by inverse Fourier transform.

  Note that the optical transfer characteristics can include not only the imaging optical system but also factors that affect the optical transfer characteristics during the imaging process. For example, characteristics of an optical low-pass filter having birefringence, an aperture shape of an image sensor of a light source, and spectral characteristics of various wavelength filters can be given. An optical low-pass filter having birefringence suppresses a high-frequency component with respect to the frequency characteristic of the optical transfer characteristic, and the image pickup element causes a blur image on the image pickup element to change due to the inclination thereof. It is one of. It is more desirable to perform image restoration processing based on the broad optical transfer characteristics including these.

  Also, the advantage of convolution processing of the recovery filter on the image in real space is that it is not necessary to perform Fourier transform or inverse Fourier transform of the image in the image recovery processing step, so that faster processing is possible. is there. The filter shown in FIG. 8A is a square array filter (the number of vertical and horizontal taps is the same), but is not limited thereto, and the number of vertical and horizontal taps of the recovery filter can be arbitrarily changed.

  As described above, the ideal recovery filter has been described using an equation. However, since an actual image has a noise component, using a recovery filter based on the inverse function of the optical transfer function (OTF) as described above causes recovery. The noise component will be amplified. In order to suppress the amplification of the noise component, an applied recovery filter such as a Wiener filter may be used.

  In addition, when performing the image restoration process, the aberration to be restored is not particularly limited, but the aberration to be restored by the image restoration process is Seidel aberration (spherical aberration, coma aberration, astigmatism, field curvature). , Distortion aberration), spherical aberration, coma aberration, astigmatism, and field curvature. It is preferable not to take into account aberrations that require a large amount of correction, such as distortion. This is because as the geometric correction amount increases, the coefficient value of the recovery filter fluctuates more and undesired artifacts such as ringing tend to appear in the recovered image.

  Other effects obtained by setting the target of recovery performed by the image recovery processing to spherical aberration, coma aberration, astigmatism, and field curvature are data rather than including a distortion correction component in the recovery filter that is two-dimensional data. The effect that the amount can be reduced is obtained. The reason is that the data necessary for performing geometric distortion correction only needs to be one-dimensional data representing the degree of image expansion / contraction. Therefore, the recovery filter, which is two-dimensional data, includes a component for correcting distortion. This is because the data capacity can be reduced. This is particularly effective when an image restoration process is performed with a specific image processing apparatus on an image having various aberrations as in the first embodiment.

  The image restoration process described above can also be applied to an apparatus that does not have an imaging optical system. For example, a scanner (reading device) or an X-ray imaging device that performs imaging by bringing an imaging element into close contact with a subject surface. These do not have an imaging optical system typified by a lens, but the output image deteriorates due to image sampling or the like by the imaging device. Since this deterioration characteristic is a transfer characteristic (transfer function) of the apparatus, it does not depend on the imaging optical system, but corresponds to the above optical transfer characteristic. Therefore, a recovery image can be generated based on transfer characteristics without having an imaging optical system.

(Modification 1)
A modification of the first embodiment will be described with reference to FIG. In the first modification, the optical transfer characteristic correction unit 950 performs an operation on the optical transfer characteristic filter acquired by the optical transfer characteristic acquisition unit 930, so that it is closer to an ideal recovery filter corresponding to actual shooting conditions. A correction recovery filter (corrected optical transfer characteristic) is generated. In the first modification, unlike the first embodiment, each processing subject is described as being performed by different means. However, an arithmetic unit in the image processing apparatus 990 may perform each processing.

  The optical transfer characteristic acquisition unit 930 selects a specific pixel from each pixel of the image in order to perform an interpolation calculation process for each pixel of the image acquired by the imaging device. The number of pixels to be selected may be plural or singular. Also. For a specific pixel, a common pixel position may be determined for each shooting condition, and the pixel position may be selected, or the pixel position to be selected may be changed based on specific information such as EXIF information. Next, the optical transfer characteristic acquisition unit 930 acquires an optical transfer characteristic filter (hereinafter referred to as a recovery filter) corresponding to the selected specific pixel. Thus, by obtaining a recovery filter corresponding to a specific pixel, the processing speed can be improved as compared with the case where the step of obtaining the recovery filter for each pixel is repeated for all pixels.

  Next, the optical transfer characteristic correcting unit 950 generates a recovery filter at an arbitrary pixel position by bilinear interpolation using the recovery filter acquired for each specific pixel. By using a correction recovery filter generated by interpolation, the accuracy of image recovery can be improved. This is because the correction recovery filter generated by interpolation generally becomes a recovery filter closer to the actual optical transfer characteristic than the recovery filter acquired for each specific pixel before interpolation.

Then, the image restoration unit 960 generates a restored image by performing a two-dimensional convolution process (calculation) between the correction restoration filter corrected by the optical transfer characteristic correction unit 950 and the image. The signal processing unit 970 performs various signal processing on the recovered image and outputs an output image.
By performing the processing as described above, it is possible to improve the image restoration accuracy while reducing the amount of data stored in advance.

  The processing for acquiring a recovery filter for a specific pixel of an image and generating a correction recovery filter for a pixel between the first specific pixel and the second specific pixel has already been described as the interpolation calculation. Generally, an image acquired via an optical system has a degree of image deterioration (aberration) that increases from the center of the image toward the periphery of the image. In particular, coma, astigmatism, and field curvature greatly occur in the peripheral portion of the image. For this reason, the recovery filter used for the interpolation calculation is not a recovery filter for specific pixels evenly distributed in the screen, but a recovery filter for specific pixels distributed more finely in the peripheral part than in the central part of the image. prepare. As a result, more accurate image recovery can be performed.

  The number of recovery filters in a quarter area in one screen is preferably about 5 to 60. Below this lower limit value, the difference between the corrected recovery filter after the calculation by the optical transfer characteristic correcting means and the ideal recovery filter becomes large, and the image recovery accuracy decreases. If the upper limit is exceeded, the amount of calculation performed by the calculating means increases (the number of times the recovery filter is acquired increases), so the processing speed decreases. In other words, more recovery filters are required in regions where aberrations appear greatly (the amount of aberration changes greatly). Therefore, the present invention prepares mainly a recovery filter having a large spread of the point image intensity distribution, which is stored in advance in the storage area. This device improves the recovery accuracy and processing speed of image recovery. In the above description, the recovery filter corresponding to the 1/4 region of the image is prepared in consideration of the rotational symmetry (line symmetry) of the imaging optical system, but the present invention is not limited to this.

  In addition, as the interpolation calculation, any interpolation calculation such as linear interpolation, bilinear interpolation, bicubic interpolation, or the like can be used according to the processing speed and interpolation accuracy. Interpolation calculation determining means for determining the type of interpolation calculation may be provided. The type of interpolation calculation is determined by a balance between the accuracy of the interpolation process and the time required for the calculation. For example, when it is desired to prioritize the accuracy of interpolation processing, bicubic interpolation is selected, and when priority is given to processing speed, bilinear interpolation is selected.

  In addition, although the interpolation calculation for obtaining the recovery filter value in the pixel between the first specific pixel and the second specific pixel by the interpolation processing has already been described, the interpolation calculation is not limited to the pixel position. Good. For example, the interpolation calculation may be performed with respect to the focal length, Fno, object distance, and the image stabilization state (the image stabilization lens group shift and tilt states). For example, with respect to the focal length, a specific focal length is selected from discrete focal lengths from the wide end to the tele end of the optical system. Next, the optical transfer characteristic of the selected focal length is acquired. When the focal length included in the photographing information is included in the specific focal length, image recovery is performed using the acquired optical transfer characteristics. If it is not included, an optical transfer function of the photographing focal length is created by interpolation using the optical transfer characteristics of a specific focal length on the wide side and the tele side, and image restoration is performed using this.

  The above is interpolation for one axis of the focal length, but in the space with discrete focal length, Fno, object distance, and anti-vibration state as axes, the optical transfer characteristic of a specific coordinate is used to set the arbitrary coordinate. An optical transfer characteristic may be created. For example, in the case of a three-dimensional space (focal length, Fno, object distance), the optical transfer characteristic of an arbitrary coordinate is created by plane interpolation using the optical transfer characteristic of a specific coordinate surrounding the arbitrary coordinate. I can do it.

  In addition to the interpolation calculation, another calculation for the optical transfer characteristic correction unit 950 to perform the calculation for the recovery filter includes a calculation for correcting each recovery filter acquired by the optical transfer characteristic acquisition unit 930. For example, an operation for proportionally multiplying the value of the recovery filter can be mentioned. Since this calculation can change the luminance level of the image after recovery by the recovery filter, it can be used when white balance is to be adjusted.

  As another calculation, if another recovery filter is convolved (convolutionally integrated) with the recovery filter acquired by the optical transfer characteristic acquisition means 930, the recovery function has a composite action of both recovery filters. It is also possible to create a filter.

  Further, as another calculation, there is a function calculation calculation that obtains a corrected recovery filter by using a recovery filter corresponding to a specific pixel. In the functionalization calculation, a coefficient of a function having a pixel position as a variable is obtained by fitting using a recovery filter for discrete pixel positions. By substituting an arbitrary pixel position into this function, a recovery filter at that position can be generated. This calculation is effective in reducing the processing time because once a function is created, a recovery filter can be generated simply by substituting variables thereafter.

  The functionalization calculation will be described more specifically. Based on the recovery filter acquired by the optical transfer characteristic acquisition means 930, a functionalization operation using each pixel position as a variable is executed. First, recovery filters are acquired for a plurality of pixel positions. In this case, a common pixel position may be determined for each photographing condition, or the pixel position may be changed based on specific information. Since the acquired recovery filter is acquired for each determined pixel position, fitting is performed using a model function having the pixel position as a variable using these recovery filters. Hereinafter, this model function is referred to as a fitting function. By substituting an arbitrary pixel position into this function, a recovery filter at that position can be generated. The recovery filter fitted at an arbitrary pixel position is closer to the actual recovery filter than the recovery filter nearest to the pixel position among the discrete recovery filters stored in advance in the storage area. As a result, the accuracy of the image restoration process is improved. If the fitting function is used, the restoration filter for each pixel is determined by substituting the pixel position into the function, so that even when the processing capability of the CPU or the like is not high, it is possible to perform image restoration processing at high speed. . It is also possible to execute this processing for the focal length, the object distance, Fno, and the vibration-proof lens position.

  The above calculation is an example, and the calculation according to the present invention can be performed to bring the recovery filter acquired by the optical transfer characteristic acquisition unit 930 closer to the ideal recovery filter or to improve the processing speed. It is not restricted to the said calculation.

(Embodiment 2)
Hereinafter, Embodiment 2 will be described with reference to FIG. In the second embodiment, a series of processing is performed in the imaging apparatus. The difference from the first embodiment is that the specific information is information capable of identifying the focus type, and that an appropriate optical transfer characteristic filter is stored in the storage unit for each focus type.

  The imaging optical system attached to the imaging apparatus of Embodiment 2 is a negative lead type rear focus lens (first imaging optical system) (not shown). The storage unit 208 of the imaging device has an optical transfer characteristic filter group No. that can generate a recovered image satisfactorily even if different negative lead type rear focus lenses (second imaging optical system) are attached to the imaging device. . 1 (third optical transfer characteristic) is stored. In addition, the optical transfer characteristic filter group No. corresponding to a plurality of negative lead type front focus lenses (third imaging optical system) is provided. 2 is stored. The optical transfer characteristic filter group No. 1, no. Reference numeral 2 denotes a filter group in which a plurality of filters are combined into one set.

  Since the filter is two-dimensional data, it is difficult to hold a large amount. Of course, if a large number of filters can be held, it is possible to improve the accuracy of recovery. However, it is necessary to devise in order to efficiently store the filters in a limited memory capacity such as an imaging device. Therefore, like the first embodiment, the inventor makes use of the knowledge through various optical designs that the optical transfer characteristics are similar for each focus type of the lens, and if the focus type is the same, the identification code of the lens is different. Invented an imaging device that uses the same filter. As a result, it is possible to reduce the capacity of the memory necessary for image restoration while suppressing the deterioration of the image quality of the restored image. In particular, the imaging apparatus is useful because the memory capacity is often limited as compared with the information processing apparatus.

  Preferably, a plurality of filters are stored for each of the photographing conditions of the focal length, Fno, and photographing distance, and the image height under the photographing conditions.

  Hereinafter, the operation of the imaging apparatus of FIG. 10 will be described. First, information identifying the focus type of the interchangeable lens stored in the image a2 captured by the negative lead type rear focus lens of the image capturing apparatus 100 and the memory of the interchangeable lens is acquired as the specific information a1. In the second embodiment, if the imaging apparatus is integrated with an optical system such as a compact camera, focus type information stored in advance in the memory of the compact camera may be used as the specific information. Of course, in the system as in the first embodiment, the imaging apparatus may write the lens focus type information into the EXIF information attached to the image.

  In the image processing unit 204, the imaging apparatus 100 refers to the specific information a <b> 1 and stores the optical transfer characteristic No. from the storage unit 208. 1 is acquired. Next, the imaging apparatus applies the optical transfer characteristic filter No. 1 to the image a2. A restored image is generated by convolving 1. Finally, the imaging apparatus outputs an output image a3 by performing camera signal processing such as demosaicing on the recovered image. Note that the order of recovery processing and camera signal processing may be changed.

The storage unit 208 includes an optical transfer characteristic filter group No. The filter in 1 preferably stores a filter obtained by averaging optical transfer characteristic filters of a plurality of negative lead type rear focus type lenses. As a result, when recovering images picked up by various negative lead type rear focus lenses, the deviation from the ideal optical transfer characteristic filter is reduced, and the accuracy is improved.
The processing is the same even if a negative lead type front (front lens) focus lens is attached to the imaging apparatus 100 in place of the rear focus type interchangeable lens.

  In addition, the imaging device may perform a correction process on the optical transfer characteristic filter acquired in the imaging device.

  The optical transfer characteristics stored in the storage unit 208 of the imaging apparatus are not limited to filters having two-dimensional pixel data, and may be optical transfer functions, point image intensity distribution functions, wavefront aberrations, pupil functions, and the like. However, when the capacity of the main memory (RAM capacity) in the imaging apparatus is small, high-speed processing cannot be performed as with an information processing apparatus with a large capacity. Therefore, data stored in the storage unit 208 is an optical transfer characteristic filter. It is preferable that If the stored optical transfer characteristic is a filter, it is not necessary to perform an operation with a large amount of calculation such as Fourier transform when executing the image restoration process, and thus high-speed processing is possible.

  The filter group in this specification is a set of filter groups applied to one image, and the plurality of filter groups means that a plurality of sets are stored in the storage unit. is there.

(Embodiment 3)
With reference to FIG. 11, an image processing system in which the program of the present invention is installed will be described as a third embodiment. The image processing system according to the third embodiment is different from the first embodiment in that the storage unit 860 is included in the external device 880 and the optical transfer characteristic stored in the storage unit 860 is not the recovery filter itself.
The imaging devices 1 and 2 are imaging devices having a negative lead type imaging optical system. The imaging device 3 is an imaging device having a positive lead type imaging optical system.

  The storage means 860 stores in advance optical transfer functions No. 1 that can be commonly used by the imaging apparatus 1 and the imaging apparatus 2 based on the similarity of the optical transfer functions. 1 (optical transfer characteristics) and the optical transfer function No. corresponding to the imaging device 3. 2 (optical transfer characteristic) is stored.

  Here, how to store the optical transfer function of the storage unit 860 will be described. The storage means 860 stores optical transfer characteristics discretely arranged in an optical transfer characteristic expression space having a plurality of optical transfer characteristic expression vectors. Here, the optical transfer characteristic expression vector is a vector that can express the property of the optical transfer characteristic.

  Examples of the optical transfer characteristic expression vector include the following.

  From the viewpoint of aberration, the imaging characteristics under a certain photographing condition can be expressed by adding a spherical aberration component, an astigmatism component, a coma aberration component, and the like. A mathematical description of this is the Zernike polynomial, which is known as wavefront aberration expressed by an orthogonal coordinate space.

  Further, when viewing the aberration from the viewpoint of the point spread (PSF), spherical aberration is a rotationally symmetric spreading component of the point spread, and astigmatism is two directions orthogonal to the point spread (for example, meridional direction). And the sagittal direction) aspect ratio component.

  Further, coma aberration can be expressed as an asymmetric component in the symmetry axis of the point image intensity distribution that is line symmetric. Examples of the reference point of the asymmetric component include the intersection of the light beam passing through the center of the light beam and the imaging surface, the intersection of the light beam passing through the center of the stop and the imaging surface, or the center of gravity of the point image intensity distribution. An example of a point image intensity distribution in which coma aberration has occurred is shown in FIG. The coma aberration is asymmetric with respect to the x1 axis and symmetric with respect to the x2 axis. In FIG. 13, x1 and x2 indicate coordinate axes that pass through the center of the point image distribution intensity on the image plane and are orthogonal to each other, and θ indicates an azimuth angle with respect to the x1 axis. The center of the point image intensity distribution is the intersection of the principal ray at the reference wavelength and the image plane, or the center of gravity of the point image at the reference wavelength.

  The lateral chromatic aberration can be expressed as a center-of-gravity shift component of point images of different color components. The component is, for example, a color component such as R, G, or B.

  Expression based on such aberration or point image intensity distribution is defined as an optical transfer characteristic expression vector. Further, if the size of the optical system is doubled, the size of the point image is also doubled. Therefore, a component that defines the size of the point image as a proportional multiple can also be defined as one of the optical transfer characteristic expression vectors.

  FIG. 12 schematically shows an optical transfer characteristic expression space. FIG. 12A is a schematic diagram of a space expressing optical transmission characteristics with the spherical aberration component, astigmatism component, and coma aberration component as coordinate axes. FIG. 12B shows a part of a cross section parallel to the axes of the astigmatism component and the coma aberration component when the astigmatism is almost zero. The horizontal axis represents the coma aberration component, and the vertical axis represents the coma aberration component. The astigmatism component and the point image intensity distribution (PSF) arranged discretely are shown.

  The storage unit 860 stores a point image intensity distribution (optical transfer characteristic) corresponding to each point (black circle) in the optical transfer characteristic expression space. Of course, the optical transfer characteristic corresponding to the state of the position deviated from the lattice point may be stored.

  By using such an optical transfer characteristic expression vector, the storage unit 860 does not have to store a recovery filter corresponding to all the optical transfer characteristics that each imaging device can take. That is, a proximity point may be selected from representative points (specific optical transfer characteristics), and a recovery filter may be generated from the optical transfer characteristics of the proximity point. For example, when the optical transfer characteristic of the imaging device is a point indicated by a large white circle in FIG. 12, among the optical transfer characteristics stored in the storage area, the optical transfer characteristic located at the minimum distance from the point of the actual optical transfer characteristic. A recovery filter may be generated based on (a small white circle in FIG. 12A). Alternatively, the representative point and the representative point may be interpolated. As a result, an optical transfer characteristic corresponding to the actual optical transfer characteristic can be generated, so that the amount of data can be reduced as compared with a case where a recovery filter corresponding to each photographing condition is stored.

  It should be noted that the number of grid divisions is preferably such that the minimum value (zero) to the maximum value is divided from 3 to 20 with respect to a specific coordinate axis. When the number of divisions (the number of discrete values) is less than 3, the difference between the optical transfer characteristics stored in advance and the actual shooting conditions or the optical transfer characteristics of the image pickup apparatus increases, and the image restoration effect is reduced. On the other hand, when the number of divisions exceeds 20, the number of optical transfer characteristics stored in advance increases, and the data capacity reduction effect becomes low.

  FIG. 12A shows a three-dimensional optical transfer characteristic expression space represented by three parameters (spherical aberration, coma aberration, astigmatism). For example, four or more including a chromatic aberration component of magnification and the like are added. It may be a four-dimensional or more optical transfer characteristic expression space that represents the above state. By using a space that expresses optical transfer characteristics with four or more aberration components as axes, image restoration can be performed with higher accuracy.

  By defining the optical transfer characteristic expression space as described above, it is possible to suppress duplication of the same optical transfer characteristic and to construct an optical transfer characteristic expression space that covers all possible optical transfer characteristics. In addition, the data capacity of the image processing system can be reduced. As a result, it is possible to efficiently obtain an image restoration effect while keeping the data amount of the image processing system small.

  In the third embodiment, the form using the point image intensity distribution as the optical transfer characteristic has been described. However, the optical transfer characteristic stored in the storage unit 860 may be the recovery filter itself as in the first embodiment. Alternatively, any one of Seidel aberrations corresponding to the imaging conditions of the imaging device, any of a point spread function, an optical transfer function, a recovery filter, a wavefront aberration, and a pupil function may be used.

  Returning to the description of FIG. The specific information 802a from the imaging device 1 is given as coordinates of the optical transfer characteristic expression space of the storage unit 860. The optical transfer characteristic acquisition means 810 passes this specific information to the external device 880, and acquires the optical transfer function corresponding to the designated coordinates from the optical transfer function in the storage area. At this time, if there is a calculation unit in the external device, an optical transfer function obtained by performing an interpolation process on the specified optical transfer function may be acquired.

  For example, assume that the state indicated by a large white circle in FIG. 12A is the coordinates specified by the specific information. When there is an optical transfer characteristic corresponding to the coordinates designated by the specific information or a lattice point in the vicinity thereof, the optical transfer characteristic of the lattice point can be acquired and used for image restoration processing.

  When there is no grid point with optical transfer characteristics at the coordinates specified by the specific information or in the vicinity (or nearest), the optical transfer characteristics that are closest to the coordinates specified by the specific information Interpolation processing is performed to obtain optical transfer characteristics at the above coordinates. By performing the interpolation process, the difference between the actual optical transfer characteristic of the imaging apparatus and the optical transfer characteristic specified by the specific information can be further reduced, and more accurate image restoration process can be performed.

As another method, the distance in the optical transfer characteristic expression space of the coordinates specified by the specific information and the optical transfer characteristic stored in advance may be calculated to obtain the one with the shortest distance.
Alternatively, weighting according to the direction in the optical transfer characteristic expression space is performed, and the product of the distance in the optical transfer characteristic expression space and the direction weight is used as an evaluation function, and the optical transfer characteristic for the interpolation processing is acquired based on the evaluation function. You can also.

  As another method, for example, an identification number is assigned to the optical transmission characteristic stored in the storage area, and an optical transmission characteristic that matches the identification number input as the specific information is acquired. Alternatively, a method of acquiring optical transfer characteristics by associating specific information added (attached) to an image with an address on a storage area is also conceivable. Alternatively, a plurality of images may be analyzed to obtain optical transfer characteristics.

  As described above, the optical transfer characteristic acquisition means is to acquire the optical transfer characteristic specified by the specific information, and any method can be used without being limited to the above method as long as this method is achieved. It is.

  Then, the recovery filter generation unit 820 performs an inverse Fourier transform of the optical transfer function acquired by the optical transfer characteristic acquisition unit 810 to generate a recovery filter. When the optical transfer characteristic is given by a point image intensity distribution, an optical transfer function, or a wavefront aberration, a recovery filter can be generated by performing Fourier transform or inverse Fourier transform.

  On the other hand, the image restoration process 830 is performed on the image obtained by performing the camera signal processing 840a on the image obtained from the imaging apparatus, using the restoration filter generated by the restoration filter generation unit. Then, the image processing device 870 performs camera signal processing 840b again on the recovered image obtained by the image recovery processing 830, and outputs an image.

  The above processing is also performed on the imaging device 2 and the imaging device 3. Since the main processing flow is the same as that shown in FIG.

  The present invention pays attention to the similarity (indicating similar imaging characteristics) of the optical transfer characteristics of each imaging device depending on the lens type or focus type of each imaging device. A recovery filter representing similar optical transfer characteristics is stored in the storage means in advance, so that the data capacity can be reduced. Then, the optical transfer characteristic acquisition means acquires the optical transfer characteristic specified by the specific information, and performs image recovery based on the acquired optical transfer characteristic, thereby reducing the data capacity and obtaining a high-quality recovered image. It is possible to get.

  As described in the third embodiment, when the optical transfer characteristic is not the recovery filter itself, the recovery filter generation unit 820 generates a recovery filter based on the optical transfer characteristic. An operation such as Fourier transform or inverse Fourier transform may be used to generate the recovery filter. Alternatively, a recovery filter corresponding to the optical transfer function may be stored one-to-one in the storage unit 860 or another storage area, and the recovery filter may be used.

  When the optical transfer characteristics stored in advance are interpolated to obtain the optical transfer characteristics used for the image restoration process, the interpolation accuracy can be improved by using the point image intensity distribution or the optical transfer function. The reason for this will be described again with reference to FIG. FIG. 8A is a schematic diagram of an 11 × 11 tap recovery filter. As shown in FIG. 8B, the recovery filter has a large fluctuation between taps, but the point image intensity distribution has a smooth fluctuation in intensity as shown in FIGS. 5A and 5B. Similarly, the optical transfer function has a smooth intensity fluctuation. For this reason, when performing an interpolation process, the interpolation accuracy can be improved by using a point image intensity distribution or an optical transfer function.

  However, when a point image intensity distribution or an optical transfer function is used for the interpolation processing, it is necessary to perform Fourier transform or inverse Fourier transform to convert it into a recovery filter. For this reason, although processing takes time, it is possible to create a recovery filter corresponding to the degree of recovery by performing Fourier transform or inverse Fourier transform. On the other hand, when the recovery filter itself is used as the optical transfer characteristic, it is not necessary to perform Fourier transform or inverse Fourier transform, so that high-speed processing is possible. And trade-off time become important.

  It is also conceivable that the optical transfer characteristic is not a vector but a function that can express the optical transfer characteristic with F (x, y, z). However, compared to this case, it is more preferable to store the optical transfer characteristic as a vector (expression) as shown in the present embodiment because the amount of data can be reduced in many cases. The reason is that when the function is used, the function becomes complicated and the amount of data may increase depending on the accuracy.

  As an external storage means, a storage area of an external device or an area on a network can be used.

  In another embodiment, even in the same imaging device, depending on the shooting conditions, there may be similarities in the optical transfer characteristics under the shooting conditions. In such a case, an image using the same optical transfer characteristic filter is used. A form of recovery is also possible. In this case, it is preferable to store a correspondence table in which different shooting conditions (specific information) specify the same optical transfer characteristics in the storage unit in the imaging apparatus. In addition, the program may be configured to acquire the same optical transfer characteristics as long as there is a difference between a plurality of different imaging conditions (difference in specific information).

  In the conventional technique, in order to perform image restoration according to the optical transfer characteristic that changes depending on the imaging condition of the imaging optical system, a large amount of data corresponding to each imaging condition must be stored in advance. For example, a large number of recovery filters are stored (stored) for each shooting condition such as the zoom position (focal length), object distance, aperture, pixel position (image height), image stabilization lens position, etc. There must be. However, as in this embodiment, even if the imaging conditions are different, if the optical transmission characteristics under the imaging conditions are similar, the optical transmission is performed by performing image restoration using the same optical transmission characteristics filter. The data capacity required for the characteristic filter can be reduced.

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

100, 101, 102 Imaging device 200 Image processing device 310 CPU
320 storage means

Claims (17)

A specific information acquisition step of acquiring specific information for specifying an optical transfer characteristic for recovering an image;
A first optical transmission characteristic used in common for the first and second photographed images photographed under different photographing conditions and a second optical transmission characteristic used for the third photographed image are stored. A characteristic acquisition step of acquiring an optical transfer characteristic specified by the specific information from the storage means;
A program causing an information processing apparatus to execute a recovery step of generating a recovery image using the optical transfer characteristic acquired in the characteristic acquisition step. A specific information obtaining step for obtaining a photographed image photographed under a certain photographing condition and specific information corresponding to the photographing condition;
A characteristic acquisition step of acquiring a specific optical transfer characteristic based on the specific information from a storage means storing a plurality of optical transfer characteristics;
A recovery step of recovering the captured image using the specific optical transfer characteristic acquired in the characteristic acquisition step;
Is a program that causes an information processing apparatus to execute
One of the plurality of optical transmission characteristics stored in the storage means corresponds to a plurality of different pieces of specific information.   The program according to claim 1, wherein the first and second captured images are images captured by different imaging devices.   The program according to claim 1, wherein the first and second photographed images are images photographed under different photographing conditions.   The optical transfer characteristic is any one of Seidel aberration corresponding to the imaging condition, a point spread function, an optical transfer function, a recovery filter, a wavefront aberration, and a pupil function. 4. The program according to any one of 4 above. The optical transfer characteristic is
Aspect ratio component of point image intensity distribution in real space,
An asymmetric component of the point spread distribution that is symmetric about a specific direction in real space,
A rotationally symmetric spread component of the point image intensity distribution in real space,
The center-of-gravity component for color components with different point image intensity distributions in real space,
A component that defines the size of the point image intensity distribution in real space as a proportional multiple;
6. The program according to claim 1, wherein the program is an optical transfer characteristic corresponding to a coordinate in an optical transfer characteristic expression space with any of the axes as an axis.   The program according to claim 6, wherein the specific information is coordinates of the optical transfer characteristic expression space.   The program according to any one of claims 1 to 7, wherein the specific information includes any one of a focal length, an F number, an object distance, and an image height.   The program according to claim 1, wherein the specific information includes information capable of identifying an imaging apparatus or an imaging optical system. An interpolation step for interpolating the optical transfer characteristic acquired by the characteristic acquisition step;
The program according to claim 1, wherein the restoration step restores the captured image using the optical transfer characteristic interpolated in the interpolation step.   The recovery filter generation step of generating a recovery filter used when recovering the captured image based on the optical transfer characteristic acquired by the characteristic acquisition step. Program.   The program according to claim 1, further comprising a storage step of storing the optical transfer characteristic in the storage unit. Specific information acquisition means for acquiring specific information for specifying optical transfer characteristics;
A first optical transmission characteristic used in common for the first and second photographed images photographed under different photographing conditions and a second optical transmission characteristic used for the third photographed image are stored. Storage means,
Characteristic acquisition means for acquiring an optical transfer characteristic specified by the specific information from the storage means;
An imaging apparatus comprising: a recovery unit that recovers an image using the optical transfer characteristic acquired by the characteristic acquisition unit and generates a recovered image. Specific information acquisition means for acquiring specific information for specifying optical transfer characteristics;
A first optical transmission characteristic used in common for the first and second photographed images photographed under different photographing conditions and a second optical transmission characteristic used for the third photographed image are stored. Characteristic acquisition means for acquiring the optical transfer characteristic specified by the specific information from the storage means, from the storage means;
An image processing apparatus comprising: recovery means for recovering an image using the optical transfer characteristic acquired by the characteristic acquisition means and generating a recovered image. A specific information acquisition step for acquiring specific information for specifying an optical transfer characteristic;
A first optical transmission characteristic used in common for the first and second photographed images photographed under different photographing conditions and a second optical transmission characteristic used for the third photographed image are stored. A characteristic acquisition step of acquiring an optical transfer characteristic specified by the specific information from the storage means;
An image processing method including a recovery step of recovering an image using the optical transfer characteristic acquired in the characteristic acquisition step and generating a recovered image. An image processing system having a plurality of imaging devices and an image processing device that performs image restoration processing using optical transfer characteristics,
The plurality of imaging devices have output means for outputting specific information and image data for specifying optical transfer characteristics from imaging conditions to the image processing device,
The image processing apparatus includes an optical transfer characteristic acquisition unit that acquires a recovery optical transfer characteristic specified by the specific information from a plurality of optical transfer characteristics stored in advance in a storage area of the image processing apparatus;
An image processing system comprising recovery processing means for generating a recovered image by recovering the image data using the recovery optical transfer characteristic. When the optical transfer characteristics of different imaging optical systems show similar characteristics, the representative optical transfer characteristic representing the optical transfer characteristic is stored in a storage device,
A storage step for storing the optical transfer characteristics if the optical transfer characteristics do not show similar characteristics;
An image processing method comprising: a recovery step of recovering a captured image captured by the imaging optical system using the representative optical transfer characteristic stored in the storage step.