JP5361582B2 - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Description

  The present invention relates to a technique for synthesizing images acquired by a plurality of imaging optical systems in a complementary relationship.

  An ideal lens re-images many rays from a point on the object. On the other hand, with an actual lens, it is difficult to form a light beam at one point. For this reason, the light emitted from one point on the object is distributed with a certain extent on the sensor surface. This causes blurring of the image and causes a reduction in resolution. Deviations from the ideal imaging state are classified according to the characteristics of how rays are displaced or blurred, such as chromatic aberration, astigmatism, and field curvature. Various designs are made in the lens design so as to suppress these aberrations, but the aberrations cannot be completely eliminated, and the lens is designed in such a balance that blur is minimized. Patent Document 1 proposes a method of synthesizing an image photographed with a telephoto lens at the center of an image photographed with a wide-angle lens. Since the image taken with the telephoto lens has a higher resolution at the center of the image, the resolution at the center is improved by the synthesis.

JP 2001-103466 A

  According to Patent Document 1, the resolution of the central portion is improved, but the resolution of the peripheral portion remains the same as that of a normal wide-angle lens. Like a normal lens, a wide-angle lens is designed with a balance of resolution at the center and the periphery, so the resolution at the periphery is somewhat blurred. That is, in Patent Document 1, there is a possibility that the resolution in the central part is good, but no improvement effect can be expected with respect to the resolution in the peripheral part. Also, it is not intended to improve the image quality by superimposing the image on the entire image, and this method creates a step in the boundary area between the central part and the peripheral part, which deteriorates the image quality. Is also expected. In addition, it is difficult to synthesize the entire image so as to suppress chromatic aberration, and it is also difficult to synthesize an image having a high resolution in the meridional direction and an image having a high resolution in the sagittal direction.

  The present invention provides an image processing apparatus and an image processing method for generating an image that has a high resolution for the entire image and that does not cause image quality deterioration due to a step in the image.

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement. That is,
Means for acquiring a first photographed image of a subject photographed by the first imaging unit;
Means for acquiring a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. Ratio acquisition means for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. Combining means for generating a combined image obtained by combining the image ,
The predetermined image component is an image height;
The ratio acquisition means acquires the ratio information with respect to the image height,
The synthesizing unit synthesizes the corresponding pixels of the first captured image and the second captured image according to the ratio information with respect to the image height of each pixel .

  It is possible to generate an image that has high resolution for the entire image and that does not deteriorate image quality due to a step in the image.

An example of an imaging device using an imaging optical system in a complementary relationship Flow chart of image processing in the first embodiment The figure which shows the calculation method of MTF and a coefficient of the imaging optical system in 1st Embodiment The figure which shows the MTF characteristic of the imaging optical system in 2nd Embodiment Flow chart of image processing in the second embodiment The figure which shows the characteristic of the image and processing area which were acquired in 2nd Embodiment The figure which shows the characteristic of MTF and the frequency characteristic of a synthesized image in 3rd Embodiment. Flow chart of composition processing in the third embodiment An example of a device configuration for realizing the functions of the present invention The figure explaining the imaging optical system in 4th Embodiment

  In the present invention, images having excellent resolution from various viewpoints are obtained by acquiring captured images of the same subject using imaging optical systems having complementary transfer characteristics and combining them based on the transfer characteristics of the imaging optical system. Is generated. In this specification, the imaging optical system and the imaging device are collectively referred to as an imaging unit. To explain intuitively, each of the imaging optical systems in a complementary relationship compensates for each other's disadvantages. For example, a set of lenses in which when a certain lens has a low resolution at the image end and another lens has a high resolution at the image end is an imaging optical system in a complementary relationship. In an imaging optical system in a complementary relationship, it is sufficient that the resolution of any one imaging optical system is high for various conditions. Neither of the images with high resolution at the image edge and low resolution at the image center and the two images with high resolution at the image center and low resolution at the image edge are not acceptable image quality. Therefore, in the present invention, an image taken with the complementary imaging optical system is synthesized based on the transfer characteristics of the individual imaging optical systems to obtain an image with high resolution from various viewpoints.

  Hereinafter, specific embodiments of an imaging apparatus using the image processing method of the present invention will be described with reference to the drawings. In the following embodiments, as an example, a method for suppressing a reduction in resolution caused by aberrations for a predetermined image component (color, image height, spatial frequency) will be described. However, embodiments of the present invention are not limited to these. For example, a configuration in which a plurality of aberrations are suppressed may be employed, and aberrations with respect to other image components may be suppressed.

  The first embodiment deals with a trade-off related to chromatic aberration. The refractive index of the lens varies depending on the wavelength of light. Therefore, the imaging state differs for each wavelength of light. When trying to collect light of a certain wavelength at one point, light of other wavelengths is not condensed and is blurred. As described above, a trade-off relationship occurs in the resolution for each color due to chromatic aberration.

  The second embodiment deals with a trade-off related to field curvature aberration. The field curvature aberration is a phenomenon that causes a difference in focus between the center and the periphery of the image plane. Considering the case where the focus is in the center, the light to be imaged on the edge region of the image is imaged in front of or behind the sensor. Since the image is formed in front of the sensor, light is blurred on the sensor. When the focus is in the center of the image, the field curvature aberration is a phenomenon in which the image end is blurred with respect to the image center. The center is blurred. Thus, a trade-off relationship occurs between the resolution of the image edge and the center of the image due to the field curvature aberration.

  In the third embodiment, a trade-off related to astigmatism is handled. Rays from a point on the object pass through every point on the lens. Consider a group of rays passing through two cross sections of these rays. The first cross section is a cross section including the optical axis, and the second cross section is a cross section perpendicular to the first cross section and parallel to the optical axis. The first section is called the meridional plane, and the second section is called the sagittal plane. Usually, light beams included in each cross-section are imaged at different positions due to the difference in incident angle. For example, if the light beam on the meridional surface forms an image on the sensor, the light beam on the sagittal surface shifts and forms an image before and after the sensor surface. The shift of the imaging position with respect to the rays on the sagittal plane causes a circumferential blur on a concentric circle with the center of the image as the center of the circle. On the other hand, when the image forming position of the light beam included in the meridional plane deviates from the sensor plane, the radial blur occurs from the center of the image. As described above, astigmatism causes a trade-off between the resolution in the circumferential direction and the radial direction.

[First Embodiment]
In the first embodiment, a method for obtaining an image with little chromatic aberration based on the transfer characteristics of an imaging optical system using an imaging optical system in a complementary relationship will be described. Here, the transfer characteristic is what is called OTF (Optical Transfer Function) by those skilled in the art. OTF represents the response of the optical system to various frequency patterns. OTF is a measurable quantity and is usually expressed as a function of two-dimensional spatial frequency. OTF is used as an index of resolution, and the closer the MTF, which is the absolute value of OTF, to 1, the higher the resolution. When the MTF is 1 for all spatial frequencies, ideal image formation occurs and no image blur occurs. Normally, MTF is an aberration and takes a value smaller than 1 except for the DC component. The MTF also differs depending on the color, and also differs depending on the image position such as the center / edge of the image. As described above, the MTF is a characteristic value that represents characteristics relating to the resolution of the imaging optical system.

  FIG. 1 shows an example of an imaging apparatus using an imaging optical system in a complementary relationship according to the first embodiment. A light beam 101 from a subject (not shown) is split into two optical paths by a half mirror 102. One light beam is guided to the image sensor 106 by the imaging optical system 104 (first imaging unit). The other light beam is reflected by the mirror 103 and guided to the image sensor 107 by the imaging optical system 105 (second imaging unit). Here, the imaging optical system 104 and the imaging optical system 105 are imaging optical systems in a complementary relationship. The imaging optical system 104 and the imaging optical system 105 in the first embodiment have different wavelengths of light that increase the resolution. For example, the imaging optical system 104 has a high resolution with respect to light on the short wavelength side, and instead has a low resolution with respect to light on the long wavelength side. Further, the imaging optical system 105 has a low resolution with respect to the light on the short wavelength side, and instead has a high resolution with respect to the light on the long wavelength side. A person skilled in the art can conceive the configuration of the imaging optical systems 104 and 105 as described above. For example, when designing a lens, an appropriate optical arrangement may be performed by setting high resolution targets for the two lenses on the short wavelength side and the long wavelength side, respectively. In the present embodiment, the two imaging optical systems 104 and 105 are cited as the imaging optical systems having a complementary relationship. However, the wavelength may be divided into three, such as short wavelength, medium wavelength, and long wavelength, and may be configured by three imaging optical systems having high resolution at each wavelength, or by an arbitrary number of imaging optical systems. It may be configured.

  The image sensors 106 and 107 are preferably CCDs, CMOSs, etc., and form image information as electrical signals. By reading signals from the image sensors 106 and 107, captured images 110 (first captured images) and 111 (second captured images) of the same subject are obtained. Here, the captured image 110 has a high resolution for blue, which is light on the short wavelength side, and a low resolution for red, which is the light on the long wavelength side. In the captured image 111, the resolution is low for blue, which is light on the short wavelength side, and the resolution is high for red, which is light on the long wavelength side. That is, both of the captured images 110 and 111 have merits and demerits, and the resolution is high for a specific color (predetermined image component), but the resolution is low for the other colors. The image synthesis unit 112 synthesizes the captured images 110 and 111 based on the coefficients stored in the coefficient storage unit 133 to generate a synthesized image 113. The synthesis method according to the first embodiment will be described later. The synthesized image 113 synthesized by the image synthesizing unit 112 is an image with high resolution with respect to light of both short wavelength and long wavelength. The composite image 113 is stored in the image storage unit 135.

<Image Composition Method According to First Embodiment>
An image composition method executed by the image composition unit 112 according to the first embodiment will be described with reference to the flowchart of FIG. First, in step S201, the captured images 110 and 111 are acquired from the image sensors 106 and 107. In step S202, the coefficient for the synthesis process is read from the coefficient storage unit 133 (ratio acquisition means). In the first embodiment, the coefficients are coefficients WaR, WaG, WaB for the imaging optical system 104 and coefficients WbR, WbG, WbB for the imaging optical system 105.

  A method for calculating coefficients for the synthesis process will be described. 3A and 3B show graphs of MTFs with respect to the wavelengths of light of the imaging optical system 104 and the imaging optical system 105, respectively. The horizontal axis represents the wavelength of light, and the vertical axis represents the MTF of each optical system with respect to a typical spatial frequency. Any typical spatial frequency may be adopted, but it may be, for example, 10 cycles / mm. In the present embodiment, the imaging optical system 104 has a high resolution at a short wavelength and a low resolution at a long wavelength. For this reason, in FIG. 3A, the MTF becomes lower on the longer wavelength side. Further, as shown in FIG. 3B, in the imaging optical system 105, the MTF is lower on the shorter wavelength side. Such data can be calculated by those skilled in the art from lens design values and the like. Coefficients WaR, WaG, WaB, WbR, WbG, WbB are calculated based on the following equation from the data of wavelength versus MTF shown in FIGS.

WaR = ∫MTFa (λ) × R (λ) dλ
WaG = ∫MTFa (λ) × G (λ) dλ
WaB = ∫MTFa (λ) × B (λ) dλ
WbR = ∫MTFb (λ) × R (λ) dλ
WbG = ∫MTFb (λ) × G (λ) dλ
WbB = ∫MTFb (λ) × B (λ) dλ
Here, R (λ), G (λ), and B (λ) are sensitivities (spectral sensitivity characteristics) with respect to light of wavelength λ of the RGB channels of the imaging unit. Integration is done over wavelength. By such integration, the coefficients of the imaging optical systems 104 and 105 for the R, G, and B color channels are calculated. The larger the coefficient, the higher the resolution for the corresponding channel color. A conceptual diagram of the above calculation is shown in FIG. As shown in FIG. 3C, the imaging optical system 104 has a low MTF because the MTF is high on the short wavelength side, while WaB is large because the MTF is low on the long wavelength side. On the contrary, in the imaging optical system 105, WaR is large because Wafer is large because MTF is low on the short wavelength side, whereas WaB is small.

  In step S203, image synthesis is performed using the coefficients read from the coefficient storage unit 133. Images corresponding to the RGB components of the captured image 110 are IaR, IaG, and IaB, and images corresponding to the RGB components of the captured image 111 are IbR, IbG, and IbB. If the image corresponding to the RGB plane of the composite image is IR, IG, IB, the image synthesis is performed based on the following equation.

IR = (WaR × IaR + WbR × IbR) / (WaR + WbR)
IG = (WaG × IaG + WbG × IbG) / (WaG + WbG)
IB = (WaB × IaB + WbB × IbB) / (WaB + WbB)
That is, the weighted average of Ia and Ib is calculated using the coefficients Wa and Wb. That is, Wa and Wb are ratio information indicating the synthesis ratio of Ia and Ib. This calculation can be similarly applied when there are three or more imaging optical systems. According to the example of FIG. 3A and FIG. 3B, the R plane of the composite image is mainly configured by the captured image 111 having a high resolution with respect to R. The B plane of the composite image is mainly composed of the captured image 110 having a high resolution for B. The G plane has the same resolution for both the captured images 110 and 111, and is composed of images obtained by combining IaG and IbG at a similar ratio. Through the processing described above, it is possible to obtain a composite image 113 with high resolution and low chromatic aberration for any color from the captured images 110 and 111 acquired by the imaging optical systems 104 and 105 that have a complementary relationship with respect to chromatic aberration.

<Second Embodiment>
As a second embodiment of the present invention, a method for obtaining images with high resolution at various image heights (predetermined image components) using an imaging optical system in a complementary relationship will be described. The image height is a distance from the center of the image. In general, the lens is designed such that the lower the image height, the higher the resolution. The image processing apparatus according to this embodiment may have the configuration shown in FIG. 1 as in the first embodiment. The differences from the first embodiment are the optical characteristics of the imaging optical systems 104 and 105, the processing contents of the image composition unit 112, and the coefficients stored in the coefficient storage unit 133.

  The imaging optical system 104 has a characteristic that the resolution is high at a low image height and the resolution is low at a high image height. In contrast, the imaging optical system 105 has a characteristic that the resolution is low at a low image height and the resolution is high at a high image height. FIG. 4 shows a graph of MTF with respect to image height for the imaging optical systems 104 and 105. Those skilled in the art can design two imaging optical systems having the above characteristics by optical CAD or the like. Specifically, the design may be performed with the resolution targets of the two lenses being set high on the low image height side and the high image height side, respectively. As an example, the imaging optical system 104 may be designed so as to form an image on the sensor surface with respect to a light beam having a low image height without worrying about a reduction in resolution that occurs at a high image height due to field curvature aberration. Conversely, the imaging optical system 105 may be designed so that the imaging surface of the light beam having a high image height matches the sensor surface. The imaging optical system 105 is low in resolution because the imaging surface is separated from the sensor surface at low image heights, but the resolution can be increased at high image heights accordingly.

Next, an image composition processing flow executed by the image composition unit 112 in the second embodiment will be described with reference to FIG. In step S501, the captured images 110 and 111 are acquired. In step S502, one of the pixels in the captured images 110 and 111 is selected as a processing target. For example, selection may be made in raster order. As an example, a case where the position (X, Y) is selected will be described. In step S503, the image height h at the image position to be processed is calculated. Specifically, when the pixel position at the center of the image is (Xc, Yc) and the current image position is (X, Y), the image height h = √ {(X-Xc) ² + (Y-Yc) ^The image height can be calculated as ² }. In step S504, the coefficients Wa (h) and Wb (h) corresponding to the image height h are read from the coefficient storage unit 133. The coefficients Wa (h) and Wb (h) according to the second embodiment may be calculated as follows and stored in the coefficient storage unit 133. That is, Wa (h) and Wb (h) are respectively defined by the MTF value for each image height. FIGS. 4A and 4B show MTF values Wa (h) and Wb (h) with respect to typical spatial frequencies for the respective image heights for the imaging optical systems 104 and 105, respectively. Any typical spatial frequency may be adopted, but it may be, for example, 10 cycles / mm. As shown in FIG. 4 (a), the imaging optical system 104 has a high resolution at a low image height, so the Wa at a low image height is large, and at a high image height, the resolution is low, so a Wa at a high image height. It is getting smaller. Further, as shown in FIG. 4B, the imaging optical system 105 has a low resolution at a low image height and a low Wb at a low image height, and a high resolution at a high image height results in a high Wb at a high image height. It is getting bigger.

Next, in step S505, the pixel value Ia (X, Y) of the captured image 110 and the pixel value Ib (X, Y) of the captured image 111 and the coefficients Wa (h) and Wb (h) for the position (X, Y). ) To calculate in the same manner as in the first embodiment. Specifically, the pixel value I (X, Y) of the composite image 113 corresponding to the position (X, Y) is calculated by the following equation.
I (X, Y) = {Wa (h) × Ia (X, Y) + Wb (h) × Ib (X, Y)} / {Wa (h) + Wb (h)}
The calculated pixel value I (X, Y) may be stored in the image storage unit 135. Based on the above calculation formula, the low image height portion of the composite image 113 is mainly composed of the captured image 110 having a low image height and a high resolution. The high image height portion of the composite image 113 is mainly composed of the captured image 111 having a high image height and high resolution.

  In step S506, it is determined whether there is a pixel that has not yet been selected. If there is a pixel that has not yet been selected, the process returns to step S502 to select the next pixel. If all the pixels in the captured image 110 have been selected, the process ends. Through the above processing, it is possible to obtain a composite image with various image heights, high resolution, and low field curvature aberration.

<Third Embodiment>
In the present embodiment, a method for obtaining an image having a high resolution in both directions by using an imaging optical system having a high resolution in the meridional direction and an imaging optical system having a high resolution in the sagittal direction will be described. Usually, it is difficult to increase the resolution in both directions with a single imaging optical system. The image processing apparatus according to this embodiment may have the configuration shown in FIG. 1 as in the first embodiment. The differences from the first embodiment are the optical characteristics of the imaging optical systems 104 and 105, the processing contents of the image composition unit 112, and the coefficients stored in the coefficient storage unit 133.

  In this embodiment, the imaging optical system 104 has a high resolution in the meridional direction and a low resolution in the sagittal direction. In this embodiment, the imaging optical system 105 has a low resolution in the meridional direction and a high resolution in the sagittal direction. The imaging optical systems 104 and 105 are imaging optical systems in a complementary relationship. The characteristics of the imaging optical systems 104 and 105 will be described with reference to FIG. FIG. 6B is an example of a captured image when the chart shown in FIG. 6A is captured using the imaging optical system 104. Since the imaging optical system 104 has a high resolution in the meridional direction, the radial direction from the center of the screen is not so blurred and clear. Therefore, in FIG. 6 (b), a clear image is obtained with respect to the curves forming concentric circles. On the other hand, since the imaging optical system 105 has a low resolution in the sagittal direction, the image is blurred in the circumferential direction. Therefore, in FIG. 6B, the radial line segment image is blurred in the circumferential direction. On the other hand, FIG. 6C is an example of a photographed image when the chart shown in FIG. 6A is photographed using the imaging optical system 105. Contrary to FIG. 6 (c), since the resolution in the meridional direction is low, a concentric curve is blurred in the radial direction. On the contrary, since the resolution in the sagittal direction is high, the radial line segment image is clear.

<Third Embodiment / Synthesis Method Example 1>
Hereinafter, processing of the image composition unit 112 according to the present embodiment will be described. In the synthesis of the captured images 110 and 111, image synthesis is sequentially performed for each area of the synthesized image. For the purpose of explanation, for example, consider a case where image synthesis within the region 802 is performed among the synthesized images 801 shown in FIG. A coefficient calculation method for synthesis corresponding to the region 802 will be described. An example of the MTFa of the imaging optical system 104 corresponding to the region 802 is shown in FIG. 7A, and an example of the MTFb of the imaging optical system 105 is shown in FIG. 7B. 7 (a) and 7 (b), the vertical axis and the horizontal axis mean the vertical and horizontal spatial frequencies (predetermined image components), respectively. A halftone dot portion indicates a region where the MTF is high. The imaging optical system 104 has a high resolution in the meridional direction, that is, in the radial direction. The radiation direction of the photographed image 110 photographed using the imaging optical system 104 corresponds to the direction connecting the upper left and lower right of the region 802, and FIG. 7A shows a state in which the MTF takes a high value around this direction. Has been. Conversely, since the imaging optical system 105 has a high resolution in the sagittal direction, the MTF takes a high value around the direction connecting the lower left and the upper right in the region 802 of the captured image 111 captured using the imaging optical system 105. This is shown in FIG. 7 (b). In Example 1 of the third embodiment / synthesis method, MTFa and MTFb, which are MTFs of the imaging optical systems 104 and 105, are nothing but coefficients for synthesis. Note that since the MTF of the imaging optical system varies slightly depending on the aperture, object distance, zoom position, etc., it is desirable to store a value corresponding to the imaging condition in the coefficient storage unit 133 or to change the coefficient. . For example, there is a method of acquiring the photographing condition by the system controller 132 of FIG.

FIG. 8A shows a flow of the synthesis process in Example 1 of the third embodiment / synthesis method. In step S1001, captured images 110 and 111 are acquired. Subsequently, the photographed images 110 and 111 are divided into a plurality of regions by the same method to obtain partial images (first partial image and second partial image) (S1002) (first dividing means and second dividing means). . Any division method may be used. For example, the image may be divided into 25 areas by dividing the area into 5 parts in the vertical direction and 5 parts in the horizontal direction. In the subsequent processing, it is assumed that the composite image is sequentially calculated for each divided region. In step S1003, one of the divided areas is selected in order. Next, Fourier transformation (orthogonal spatial frequency transformation or orthogonal transformation) is performed on the selected region of the captured images 110 and 111 (S1004). The Fourier transform of the selected region of the captured image 110 is Fa (u, v) (first frequency space image), and the Fourier transform of the same region of the captured image 111 is Fb (u, v) (second frequency space image). To do. Here, u represents the spatial frequency in the horizontal direction, and v represents the spatial frequency in the vertical direction. In step S1005, the Fourier transform FI (third frequency space image) of the composite image in the selected divided region is calculated by combining each component of the first and second frequency space images based on the following equation. MTFa (u, v) represents the MTF of the imaging optical system 104 when the horizontal spatial frequency is u and the vertical spatial frequency is v in the selected region. This can be calculated from the MTF for the spatial frequency in the meridional direction and the sagittal direction. For example, for each divided region, the coordinate system of the meridional direction and the sagittal direction may be approximated to an orthogonal coordinate system, and then the coordinate system may be rotated so as to become a vertical and horizontal coordinate system. The same applies to MTFb (u, v).
FI = {MTFa (u, v) × Fa (u, v) + MTFb (u, v) × Fb (u, v)} / {MTFa (u, v) + MTFb (u, v)}

  FIG. 7C shows a conceptual diagram of the FI configured by the above calculation formula. According to the above calculation formula, as the FI at a certain spatial frequency, an image obtained by the imaging optical system having the higher MTF at the frequency is mainly used. Therefore, the frequency characteristic of the Fourier transform FI of the composite image is a characteristic obtained by adding the high MTF portions in FIGS. 7 (a) and 7 (b). As a result, it can be seen that the composite image has a high MTF and high resolution in both the sagittal direction and the meridional direction. As can be seen from FIG. 7C, the Fourier transform FI of the composite image has relatively low values in the vertical direction and the horizontal direction. If this is unsatisfactory, a new imaging optical system that complements the low response frequency in FIG. 7C may be added, and the low response frequency may be compensated by a similar synthesis method. In step S1006, the Fourier transform FI of the composite image of the selected region is inverse Fourier transformed to obtain a composite image (third partial image) of the selected region. Each composite partial image constitutes a composite partial image group. The obtained composite image may be stored in the image storage unit 135 sequentially. In step S1007, it is determined whether or not all partial areas have been selected. If not, the process returns to step S1003 to select the next partial area. If selected, the composite partial images stored in the image storage unit 135 are integrated to generate a composite image (S1008).

<Third Embodiment / Synthesis Method Example 2>
Similar to the first embodiment and the example 1 of the composition method, image composition is sequentially performed for each region of the image. In the third embodiment / example 1 of the synthesizing method, Fourier transformation is performed on the captured image, and weighted averaging is performed in the frequency space. However, this method has an advantage that the synthesis process is intuitively easy to understand, but has a high mounting load. Therefore, in the synthesis method example 2, the captured image is not Fourier-transformed and synthesized in the frequency space, but is filtered in the real space to obtain a result equivalent to the third embodiment / synthesis method example 1. .

The calculation formula of the synthesis according to the third embodiment / example 1 of the synthesis method will be described again.
FI = {MTFa (u, v) × Fa (u, v) + MTFb (u, v) × Fb (u, v)} / {MTFa (u, v) + MTFb (u, v)}
When the above equation is inverse Fourier transformed, the following equation is obtained.
I = fa * Ia + fb * Ib
Here, I is a composite image, and Ia and Ib are captured images 110 and 111, respectively. * Indicates convolution calculation. Fa and fb are respectively expressed by the following equations.
fa = (Inverse Fourier transform of MTFa (u, v) / {MTFa (u, v) + MTFb (u, v)})
fb = (MTFb (u, v) / {MTFa (u, v) + MTFb (u, v)} inverse Fourier transform)
The coefficients to be used for image composition are none other than fa (first composition ratio) and fb (second composition ratio) expressed by the above two equations, and fa and fb are the composition ratio of Ia and Ib. give. What is necessary is just to synthesize | combine Ia and Ib in the ratio of this synthetic | combination ratio. Such coefficients are stored in the coefficient storage unit 133 of FIG. 1 for each photographing condition and position in the image, and the image composition unit 112 reads these coefficients at the time of image composition.

  FIG. 8B shows an image composition processing flow according to the second embodiment / example 2 of the composition method. The process in FIG. 8B is performed by the image composition unit 112 unless otherwise specified. In step S1201, captured images 110 and 111 are acquired. In step S1202, the above-described coefficients fa and fb are read from the coefficient storage unit 133 in accordance with the shooting conditions / pixel positions, and convolution is performed on each of the shot images 110 and 111. In step S1202, convolution is performed for each pixel of the captured image, and attention is paid to using the coefficient fa or fb corresponding to each pixel position. fa and fb give the combined ratio of Ia and Ib according to the spatial frequency component around each pixel and the resolution of the imaging optical systems 105 and 106 with respect to the spatial frequency. In step S1203, the convolved images fa * Ia and fb * Ib are added to obtain a composite image 113. The composite image 113 is mathematically equivalent to the image obtained in the third embodiment / composition method example 1, and therefore has high resolution in both the meridional direction and the sagittal direction. As described above, according to the third embodiment, it is possible to obtain a composite image with high resolution in both the meridional direction and the sagittal direction using images captured using the imaging optical systems 104 and 105 having a complementary relationship.

<Fourth Embodiment>
In the present embodiment, image synthesis is performed using a single imaging optical system having variable optical characteristics. Even with a single imaging optical system, an optical system having a substantially complementary relationship can be configured by making optical characteristics variable. As an example, a method of configuring an imaging optical system having a complementary relationship in terms of image height using a single imaging optical system will be described with reference to FIG. In the present embodiment, an image of a region having a high image height is imaged as much as possible on the front imaging image plane (second sensor position 1404). In addition, an image in a region having a low image height is formed as much as possible on the deep imaging image plane (first sensor position 1403). In this configuration, the entire captured image of the image is imaged like an imaging image plane 1402. An optical arrangement having the above characteristics can be designed by those skilled in the art. Two images are obtained by performing imaging twice by changing the driving of the sensor position with the imaging optical system having the above characteristics. At the first sensor position 1403 (first configuration), the image formation surface and the sensor surface approach each other at a low image height, so that the resolution is low and the image height is high. At the second sensor position 1404 (second configuration), the image formation image surface and the sensor surface are close to each other at a position where the image height is high, so that the resolution is high at a high image height. That is, by changing the sensor position, two types of optical characteristics are obtained, and the two types of optical characteristics are complementary with respect to the resolution related to the image height. A processing method for synthesizing two images obtained using such an imaging optical system and obtaining an image with high resolution in terms of image height is the same as in the second embodiment. In this embodiment, since a single image pickup optical system is used, the number of components constituting the image pickup optical system is small, and there is an advantage that the image processing apparatus according to this embodiment can be realized at low cost. Of course, it is also possible to combine images taken at three or more sensor positions.

<Other embodiments>
In the first to third embodiments described above, the coefficients used for image composition are calculated in advance for each combination of the imaging optical systems 104 and 105. That is, a coefficient pair corresponding to each of a plurality of combinations of the imaging optical systems 104 and 105 is stored in the coefficient storage unit 133. Also in the fourth embodiment, the coefficient used for synthesizing the captured images obtained at the first sensor position 1403 and the second sensor position 1404 is stored in the coefficient storage unit 133 in advance. However, when the imaging optical systems 104 and 105 are set, for example, when the imaging optical systems 104 and 105 are replaced, an appropriate processing unit such as the image synthesis unit 112 acquires MTF data from the memory or the like included in the imaging optical systems 104 and 105. Then, the coefficient may be calculated and stored in the coefficient storage unit 133. With this configuration, the present invention can be applied even when the imaging optical system is freely combined.

  The present invention can also realize processing equivalent to that of the first to fourth embodiments with a computer program. An example of the device configuration is shown in FIG. In this case, each of the components including FIG. 1 may be functioned by a function or a subroutine executed by the CPU (arithmetic unit 1305). In general, the computer program is stored in a computer-readable storage medium 1304 such as a CD-ROM. The program can be executed by being set in a reading device (such as a CD-ROM drive) of the computer and copied or installed in the system. Therefore, it is obvious that such a computer readable storage medium is also within the scope of the present invention. Specifically, a captured image is acquired by an imaging optical system having a complementary relationship, and a coefficient calculated from a transfer function of each imaging optical system is provided to a computer via a recording medium 1304, a network, or the like, and a computing device 1305 of the computer. The image composition process may be performed as described above.

Claims (12)

Means for acquiring a first photographed image of a subject photographed by the first imaging unit;
Means for acquiring a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. Ratio acquisition means for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. Combining means for generating a combined image obtained by combining the image,
The predetermined image component is an image height;
The ratio acquisition means acquires the ratio information with respect to the image height,
The image synthesizing unit synthesizes the corresponding pixels of the first captured image and the second captured image in accordance with the ratio information with respect to the image height of the pixels.   The image processing apparatus according to claim 1, wherein the ratio acquisition unit sets the ratio of the resolution of the first imaging unit and the resolution of the second imaging unit as the synthesis ratio. Means for acquiring a first photographed image of a subject photographed by the first imaging unit;
Means for acquiring a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. Ratio acquisition means for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. Combining means for generating a combined image obtained by combining the image,
The predetermined image component is a spatial frequency in a meridional direction and a sagittal direction,
The ratio acquisition means acquires the ratio information for the spatial frequency in the meridional direction and the sagittal direction,
The synthesis means includes
First dividing means for dividing the first photographed image into a first partial image;
A second dividing unit that divides the second captured image in the same manner as the first dividing unit, and sets the second captured image as a second partial image;
Means for orthogonally transforming the first partial image to generate a first frequency space image;
Means for orthogonally transforming the second partial image to generate a second frequency space image;
Means for converting the ratio information for spatial frequencies in the meridional direction and sagittal direction into the ratio information in the frequency space;
Means for combining a component of the first frequency space image and a corresponding component of the second frequency space image according to the ratio information in the frequency space to generate a third frequency space image;
Means for converting the third frequency space image into a real space and generating a third partial image;
An image processing apparatus comprising: means for integrating the third partial images to generate a composite image. Means for acquiring a first photographed image of a subject photographed by the first imaging unit;
Means for acquiring a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
Means for obtaining, for each pixel of the first photographed image, a filter that gives a composition ratio for the pixel, and calculating a first composition ratio by a convolution operation using the filter;
Means for obtaining, for each pixel of the second photographed image, a filter that gives a composition ratio for the pixel, and calculating a second composition ratio by a convolution operation using the filter;
Combining means for generating a combined image by combining corresponding pixels of the first captured image and the second captured image at a ratio of the first combined ratio and the second combined ratio; Characterized by comprising,
Each of the filters provides a synthesis ratio according to a spatial frequency component around a pixel and the resolution of the first and second imaging units with respect to the spatial frequency. Driving means for driving the imaging unit so that the imaging unit has the first configuration or the second configuration;
Means for acquiring a first photographed image of a subject photographed by the image pickup unit of the first configuration;
Means for acquiring a second photographed image of the subject photographed by the image pickup section of the second configuration in which an optical axis overlaps with the image pickup section of the first configuration;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. Ratio acquisition means for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. Combining means for generating a combined image obtained by combining the image,
The predetermined image component is an image height;
The ratio acquisition means acquires the ratio information with respect to the image height,
The image processing apparatus characterized in that the synthesizing unit synthesizes corresponding pixels of the first captured image and the second captured image in accordance with the ratio information with respect to each image height of the pixel. The image processing apparatus according to claim 5 , wherein the driving unit changes a distance between an imaging element of the imaging unit and a lens of the imaging unit. A method performed by an image processing apparatus,
Obtaining a first photographed image of a subject photographed by the first imaging unit;
Obtaining a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. A ratio acquisition process for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. And a synthesis step for generating a synthesized image obtained by synthesizing the image,
The predetermined image component is an image height;
In the ratio acquisition step, the ratio information for the image height is acquired,
In the composition step, the corresponding pixels of the first photographed image and the second photographed image are synthesized in accordance with the ratio information with respect to the respective image heights of the pixels. A method performed by an image processing apparatus,
Obtaining a first photographed image of a subject photographed by the first imaging unit;
Obtaining a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. A ratio acquisition process for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. And a synthesis step for generating a synthesized image obtained by synthesizing the image,
The predetermined image component is a spatial frequency in a meridional direction and a sagittal direction,
In the ratio acquisition step, the ratio information for the spatial frequency in the meridional direction and the sagittal direction is acquired,
The synthesis step includes
A first dividing step of dividing the first photographed image into a first partial image;
A second dividing step of dividing the second photographed image in the same manner as in the first dividing step to form a second partial image;
Generating a first frequency space image by orthogonally transforming the first partial image;
Generating a second frequency space image by orthogonally transforming the second partial image;
Converting the ratio information for the spatial frequency in the meridional direction and sagittal direction into the ratio information in the frequency space;
Combining a component of the first frequency space image and a corresponding component of the second frequency space image according to the ratio information in the frequency space to generate a third frequency space image;
Transforming the third frequency space image into real space to generate a third partial image;
And a step of integrating the third partial images to generate a composite image. A method performed by an image processing apparatus,
Obtaining a first photographed image of a subject photographed by the first imaging unit;
Obtaining a second photographed image of the subject photographed by a second image sensing unit whose optical axis overlaps with the first image sensing unit;
For each pixel of the first photographed image, obtaining a filter that gives a composition ratio for the pixel, and calculating a first composition ratio by a convolution operation using the filter;
For each pixel of the second captured image, obtaining a filter that gives a composition ratio for the pixel, and calculating a second composition ratio by a convolution operation using the filter;
A synthesizing step of synthesizing corresponding pixels of the first photographed image and the second photographed image at a ratio of the first composition ratio and the second composition ratio to generate a composition image; Characterized by comprising,
Each of the filters provides a synthesis ratio according to a spatial frequency component around a pixel and a resolution of the first and second imaging units with respect to the spatial frequency. A method performed by an image processing apparatus,
A driving step of driving the imaging unit to make the imaging unit the first configuration or the second configuration;
Obtaining a first photographed image of a subject photographed by the imaging unit of the first configuration;
Obtaining a second photographed image of the subject photographed by the image pickup unit of the second configuration in which an optical axis overlaps the image pickup unit of the first configuration;
A combination ratio of the first captured image and the second captured image with respect to the image component, obtained from the resolution of the first imaging unit and the resolution of the second imaging unit with respect to a predetermined image component. A ratio acquisition process for acquiring ratio information;
The first captured image and the second captured image are synthesized by combining the predetermined image component of the first captured image and the predetermined image component of the second captured image according to the ratio information. And a synthesis step for generating a synthesized image obtained by synthesizing the image,
The predetermined image component is an image height;
In the ratio acquisition step, the ratio information for the image height is acquired,
In the composition step, the corresponding pixels of the first photographed image and the second photographed image are synthesized in accordance with the ratio information with respect to the respective image heights of the pixels. A computer program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 6 . A computer-readable storage medium storing the computer program according to claim 11 .

Images