WO2006064751A1 - Multi-eye imaging apparatus - Google Patents

Abstract

A multi-eye imaging apparatus including a plurality of imaging systems (106a,106b) that have their respective optical systems (107a,107b) and imaging elements (108a,108b) and also have their respective different optical axes. The plurality of imaging systems (106a,106b) include a first imaging system (106b), which has a pixel shifting means (101) for changing the positional relationship between the imaging element (108b) and an image focused thereon, and a second imaging system (106a) in which the positional relationship between the imaging element (108a) and an image focused thereon is fixed in a time sequence of image pickup.

Description

 Specification

 Compound eye imaging device

 Technical field

 The present invention relates to a compound eye imaging apparatus having a pixel shifting function.

 Background art

 [0002] An imaging device used for a portable device is required to achieve both high resolution and small size. The downsizing of the imaging device is limited by the size and focal length of the imaging optical lens and the size of the imaging device.

 [0003] Generally, since the refractive index of light differs depending on the wavelength, it is not possible to form a scene including information on all wavelengths on a photographing surface with a single lens. For this reason, the optical system of a normal imaging device has a configuration in which a plurality of lenses are stacked in order to form red, green, and blue wavelengths of light on the same imaging surface. In this configuration, the optical system of the imaging device is inevitably long and the imaging device is thick. Therefore, a compound eye type imaging apparatus using a single lens with a short focal length has been proposed as an effective technique for downsizing, in particular, thinning of the imaging apparatus (for example, Patent Document 1).

[0004] In a compound eye type color image capturing apparatus, an imaging optical system is arranged in a plane with a lens that handles light of a blue wavelength, a lens that handles light of a green wavelength, and a lens that handles light of a red wavelength. The imaging region is provided for each lens.

 [0005] In this imaging area, a single imaging element may be divided into a plurality of areas simply by arranging a plurality of imaging elements side by side. In this configuration, since the wavelength of light that each lens is responsible for is limited, a subject image can be formed on the imaging surface by a single lens, and the thickness of the imaging device can be significantly reduced.

FIG. 19 is a schematic perspective view of the main part of an example of a conventional compound eye type imaging apparatus. Reference numeral 1900 denotes a lens array, which is formed by three lenses 1901a, 1901b, and 1901c. Reference numeral 1901a denotes a lens that handles light of a red wavelength, and converts the formed subject image into image information in an imaging region 1902a in which a red wavelength separation filter (color filter) is attached to the light receiving unit. Similarly, 1901b is a lens that handles light of the green wavelength. The image is converted into green image information in the imaging area 1902b, and 1901c is a lens corresponding to light of a blue wavelength, and is converted into blue image information in the imaging area 1902c.

[0007] By superimposing and synthesizing these images, a color image can be obtained. In addition, it is not necessary to limit the number of lenses to three.

 [0008] As described above, the compound-eye imaging device can reduce the thickness of the imaging device, but when the images of each color are simply superimposed and synthesized, the images are synthesized according to the number of pixels of the image separated into each color. The resolution of the image will be determined. For this reason, there is a problem that the resolution is inferior to that of an ordinary Bayer image pickup device in which green, red and blue filters are arranged in a staggered manner.

 On the other hand, in order to improve the resolution of the imaging apparatus, there is a technique called “pixel shift”.

 FIG. 20 is a conceptual explanatory diagram of high resolution using pixel shifting. This figure shows a part of the enlarged portion of the image sensor. As shown in FIG. 20A, the image sensor has a photoelectric conversion unit 2101 (hereinafter referred to as “photoelectric conversion unit”) that converts received light into an electric signal, and light from a transfer electrode or the like can be converted into an electric signal. There is an invalid part 2102 (hereinafter referred to as “invalid part”). In the image sensor, the photoelectric conversion unit 2101 and the invalid portion 2102 are combined to form one pixel. These pixels are usually formed regularly at a certain interval (pitch). The part surrounded by the thick line in Fig. 20A is one pixel, and P indicates one pitch.

 An outline of pixel shifting performed using such an image sensor is shown below. First, photographing is performed at the position of the image sensor shown in FIG. 20A. Next, as shown in FIG. 20B, the photoelectric conversion unit 2101 of each pixel is moved in an oblique direction (1Z2 pitch of the pixel in both the horizontal direction and the vertical direction) so as to move to the invalid portion 2102, and shooting is performed. To do. Then, taking into account the amount of movement of the image sensor, these two captured images are combined as shown in FIG. 20C.

[0011] By this, it is possible to extract a signal from a powerful invalid portion that cannot be originally obtained as a signal. That is, the imaging state in FIG. 20C has the same resolution as that captured by the imaging element having twice the photoelectric conversion unit as compared to the imaging state for one time captured by the imaging element in FIG. 20A. Therefore, if the image shift as described above is performed, an image equivalent to an image shot using an image sensor having twice the number of pixels without increasing the number of pixels of the image sensor is obtained. An image can be obtained.

 [0012] It should be noted that the resolution is not limited to the case of shifting in the oblique direction as illustrated, but the resolution can be improved in the shifted direction when shifting in the horizontal direction and the vertical direction. For example, when the shift is combined vertically and horizontally, a resolution of 4 times can be obtained. Also, the amount of pixel shift need not be limited to 0.5 pixels, and the resolution can be further improved by finely shifting pixels so as to interpolate invalid portions.

 [0013] In the above-described example, the method of shifting the force pixels by changing the relative positional relationship between the image sensor and the incident light beam by moving the image sensor is not limited to this method. For example, the optical lens may be moved instead of the image sensor. As another method, a method using a parallel plate has been proposed (for example, Patent Document 1). In the invention described in Patent Document 1, the image formed on the image sensor is shifted by inclining parallel plates.

 [0014] In this way, the power that can improve the resolution by pixel shifting In pixel shifting, a plurality of images are photographed in time series, and then a synthesis process is performed to generate a high-resolution image. For this reason, if the images to be complemented are shifted, the resolution may deteriorate. In other words, in order to synthesize a high-resolution image from a plurality of images taken in time series, there is a blur due to movement of the imaging device during shooting due to camera shake (hereinafter referred to as “camera shake”) and a subject. It is necessary to remove blur on the subject side (hereinafter referred to as “subject blur”) due to movement.

 [0015] Therefore, in order to compensate for the reduction in resolution, which is a disadvantage of the compound eye system adopted to realize a small size and a thin shape, by pixel shifting technology, it is possible to remove or correct blurring in pixel shifting. Required.

 [0016] There have been proposed several methods for removing blur as much as possible, or several prior arts for correcting blur. One method is to shoot with the camera fixed on a tripod or the like. This method can reduce the influence of camera shake.

 [0017] Another method is to detect and correct camera shake using shake detection means such as an angular velocity sensor. A method of correcting by using both the camera shake correction mechanism and the pixel shifting mechanism has been proposed (for example, Patent Document 2 and Patent Document 3).

[0018] In the invention described in Patent Document 2, the amount of shake is detected using the shake detection means, and the shake is detected. After correcting the pixel shift direction and pixel shift amount based on the amount, the image sensor is moved to shift the pixel. By doing so, it is possible to reduce the influence of camera shake.

 [0019] Further, as described above, in Patent Document 3, it is not necessary to be limited to the method of moving the image sensor. By moving a part of the optical lens in accordance with the detected amount of shake, camera shake correction and pixel shift are performed. The same effect is obtained. Various methods have been proposed for blur detection, such as a method using an angular velocity sensor such as a vibration gai mouth, and a method for comparing motion images taken in time series to obtain a motion beta.

 [0020] Further, as another method for reducing blurring, Patent Document 3 compares a plurality of images taken in time series, and the positional relationship of the images is appropriately shifted due to camera shake or the like, thereby improving resolution. A method has been proposed in which only images having a relationship that can be expected are selected and combined. This method is all performed electronically, and can reduce the size of an imaging apparatus that does not require the provision of a mechanical mechanism for correcting camera shake.

 [0021] The method of fixing with a tripod while damaging it is not practical because it is necessary to always carry the tripod.

 [0022] In addition, the method described in Patent Documents 2 and 3, which detects camera shake, and performs camera shake correction and pixel shifting, requires a new sensor and requires a complicated optical system. , Disadvantageous for miniaturization and thinning.

 [0023] On the other hand, the method of comparing a plurality of images taken in time series described in Patent Document 3 and selecting only the images suitable for synthesis does not require a new sensor, but there is a need for camera shake, etc. Therefore, it is expected that the image will come to an appropriate position by chance, and the resolution is not necessarily improved.

 Patent Document 1: JP-A-6-261236

 Patent Document 2: Japanese Patent Laid-Open No. 11-225284

 Patent Document 3: Japanese Patent Laid-Open No. 10-191135

 Disclosure of the invention

[0024] The present invention solves the above-described conventional problems, and in a compound eye imaging device that performs pixel shifting, a compound eye that can prevent a decrease in the effect of pixel shifting even when there is camera shake or subject blurring. An object is to provide an imaging device. In order to achieve the above object, the compound eye imaging apparatus of the present invention is a compound eye imaging apparatus including a plurality of imaging systems each including an optical system and an imaging element and having different optical axes, and the plurality of imaging systems. Includes a first imaging system having a pixel shifting unit that changes a relative positional relationship between an image formed on the image sensor and the image sensor, an image formed on the image sensor, and the image sensor. The second imaging system is characterized in that the relative positional relationship is fixed in time-series imaging.

 Brief Description of Drawings

FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention.

 FIG. 2 is a flowchart showing the overall operation of the imaging apparatus according to Embodiment 1 of the present invention.

FIG. 3 is a diagram showing a positional relationship between a comparison source region and an evaluation region according to an embodiment of the present invention.

 FIG. 4 is a diagram showing image movement due to camera shake in the embodiment of the present invention.

 FIG. 5 is a diagram for explaining adjustment of a pixel shift amount in one embodiment of the present invention.

 FIG. 6 is a configuration diagram of an image pickup optical system, a pixel shifting unit, and an image pickup element according to Embodiment 1 of the present invention.

 FIG. 7 is a block diagram showing a configuration of an imaging apparatus according to Embodiment 2 of the present invention.

 FIG. 8 is a flowchart of the overall operation of the imaging apparatus according to Embodiment 2 of the present invention.

 FIG. 9 is a diagram for explaining parallax in an embodiment of the present invention.

 FIG. 10 is a diagram for explaining a method for selecting an optimum image in one embodiment of the present invention.

 FIG. 11 is another diagram for explaining a method for selecting an optimum image in the embodiment of the present invention.

 FIG. 12 is a diagram showing an image stored in the image memory after pixel shifting once in Embodiment 2 of the present invention.

 FIG. 13 is a diagram showing images taken in time series with the second imaging system without pixel shift stored in the image memory in Example 3 of the present invention.

 FIG. 14 is a diagram showing an image photographed in Example 3 of the present invention and a subject group determined by subject determination means.

FIG. 15 is a configuration diagram of an image pickup optical system, a pixel shifting unit, and an image pickup device according to Example 5 of the present invention. FIG. 16 is a plan view of a piezoelectric fine movement mechanism according to an embodiment of the present invention.

 FIG. 17 is a diagram showing an example of an arrangement of an optical system according to an embodiment of the present invention.

 FIG. 18 is a flowchart of the entire operation in the imaging apparatus according to Embodiment 3 of the present invention.

 FIG. 19 is a schematic perspective view of a main part of an example of a conventional compound-eye imaging device.

 FIG. 20 is a conceptual explanatory diagram of high resolution using conventional pixel shifting.

 BEST MODE FOR CARRYING OUT THE INVENTION

[0027] According to the present invention, the image pickup apparatus using the compound eye type optical system is reduced in size and thickness, and images taken in time series by the second image pickup system that does not shift pixels are compared. The amount of camera shake (the amount of camera shake) can be detected. Using this amount of camera shake, camera shake can be corrected for images shot with the first imaging system that shifts pixels. That is, it is possible to achieve both downsizing and thinning of the imaging device and high resolution.

[0028] In the compound eye imaging device of the present invention, the amount of blur is compared by comparing the image memory storing the image information of a plurality of frames taken in time series with the image information of the plurality of frames stored in the image memory. It is preferable that the camera further includes a blur amount deriving unit that derives the image and an image synthesizing unit that synthesizes the images of the plurality of frames stored in the image memory.

[0029] Further, it is preferable that the amount of change in the positional relationship by the pixel shifting unit is determined based on the amount of blur obtained by the blur amount deriving unit. According to this configuration, the pixel shift amount can be adjusted according to the amount of camera shake, which is advantageous for improving the resolution.

 [0030] In addition, the amount of change in the positional relationship by the pixel shifting unit may be fixed. According to this configuration, it is not necessary to derive the blur amount during shooting and adjust the pixel shift amount, and the time-series shooting time interval can be shortened. This reduces camera shake and enables shooting even when the subject moves quickly.

[0031] Further, the present invention further includes a parallax amount deriving unit that obtains the magnitude of the parallax for the image forces captured by the plurality of imaging systems having different optical axes, and the image synthesizing unit is obtained by the parallax amount deriving unit. It is preferable that the image is corrected and synthesized based on the parallax amount and the blur amount obtained by the blur amount deriving unit. According to this configuration, when correcting an image, blur correction is performed. In addition, since the parallax correction depending on the distance of the subject is also corrected, the resolution of the synthesized image can be further increased. That is, it is possible to prevent a decrease in resolution depending on the distance of the subject.

 [0032] Further, based on the blur amount obtained by the blur amount deriving unit and the parallax amount obtained by the parallax amount deriving unit, an image captured by the first imaging system stored in the image memory It is preferable that the image processing apparatus further includes an optimum image selection unit that selects image information used for the composition of the image composition unit from the information and image information captured by the second imaging system. According to this configuration, the first and second imaging systems can obtain images before and after blurring, images with parallax, and images with pixel shifts, so images that are suitable for improving resolution without depending on chance. You can choose.

 [0033] Further, it further includes means for discriminating different subjects, the blur amount deriving unit derives a blur amount for each of the different subjects, and the image composition unit composes an image for each of the different subjects. It is preferable. According to this configuration, by deriving the blur amount for each subject, the resolution can be improved even when the entire image does not move uniformly due to the subject moving.

 [0034] Further, the image processing apparatus further includes means for dividing the image information into a plurality of blocks, the blur amount deriving unit derives a blur amount for each of the plurality of blocks, and the image synthesizing unit includes the plurality of blocks. It is preferable to synthesize an image every time. Also with this configuration, it is possible to improve the resolution when the amount of movement of the subject exists. Furthermore, it is not necessary to detect the subject, and the processing time can be shortened.

 [0035] Further, the plurality of imaging systems having different optical axes include an imaging system that handles red, an imaging system that handles green, and an imaging system that handles blue, and each of the imaging systems corresponding to the respective colors. Among them, the number of imaging systems corresponding to at least one color is two or more, and the two or more imaging systems that handle the same color include the first imaging system and the second imaging system. It is preferable that According to this configuration, a color image with improved resolution can be obtained.

 Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

 [0037] (Embodiment 1)

FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus according to the first embodiment. System The control unit 100 is a central processing unit (CPU) that controls the entire imaging apparatus. The system control means 100 controls the pixel shifting means 101, the transfer means 102, the image memory 103, the blur amount deriving means 104, and the image composition means 105.

 [0038] A subject to be photographed (not shown) is photographed by the first imaging system 106b having the pixel shifting means 101 and the second imaging system 106a having no pixel shifting function. By the imaging optical system 107a and the imaging optical system 107b, the subject forms an image on the imaging elements 108a and 108b and is converted into image information as a light intensity distribution.

 [0039] The pixel shifting means 101 shifts the relative positional relationship between the object image formed on the image sensor 108b by the image pickup optical system 107b and the image sensor 108b in the in-plane direction of the image sensor 108b. It is. That is, the pixel shift means 101 can change the relative positional relationship between the image sensor 108b and the incident light beam incident on the image sensor 108b in time-series imaging.

 On the other hand, the positional relationship between the imaging optical system 107a and the imaging element 108a is set so as not to deviate in the in-plane direction of the imaging element 108a. Therefore, the relative positional relationship between the subject image formed on the image sensor 108a by the imaging optical system 107a and the image sensor 108a is fixed in time-series imaging. That is, in the second imaging system 106a, the relative positional relationship between the imaging device 108a and the incident light incident on the imaging device 108a is fixed in time-series imaging.

 [0041] The transfer means 102 transmits the image information photoelectrically converted by the image sensors 108a and 108b to an image memory 103 that stores an image.

 [0042] The first imaging system 106b and the second imaging system 106a are individually driven, and each image is sequentially transferred to the image memory 103 and stored. As will be described later, the pixel shift amount is adjusted while detecting the blur amount using the image captured by the second imaging system 106a. For this reason, the second imaging system 106a can be driven at high speed. That is, the second imaging system 106a can increase the number of times of capturing images per unit time.

[0043] The blur amount deriving unit 104 derives a blur amount by comparing image information captured at different times (in time series) by the optical system without shifting the pixels, in the second imaging system 106a. It is. The details will be described later. The first imaging system 106 so as to correct this blur amount. The pixel shift amount of b is set, and the pixel shifted image is stored in the image memory 103.

The image synthesizing unit 105 synthesizes images captured by the first imaging system 106b and the second imaging system 106a and stored in the image memory 103, and generates a high-resolution image.

FIG. 2 is a flowchart showing the overall operation of the imaging apparatus according to the present embodiment.

 Shooting starts in response to the shooting start command in step 200. When shooting starts, first, shooting pre-processing in step 201 is performed. This calculates the optimal exposure time and performs the focusing process.

 [0046] For example, when the distance between the subject and the imaging device changes, there is a phenomenon in which the imaging distance changes and the image is blurred. In order to correct this phenomenon, distance adjustment (focusing) between the imaging optical system and the imaging element is performed. Focusing uses the characteristic that maximizes the contrast of the image taken when the image is in focus, and the focusing distance between the imaging optical system and the image sensor (imaging distance) is a focusing actuator (not shown). It can be realized by changing.

 [0047] Note that it is not always necessary to use contrast for focusing, and focusing may be performed by measuring the distance of the subject using a laser or radio wave.

 In addition, it is necessary to adjust the optimal exposure time in consideration of ambient ambient light and the like. These include a method of detecting the brightness with an illuminance sensor and setting the exposure time, and a method of providing a preview function for capturing an image before starting shooting. The method of setting the preview function is to convert the image captured before the start of shooting to brightness information by gray scale. If the histogram is biased to white (bright), it is judged as overexposed (exposure time is too long), and if the histogram is biased to black (dark), it is underexposed (exposure time is too short). And adjust the exposure time.

 In addition, when the preview function is provided, by performing this preprocessing before the shooting start command, the time from the shooting start command to the start of exposure can be shortened.

 [0050] Next, in step 202, imaging by pixel shifting is performed. This photographing is performed by repeating each process from step 203 to step 208.

Step 203 is an exposure process of the second imaging system 106a, and step 204 is a process of transferring an image captured by the second imaging system 106a to the image memory 103. Images taken at different times by the second imaging system 106a are transferred to the image memory 103. [0052] In step 205, the images stored in the image memory 103 are compared to determine the amount of blur (the amount of blur of the imaging device). In step 206, based on the pixel shift amount adjusted to reflect the blur amount obtained in step 205, the first imaging system 106b shifts the pixel and takes an image. Step 207 is an exposure process of the first imaging system 106b, and step 208 is a process of transferring an image photographed by the first imaging system 106b to the image memory 103.

 [0053] Among these processes, the blur amount derivation will be specifically described first. As described above, if images are taken at different times, the image may be blurred due to camera shake or subject shake during that time. In order to utilize the invalid portion of the pixel by pixel shifting, it is necessary to determine the amount of pixel shifting considering this blur.

 [0054] Therefore, in step 202, do not perform pixel shifting immediately before pixel shifting! Capture images taken at different times with the second imaging system 106a, calculate the blur amount, and reflect it in the pixel shift amount. I am letting.

 In the blur amount deriving process in step 205, the camera shake amount of the imaging apparatus is obtained as described above. Hereinafter, the specific method will be described. When there is camera shake or subject shake, the subject appears and moves in the time-series images.

 [0056] If the time interval is short, the shape of the subject does not change, and the position can be regarded as moving. For this reason, of two images with different shooting times, one is the comparison source image and the other is the comparison destination image, and it is examined to which part of the comparison destination image the predetermined area of the comparison source image has moved. Thus, it is possible to determine how the image has moved.

 More specifically, in order to check which region of the comparison target image a specific region in the comparison source image (hereinafter referred to as “comparison source region”) corresponds to the comparison source image, Set an evaluation area of the same size as the area, and evaluate how similar the comparison source area and evaluation area are! Thereafter, evaluation areas are sequentially set at different positions, and the movement destination of the comparison source area is searched while performing the above-described evaluation in each evaluation area. In this case, the evaluation area most similar to the comparison source area becomes the movement destination of the comparison source area.

[0058] Since the image captured by the image sensor can be regarded as a set of light intensities corresponding to each pixel, the light from the Xth pixel in the horizontal direction to the right and the yth pixel in the vertical direction from the top left is the origin. If the intensity is I (x, y), the image can be considered as a distribution of this light intensity I (x, y). it can.

 FIG. 3 shows the positional relationship between the comparison source region 301 and the evaluation region 302. In the example in Fig. 3, the comparison source area is set to a rectangular shape where the upper left pixel position of the comparison source area is (xl, yl) and the lower right pixel position is (x2, y2). is doing. In this case, in the evaluation area (m, n) moved by m pixels to the right and n pixels downward from the comparison source area, the upper left pixel is (xl + m, yl + n) and the lower right position is It can be represented by a region of (x2 + m, y2 + n).

 [0060] The correlation between the evaluation region and the comparison source region (how much the evaluation value R (m, n) representing the force is, as shown in (Equation 1)), is the absolute difference in light intensity at each pixel. It is represented by the sum of values.

 [0061] [Equation 1]

V, — _y, J¾ 一 J¾ y = y \ ^{χ = χ} ι

As the comparison source region and the evaluation region are similar, the difference in light intensity between the corresponding pixels in both regions becomes smaller. For this reason, the evaluation value R (m, n) shows a smaller value as the correlation between the light intensity distribution (image) of the comparison source region and the evaluation region becomes larger (similar).

 [0063] Since the correlation of the regions is compared, m and n need not be limited to integers, and the original light intensity I (X, y) force is also interpolated between the pixels I '(X, y ) And calculating the evaluation value R (m, n) from (Equation 1) based on I '(X, y) Can do. As a method of data interpolation, linear interpolation, nonlinear interpolation, or deviation method may be used!

From the above, in order to derive the blur amount, the values of m and n are changed, and an evaluation region whose evaluation value is most similar to the comparison source region is searched with subpixel accuracy. In this case, the blur direction of camera shake and subject blur is not limited to a specific direction, so the values of m and n need to be considered including negative values (evaluation of areas moved leftward or upward). There is. [0065] Further, although m and n may be changed so that the entire range of the comparison target image can be evaluated, the image of the subject moves greatly due to camera shake or the like, and falls outside the light receiving range of the image sensor. Since it cannot be synthesized as an image, it is generally preferable to limit m and n to a predetermined range to shorten the calculation time. The combination of m and n where the evaluation value R (m, n) found in this way is the minimum value is the amount of blur indicating the position of the comparison target image area corresponding to the comparison source area.

 [0066] Note that the comparison source region need not be limited to a rectangle, and an arbitrary shape can be set. In addition, the calculation of the evaluation value does not need to be limited to the sum of absolute values of the differences in light intensity. The evaluation value may be calculated.

 [0067] The comparison method using the correlation of the images can be used for obtaining the amount of parallax described later, and can also be used for calibration of the pixel shifting means. For example, by taking an image before and after pixel shifting by the pixel shifting means and evaluating the amount of shift of the image, the actuator used for pixel shifting is moving accurately due to the surrounding environment (temperature and deterioration over time). Can confirm. By such processing, pixel shifting by the actuator can be ensured.

 Hereinafter, camera shake will be described more specifically with reference to FIG. FIG. 4 is a diagram showing image movement due to camera shake in the present embodiment. This figure shows an example of shooting an image of a landscape with little subject movement! FIG. 4A is a diagram when the subject and the camera are moved in parallel, and FIG. 4C shows the change in the image between shooting times 1 and 2 in this case. FIG. 4B is a view when the camera is rotated in the horizontal direction, and FIG. 4C shows a change in the image between shooting times 1 and 2 in this case.

 [0069] In both cases where the imaging device moves in parallel as shown in FIG. 4A and in the case where the imaging device rotates as shown in FIG. 4B, the image can be considered to be translated in the plane. Thus, the effect on the image is greater when the optical axis is rotated and the optical axis is deviated than when translated. Figure 4B shows the same result when the camera rotates in the vertical direction. By correcting the translation of the image due to the translation or rotation of the imaging apparatus, it is possible to correct camera shake.

[0070] Note that, when the imaging device rotates, it can be considered that the image moves in parallel, Strictly speaking, since the distance between the subject and the lens changes in part, the image will be slightly distorted. If these slightly distorted images are simply superimposed, the originally overlapping parts do not overlap, and the effect of increasing the resolution by shifting the pixels is reduced.

 Therefore, the resolution is further improved by detecting and correcting image distortion due to this rotation. In addition, only calculating the amount of image blur for one specific evaluation area can only determine the parallel movement of the image, so multiple evaluation areas are set and the amount of blur at each location is calculated. Thus, the amount of camera shake and image distortion in each evaluation region can be obtained. By deforming the superimposed image according to the distortion of the image, it is possible to prevent image deterioration and obtain a high-resolution image.

 [0072] Next, the adjustment of the pixel shift amount will be specifically described. FIG. 5 is a diagram for explaining adjustment of the pixel shift amount. This figure shows an enlarged part of a part of the image sensor, and shows the assumed pixel shift vector 400, the blur vector 401 detected by the blur derivation means, and the actual pixel shift vector 402. ing.

 [0073] When there is no camera shake, in order to effectively use the invalid portion 405 on the right side of the photoelectric conversion unit 404, it is necessary to shift 0.5 pixels in the X direction and 0 pixels in the Y direction as in the vector 400 There is. On the other hand, an example is shown in which the image is shifted by 401 °, the camera shake is less than J, the X direction is less than 1.25 pixels, and the lower direction is less than 1.5 pixels. In this case, if the pixel shift is performed without adjusting the pixel shift amount, that is, if the pixel shift is performed by 0.5 pixel shift in the X direction as in the vector 400, the position where the vector 400 and the vector 401 are combined 403 Then, the next shooting will be performed. In this case, a portion different from the right side portion of the photoelectric conversion unit 404 originally intended to be used is photographed.

[0074] Here, the force that slightly shifts the optical axis due to movement due to camera shake is very small. For this reason, an image in which the amount of deviation in the X direction and the 方向 direction of the vector 401 is an integer pitch (integer multiple of one pixel pitch) can be regarded as the same as an image obtained by shifting pixel coordinates by an integer pixel. it can. In other words, shooting at shooting time 2 by the second imaging system 106a without pixel shifting is the same as shooting an image that was already shot at shooting time 1 with different pixels. Therefore, in this case, the first pixel shift In the imaging system 106b, as in the case where there is no camera shake at all, by shifting 0.5 pixels in the X direction as in the vector 400, the invalid part 405 on the right side of the photoelectric conversion unit 404 can be photographed, and the pixels are shifted. The effect of can be obtained.

 That is, it is the portion of the hand movement that is less than the integer pitch (below the decimal point) that affects the pixel shifting effect.

 [0076] Therefore, if a new pixel shift vector is set so as to be the same as the shift amount of the partial force vector 400 equal to or smaller than the integer pitch in the hand shake, the pixel shift effect can be obtained. In the above example, the portion of the blur vector 401 that is less than or equal to the integer pitch in the X direction is 0.25 pixels, and the portion that is less than or equal to the integer pitch in the Y direction is 0.5 pixels. In this case, a new pixel shift vector should be set so that the portion below the integer pitch in the X direction is 0.5 pixels and the partial force is less than the integer pitch in the Y direction!

 [0077] Therefore, the pixel shift vector is set to 0.25 pixels in the X direction and 0.5 pixels in the Y direction as in the vector 402 in FIG. The positional relationship is the same as in the case of pixel shifting using the element shifting vector 400. That is, according to the present embodiment, since the pixel shift vector is adjusted in accordance with the blur vector, it is possible to always obtain the pixel shift effect.

 [0078] After a series of steps in step 202 is repeated until the set number of image shifts is completed, the images stored in the image memory are combined in step 209, and the images are output in step 210 to complete the shooting. . A concrete example is shown below.

 [0079] (Example 1)

 FIG. 6 shows the configuration of the imaging optical system, the pixel shifting means, and the imaging device according to the first embodiment. As an imaging optical system, two aspherical lenses 601a and 601b having a diameter of 2.2 mm were used. The optical axis of the lens is almost parallel to the Z axis in Fig. 6, and the distance between them is 3 mm.

[0080] In the first imaging system that performs pixel shifting, a glass plate 602 is provided on the optical axis of the lens 601b. The glass plate 602 can be tilted with respect to the X-axis and Y-axis by a piezoelectric actuator and a tilt mechanism (not shown). In this embodiment, the pixel is shifted by 1/2 (1.2 m) of the pixel pitch in the horizontal direction (X-axis direction) to double the number of pixels. The glass plate 602 has an optical glass with a width (X-axis direction) of 2 mm, a height (Y-axis direction) of 2 mm, and a thickness (Z-axis direction) of 500 μm. BK7 is used.

 As the image sensor 603, a monochrome CCD 603 having a pitch of 2.4 m between adjacent pixels was used.

 The light receiving surfaces of the glass plate 602 and the image sensor 603 are substantially parallel to the XY plane in FIG. Further, the image sensor 603 is divided into two regions 603a and 6003b so as to correspond to each optical system on a one-to-one basis. By providing a readout circuit and a drive circuit for each of the regions 603a and 603b of the image sensor 603, the images of the regions 603a and 603b can be individually read out.

 [0082] When the apparatus according to this example was held by hand and shot, the resolution was improved in an environment where the exposure time was short and the movement of the subject was short, such as an outdoor scenery in fine weather.

 In this embodiment, a method of inclining the glass plate as the pixel shifting means is used. The method is not limited to this method. For example, an image sensor or lens may be physically powered by a predetermined amount using an actuator using a piezoelectric element, an electromagnetic actuator, or the like. Even when other means are used as the pixel shifting means in this way, the configuration shown in FIG. 6 is the same except for the glass plate 602.

 Further, in this embodiment, one image sensor is divided into two different regions, but an image sensor that uses two different image sensors so as to correspond one-to-one with each optical system may be used. This form may be any form as long as the plurality of imaging areas correspond one-to-one with each optical system.

 [0085] (Embodiment 2)

 FIG. 7 shows a configuration of the imaging apparatus according to the second embodiment. The main difference from the first embodiment is that the second embodiment is that a parallax amount deriving means 700 is added, and the imaging element 70 1 is a body, which is substantially the same time as the first imaging system. The second imaging system is photographed, and optimum image selection means 702 for selecting an image to be combined based on the parallax amount and the blur amount is added. Explanation of overlapping parts with the first embodiment is omitted.

FIG. 8 shows a flowchart of the overall operation of the imaging apparatus according to the present embodiment. The imaging start command in step 200 and the imaging pre-processing in step 201 are the same as in the first embodiment.

[0087] In step 800, photographing by pixel shifting is performed. Step 800 is the same as Step 801 The exposure process of the image sensor, the transfer process of the image of the image sensor in step 802 to the image memory 103, and the pixel shift process in step 803 are repeated.

 [0088] Since the image sensor 701 is configured to be shared by the first image pickup system 106b and the second image pickup system 106a, images are taken at almost the same timing. In addition, the pixel shift amount is a fixed value regardless of the amount of camera shake, and is set for pixels that are set so that invalid pixels can be used effectively when there is no camera shake (for example, 0.5 pixels).

 That is, compared with step 202 of FIG. 2 of the first embodiment, step 800 is a step of taking in an image and deriving a blur amount in the second imaging system 106a in order to adjust the pixel shift amount ( Step 205) in FIG. 2 is omitted. For this reason, the interval between shooting time 1 for shooting without pixel shifting and shooting time 2 for shooting with pixel shifting can be shortened. As a result, camera shake is reduced, and shooting can be performed even when the subject moves faster than in the first embodiment.

 [0090] After the pixel-shift imaging in step 803 is completed, the images stored in time series among the images stored in the image memory 103 in step 804 are processed in the same manner as in step 205 in the first embodiment. Compare and find the amount of blur. If there is a movement of the subject, the amount of blur will not be uniform in the image, and if the amount of blur is determined and superimposed together, it will not overlap exactly and resolution will not improve depending on the location.

 Therefore, by dividing the image into blocks and obtaining the amount of blur for each of the divided blocks, the resolution of the entire image can be improved. For this division, it is not necessary to limit to the rectangle, separately detect the subject, and divide each subject to detect the amount of blur.

 [0092] Next, in step 805, images taken by imaging systems with different optical axes at the same time are compared to determine the amount of parallax. When shooting with an imaging system having different optical axes, the relative position of the subject image formed on the image sensor changes according to the distance of the subject that the image forming position is not only the distance between the centers of the lenses.

This difference is called parallax. FIG. 9 is a diagram for explaining parallax. In FIG. 9, for the sake of simplicity, two imaging optical systems 1301a and 130 lb having the same characteristics are installed at a distance D, and the imaging surfaces of the imaging optical systems are denoted by 1302a and 1302b, respectively. At this time, the imaging optical systems 1301a and 1301b observe the same subject from different positions. For this reason, parallax occurs between images formed on the imaging surfaces 1302a and 1302b. The parallax amount Δ is given by (Equation 2) below. D is the distance between the optical axis of the imaging optical system 1301a and the optical axis of the imaging optical system 1301b, f is the focal length of the imaging optical systems 1301a and 1301b, and A is the distance between the subject and the imaging planes 1302a and 1302b.

 [0095] [Equation 2]

Δ = D · f / (A— ί)

[0096] When it can be assumed that a subject with sufficiently large wrinkles is at infinity, the parallax amount Δ can be expressed as D'fZA, and Δ can be considered as 0. In this case, the images taken by the imaging optical systems 1301a and 130 lb can be regarded as the same. Therefore, if the distance D between the lens centers is corrected, the composition process can be performed as it is.

 However, when A is small, the parallax amount Δ becomes a finite value and cannot be ignored! /.

 That is, the images captured by the imaging optical system 1301a and the imaging optical system 1301b are images that have a shift due to parallax depending on the distance of the subject and cannot be regarded as the same. Therefore, it cannot be combined as it is.

 In order to correct this parallax, it is necessary to obtain the parallax for each subject. For this parallax, images with different optical axes taken at the same time may be divided into blocks, and the position of the corresponding block may be checked. This process can be realized by comparing the images and comparing the images using (Equation 1) and searching for a highly correlated place, as in the case of obtaining the blur amount. it can.

 [0099] Although the lens center distance D may be calculated from the lens distance distance, a subject that is a marker is placed at infinity, and the position where the image is formed is regarded as the center of the lens. It may be calculated.

[0100] Further, the method of dividing the block is not limited to this method, and the block may be divided by changing the number of pixels or the shape. Unlike blur derivation, the direction in which the parallax occurs is the origin of the image sensor (image sensor And the intersection of the corresponding optical system with the optical axis of each optical system), the combination of m and n in (Equation 1) is limited according to the direction when detecting the visual difference. Just do it.

 [0101] Next, in step 806, an image is selected that has a combination that improves resolution when combined based on the amount of blur and the amount of parallax. As described above, the resolution is improved by shifting the pixels as long as the pixels to be overlapped are shifted so as to use the invalid portion. It can be used as well.

 [0102] FIG. 10 is an explanatory diagram of a method for selecting an optimum image. The shaded area in this figure is a subject image formed on the image sensor. At time 1, subject images 1001a and 1001b are formed in the imaging region 1000a of the second imaging system and the imaging region 1000b of the first imaging system. It is assumed that the subject is on the center line of the second imaging system. At this time, due to the parallax, the subject image 1001b is formed at a position shifted by Δ on the imaging region 1000b.

 [0103] When the image of each imaging area is transferred to the image memory 103, it is stored as two-dimensional data. When the upper left point of each image area is the origin and the position of the subject image is indicated by coordinates, the upper left coordinate of the subject image 1001a is (ax, ay), and the upper left coordinate of the subject image 1001b is parallax Δ Since it is shifted by the amount, (ax + Δ, ay) is obtained.

 [0104] Next, at time 2, the second imaging area is 1002a, the first imaging area is 1002b, and the subject images at that time are 1003a and 1003b. The first imaging system was moved 0.5 pixels to the right by pixel shifting means. The subject image 1003a on the imaging region 1002a is imaged at a position shifted (bx, by) from the origin.

[0105] If there is no movement of the subject, this shift amount is shifted due to camera shake. When the images in each image area are transferred to the image memory and displayed in coordinates, the coordinates at the upper left of the subject image 1003a are (ax + bx, ay + by). Since the imaging region 1002b is shifted in pixels, the origin of coordinates is shifted 0.5 pixels to the right. For this reason, compared to the imaging area 1002a of the second imaging system, the origin of coordinates is closer to the subject image 1003b by 0.5 pixels in the imaging area 1002b of the first imaging system. As with time 1, the subject image 1003b is shifted to the right by the amount of parallax Δ. Therefore, the covered The upper left coordinates of the subject image 1003b are (ax + bx + Δ—0.5, ay + by).

FIG. 11 is another explanatory diagram of the optimal image selection method. The amount of deviation bx and the amount of parallax Δ can be divided into a case where it is close to an integer pitch and a case where it is close to a value obtained by adding a 0.5 pixel pitch to an integer pitch. When the amount of deviation bx and the amount of parallax Δ are expressed as values less than or equal to an integer pitch, bx = 0 and Δ = 0 for integer pitches, and bx = 0 for values obtained by adding 0.5 pixel pitches to integer pitches. .5, Δ = 0.5.

[0107] In FIG. 11, bx and Δ indicate values equal to or smaller than the integer pitch. Each value in FIG. 11 is obtained by calculating the value of each X coordinate of the subject with 0 as the X coordinate value ax of the reference imaging region 1000a. In FIG. 11, 0 indicates that the positional relationship between the pixel of the imaging element that converts the subject into an image and the subject is shifted by an integer pitch compared to the case of the reference imaging region 1000a. . 0.5 indicates that the pixel pitch is shifted by 0.5. The image corresponding to the portion indicated by 0.5 is an image that can effectively use the invalid portion.

 Here, as can be seen from FIG. 11, the calculated value of the X coordinate is 0.5 for the four images, regardless of the combination of the amount of parallax Δ and the amount of camera shake bx. There is an image. Therefore, in any combination, it is possible to obtain an image that effectively uses the invalid portion. That is, the resolution can be improved regardless of camera shake or the distance of the subject.

 Note that both the camera shake amount and the parallax amount do not change digitally in units of 0.5 pixels, but actually change gradually and continuously. For this reason, the part where the values of bx and Δ in FIG. 11 are 0.5 may be a value close to 0.5 (for example, a value from 0.3 to 0.7). Also, the part set to 0 may be a value close to 0 (for example, a value less than 0.3 or a value greater than 0.7). On the other hand, image data needs to be arranged on a grid. So, when you overlay and combine images, you can do linear interpolation processing!ヽ.

 [0110] In the present embodiment, the pixel pitch in the oblique direction may be used as a reference when the optimum image is selected based on the horizontal pixel pitch of the image sensor. Also, pixel pitch standards can be mixed depending on the situation.

 [0111] (Example 2)

Hereinafter, Example 2 according to Embodiment 2 will be described. Configuration on appearance of Example 2 6 has the same configuration as that of FIG. 6 of the first embodiment, and the optical system and the pixel shifting mechanism of the second embodiment are also the same as those of the first embodiment.

[0112] The second embodiment is different in that the image sensor 603 exposes at approximately the same time and transfers an image, and the driving amount of the pixel shifting mechanism is fixed.

[0113] As a first imaging system for pixel shifting, an optical glass BK7 (602 in the figure) with a thickness of 500 m is provided on the optical axis of the lens 601b, and tilted by about 0.4 degrees using a piezoelectric actuator and tilting mechanism. Thus, the subject image is shifted in the horizontal direction (X-axis direction) by a pixel pitch of 1Z2 (1.2 / zm) to double the number of pixels.

[0114] With this configuration, an image stored in the image memory after one pixel shift is shown in FIG.

. First image capturing time 1 time taken of the time taken to the ^second image after the pixel shift (after inclining the glass plate) and shooting time 2.

[0115] In this example, the movement of the subject is sufficiently small! Therefore, there is no subject blur between the image 701 taken at shooting time 1 and the image 703 taken at shooting time 2. When there is blurring, the entire image moves due to camera shake between different times 1 and 2.

[0116] Therefore, it is assumed that the entire image moves uniformly, and the image 701 taken at the second imaging system without pixel shifting is compared with the image 703 taken at the imaging time 2 by the same imaging system. The amount of camera shake was derived. More specifically, the central portion of image 701 (e.g. 100 X

The region of 100 pixels) force The region of the image 703 moved to was evaluated by the image comparison method using the above (Equation 1), and the amount of camera shake was derived. As a result, the amount of blur was 2.2 pixels in the upper direction of the screen and 2.5 pixels in the horizontal direction.

[0117] In this case, the value by by the integer pitch or less out of the two pixels of the amount of blurring on the screen is 0.2 pixels, and can be regarded as by = 0. Amount of lateral blur 2. A value bx that is less than or equal to the integer pitch among 5 pixels is 0.5 pixels, and bx = 0.5.

[0118] The size of the area to be compared need not be limited to a square, and may be set arbitrarily.

[0119] Further, the parallax amount was obtained from the captured image 701 and the captured image 702 at the imaging time 1 by the parallax amount deriving means. As a result, since the subject distance is long, the parallax amount is 0.1 pixel or less in any region of the image, and can be regarded as Δ = 0. Ie The distribution of the difference amount is ignored, and the whole can be regarded as uniform parallax.

 [0120] Based on the amount of blur and the amount of parallax, the optimum image selection means selects an image to be synthesized. The above result corresponds to the part of FIG. 11 where Δ = 0 and bx = 0. The optimal image selection means is to select a combination of images corresponding to the apportionment of 0 and apportionment of 0.5 to the IJ of A = 0, bx = 0.5 in Fig. 11! become.

 [0121] In this case, since it is the value power of three images, a plurality of combinations can be selected. In this way, when there are a plurality of combinations, if the combination at the same time is selected, the blurring of the subject is reduced, and a higher resolution image can be obtained.

 [0122] Note that, in the example of Fig. 11 described above, the force by = 0.5 described when the value by less than the integer pitch of the blur amount in the Y-axis direction is considered by = 0. Also good. In this case, an image that contributes to improving the resolution was taken at a position corresponding to the invalid portion on the lower side of the photoelectric conversion unit or at a position corresponding to the invalid portion on the lower right side of the photoelectric conversion unit. Become an image.

 In this embodiment, the method of tilting the glass plate is used as the pixel shifting means. The force is not limited to this method. For example, an image sensor or lens may be physically powered by a predetermined amount using an actuator using a piezoelectric element, an electromagnetic actuator, or the like.

 [0124] In this embodiment, one image sensor is divided into two different areas. However, two different image sensors may be used so as to correspond to each optical system in a one-to-one relationship. The form of the image sensor may be any form as long as the plurality of image areas correspond to each optical system on a one-to-one basis.

 [0125] (Example 3)

 The present embodiment is different from the second embodiment in that there is a moving amount of a subject to be imaged (for example, a person or an animal). In this embodiment, the subject is moved to another location while the first image is shot, the data is stored in the memory and the second shot is taken, and the first image is captured. In the second picture and the second picture, the scene where a part of the subject moves to another location is shot.

[0126] Since the basic configuration is the same as that of the second embodiment, the description of the overlapping parts is omitted. If there is a movement of the subject, the entire image will not move uniformly. As in Example 2, the total movement cannot be estimated from the movement of a partial area of the image! /.

[0127] Therefore, in the third embodiment, block dividing means for dividing an image into a plurality of blocks is provided, and the amount of blur is derived for each block. The block dividing means is controlled by the system control means 100 and divides the entire first image taken by the second imaging system 106a without pixel shifting into 10 × 10 pixel blocks. The blur amount deriving unit 104 checks, for each block, which position in the second image each of the divided images corresponds to. (Equation 1) was used to derive the amount of image movement.

FIG. 13 shows images taken in time series by the second imaging system 106a without shifting the pixels stored in the image memory according to the present embodiment. FIG. 13A shows an image taken at shooting time 1, and FIG. 13B shows an image taken at shooting time 2. Fig. 13C shows the amount of image movement derived for each block.

In FIG. 13C, the block indicated by A is a block from which a 10.1 pixel blur is derived on the right in FIG. 13A, and the block indicated by B is 8.8 pixels on the left in FIG. 13A. This is the block from which the blur is derived. The amount of camera shake and the movement of the subject are integrated.

 [0130] Similarly, the parallax can be divided into blocks and obtained for each block. From the sum of the blur amount and the parallax amount, as in the case of the second embodiment, the one having an integer pitch (or close to the integer pitch) and the 0.5 pixel pitch (or 0.5 pixel pitch). By selecting images with a layout close to (2), it is possible to select an image with improved resolution when combined.

 In this way, by combining the optimal images selected for each block, the resolution of the entire image can be improved even when the movement of the subject is large.

 [0132] Note that only the camera shake is corrected at the user's selection! /, And the image processing is performed so as not to intentionally correct the subject shake, thereby enhancing the dynamic feeling of the moving scene. It can be done by providing a correction mode.

[0133] Also, when the subject moves, there is a portion where a part of the subject is hidden in the images taken in time series (block indicated by X in Fig. 13C). In such a case, only that part of the image taken at a certain time without combining multiple images. By selecting only the image, a natural finish image can be obtained.

 [0134] Further, since the pixel shifting technique is a technique for improving the resolution, there is no effect on a smooth surface of the subject to be photographed or a fine pattern below the resolution performance of the lens. On the other hand, when shifting pixels, by reducing the time taken between shots, camera shake and subject shake are reduced, and resolution is improved.

 [0135] Therefore, when an image divided into blocks is analyzed, and the effect of pixel shifting is an infinite image, the shooting interval can be shortened by stopping the processing for that block. In general, high-frequency components can be seen in high-resolution parts when Fourier transform is performed. Therefore, after the image is captured and divided into blocks, the frequency components of the image are analyzed, and if it is below a predetermined condition, the derivation of the blur amount and the parallax calculation for that portion may be stopped.

 [0136] Further, there is a time for exposure and a time for transferring an image from the image sensor to the image memory between the photographing. Although exposure is performed in a lump and cannot be omitted, the processing time can be shortened by transferring only the necessary blocks to the image memory.

 [Example 4]

 The present embodiment is different from the third embodiment in that the present embodiment uses subject discrimination means for discriminating different subjects in the image. By using the subject discrimination means, it is easy to derive the blur amount for each subject. For this reason, the amount of blur can be accurately derived even when the amount of blur is different in an image where there is subject blur in addition to camera shake.

 [0138] Also, when the blur amount is derived by dividing an image into blocks as in the third embodiment, the blocks can be divided for each subject or the block size can be changed for each subject. It is also possible to selectively synthesize only a specific subject when synthesizing images.

[0139] As the subject discrimination means, a means for measuring the distance to the subject using radio waves or the like to identify different image areas, a means for discriminating different subjects by performing edge detection or the like by image processing, and a parallax amount are used. There is a method of extracting a subject from an image. In addition, the specific means is not limited as long as different subjects in the image can be distinguished. The basic configuration of the present embodiment is the same as that of the second embodiment, and therefore overlapped. The description is omitted here.

 [0140] Fig. 14 is a diagram showing an image photographed in the present example and a subject group discriminated by the subject discriminating means. In this example, the captured image was divided into 10 × 10 pixel blocks (horizontal 11 × vertical 9), and the distance to the subject was measured by radio waves for each block, and different subjects were identified. In subject discrimination, those within a certain error range in distance measurement were discriminated as the same subject. In this example, the error range was 5%.

 [0141] Fig. 14A is an image taken without pixel shifting with the second imaging system 106a at shooting time 1, and Fig. 14B is an image shot with no pixel shifting with the second imaging system 106a at shooting time 2. Is. In addition, the distance (unit: meters) measured by radio waves for each block is shown. For this distance, the distance A may be calculated for each block by the above (Equation 1) using the parallax Δ obtained for each block.

 [0142] Before shooting at shooting time 1, the distance of the subject was measured by radio waves. As shown in Fig. 14A, two groups of subjects could be identified. One is subject group 1 at a distance of approximately 5 meters, and the other is subject group 2 at a distance of approximately 2 meters. Each subject group is identified by the distance within the 5% error range.

 [0143] Before shooting at shooting time 2, the distance of the subject was measured by radio waves, and each subject group was identified as shown in Fig. 14B. In this embodiment, the amount of blurring before and after pixel shift is derived for each subject group.

 [0144] When the blur amount of each subject group was derived by the blur amount deriving means, for the subject group 1, a blur of 10.3 pixel pitch was derived in the left direction in the figure. This blur is shown as one block blur in the figure. With regard to subject group 2, the subject blurring protruded from a part of the image where the subject blurring was large, so it was difficult to accurately derive the blurring amount of the subject group as a whole.

 [0145] Therefore, in this embodiment, the blur correction of subject group 1 is performed on the image taken at shooting time 2! / And the image composition is performed. The same method as in Example 2 was used as the method for selecting an image by the optimum image selection means.

[0146] More specifically, a value that is less than or equal to an integer pitch among the 10.3 pixel pitch blur of subject group 1 is 0.3 pixels, and bx in FIG. it can. Note that since the subject group 1 is moving in the left direction, the value of bx can be set to a negative value of 0.5. In this case, it becomes 0.5 force 5 in Table 11. Also, when Δ = 0, bx = -0.5, the value of ax + bx + Δ—Ο.5 is a force of 1. Since this is an integer pitch, it becomes 0, and bx = 0.5 The same.

 That is, the positive / negative difference of bx is a difference in whether the position of the invalid pixel to be effectively used is the force left which is the right of the photoelectric conversion unit, and the contribution to the resolution is the same.

 As in this embodiment, by discriminating different subjects using the subject discriminating means, the blur amount can be derived for each subject, so that the blur amount of the image can be corrected accurately.

 [0150] Also, the image may partially protrude from the shooting range due to camera shake and subject blur. V If the image cannot be recognized! /, Increase the resolution by shifting the pixel to the image area. Instead, you only need to select one of the images you have taken.

 [0151] (Example 5)

 FIG. 15 shows configurations of the imaging system, the pixel shifting means, and the imaging device according to the present embodiment. As the imaging optical system, an aspherical lens 1101a with a diameter of 2 mm: LlOld was used. The optical axis of the lens is almost parallel to the Z axis in Fig. 15, and the distance is 2.5 mm. Color filters 1102a to 1102d are provided in front of each lens (subject side) as wavelength separation means that transmits only a specific wavelength. 1102a and 1102d are color filter letters that transmit green, 1102b is a color filter that transmits red, and 1102c is a color filter that transmits blue.

 [0152] 1103a to 1103d are four image sensors corresponding to each lens on a one-to-one basis, using a common drive circuit and operating in synchronization. A color image can be obtained by combining images taken by each optical system (color component). The pixel pitch of the image sensor is 3 μm in this example.

 [0153] Further, each lens and the image sensor are installed in parallel with the X axis in FIG. 15 and at equal intervals, and the light receiving surface of each image sensor is substantially parallel to the XY plane in FIG. .

Reference numeral 1104 denotes a piezoelectric fine movement mechanism serving as a pixel shifting means. As the first imaging system to shift pixels, the image sensors 1103a to 103c are attached to the piezoelectric fine movement mechanism 1104, and the X direction in the figure. Enabled to drive in Y direction. The 1103d is independent of the piezoelectric fine movement mechanism, and does not perform pixel alignment. It becomes the second imaging system.

 FIG. 16 is a plan view of the piezoelectric fine movement mechanism 1104. Image sensors 1103a to 1103c are installed on the stage 1201 at the center. The stage 1201 is finely moved in the X-axis direction in the figure by the laminated piezoelectric elements 1202a and 1202b, and the stage fixing frame 1202 is finely moved in the Y-axis direction in the figure by the laminated piezoelectric elements 1203a to 1203d. . As a result, the image sensor can be finely moved independently in two axial directions perpendicular to each other in the horizontal plane of the image sensor.

 [0156] In this example, four images were shot for each image sensor while shifting pixels by one shooting command. By capturing the first image, four images corresponding to the four image sensors 1103a to 1103d are obtained. Each of the three image sensors 1103 & 1103 was configured to shoot while moving by 0.5 pixel pitch (1.5 m) in the direction and Y direction. Specifically, take the first picture without shifting the pixels, move the picture by 0.5 pixels in the X direction, take the second picture, and then keep the position in the X direction. The third shot was taken with a 0.5 pixel pitch shift in the Y direction, and finally the fourth shot was taken with a power of 0.5 pixel pitch in the X direction while maintaining the position in the Y direction. By combining these four images, a high-resolution image was obtained.

 [0157] First, the amount of blur at each shooting time was derived for each of a plurality of images taken in time series using the lens l lOld of the second imaging system without pixel shifting. In addition, the parallax amount is derived from the first image captured by the first imaging system with the green power filter 1102a and the second imaging system with the green color filter 1102d. Asked. This is because the amount of parallax that makes it easy to compare the direction images of images taken using the same color filter can be obtained more precisely.

 [0158] Next, based on the derived blur amount and parallax amount, an image to be combined by the optimal image selection means was selected, and images of each color were combined. In order to generate a color image, luminance data of the three primary colors is required for each pixel. Since the green image data is included in both the first imaging system and the second imaging system, the resolution can be improved.

[0159] On the other hand, for red and blue images, there are no images taken without pixel shifting, so images that are shifted by 0.5 pixels depending on the amount of blur or parallax (using the invalid portion). An image may not be obtained and resolution may not be improved.

 [0160] However, since the human eye generally receives a lot of information about green, even when the resolution of blue and red is worse than green, when shooting natural scenery or people Not easily affected. In addition, it is known that there is a strong correlation between green, blue, and red images in the local region of the image. Using this characteristic, the interpolation part of the blue and red images can be estimated from the green image. It is also possible to do.

 [0161] In addition, if all of the green, red, and blue colors are provided with an imaging optical system without pixel shifting, invalid portions can be used in the image selected by the optimum image selection means. An image that is shifted by 0.5 pixels can be included reliably, and a high-resolution image can be reliably obtained.

 [0162] In this example, the force of arranging four optical systems on one straight line is not necessarily limited to this arrangement. FIG. 17 shows another example of the arrangement of the four optical systems. FIG. 17A is an example in which four optical systems are arranged at the vertices of a rectangle. GO, G1 are green, R is red, and B is blue, indicating the wavelength separation means (color filter).

 FIG. 17B is a diagram for explaining the derivation of the amount of parallax in the arrangement of FIG. 17A. For deriving the amount of parallax, a green imaging system arranged on a diagonal line is used. The parallax of the other red and blue imaging systems is an orthogonal component of the parallax amount in the green imaging system because the four optical systems are arranged in a rectangular rectangle.

 [0164] In this embodiment, a color filter is provided in front of the lens for wavelength separation. A color filter is provided between the lens and the image sensor, or a color filter is directly formed on the lens. May be.

 [0165] The color filter is not necessarily limited to the three primary colors R, G, and B. The complementary color filter is used to separate the wavelengths and invert the color information by image processing.ヽ.

 [0166] Furthermore, the wavelength separation means is not limited to a color filter. For example, when a mechanism for tilting using a glass plate is used as the pixel shifting means, colored glass can be used as the glass plate. As described above, any specific means may be used as the wavelength separation means as long as it is a means for separating only a predetermined wavelength component.

[0167] In addition, images taken with an optical system that handles green were compared to derive the parallax and blur amount. As explained in the example, it is possible to obtain the same result by arranging the wavelength separation means of the same color which need not be limited to green in the first imaging system and the second imaging system.

 [Embodiment 3]

 FIG. 18 is a flowchart illustrating the overall operation of the imaging apparatus according to the third embodiment. In the second embodiment, the pixel shift operation method is first determined and a predetermined number of times of shooting is performed. Embodiment 3 has a configuration in which the number of shots varies depending on the shot image.

 [0169] Fig. 18 [Steps 1500, 1501, 1503, 1504ί] are the same as steps 200, 201, 801, 802 in Fig. 8. The configuration in FIG. 18 is different from the configuration in FIG. 8 in the subsequent configuration. In the flowchart of FIG. 18, among the processing steps 1502 for repeating pixel shifting and photographing, the amount of blur is obtained in step 1505, and the image to be synthesized is selected in step 1506.

 [0170] Depending on the amount of blur and the amount of parallax, a plurality of images with a shift of 0.5 pixel pitch necessary for composition can be obtained in one shooting. For this reason, when the pixel shift operation determined first is performed, an image having the same positional relationship is captured, and an image that does not contribute to high resolution is captured.

 [0171] Therefore, after selecting an image in step 1506, in step 1507, an image lacking in composition is found, and the amount of deviation is determined so that the image can be obtained. In step 1508, pixel shifting is executed. To do.

 Step 1502 is repeated by repeating a series of steps in step 1502 until an image necessary for composition is obtained. Thereafter, the amount of parallax is derived in step 1509, the images stored in the image memory are combined in step 1510, the image is output in step 1511, and the shooting is completed.

 [0173] With such processing, the number of pixel shifts can be reduced, and the influence of camera shake and subject movement can be minimized and a higher resolution image can be obtained. Industrial applicability

[0174] As described above, according to the present invention, even if camera shake or subject shake occurs when pixel shifting is performed, it is possible to prevent a reduction in pixel shifting effect and obtain a high-resolution image. . For this reason, the present invention is useful for an imaging device in, for example, a digital still camera or a mobile phone.

Claims

The scope of the claims

 [1] A compound-eye imaging device comprising a plurality of imaging systems each including an optical system and an imaging element and having different optical axes,

 The plurality of imaging systems are:

 A first imaging system having a pixel shifting means for changing a relative positional relationship between an image formed on the imaging element and the imaging element;

 The relative positional relationship between the image formed on the image pickup device and the image pickup device includes a second image pickup system that is fixed in time-series shooting. Compound eye imaging device.

 [2] Image memory that stores image information of multiple frames taken in time series,

 A blur amount deriving unit for deriving a blur amount by comparing the image information of the plurality of frames stored in the image memory;

 The compound eye imaging apparatus according to claim 1, further comprising: an image synthesizing unit that synthesizes the images of the plurality of frames stored in the image memory.

3. The compound eye imaging apparatus according to claim 2, wherein the amount of change in the positional relationship by the pixel shifting unit is determined based on the amount of blur calculated by the amount of blur deriving unit.

4. The compound eye imaging device according to claim 1, wherein a change amount of the positional relationship by the pixel shifting means is fixed.

 [5] The apparatus further includes a parallax amount deriving unit that obtains the magnitude of parallax from images taken by a plurality of imaging systems having different optical axes,

 The compound-eye imaging device according to claim 2, wherein the image synthesis unit corrects and synthesizes an image based on the parallax amount obtained by the parallax amount derivation unit and the blur amount obtained by the blur amount derivation unit.

[6] Image information captured by the first imaging system stored in the image memory based on the blur amount obtained by the blur amount deriving unit and the parallax amount obtained by the parallax amount deriving unit; 6. The compound eye imaging apparatus according to claim 5, further comprising optimum image selection means for selecting image information used for synthesis of the image synthesis means from image information photographed by the second imaging system.

[7] It further comprises means for distinguishing different subjects,

 The blur amount deriving unit derives a blur amount for each different subject,

 3. The compound eye imaging apparatus according to claim 2, wherein the image synthesizing unit synthesizes an image for each of the different subjects.

 [8] It further comprises means for dividing the image information into a plurality of blocks,

 The blur amount deriving unit derives a blur amount for each of the plurality of blocks,

 The compound eye imaging apparatus according to claim 2, wherein the image synthesizing unit synthesizes an image for each of the plurality of blocks.

 [9] The plurality of imaging systems having different optical axes are:

 An imaging system that handles red,

 An imaging system that handles green,

 It consists of an imaging system that handles blue,

 Among the imaging systems corresponding to each color, the number of imaging systems corresponding to at least one color is two or more,

 2. The compound eye imaging apparatus according to claim 1, wherein the two or more imaging systems that handle the same color include the first imaging system and the second imaging system.