JP2010010958A - Multi-band image pickup method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a multi-band image pickup method for imaging an object with the number of imaging bands of four or more colors without reducing spatial resolution to obtain a color image of three or more colors, and achieving successful color reproducibility, and a multi-band imaging apparatus. <P>SOLUTION: First imaging is performed to an object image with a single plate imaging device (5) having a color filter (11) comprising five colors of R, O, G, C, B. Next, a control part 4 shifts the imaging device (5) by a shift drive part (6), and performs second imaging so that each light which forms images in filters of R, B, G of the color filter (11) forms images at positions of filters of O, C, G of the color filter 11, respectively. An image processing part (7) generates two three-band Bayer images based on picked-up images obtained by the first imaging and the second imaging, and generates a multi-band image of three or more colors from the three-band Bayer images. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a single-plate type imaging apparatus that images a subject with the number of imaging bands of four or more colors.

  Conventionally, filters of three primary colors are used in an imaging apparatus. However, since these three primary color filters do not necessarily have good color reproducibility, attempts have been made to improve color reproducibility by increasing the number of color filters and making them multiband.

  For example, as a multiband method, a plurality of color filters provided in the optical path of the imaging optical system are switched, images corresponding to the respective colors are sequentially captured, and then the respective images are combined to obtain a color image ( For example, see Patent Document 1) or incident light from a subject is separated by a half mirror or a dichroic mirror, and spectral images of three colors having different wavelengths are acquired from the separated incident lights, for a total of six color images. (For example, refer to Patent Document 2), a single-plate type image pickup apparatus that uses an image sensor having filters of four or more colors (for example, refer to Patent Document 3), and the like. Are known.

  In particular, a single-plate type imaging device using four or more color filters has the advantage that it can be downsized and the imaging time can be shortened because no filter replacement mechanism or optical path division is required. .

  For example, in the imaging device described in Patent Document 3, as shown in FIG. 16, light from a subject is collected by a lens 101, passes through a six-color filter 102 regularly arranged, and then captured by an imaging device 103. Receives light and converts it to an analog signal. Thereafter, the analog signal is converted into a digital signal by the A / D conversion unit 105 via the gain adjustment unit 104. Further, a signal of one color per pixel obtained by the image sensor 103 is converted into a digital signal of six colors per pixel by interpolating in the multiband color interpolation processing unit 106, and exposure / color adjustment and image output format conversion are performed. Through the unit 107, the image output interface 108 outputs the image signals of six colors. The exposure / color adjustment and image output format conversion unit 107 determines the gain of each color signal of the gain adjustment unit 4 so as to obtain an image signal with appropriate exposure, and based on the signal level distribution of each color signal, and the like. The exposure time of the image sensor 3 is determined.

JP 2006-314043 A JP 2004-172832 A JP 2005-286649 A

  However, in the multiband image pickup apparatus using the color filter as described above, one color filter corresponds to each pixel of the image pickup element, and thus signals of other colors cannot be obtained for the pixel. For this reason, as in the above-described example, other color signals of the pixel are derived from the surrounding pixel signals by interpolation processing. As a result, there is a problem that the spatial resolution of each band image is lowered and the occurrence of false colors and the like is increased as the number of bands is increased in more wavelength regions.

  Accordingly, an object of the present invention made by paying attention to such a point is to provide a color image capable of capturing a subject with four or more imaging bands and obtaining a color image of three or more colors without reducing the spatial resolution. An object is to provide a multiband imaging method and apparatus excellent in reproducibility.

  The invention of the multiband image capturing method according to claim 1, which achieves the above object, is a method for capturing a subject image with a single-plate image sensor having color filters of four or more colors. After the image is taken, each light imaged on the first color filter of the color filter is imaged at the position of the second color filter different from the first color of the color filter. And at least three or more colors based on a captured image obtained by shifting the subject image by a predetermined shift amount to perform the second imaging, and the first imaging and the second imaging. And a multiband image generation step of generating a multiband image.

  According to a second aspect of the present invention, in the multiband image capturing method according to the first aspect, the imaging step includes a plurality of sets in which the first imaging and the second imaging are performed under different imaging conditions. An imaging step, and a selection step of selecting a set of captured images from a plurality of sets of captured images obtained in the plurality of sets of imaging steps, wherein the multiband image generation step includes one set selected in the selection step A multiband image of at least three colors or more is generated based on the captured image.

  The multiband image pickup device according to claim 3 is a multiband image pickup device that picks up a subject image with a single-plate image pickup device having four or more color filters via an image pickup optical system. The subject image is selectively shifted so that each light imaged on the first color filter forms an image at the position of the second color filter different from the first color of the color filter. A shift unit and an image processing unit that generates a multiband image of at least three colors based on two captured images obtained by imaging with the imaging element before and after the shift of the subject image by the shift unit Are provided.

  According to a fourth aspect of the present invention, in the multiband image capturing apparatus according to the third aspect, the shift unit relatively shifts the imaging optical system and the imaging element within an imaging plane. It is what.

  According to a fifth aspect of the present invention, in the multiband image pickup device according to the third or fourth aspect, the color filter is different from the first color and the second color before the shift. Each light imaged on the filter of the three colors is arranged so as to form an image at the position of the filter of the same color as the third color after the shift.

  According to a sixth aspect of the present invention, in the multiband image capturing apparatus according to any one of the third to fifth aspects, the image processing unit captures two images obtained by capturing images before and after the shift, respectively. Multi-band image synthesis that generates two images having a number of bands smaller than the number of imaging bands based on the image and generates a multi-band image by performing demosaicing processing on each of the two images It is characterized by providing a part.

  According to the present invention, a single-plate image sensor having four or more color filters is used, and after taking a subject image for the first time, each light imaged on the first color filter of the color filter is The subject image is shifted by a predetermined shift amount so as to form an image at the position of the second color filter different from the first color of the color filter, and the second imaging is performed. Since a multiband image of at least three colors is generated based on the captured image obtained by the second imaging, the spatial resolution is lowered as compared with the case of imaging with the three color imaging bands. The subject can be imaged with the number of imaging bands of 4 colors or more, and a color image of 3 colors or more can be obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First embodiment]
FIG. 1 is a block diagram showing a schematic configuration of a multiband image pickup apparatus according to the first embodiment of the present invention. This multiband image pickup apparatus 1 uses an image pickup device 5 having a color filter 11 composed of five color filters, and generates a multiband image by synthesizing two images picked up by shifting the image pickup device 5. Is.

  The multiband image pickup apparatus 1 includes an image pickup lens 2, a release button 3, an image pickup device 5, a shift drive unit 6, an image processing unit 7, an image storage unit 8, and a control unit 4 that controls the overall operation.

  When the release button 3 is pressed, the control unit 4 instructs the imaging device 5 to perform the first imaging, and immediately after this imaging is completed, the control unit 4 controls the shift amount by the shift driving unit 6. Then, the image pickup device 5 is shifted, and then the image pickup device 5 is instructed to take a second image. Thereby, the first imaging and the second imaging are performed continuously in a short time. Here, the control unit 4 and the shift driving unit 6 constitute a shift unit. Note that the shift drive unit 6 can be configured using a known mechanism such as an ultrasonic motor or a piezo element that is used to shift the image sensor with camera shake correction.

  Next, the pixel configuration and shift amount of the color filter 11 of the image sensor 5 will be described.

  The color filter 11 is configured by repeatedly arranging filters in a unit arrangement shown in FIG. 2A on a plane. In this figure, R represents a filter that transmits red, O is orange, G is green, C is cyan, and B is blue (hereinafter, red is appropriately R, orange is O, green is G, and cyan). Is denoted by C and blue is denoted by B). As shown in FIG. 2A, this unit array is a 16-pixel array of 4 rows and 4 columns. From the left, the first row is RGOG, the second row is GBGC, and the third row is the first row. Similarly to the RGOG, the fourth row has GBGC filters arranged in the same manner as the second row. In the following, a filter with a unit array of 4 rows and 4 columns as described above is represented as RGOG-GBGC-RGOG-GBGC. Here, R, O, and G are the first color, the second color, and the third color, respectively.

  The control unit 4 shifts the image sensor 5 by two pixels in the horizontal direction of the image sensor (hereinafter referred to as the horizontal direction) by the shift driving unit 6 according to the size of the pixel of the filter. FIG. 2B shows a state in which the color filter 11 is shifted to the left from the state of FIG. With this shift, the O filter is positioned at the position of the R filter before the shift of the subject image, and the R filter is positioned at the position of the O filter. For the B filter and C filter, the same positional displacement occurs before and after the shift. On the other hand, the G filter is located after the shift at the position where the G filter was present before the shift.

  By shifting the image sensor 5 in this manner, the position of the pixel on which the subject 31 forms an image on the image sensor 5 is displaced as shown in FIG. FIG. 3 is a diagram illustrating an example of the shift of the image sensor 5. Here, FIG. 3A shows before the shift, and FIG. 3B shows the after the shift. In FIG. 3A, the head portion of the subject 31 indicated by the arrow forms an image on the O filter, whereby only the O component image signal that passes through the O filter is extracted. Next, as shown in FIG. 3B, when the image pickup device 5 is shifted by two pixels in the direction indicated by the arrow, the head portion of the same subject 31 is imaged on the R filter, so that the R component transmitted through the R filter Only detected.

  FIG. 4 is a functional block diagram illustrating a schematic configuration of the image processing unit 7 according to the first embodiment. The image processing unit 7 includes a multiband image processing unit 41, a multiband image storage unit 42, and a color conversion unit 43. The multiband image processing unit 41 synthesizes a 5-band multiband image corresponding to each of the RGBOC colors based on the image signals of the two captured images obtained by the above two imaging operations. The synthesized multiband image is temporarily stored in the multiband image storage unit 42. The color conversion unit 43 converts the 5-band multiband image stored in the multiband image storage unit 42 to generate a 3-band multiband image.

  FIG. 5 is a functional block diagram illustrating a schematic configuration of the multiband image processing unit 41. The multiband image processing unit 41 includes a first captured image storage unit 51 that stores a first captured image captured by the image sensor 5, a second captured image storage unit 52 that stores a first captured image, and a first captured image. It is composed of an alignment processing unit 53 that aligns the captured image and the second captured image, and a multiband image combining unit 54 described later.

  When the multiband image capturing apparatus 1 captures an image, a shift in the pixel position of the captured image due to camera shake occurs in addition to the shift in the pixel due to the shift of the image sensor 5 described above. The alignment processing unit 53 calculates a horizontal shift of the pixel position due to these, corrects the shift, and performs processing for processing by the multiband image combining unit 54 on the second captured image. Is.

Specifically, the alignment processing unit 53 extracts two G component extraction units 61 and 62 corresponding to the first captured image and the second captured image, respectively, as schematically illustrated in FIG. A registration amount calculation unit 63 that calculates a correction amount for alignment based on the G component, and a pixel value of a desired color pixel at each pixel position of the second captured image is calculated based on the alignment correction amount. An alignment calculation unit 64 is included.

  The alignment amount calculation unit 63 performs demosaicing processing on each of the first captured image and the second captured image from the G components extracted by the G component extraction units 61 and 62, and generates an image using only the G component. . Further, for the two generated G images, the correlation is calculated while relatively displacing in the horizontal direction, and the pixel position of the second captured image with respect to the first captured image at the relative position of both images where the value is maximized. The shift amount is represented by the number of pixels, with the shift direction of the image sensor 5 by the shift drive unit 6 described above being positive. Since the alignment correction can be performed by aligning the pixel coordinates of the second captured image with the pixel coordinates of the first captured image in accordance with the shift amount, the shift amount is also referred to as the alignment correction amount. Of this amount of deviation, two pixels are generated by shifting the image sensor 5, so that the amount of shift indicating the deviation actually caused by camera shake is the value obtained by subtracting two pixels from this amount of deviation. Become.

  The calculation processing of the alignment correction amount (deviation amount) will be described more specifically with reference to FIG. FIG. 7 shows a part of the color filter 11 of the image sensor 5, where R, O, G, C, and B indicate the color of the filter of each pixel, and the right side of each R, O, G, C, and B The two numbers attached to indicate the coordinates of the pixel filter on the color filter 11.

  FIG. 7A shows the first imaging, and FIG. 7B shows the second imaging when there is no camera shake, that is, when the shift amount is zero. In this case, the subject image of the second photographic image is shifted to the right by two pixels and is captured by shifting the two pixels to the left of the image sensor 5 by the shift driving unit 6 as compared with the first photographic image. Become. Focusing on the G component, each G pixel of G12, G14, G21, G23, G32, G34, G41, and G42 in FIG. 7A is shifted by 2 pixels in the horizontal direction, G14 in FIG. This corresponds to G pixels G16, G23, G25, G34, G36, G43, and G45. Therefore, even in an image obtained by demosaicing both images, the correlation at the position shifted by 2 pixels is the highest.

  In this way, it is possible to detect that the specific pixel position of the first captured image corresponds to the position two pixels to the right of the same pixel position of the second captured image using the G pixel component. The subject image formed on the unit array with R11 and C44 on the diagonal line in (a) corresponds to the object image formed on the unit array with O13 and B46 on the diagonal line in FIG. 7B. To do. That is, in the second shot image, an image signal of the OGRG-GCGB-OGRG-GCGB unit array for the same subject image is obtained with respect to the RGOG-GBGC-RGOG-GBGC unit array of the first shot image. . As will be described later, the multiband image synthesis unit 54 synthesizes a multiband image using this pixel arrangement.

  On the other hand, FIG. 7C shows an example in which, in addition to the shift of the image sensor 5 in the left direction, a shift of one pixel further occurs due to camera shake or the like, resulting in a total shift of three pixels. Similarly to the case of FIG. 7B, by using the G pixel component, it is possible to calculate that the shift amount of FIG. 7C is 3 pixels, that is, the shift amount is 1 pixel. Therefore, in this case, the second captured image is shifted by 3 pixels on the right side with respect to the first captured image on the color filter 11.

  However, in this example, with respect to the unit array of RGOG-GBGC-RGOG-GBGC of the first captured image, the same subject image of the second image corresponds to the GRGO− with G14 and G17 as diagonal vertices. The sequence is CGBG-GRGO-CGBG. Since this arrangement is incompatible with the processing by the multiband image composition unit 54 described later, there is an OGRG-GCGB-OGRG-GCGB pixel arrangement at the position of the 4-by-4 pixel arrangement including G14 and G47. As a result, the pixel value of each pixel is calculated by interpolation or the like in the alignment calculation unit 64 of FIG.

  For example, when there is an O filter at the position of G14 in FIG. 7C, the pixel values are the peripheral O pixels, for example, the pixel values of O13 and O17, and between the respective O pixels and G14. It can be calculated by interpolation based on the distance. Similarly, the pixel value in the case where there is a G filter at the position of R15 can be calculated by interpolation using peripheral G pixels, for example, the pixel values of G14 and G16. Thereafter, by performing interpolation in the same manner, the alignment calculation unit 64 corresponds to an image having the unit image of RGOG-GBGC-RGOG-GBGC of the first captured image, and OGRG-GCGB-OGRG of the second captured image. -The pixel value of each pixel having GCGB as a unit array can be calculated.

  If the shift amount is a multiple of 4, an image having OGRG-GCGB-OGRG-GCGB as a unit array can be obtained by simply shifting the pixel coordinates in accordance with the shift amount. No interpolation processing at 64 is required. In the above example, the shift amount is 3 pixels, but even when the shift amount is not an integral multiple of the pixel size, the pixel value can be calculated by interpolation in the alignment calculation unit 64 as described above. .

  As described above, the alignment processing unit 53 receives the pixel values of the respective colors of the captured images from the first captured image storage unit 51 and the second captured image storage unit 52 and inputs RGOG-GBGC-RGOG-GBGC as a unit array. Output the image signal of the first shot image and align the second shot image with the first shot image, and further process the image signal of the second shot image, An image signal having OGRG-GCGB-OGRG-GCGB as a unit array corresponding to the pixel position of the first captured image is output.

  Next, the configuration of the multiband image composition unit 54 shown in FIG. 5 will be described. The multiband image composition unit 54 includes a 3-band Bayer image creation unit 55, demosaicing units 56, 57, and a 5-band image composition unit 58. The three-band Bayer image creation unit 55 stores the first shot image having RGOG-GBGC-RGOG-GBGC of the first shot image transmitted from the alignment processing unit 53 as a unit array, and GRGO-CGBG-GRGO-CGBG. The second captured image as the unit array is synthesized to generate an image by two 3-band Bayer arrays of RGB and OGC. For this reason, the 3-band Bayer image creation unit 55 partially replaces the O pixel and C pixel of the first captured image and the R pixel and B pixel located at the same position in the second captured image, respectively.

  The demosaicing units 56 and 57 generate each RGB image and each OGC image by performing demosaicing processing on each Bayer array image generated by the three-band Bayer image creation unit 55. The 5-band image synthesis unit 58 creates 5-band ROGCB images from the RGB images and OGC images generated by the demosaicing units 56 and 57.

  Next, a subject imaging operation using the multiband image capturing apparatus 1 according to the present embodiment will be described using the flowchart shown in FIG.

  First, when the release button 3 in FIG. 1 is pressed by the user, the control unit 4 performs the first imaging with the imaging element 5 (step S1). Next, the control unit 4 controls the shift amount by the shift driving unit 6 (step S2), shifts the image sensor 5 by two pixels in the horizontal direction (step S3), and again captures the second image by the image sensor 5. (Step S4). The first and second imaging are performed continuously in a short time.

  The first and second shot images are respectively supplied to the image processing unit 7 and temporarily stored in the first shot image storage unit 51 and the second shot image storage unit 52, respectively, and the image processing unit shown in FIG. The multiband image processing unit 41 in FIG. 7 performs alignment processing (step S5). As a result of this alignment processing, as described above, the second shot image has a unit array of GRGO-CGBG-GRGO-CGBG for the corresponding position of the first shot image with RGOG-GBGC-RGOG-GBGC as the unit array. Is output as an image. 5 generates an RGB Bayer image and an OGC Bayer image from these two photographed images, and performs demosaicing processing on these by the demosaicing units 56 and 57. Two 3-band images of the image and each OGC image are generated, and a 5-band multiband image of ROGCB is generated from these two 3-band images (step S6).

  The 5-band multiband image generated by the multiband image composition unit 54 is temporarily stored in the multiband image storage unit 42 of FIG. 4 (step S7). Thereafter, the 5-band multiband image is color-converted by the color conversion unit 43 into, for example, a color image of the three primary colors R, G, and B (step S8). The control unit 4 supplies the color image created thereby to the image storage unit 8 of FIG. 1 and stores it in a storage medium (not shown) (step S9). The control unit 4 confirms completion of all processing and ends the processing.

As described above, according to the present embodiment, the image sensor 5 having the ROGCB five-color color filter 11 having a predetermined unit array is used, and 2 before and after the shift of the image sensor 5 by the shift driving unit 6. Since two images of RGB and OGC are generated from the two photographed images, and compared with the conventional three-band imaging from the two three-band images, Generation of false colors due to a decrease in spatial resolution can be suppressed, and a 5-band image with excellent color reproducibility can be generated. In addition, by using the 5-band image, an RGB 3-band image having excellent color reproducibility can be generated.

  In addition, in the unit array of the color filters 11 arranged in 4 rows and 4 columns, half of the 8 pixels are G pixels, and the alignment processing is performed using the G component before and after the shift of the image sensor 5. Even if there is camera shake or movement of the subject between the first shot image and the second shot image, it becomes possible to correct the displacement of the subject image on the image sensor, and the false color Occurrence can be suppressed.

  Further, since the G component with high luminance is included in the color filter 11 in the same arrangement as in the conventional three-band Bayer arrangement, that is, in a ratio of half of all pixels, the G component is used. Conventional processing, for example, blur correction can be performed in the same manner as in the past.

[Second Embodiment]
FIG. 9 is a block diagram showing a schematic configuration of a multiband image pickup apparatus according to the second embodiment of the present invention. In this embodiment, in the multiband image pickup apparatus 1 according to the first embodiment, the image pickup device 5 is not shifted to shift the position of the pixel on which the subject image is formed, but an optical element is used. A certain imaging lens 2a is shifted. For this reason, in the multiband image pickup apparatus 1 shown in FIG. 1, the shift driving unit 6 is not provided, and the image pickup lens 2 is an image pickup lens 2a incorporating a shift mechanism for displacing the lens position in the horizontal direction. A shift control unit 9 for controlling the lens shift by the shift mechanism is provided, and the shift control unit 9 is controlled by the control unit 4 to shift the imaging position of the subject on the image sensor 5 by two pixels. As the shift mechanism of the imaging lens 2a, for example, a known mechanism such as an ultrasonic motor that is used as a lens driving unit in camera shake correction or the like can be used.

  By shifting the imaging lens 2a in this way, the position of the pixel on which the subject 31 forms an image on the imaging device 5 is displaced as shown in FIG. FIG. 10 is a diagram illustrating an example of the shift of the imaging lens 2a. Here, FIG. 10A shows before the shift, and FIG. 10B shows the after the shift. In FIG. 10A, the head portion of the subject 31 indicated by the arrow forms an image on the R filter, whereby only the R component pixel value that passes through the R filter is extracted. Next, when the imaging lens 2a is shifted downward in FIG. 10, as shown in FIG. 10 (b), the head of the same subject is focused on the O filter, so that the O component transmitted through the O filter Pixel value only is detected.

  Other configurations and operations are the same as those in the first embodiment. As a result, a 5-band multi-band image can be picked up as in the first embodiment, and an RGB 3-band image having excellent color reproducibility can be generated by using the 5-band image. You can also.

  As described above, according to the present embodiment, the subject image formed on the image sensor 5 is shifted by shifting the imaging lens 2a, so that the in-lens shift mechanism is known for camera shake correction. By using this mechanism, the same effect as in the first embodiment can be obtained.

[Third Embodiment]
FIG. 11 is a block diagram showing a schematic configuration of a multiband image pickup apparatus according to the third embodiment of the present invention. In this embodiment, the first and second imaging described in the first embodiment are set as one set, and a plurality of sets of images are captured by changing the shutter speed, and a set of a plurality of captured images captured by this is taken. The image is evaluated, a set of photographed images determined to be optimal from a predetermined condition is selected, a multiband image is generated, and the generated multiband image is output.

  For this reason, in this embodiment, a mode setting unit 10 is further provided in the first embodiment shown in FIG. This mode setting unit 10 can select each mode of “blur prevention priority”, “S / N priority”, “AUTO”, and “none” by the user's operation. When the mode is set by the mode setting unit 10, this setting mode is detected by the control unit 4. Thereafter, when the release button 3 is pressed by the user, the control unit 4 operates the image sensor 5 and the shift driving unit 6 in accordance with the set mode.

Each mode is used for the following purposes in capturing a multiband image.
(1) Camera shake prevention priority: When shooting a subject that tends to blur, in addition to the appropriate shutter speed calculated based on the image signal obtained from the image sensor 5, two sets of images are taken at a higher shutter speed. A photographed image with less blur is selected from the three sets of photographed images.
(2) S / N priority: For a relatively stationary subject, in addition to the appropriate shutter speed, two sets of images are taken at a slower shutter speed, and S / N is taken from a total of three sets of captured images. Select an image with a good (Signal to Noise) ratio.
(3) AUTO: The above-mentioned blur prevention priority or S / N priority is not specified, and the image quality is comprehensively determined from a total of three sets of multiband captured images of the appropriate shutter speed, faster shutter speed, and slower shutter speed. Choose a good one.
(4) None:
As in the first embodiment, only one set is imaged at the appropriate shutter speed.

  FIG. 12 is a block diagram showing a schematic configuration of the multiband image processing unit 41 in the present embodiment. As shown in FIG. 12, in this embodiment, an evaluation unit 59 for evaluating the above-described plural sets of captured images is provided in the configuration of the multiband image processing unit 41 in the first embodiment. The evaluation unit 59 is connected to the alignment processing unit 53, receives data such as a first captured image, a second captured image, and a deviation amount from the alignment processing unit 53, and performs evaluation described later. A set of photographed images having the best evaluation result is identified, and the identification result is returned to the alignment processing unit 53.

    Since the other configuration of the present embodiment is the same as that of the first embodiment, description thereof is omitted.

  Next, the imaging operation of the multiband image capturing apparatus 1 according to the present embodiment will be described using the flowcharts shown in FIGS.

  FIG. 13 is a flowchart illustrating an imaging operation performed by the multiband image capturing apparatus according to the present embodiment. In the present embodiment, first, multiband images are captured according to the setting mode, and two captured images before and after the shift are acquired (step S11). Thereafter, using the acquired two captured images, a 5-band multi-band image is synthesized (step S12), and the 5-band multi-band image is temporarily stored (step S12), as in the first embodiment. In step S13, the conversion unit 43 performs color conversion, for example, to a color image using the three primary colors R, G, and B (step S14).

  Hereinafter, the captured image acquisition process in step 11 will be described in more detail with reference to the flowchart shown in FIG. First, prior to imaging, the control unit 4 identifies the imaging mode set via the mode setting unit 10 in FIG. 11 (step S21).

  Next, when detecting that the release button 3 is pressed, the control unit 4 calculates an appropriate shutter speed based on an image signal obtained from the image sensor 5. When the mode is “blur prevention priority”, two different shutter speeds higher than the appropriate shutter speed are calculated. When the mode is “S / N priority”, two different shutter speeds lower than the appropriate shutter speed are calculated. When the mode is “AUTO”, one shutter speed that is higher and lower than the appropriate shutter speed is calculated. Accordingly, three shutter speeds are calculated in each of the “blur prevention priority”, “S / N priority”, and “AUTO” modes (step S22). When the mode is “none”, only the appropriate shutter speed is calculated.

  Thereafter, the control unit 4 starts imaging. First, the control unit 4 performs first imaging at an appropriate shutter speed (step S23). Next, as in the first embodiment, the control unit 4 controls the shift amount by the shift drive unit 6 (step S24), shifts the image sensor 5 by two pixels in the horizontal direction (step S25), and Second imaging is performed (step S26). The first and second captured images are supplied to the respective image processing units 7 and temporarily stored in the first captured image storage unit 51 and the second captured image storage unit 52 in FIG. 12 respectively (step S27). .

  Next, if the mode setting is “none”, the control unit 4 performs the first and second shot image alignment processing (step S32), as in the first embodiment, and the multi-function of FIG. It outputs to the band image synthetic | combination part 54 (step S33).

  On the other hand, when the mode is “blur prevention priority”, “S / N priority” or “AUTO”, the set of captured images is evaluated, and the evaluation result is calculated as an evaluation value (step S29). Next, the control unit 4 sequentially sets the other two shutter speeds calculated in step S22, and repeatedly executes steps S23, S24, S25, S26, S27, and S29 in FIG. For each of the three shutter speeds, two imaging operations are completed, and when the calculation of the image evaluation value is completed (step 30), the evaluation unit 59 compares the three image evaluation values corresponding to the three shutter speeds, A set of images with the maximum evaluation value is selected (step S31).

  Thereafter, the evaluation unit 59 supplies information on the set of images selected above to the alignment processing unit 53, and the alignment processing unit 53 performs an alignment process on the set of images as in the first embodiment. (Step S32), the first captured image and the second captured image are output to the multiband image composition unit 54 (step S33).

  Next, the calculation of the evaluation value for the set of photographed images in step S29 will be described with reference to the flowchart shown in FIG. The alignment processing unit 53 in FIG. 12 uses a method similar to that described in the first embodiment to shift the amount of deviation from each captured image stored in the first captured image storage unit 51 and the second captured image storage unit 52. Is calculated (step S41).

  The alignment processing unit 53 supplies the calculated shift amount and the first and second captured images to the evaluation unit 59. In the evaluation unit 59, for example, an S / N evaluation defined by an ISO sensitivity, a region where a change in the pixel value of the G pixel is small, and a variance value of signals calculated for this region, the number of saturated pixels, the number of blackout pixels, etc. A value is calculated (step S42).

  Thereafter, the evaluation unit 59 calculates an image evaluation value using a predetermined evaluation formula from the deviation amount calculated in step S41 and the S / N evaluation value calculated in step S42 (step S43). Here, when the mode is “blur prevention priority”, the evaluation formula is calculated by weighting the deviation amount, and when “S / N priority”, the S / N evaluation value is weighted. A coefficient can be set for. When the calculation of the image evaluation value is completed, the evaluation unit 59 stores the image evaluation value and notifies the control unit 4 of the completion of the image evaluation value calculation.

  As described above, according to the present embodiment, a plurality of sets of multiband images are captured at a shutter speed corresponding to the imaging mode, and the captured plurality of sets of images are evaluated to obtain the best set of images. Since the selection is made, in addition to the effects of the first embodiment, it is possible to capture a multiband image suitable for each imaging scene or situation.

  It should be noted that the present invention is not limited to the above embodiment, and many changes or modifications are possible. For example, the color filter 11 of the image sensor 5 is composed of five color filters. However, the present invention is not limited to this, and filters of four colors or six colors or more can be arranged.

  In the flowchart shown in FIG. 8 of the first embodiment, after the multiband image synthesis in step S6, the multiband image stored in step S7 and the color conversion in step S8 are not performed, but directly generated in the image storage unit 8. You may make it preserve | save the multiband image of a band.

  Although the calculation of the shift amount in the alignment processing unit 53 is performed on the entire captured image, it is also possible to determine a region where priority is given to correction of the shift amount. For example, priority can be given to the background, foreground, designated subject, or automatically recognized subject. When giving priority to the automatically recognized subject, the positioning processing unit 53 has a function for subject recognition or face recognition, and priority is given to correction of the recognized subject region, for example, two G images. Can be performed by weighting the recognized subject area when calculating the correlation.

  In the third embodiment, the image evaluation value is calculated from the shift amount and the S / N evaluation value of the first captured image and the second captured image, but instead of the shift amount, the first time after alignment. You may use the correlation regarding a picked-up image and a 2nd time picked-up image. If the correlation is low, it can be determined that the difference between the first captured image after the alignment and the second captured image is large and the image quality is poor.

1 is a block diagram illustrating a schematic configuration of a multiband image capturing device according to a first embodiment of the present invention. It is a figure which shows the unit array of a color filter, and the shift of the same unit array. It is a figure explaining the displacement of the image formation position on a color filter by the shift of a color filter. It is a functional block diagram which shows schematic structure of the image process part which concerns on 1st Embodiment. It is a functional block diagram which shows schematic structure of a multiband image processing part. It is a block diagram which shows schematic structure of a position alignment process part. It is a figure for demonstrating the process in a position alignment process part. It is a flowchart which shows operation | movement of the multiband image imaging device which concerns on 1st Embodiment. It is a block diagram which shows schematic structure of the multiband image pick-up device which concerns on 2nd Embodiment of this invention. It is a figure explaining the displacement of the image formation position on a color filter by the shift of a lens. It is a block diagram which shows schematic structure of the multiband image pick-up device which concerns on 3rd Embodiment of this invention. It is a functional block diagram which shows schematic structure of the image process part which concerns on 3rd Embodiment. It is a flowchart which shows operation | movement of the multiband image pick-up device which concerns on 3rd Embodiment. It is a flowchart which shows the detail of the multiband image pick-up in the flowchart of FIG. It is a flowchart which shows the detail of evaluation value calculation in the flowchart of FIG. It is a figure which shows an example of the multiband image pick-up device of a prior art example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Multiband image pick-up device 2, 2a Image pickup lens 3 Release button 4 Control part 5 Image pick-up element 6 Shift drive part 7 Image processing part 8 Image storage part 9 Shift control part 10 Mode setting part 11 Color filter 31 Subject 41 Multiband image processing Unit 42 multiband image storage unit 43 color conversion unit 51 first shot image storage unit 52 second shot image storage unit 53 alignment processing unit 54 multiband image composition unit 55 3-band Bayer image creation unit 56, 57 demosaicing unit 58 5 Band image composition unit 59 Evaluation unit 61, 62 G component extraction unit 63 Registration amount calculation unit 64 Registration calculation unit 101 Imaging lens 102 Color filter 103 Imaging element 104 Gain adjustment unit 105 A / D conversion unit 106 Multi-band color interpolation processing Part 107 Exposure and color adjustment and Image output format converting unit 108 an image output interface

Claims (6)

A method of imaging a subject image with a single-plate image sensor having color filters of four or more colors,
After the first image of the subject image is taken, each light that forms an image on the first color filter of the color filter has a second color filter position different from the first color of the color filter. An imaging step of shifting the subject image by a predetermined shift amount so as to form an image, and performing a second imaging,
A multiband image generation step of generating a multiband image of at least three colors based on the captured images obtained by the first imaging and the second imaging;
A multiband image capturing method comprising: The imaging step includes
A plurality of sets of imaging steps in which the first imaging and the second imaging are performed under different imaging conditions, and one set of captured images is selected from the plurality of sets of captured images obtained in the plurality of sets of imaging steps. Including a selection step,
The multiband image generation step includes:
The multiband image capturing method according to claim 1, wherein a multiband image of at least three colors or more is generated based on the set of captured images selected in the selection step. In a multiband image capturing apparatus that captures a subject image with a single-plate image sensor having color filters of four or more colors through an imaging optical system,
The subject image is selectively selected so that each light imaged on the first color filter of the color filter forms an image at a position of a second color filter different from the first color of the color filter. Shift part to shift to,
An image processing unit that generates a multiband image of at least three colors based on two captured images obtained by imaging with the imaging device before and after the shift of the subject image by the shift unit. A multiband image pickup device characterized by the above.   The multiband image capturing apparatus according to claim 3, wherein the shift unit relatively shifts the imaging optical system and the imaging element within an imaging plane.   In the color filter, each light imaged on a filter of a third color different from the first color and the second color before the shift is converted into the third color after the shift. 5. The multiband image pickup device according to claim 3, wherein the multiband image pickup device is arranged so as to form an image at a position of a filter of the same color.   The image processing unit generates two images having a number of bands smaller than the number of imaging bands based on two captured images obtained by imaging before and after the shift, and generates the two images. 6. The multiband image capturing apparatus according to claim 3, further comprising a multiband image synthesizing unit that generates a multiband image by performing demosaicing processing on each of them.

Images