JP5733706B2 - Image processing apparatus, method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, method, and program for performing color reproduction processing using a multiband image.

  Filters corresponding to the original R (red), G (green), and B (blue) colors of RGB (3-band) digital cameras and digital cameras as shown in Non-Patent Document 1 and Non-Patent Document 2 A technique for capturing a multiband image of four or more bands by combining color filters that transmit only a part of the transmission wavelength and generating an RGB image or the like from the obtained multiband image is known.

  In the case of Non-Patent Document 1 and Non-Patent Document 2, for example, as shown in FIG. 14, a 3-band image 1 is obtained by an image input unit 501 combining an RGB camera 501a and a color filter 501b, and an RGB camera 502a is installed. By obtaining the three-band image 1 using the image input unit 502, two types of three-band images in which the optical aberration due to the filter thickness and the like and the corresponding point on the subject are recorded at different positions on the image are obtained. 6-band images are obtained, corresponding points search unit 503a of conventional image processing apparatus 503 detects corresponding points of 3-band image 1 and 3-band image 2, and image deformation processing unit 503b detects 3-band image 1 The multi-band image is generated by geometrically deforming one of the three-band images 2 and matching the other image with the other image. Then, the color reproduction processing unit 503c performs a color reproduction calculation using all the obtained band images, and generates an RGB image from the result (see also FIG. 15).

Masaru Hashimoto, “Archives of Ukiyo-e and High-Precision Color Reproduction Using 6-Band Spectral Images,” Humanities and Symposium 2008, pp.305-310, Dec., 2008 Masaru Tsuchida, Takahito Kawanishi, Junji Yamato, Koichi Ito, Takafumi Aoki, “Stereo one-shot six-band imaging system for color reproduction”, Image Recognition and Understanding Symposium (MIRU) 2010, Demo Session, No. DS -1, pp. 1310-1311, July 2010

  However, Non-Patent Documents 1 and 2 do not clearly indicate which of the image captured without a filter and the image captured with a filter should be the reference image / the image to be deformed. When an image taken with a filter is used as a reference image, if a corresponding point search or image transformation fails, false color generation on the color reproduction result image or an image with an unnatural appearance (color reproducibility) There is a problem that the image quality is remarkably deteriorated, such as a decrease in image quality), and that it is difficult to restore the image quality using the reference image.

  The present invention has been made to solve the above-described problems, and an object thereof is to provide an image processing apparatus, method, and program in which the accuracy of the color reproduction result is improved as compared with the conventional method.

  In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention includes a first color image photographed using a color filter that transmits a part of wavelengths of colors that can be photographed by a camera, and the color filter. Image acquisition means for acquiring a second color image photographed without use, and corresponding point detection means for detecting a plurality of corresponding points between the first color image and the second color image acquired by the image acquisition means; Image deformation means for generating a third color image obtained by deforming the first color image so that the positions of the corresponding points coincide with each other based on the plurality of corresponding points detected by the corresponding point detection means; It is determined whether or not the amount of positional deviation of the corresponding points of the second color image acquired by the image acquisition unit and the third color image generated by the image transformation unit is greater than or equal to a threshold value. When the image deformation result determining unit and the image deformation result determining unit determine that the amount of positional deviation of the corresponding point is equal to or greater than a threshold, the second color image acquired by the image acquiring unit is used to The second color image acquired by the image acquisition unit and the image deformation unit when the image deformation result determination unit determines that the amount of positional deviation of the corresponding point is less than a threshold value. And a color reproduction processing means for performing a color reproduction process of the subject using the third color image generated by the above-described method.

  An image processing method according to a second aspect of the present invention is an image processing method in an image processing apparatus including image acquisition means, corresponding point detection means, detection corresponding point number result determination means, image deformation means, image deformation result determination means, and color reproduction processing means. A first color image photographed using a color filter that transmits a part of wavelengths of colors that can be photographed by the camera and a second photographed without using the color filter. A color image is acquired, and a plurality of corresponding points between the first color image and the second color image acquired by the image acquisition unit are detected by the corresponding point detection unit, and the detection corresponding point number determination unit determines the Determining whether or not the number of corresponding points detected between the first color image and the second color image acquired by the corresponding point detecting means is greater than or equal to a threshold; If the number is equal to or greater than a certain threshold, the first color image is adjusted by the image deformation unit based on the plurality of corresponding points detected by the corresponding point detection unit so that the positions of the corresponding points coincide with each other. A deformed third color image is generated, and the corresponding point of the second color image acquired by the image acquiring unit and the corresponding point of the third color image generated by the image deforming unit are determined by the image deformation result determining unit. It is determined whether or not the amount of misregistration is greater than or equal to a threshold, and when the color reproduction processing unit determines that the amount of misregistration of the corresponding point is greater than or equal to the threshold, the image acquisition The second color image acquired by the means is used to perform subject color reproduction processing, and the image deformation result determination means determines the amount of positional deviation of the corresponding point as a threshold value. If it is determined that the full, with the third color image generated by the acquired second color image and said image modification means by said image acquiring means, performs color reproduction processing of an object. When it is determined that the number of corresponding points detected by the detected corresponding point determination unit is less than a threshold value, subject color reproduction processing is performed using the second color image acquired by the image acquisition unit.

  As described above, when a sufficient number of corresponding points cannot be obtained using the second color image captured by the second imaging unit without the color filter as a reference image, the second color image is used to Perform color reproduction processing. Further, when a sufficient number of corresponding points are obtained, a third color image obtained by modifying the first color image captured by the first imaging unit having the color filter so that the positions of the corresponding points coincide with each other. Is generated, and the color reproduction process of the subject is performed using the second color image when the positional deviation amount of the corresponding point with the third color image is equal to or larger than the threshold value. By these processes, the accuracy of the color reproduction result is improved as compared with the conventional method.

  The color reproduction process calculates (estimates) the color of the subject (for example, spectrum, spectral reflectance, color, RGB / multi-primary pixel value, etc.) and uses the calculated value as a pixel value (for example, a color reproduction result image). , Spectral image, spectral reflectance image, RGB image, etc.).

  An image processing apparatus according to a third aspect is the image processing apparatus according to the first aspect or the second aspect, wherein each of the first color image and the second color image acquired by the image acquisition means is obtained. The image processing unit further includes an image division processing unit that divides the image into small region images, and an image pasting processing unit, wherein the corresponding point detection unit is configured to output the first color image acquired by the image acquisition unit for each of the small region images. A plurality of corresponding points between the small region image and the small region image corresponding to the second color image are detected, and the image transformation unit detects the corresponding point detected by the corresponding point detection unit for each small region image. Based on the corresponding points, a small area image of a third color image is generated by deforming the small area image of the first color image so that the positions of the corresponding points coincide with each other. Small territory For each image, the displacement of the corresponding point of the small area image corresponding to the small color image of the second color image acquired by the image acquiring means and the third color image generated by the image deformation means. Determining whether or not the amount is greater than or equal to a threshold value, the color reproduction processing means, for each of the small area images, the image deformation result determination means for the small area image the positional deviation amount of the corresponding point is greater than or equal to the threshold value If it is determined that there is a color reproduction process of a subject in the small region image using the small region image of the second color image acquired by the image acquisition unit, the image deformation result determination unit When it is determined that the positional deviation amount of the corresponding point is less than a threshold value for the small area image, the small area of the second color image acquired by the image acquisition means Using the small area image corresponding to the image and the third color image generated by the image transformation means, color reproduction processing of the subject in the small area image is performed, and the image combining processing means One image is generated from each processed small area image.

  The image processing apparatus according to the third aspect of the invention generates a color reproduction image for each small area image based on the same method as the first aspect or the second aspect of the invention after dividing into small area images. The obtained color reproduction image results for each small region are combined to obtain one color reproduction image.

  An image processing apparatus according to a fourth aspect of the present invention is a first color image captured using a color filter that transmits a part of wavelengths of colors that can be captured by the camera, and a second image captured without using the color filter. Image acquisition means for acquiring a color image, image division processing means for dividing each of the first color image and the second color image acquired by the image acquisition means into small area images, and each small area image Corresponding point detecting means for detecting a plurality of corresponding points between the small area image of the first color image acquired by the image acquisition means and the corresponding small area image of the second color image; For each area image, based on the corresponding points detected for the small area image by the corresponding point detection means, the small area image of the first color image is changed so that the positions of the corresponding points coincide with each other. Generated by the image deformation means for generating a small area image of the third color image, and the small area image of the second color image acquired by the image acquisition means and the image deformation means for each of the small area images. In addition, an image deformation result determination unit that determines whether or not a positional deviation amount of the corresponding point of the small area image corresponding to the third color image is greater than or equal to a threshold value, and the image deformation result determination unit determines the corresponding point. Image estimation means for estimating the small area image corresponding to the first color image from the small area image of the second color image for each of the small area images determined to have a positional deviation amount equal to or greater than a threshold value. And the small area image of the first color image corresponding to the small area image determined by the image deformation result determination means that the positional deviation amount of the corresponding point is less than or equal to And an image pasting unit that generates an image obtained by pasting the small area images of the first color image estimated by the image estimating unit, and the second color image and the image pasting obtained by the image obtaining unit. And color reproduction processing means for performing color reproduction processing of the subject using the image generated by the matching processing means.

  In this way, for each small area image, the first image pickup having the color filter so that the positions of the corresponding points coincide with each other using the second color image taken by the second image pickup means without the color filter as the reference image. When a third color image obtained by deforming the first color image photographed by the means is generated, and the positional deviation amount of the corresponding point with the third color image is greater than or equal to the threshold value for each small region image, the second color image By performing the color reproduction process of the subject using, the accuracy of the color reproduction result is improved as compared with the conventional method.

  The program of the present invention is an image processing program for causing a computer to function as each means constituting the image processing apparatus.

  As described above, according to the image processing apparatus, method, and program of the present invention, the positions of the corresponding points coincide with each other using the second color image captured by the second imaging unit without the color filter as the reference image. As described above, when a third color image obtained by deforming the first color image photographed by the first photographing means having the color filter is generated, and the amount of positional deviation of the corresponding point with the third color image is equal to or larger than the threshold value. By performing the color reproduction process of the subject using the second color image, an effect that the accuracy of the color reproduction result is improved as compared with the conventional method can be obtained.

1 is a schematic diagram illustrating a configuration of an image processing apparatus according to a first embodiment. It is a figure which shows an example of the image input device which inputs a multiband image. (A) It is a figure which shows an example of the transmission wavelength of a color filter, (B) It is a figure which shows an example of the band obtained by a color filter. It is a figure which shows an example of a structure of the corresponding point detection part in the image processing apparatus which concerns on 1st Embodiment. 3 is a flowchart showing the contents of a color reproduction processing routine in the image processing apparatus according to the first embodiment. It is the schematic which shows the structure of the image processing apparatus which concerns on 2nd Embodiment. It is a flowchart which shows the content of the color reproduction process routine in the image processing apparatus which concerns on 2nd Embodiment. It is the schematic which shows the structure of the image processing apparatus which concerns on another example. It is the schematic which shows the structure of the image processing apparatus which concerns on 3rd Embodiment. 14 is a flowchart showing the contents of a color reproduction processing routine in the image processing apparatus according to the third embodiment. It is the schematic which shows the structure of the image processing apparatus which concerns on 4th Embodiment. It is a flowchart which shows the content of the color reproduction process routine in the image processing apparatus which concerns on 4th Embodiment. It is a figure which shows an example of the image input device which inputs a multiband image. It is the schematic which shows the structure of the conventional image processing apparatus. It is a figure which shows the principle image of spectral imaging.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
<System configuration>
An image processing apparatus according to the first embodiment will be described.

  An image processing apparatus 10 according to the first embodiment includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) that stores a program for executing a color reproduction processing routine described later. ). As shown in FIG. 1, this computer functionally includes a corresponding point detection unit 14, an image deformation processing unit 15, an image deformation result determination unit 16, a deformation result image output unit 17, a color reproduction processing unit 18, and a color It can be expressed by a configuration including the reproduction processing unit 19.

  The image processing apparatus 10 receives a three-band image 1 obtained by photographing with the image input unit 11 and an RGB (three-band) image 2 obtained by photographing with the image input unit 12. The image input unit 11 has an RGB camera 11a, and the image input unit 12 has an RGB camera 12a. Each of the RGB camera 11a and the RGB camera 12a is a digital camera including a sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and filters for each color of RGB. As shown in FIG. 2, the image input unit 11 is a color filter that transmits only a part of the transmission wavelengths of the filters corresponding to the original R, G, and B colors of the RGB camera 11 a in addition to the RGB camera 11 a. 11b (see FIG. 3A). The digital signal values of each of the six bands are obtained as the pixel values of the six-band image by the image input unit 11 and the image input unit 12 (see FIG. 3B). In the following, a 6-band image will be described as an example of a multiband image, but the multiband image is not limited to a 6-band image. The color filter in the image input unit 11 may be a band pass filter that transmits only a part of the wavelength sensitivity of the camera. The camera in the image input unit 11 may be a black and white camera.

  The corresponding point detection unit 14 of the image processing apparatus 10 takes an image without a filter based on the data of the 3-band image 1 input from the image input unit 11 and the data of the 3-band image 2 input from the image input unit 12. A plurality of corresponding points between two three-band images of the three-band image 1 and the three-band image 2 photographed using a filter are detected by using the three-band image 2 thus obtained as a reference image.

  Corresponding points between two 3-band images can be detected by, for example, phase-only correlation method (references (Takafumi Aoki, Koichi Ito, Takuma Shibahara, Kiyoshi Nagashima, “High-precision machine vision based on phase-only correlation method− Aiming at Image Sensing Technology that Crosses the Pixel Resolution Wall ”, see Fundamentals Review Vol.1 No.1)). In this case, as shown in FIG. 4, the corresponding point detection unit 14 includes black and white image generation units 112a and 112b, phase image generation units 113a and 113b, a correlation image generation unit 114, and a corresponding point detection unit 115. Can be expressed in a configuration including According to the corresponding point detection unit 14, the monochrome images are generated from the 3-band image 1 and the 3-band image 2 by the monochrome image generation units 112a and 112b, respectively, and the monochrome image generated by the phase image generation units 113a and 113b is generated. A phase image is generated from each. Then, the correlation image generation unit 114 generates a correlation image from the two generated phase images, and the corresponding point detection unit 115 shifts between the two images based on the coordinates having the maximum correlation value in the generated correlation image. Find the quantity and detect the corresponding points. In obtaining the coordinate having the maximum correlation value, it is possible to detect the shift amount and the coordinates of the corresponding point with subpixel accuracy by interpolating a Gaussian function, for example.

  The corresponding point detection unit 14 outputs the number of detected corresponding points, the coordinates of the corresponding points in the 3-band images 1 and 2, and the deviation width as a result of the detection of the corresponding points.

  The image deformation processing unit 15 deforms an image (3-band image 1) photographed with the color filter 11b by using the obtained corresponding point detection result, and creates a new 3-band image (3-band image 3). ) Is generated.

  In the deformation process by the image deformation processing unit 15, based on the corresponding point information obtained by the corresponding point detection unit 14, an image deformation parameter (horizontal / vertical shift amount in the case of projective transformation, three axes (X axis, Y Rotation angle and magnification with respect to each of the axis and the Z-axis) (when using non-linear methods such as spline interpolation, the model parameters) are calculated, and the calculated image deformation parameters are assumed to be a deformation model (projective transformation or non-linear transformation). As a result, deformation processing is performed on the 3-band image 1 so that the positions of the corresponding points coincide with those of the 3-band image 2. The method for image deformation is not particularly limited here, but for example, projective transformation (linear transformation) or nonlinear image deformation processing is performed.

  Based on the 3-band image 2 and the 3-band image 3, the image deformation result determination unit 16 determines whether image deformation has been performed correctly. In the determination process, in order to determine whether the corresponding points between the 3-band image 2 and the 3-band image 3 exist on the same coordinates, the amount of positional deviation of the corresponding points between the images is calculated again. In addition, a threshold value regarding a positional deviation amount (distance) between corresponding points is set in advance (how to determine the threshold value is not limited). For example, if the mean square error between corresponding points is equal to or less than the threshold value, the image is correctly deformed. If it is equal to or greater than the threshold value, it is determined that the image has not been correctly deformed. Note that the mean square error between corresponding points is an example of the amount of positional deviation between corresponding points.

  The deformation result image output unit 17 expands the 3-band image 2 and the 3-band image 3 obtained when the image deformation result determination unit 16 determines that the image is correctly deformed on the memory, displays the image on the monitor, Or output to a file.

  The color reproduction processing unit 18 performs color reproduction processing of the subject using only the 3-band image 2 when the image deformation result determination unit 16 determines that the image deformation has not been performed correctly, and generates a color reproduction result image. And output.

  In color reproduction processing, 3-band image 2, color reproduction parameters (information necessary for spectral reflectance estimation such as illumination light spectrum and camera sensitivity at the time of photographing, and illumination light spectrum and display device characteristics in the illumination environment to be reproduced) ), The color (spectrum, spectral reflectance, color, RGB / multi-primary color signal value) of the subject in the three-band image 2 is calculated by the Wiener estimation method or the like, and the color reproduction result image (spectral reflectance image) Or RGB / multi-primary color image).

  For example, the spectral reflectance for each pixel is estimated from the 3-band image 2 using the Wiener estimation method, and a spectral reflectance image having the estimated spectral reflectance as a pixel value is generated. For example, when an RGB image is generated from the spectral reflectance image, for example, an illumination light spectrum in a predetermined illumination environment to be reproduced and a predetermined display device are generated with respect to the generated spectral reflectance image. An RGB image may be generated using the above characteristics. Specifically, the reflected light spectrum is calculated by multiplying the value of each wavelength of the estimated spectral reflectance corresponding to each pixel by the illumination light spectrum in the lighting environment to be reproduced in advance, and the same color for each pixel. The XYZ colorimetric values are obtained by multiplying the functions. Based on the XYZ colorimetric values calculated for each pixel, based on the characteristics of the display device (the relational expression between the XYZ colorimetric values and the RGB value of the display device, and the input / output relationship regarding the display device such as the tone curve), the R value, G value, And the B value are calculated, and an RGB image is generated with the pixel value of each pixel as the calculated RGB value. A spectrum-RGB value conversion matrix may be obtained in advance from the color matching function and the characteristics of the display device, the reflection spectrum may be multiplied by the corresponding matrix, and an RGB image may be generated through correction using a tone curve.

  The color reproduction processing unit 19 performs color reproduction processing using the 3-band image 2 and the 3-band image 3 when the image deformation result determination unit 16 determines that the image deformation has been correctly performed, and the color reproduction result image Is generated and output.

  In color reproduction processing, 3-band image 2, 3-band image 3, color reproduction parameters (illumination light spectrum at the time of photographing, information necessary for spectral reflectance estimation such as camera sensitivity, and illumination light spectrum in the illumination environment to be reproduced) And the color of the subject (spectrum, spectral reflectance, color, RGB / multi-primary color signal value) from the 3-band image 2 and 3-band image 3 based on the characteristics of the display device and the display device) Then, a color reproduction result image (spectral reflectance image or RGB / multi-primary color image) is generated and output.

<Operation of image processing apparatus>
Next, the operation of the image processing apparatus 10 according to the first embodiment will be described. First, after the 3-band image 1 and the 3-band image 2 are captured by the image input unit 11 and the image input unit 12, the image processing apparatus 10 executes a color reproduction processing routine shown in FIG.

  In step S101, the data of the 3-band image 1 captured by the image input unit 11 and the data of the 3-band image 2 captured by the image input unit 12 are acquired.

  In step S102, a plurality of corresponding points between the 3-band image 1 and the 3-band image 2 acquired in step S101 are detected. In step S103, based on the corresponding point information detected in step S102, image deformation processing is performed on the 3-band image 1 acquired in step S101 to generate a 3-band image 3.

  In the next step S104, a positional deviation amount (for example, mean square error) of corresponding points between the 3-band image 2 acquired in step S101 and the 3-band image 3 generated in step S103 is calculated. In step S105, it is determined whether or not the displacement amount calculated in step S104 is greater than or equal to a threshold value. If the calculated misregistration amount is equal to or larger than the threshold, in step S106, subject color reproduction processing is performed using only the three-band image 2, the result of color reproduction processing is output, and the color reproduction processing routine is executed. finish.

  On the other hand, if it is determined in step S105 that the calculated positional deviation amount of the corresponding point is less than the threshold value, the three-band image 2 acquired in step S101 and the step S103 are generated in step S107. 3 band image 3 is output. In the next step S108, subject color reproduction processing is performed using the three-band image 2 acquired in step S101 and the three-band image 3 generated in step S103, and the result of the color reproduction processing is output. The reproduction processing routine is terminated.

  As described above, according to the image processing apparatus of the first embodiment, using the 3-band image 2 photographed by the RGB camera without a color filter as the reference image, the positions of the corresponding points match. When a 3-band image 3 is generated by deforming the 3-band image 1 photographed by an RGB camera equipped with a color filter, and the amount of positional deviation (for example, the mean square error) of the corresponding point with the 3-band image 3 is greater than or equal to a threshold value Furthermore, by performing the color reproduction process of the subject using only the three-band image 2, the accuracy of the color reproduction result and the image quality of the color reproduction result image are improved as compared with the conventional method.

  The false color around the edge when the image transformation process fails (although the area is very small) is extremely noticeable to the human eye. On the other hand, even if the color reproduction process is performed from the three primary color images, no significant deterioration is felt by humans. Therefore, even when the image deformation process fails and the generation of multiband shooting fails, at least the accuracy almost equivalent to the color reproduction result from the three-band image generally used at present is guaranteed.

[Second Embodiment]
<System configuration>
Next, a second embodiment will be described. In addition, about the image processing apparatus of 2nd Embodiment, about the structure similar to the image processing apparatus 10 of 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  As shown in FIG. 6, the computer constituting the image processing apparatus 210 according to the second embodiment functionally has a corresponding point detection unit 14, a detection corresponding point number determination unit 214, an image deformation processing unit 15, and an image deformation. The result determination unit 16, the deformation result image output unit 17, the color reproduction processing unit 18, and the color reproduction processing unit 19 can be represented.

  The detected corresponding point number determination unit 214 determines whether or not the number of corresponding points detected by the corresponding point detection unit 14 is equal to or greater than a threshold value (for example, the threshold value when the image deformation process is projective deformation is equal to or greater than six points). If the number of detected corresponding points is equal to or greater than the threshold value, it is determined that a sufficient number of corresponding points are detected for deformation of the image. On the other hand, if the detected number of corresponding points is less than the threshold value, the image It is determined that a sufficient number of corresponding points have not been detected.

  When the detected corresponding point number determination unit 214 determines that a sufficient number of corresponding points for image deformation have been detected, the image deformation processing unit 15 uses the corresponding point information obtained by the corresponding point detection unit 14 as information. Then, the 3-band image 2 photographed with the color filter 11b is deformed to generate a new 3-band image 3.

  When the detection corresponding point number determination unit 214 determines that a sufficient number of corresponding points for image deformation has not been detected, the color reproduction processing unit 18 performs color reproduction processing of the subject using only the three-band image 2. Then, a color reproduction result image is generated and output.

<Operation of image processing apparatus>
Next, a color reproduction processing routine executed by the image processing apparatus 210 according to the second embodiment will be described with reference to FIG. Note that the same processes as those in the color reproduction process routine in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  In step S101, the data of the 3-band image 1 captured by the image input unit 11 and the data of the 3-band image 2 captured by the image input unit 12 are acquired.

  In step S102, a plurality of corresponding points between the 3-band image 1 and the 3-band image 2 acquired in step S101 are detected. In step S201, it is determined whether or not the number of corresponding points detected in step S102 is greater than or equal to a threshold value. If the number of detected corresponding points is greater than or equal to the threshold, the process proceeds to step S103. On the other hand, if the number of detected corresponding points is less than the threshold, the process proceeds to step S106.

  In step S103, image deformation processing is performed on the 3-band image 1 acquired in step S101 to generate a 3-band image 3. In the next step S104, a positional deviation amount (for example, mean square error) of corresponding points between the 3-band image 2 acquired in step S101 and the 3-band image 3 generated in step S103 is calculated. In step S105, it is determined whether or not the positional deviation amount of the corresponding point calculated in step S104 is equal to or greater than a threshold value. If the calculated positional deviation amount of the corresponding point is equal to or larger than the threshold value, the color reproduction process of the subject is performed using the 3-band image 2 in step S106, the result of the color reproduction process is output, and the color reproduction process is performed. End the routine.

  On the other hand, if it is determined in step S105 that the calculated positional deviation amount of the corresponding point is less than the threshold value, the three-band image 2 acquired in step S101 and the step S103 are generated in step S107. 3 band image 3 is output. In the next step S108, subject color reproduction processing is performed using the three-band image 2 acquired in step S101 and the three-band image 3 generated in step S103, and the result of the color reproduction processing is output. The reproduction processing routine is terminated.

  As described above, according to the image processing apparatus of the second embodiment, using the 3-band image 2 photographed by the RGB camera without the color filter as the reference image, the positions of the corresponding points match. A 3-band image 3 obtained by deforming the 3-band image 1 photographed by an RGB camera equipped with a color filter is generated, and the number of corresponding points sufficient for image deformation is not detected, and the correspondence with the 3-band image 3 When the point misregistration amount is equal to or greater than the threshold value, the color reproduction process of the subject using the three-band image 2 improves the accuracy of the color reproduction result as compared with the conventional method.

  In the first embodiment and the second embodiment described above, when the image deformation result determination unit 16 determines that the image deformation has not been performed correctly, the image deformation is performed again and the image deformation is performed. You may make it redo the determination in the result determination part 16. FIG. For example, as illustrated in FIG. 8, when the image deformation result determination unit 16 determines that image deformation has not been performed correctly, the image deformation processing unit 215 performs image deformation processing again, and the image deformation result determination unit 16 is repeated until it is determined that the image has been correctly deformed. When a certain number of times is exceeded, it is determined that correct image deformation is impossible, and the color reproduction processing unit 18 performs color reproduction processing using only the three-band image 2.

[Third Embodiment]
<System configuration>
Next, a third embodiment will be described. In addition, about the image processing apparatus of 3rd Embodiment, about the structure similar to the image processing apparatus 10 of 1st Embodiment, the same code | symbol is attached | subjected and detailed description is abbreviate | omitted.

  The computer constituting the image processing apparatus 310 of the third embodiment functionally has an image division processing unit 312, a color reproduction result image generation unit 314, and an image pasting processing unit 316 as shown in FIG. Can be expressed in a configuration including

  The image division processing unit 312 divides each of the 3-band image 1 obtained by the image input unit 11 and the 3-band image 2 obtained by the image input unit 12 into small region images. The 3-band image 1 is divided into areas having the same size as the small-area image of the 3-band image 2 or a larger area than the small area obtained by dividing the 3-band image 2. When the 3-band image 1 and the 3-band image 2 are divided into small-area images, the 3-band image 1 and the 3-band image 1 In both of the three-band images 2, it is preferable that the divided small region images have an overlap.

  Further, the image division processing unit 312 preferably divides the image into small area images in which the number of pixels on one side is a power of 2, 64 pixels, 128 pixels, 256 pixels, or the like in consideration of speeding up of the calculation process.

  The color reproduction result image generation unit 314 includes a corresponding point detection unit 14, an image deformation processing unit 15, an image deformation result determination unit 16, a deformation result image output unit 17, a color reproduction processing unit 18, and a color reproduction processing unit 19. Each of the corresponding point detection unit 14, the image deformation processing unit 15, the image deformation result determination unit 16, the deformation result image output unit 17, the color reproduction processing unit 18, and the color reproduction processing unit 19 is divided by the image division processing unit 312. The same processing as in the first embodiment is performed for each combination of the corresponding small area image of the 3-band image 1 and the small area image of the 3-band image 2. The color reproduction result image generation unit 314 outputs a color reproduction result image of the small area image for each small area image. Note that the color reproduction result image generation unit 314 may perform the same processing as in the second embodiment.

  In the image deformation processing unit 15, only local corresponding points within the small area affect the image deformation parameters. Therefore, the positional deviation between the captured images is more accurately performed than using all the corresponding points obtained from the entire image. Can be corrected.

  Further, the image pasting processing unit 316 tiles the color reproduction result image for each small region image generated by the color reproduction result image generation unit 314, and the image has the size of the original image before being divided into the small region images. A color reproduction result image is generated and output by pasting the images together so that

<Operation of image processing apparatus>
Next, a color reproduction processing routine executed by the image processing apparatus 310 according to the third embodiment will be described with reference to FIG. Note that the same processes as those in the color reproduction process routine in the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  In step S101, the data of the 3-band image 1 captured by the image input unit 11 and the data of the 3-band image 2 captured by the image input unit 12 are acquired.

  In step S302, each of the 3-band image 1 and the 3-band image 2 acquired in step S101 is divided into small area images. In step S303, the combination of the small area images of the corresponding 3-band image 1 and 3-band image 2 is set as the small area image to be processed among the small area images divided in step S302.

  In step S304, the color reproduction processing routine shown in FIG. 5 described in the first embodiment is applied to each of the small area images of the 3-band image 1 and the 3-band image 2 set as the processing target. Similar processing is executed to generate a color reproduction result image for the small region image.

  In step S305, it is determined whether or not the processes in steps S303 and S304 have been executed for all the small area images. If there are small area images in which the processes in steps S303 and S304 have not been performed, Returning to step S303, the small area image is set as a processing target. On the other hand, if it is determined that the processes in steps S303 and S304 have been executed for all the small area images, the process proceeds to step S306.

  In step S306, the color reproduction result images for the respective small area images obtained in step S304 are pasted to generate and output one color reproduction result image, and the color reproduction processing routine ends.

  As described above, according to the image processing apparatus of the third embodiment, for each small region image, the position of each corresponding point using the 3-band image 2 photographed by the RGB camera without a color filter as the reference image. 3 band image 3 is generated by deforming 3 band image 1 photographed by an RGB camera equipped with a color filter so as to match, and for each small area image, the amount of positional deviation of the corresponding point with 3 band image 3 When (for example, the mean square error) is equal to or greater than a threshold value, the color reproduction result and the image quality of the color reproduction result can be improved compared to the conventional method by performing the color reproduction process of the subject using only the small area image of the 3-band image 2. improves.

  The false color around the edge when the image transformation process fails (although the area is very small) is extremely noticeable to the human eye. On the other hand, when processing a small area that has failed in image transformation processing with three primary colors, the overall area is very small, so no significant degradation is felt by humans as color reproduction processing from three primary color images. . Even when the deformation process of the small area image fails, the color reproduction accuracy from the three-band image is guaranteed at least.

  In the above-described embodiment, the color reproduction result image generation unit 314 includes the corresponding point detection unit 14, the image deformation processing unit 15, the image deformation result determination unit 16, the deformation result image output unit 17, the color reproduction processing unit 18, In addition, the case where the image is expressed by a configuration including the color reproduction processing unit 19 has been described as an example, but the present invention is not limited to this. The color reproduction result image generation unit 314 includes a corresponding point detection unit 14, a detection corresponding point number determination unit 214, an image deformation processing unit 15, an image deformation result determination unit 16, a deformation result image output unit 17, a color reproduction processing unit 18, and a color. You may make it represent with the structure containing the reproduction process part 19. FIG. In this case, the corresponding point detection unit 14, the detection corresponding point number determination unit 214, the image deformation processing unit 15, the image deformation result determination unit 16, the deformation result image output unit 17, the color reproduction processing unit 18, and the color reproduction processing unit 19 For each combination of the corresponding small area image of the 3-band image 1 and the small area image of the 3-band image 2 divided by the image division processing unit 312. I do.

[Fourth Embodiment]
<System configuration>
Next, a fourth embodiment will be described. Note that, in the image processing device according to the fourth embodiment, the same reference numerals are given to the same configurations as those of the image processing device 10 according to the first embodiment and the image processing device 310 according to the third embodiment. Detailed description is omitted.

  As shown in FIG. 11, the computer constituting the image processing apparatus 410 according to the fourth embodiment functionally includes an image division processing unit 312, a small region deformation determination unit 414, an image band number conversion processing unit 416, It can be expressed by a configuration including an image pasting processing unit 418, a deformation result image output unit 420, and a color reproduction processing unit 422. The image band number conversion processing unit 416 is an example of an image estimation unit.

  The small region deformation determination unit 414 includes a corresponding point detection unit 14, a detection corresponding point number determination unit 214, an image deformation processing unit 15, and an image deformation result determination unit 16, and includes a corresponding point detection unit 14, a detection corresponding point number determination unit 214, Each of the image deformation processing unit 15 and the image deformation result determination unit 16 is divided by the image division processing unit 312 for each combination of the corresponding small area image of the 3-band image 1 and the small area image of the 3-band image 2. The same processing as in the second embodiment is performed. For each small region image, the small region deformation determination unit 414 determines the small region image of the 3-band image 1 and the small region image of the 3-band image 3, the determination result of the detection corresponding point number determination unit 214, and the image deformation result determination. The determination result of the unit 16 is output. In addition, the small region deformation determination unit 414 may perform the same processing as in the first embodiment.

  The image band number conversion processing unit 416 includes a small area image determined by the detection corresponding point number determination unit 214 that a sufficient number of corresponding points have not been detected for image deformation, and the image deformation result determination unit 16 to correctly perform image deformation. For each of the small area images determined not to be performed, the small area image of the 3-band image 1 that would be obtained by the image input unit 11 is estimated and generated from the small area image of the 3-band image 2. Then, the small area image and the image data of the 3-band image 3 after the image transformation are replaced.

  In the estimation process of the 3-band image 1, the spectral reflectance is estimated from the small region image of the 3-band image 2, and the image input unit 11 uses the illumination light spectrum at the time of shooting and the optical characteristics of the image input unit 11. The small region image of an image equivalent to that obtained is estimated. For example, the spectral reflectance for each pixel is estimated from the small area image of the 3-band image 2 using a Wiener estimation method or the like, and a spectral reflectance image having the estimated spectral reflectance as a pixel value is generated. A reflected light spectrum to be observed is calculated by multiplying the generated spectral reflectance image by a predetermined illumination light spectrum at the time of photographing, and the camera 11a in the image input unit 11 is compared with the obtained reflected light spectrum. By multiplying a predetermined spectral sensitivity that combines the color filter 11b, a signal value that would have been obtained by the image input unit 11 is calculated for each pixel, and an image equivalent to that obtained by the image input unit 11 is obtained. The small area image is generated.

  The image pasting processing unit 418 includes the small region images of the three-band image 3 corresponding to the small region image determined to have been correctly deformed by the image deformation result determination unit 16 and the image band number conversion processing unit 416. The generated small area images are pasted together to generate a 3-band image 3 having the original image size. Alternatively, one 6-band image may be generated by further using the 3-band image 2.

  The deformation result image output unit 420 develops two types of three-band images (or one six-band image) composed of the three-band image 3 and the three-band image 2 generated by the image pasting processing unit 418 in the memory. Display on monitor or output to file.

  The color reproduction processing unit 422 performs color reproduction using two types of three-band images (or one six-band image) including the three-band image 3 and the three-band image 2 generated by the image combining processing unit 418. Similar to the processing unit 19, subject color reproduction processing is performed, and a color reproduction result image is generated and output.

<Operation of image processing apparatus>
Next, a color reproduction processing routine executed by the image processing apparatus 410 according to the fourth embodiment will be described with reference to FIG. Note that the same processing as the color reproduction processing routine in the first embodiment, the second embodiment, and the third embodiment is denoted by the same reference numeral, and detailed description thereof is omitted.

  In step S101, the data of the 3-band image 1 captured by the image input unit 11 and the data of the 3-band image 2 captured by the image input unit 12 are acquired.

  In step S302, each of the 3-band image 1 and the 3-band image 2 acquired in step S101 is divided into small area images. In step S303, the combination of the small area images of the corresponding 3-band image 1 and 3-band image 2 is set as the small area image to be processed among the small area images divided in step S302.

  In step S404, the color reproduction processing routine shown in FIG. 7 described in the first embodiment is applied to each of the small area images of the 3-band image 1 and the 3-band image 2 set as the processing target. The same processing as in steps S101, S102, S201, S103, S104, and S105 is executed, and corresponding point detection, determination of the number of detected corresponding points, image deformation, and image deformation result are determined for the small region image. .

  In step S305, it is determined whether or not the processes in steps S303 and S404 have been executed for all the small area images. If there are small area images in which the processes in steps S303 and S404 have not been performed, Returning to step S303, the small area image is set as a processing target. On the other hand, if it is determined that the processes in steps S303 and S404 have been executed for all the small area images, the process proceeds to step S406.

  In step S406, based on the determination result for each small region image obtained in step S404, the small region image determined to have not detected a sufficient number of corresponding points for image deformation, and an image deformation result determination unit For each of the small area images that are determined not to have been correctly deformed by 16, the small area image of the three-band image 1 that will be obtained by the image input unit 11 from the small area image of the three-band image 2. Is generated by substituting and substituting the small area image of the 3-band image 3 with the image data.

  In step S407, each small area image of the 3-band image 3 obtained as a result of the replacement in step S406 is pasted to generate one 3-band image 3. In the next step S408, using the 3-band image 2 acquired in step S101 and the 3-band image 3 generated in step S407, subject color reproduction processing is performed, and the result of color reproduction processing is output. The reproduction processing routine is terminated.

  As described above, according to the image processing apparatus of the fourth embodiment, for each small region image, the position of each corresponding point using the 3-band image 2 photographed by an RGB camera without a color filter as a reference image. 3 band image 3 is generated by deforming 3 band image 1 photographed by an RGB camera equipped with a color filter so as to match, and for each small area image, the amount of positional deviation of the corresponding point with 3 band image 3 When (for example, the mean square error) is equal to or greater than the threshold, an image similar to the 3-band image 1 is estimated using the small-area image of the 3-band image 2, and an image obtained by pasting the small-area images is used. By performing the subject color reproduction process, the accuracy of the color reproduction result is improved as compared with the conventional method.

  In the above embodiment, the small region deformation determination unit 414 is represented by a configuration including the corresponding point detection unit 14, the detected corresponding point number determination unit 214, the image deformation processing unit 15, and the image deformation result determination unit 16. Although the case has been described as an example, the present invention is not limited to this. The small region deformation determination unit 414 may be represented by a configuration including the corresponding point detection unit 14, the image deformation processing unit 15, and the image deformation result determination unit 16 described in the first embodiment. . In this case, the image band number conversion processing unit 416, for each of the small region images determined by the image deformation result determination unit 16 that the image deformation has not been performed correctly, from the small region image of the 3-band image 2, What is necessary is just to estimate the said small area image of the 3 band image 1 which will be obtained by the image input part 11. FIG.

  In the first to fourth embodiments, an example in which two three-band images (total six-band images) are input to the image processing apparatus has been described. However, the present invention is not limited to this as long as it is a multiband image. Of course.

  In the first to fourth embodiments, as an image input device for inputting a multiband image, an image input unit 11 having an RGB camera 11a and a color filter 11b, and an image input unit 12 having an RGB camera 12a However, the present invention is not limited to this. For example, as shown in FIG. 13, the present invention can also be applied to a multi-lens camera system in which a plurality of RGB cameras and a single color (monochrome) camera having a single color filter are combined. The color filter in the image input unit 11 may be a band pass filter that transmits only a part of the wavelength sensitivity of the camera. The camera in the image input unit 11 may be a black and white camera.

  In the first to fourth embodiments, the case where the three-band image 1 and the three-band image 2 are captured in synchronization by the two image input units has been described as an example. However, the present invention is not limited to this. Instead, the 3-band image 1 and the 3-band image 2 captured in time series from one image input unit may be used. In this case, a mechanism for switching between mounting and non-mounting of the color filter is provided so that the 3-band image 1 is captured with the color filter mounted and the 3-band image 2 is captured with the color filter removed. That's fine. Moreover, you may have three or more image input parts.

  Further, the above-described image processing apparatus has a computer system inside, but the computer system includes a homepage providing environment (or display environment) if the WWW system is used.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

10, 210, 310, 410 Image processing apparatus 11a Camera 11b Color filter 11 Image input unit 12a Camera 12 Image input unit 14 Corresponding point detection unit 15 Image deformation processing unit 16 Image deformation result determination unit 18, 19, 422 Color reproduction processing unit 19 color reproduction processing unit 214 detection-corresponding point determination unit 312 image division processing unit 314 color reproduction result image generation unit 316, 418 image combination processing unit 414 small region deformation determination unit 416 image band number conversion processing unit

Claims (7)

Image acquisition means for acquiring a first color image captured using a color filter that transmits a part of wavelengths of colors that can be captured by the camera, and a second color image captured without using the color filter;
Corresponding point detecting means for detecting a plurality of corresponding points between the first color image and the second color image acquired by the image acquiring means;
Image deformation means for generating a third color image obtained by deforming the first color image based on the plurality of corresponding points detected by the corresponding point detection means so that the positions of the corresponding points coincide with each other;
Image deformation result determination for determining whether the amount of positional deviation of the corresponding points of the second color image acquired by the image acquisition unit and the third color image generated by the image deformation unit is greater than or equal to a threshold value. Means,
When it is determined by the image deformation result determination means that the amount of positional deviation of the corresponding point is greater than or equal to a threshold value, using the second color image acquired by the image acquisition means, subject color reproduction processing is performed, When it is determined by the image deformation result determination means that the positional deviation amount of the corresponding point is less than a threshold value, the second color image acquired by the image acquisition means and the third color generated by the image deformation means. Color reproduction processing means for performing color reproduction processing of a subject using a color image;
An image processing apparatus. Further comprising a detection corresponding point number determination means for determining whether or not the number of the corresponding points detected by the corresponding point detection means is a predetermined value or more,
The image deformation means deforms the first color image so that the positions of the corresponding points coincide when the number of corresponding points is determined to be equal to or greater than a predetermined value by the detected corresponding point number determination means. Generating the third color image;
The color reproduction processing unit uses the second color image acquired by the image acquisition unit when the number of corresponding points is determined to be less than a predetermined value by the detection corresponding point number determination unit. The image processing apparatus according to claim 1, wherein color reproduction processing is performed. Image division processing means for dividing each of the first color image and the second color image acquired by the image acquisition means into small area images;
Further comprising image pasting processing means,
The corresponding point detecting means corresponds to, for each small area image, a corresponding point between the small area image of the first color image acquired by the image acquiring means and the corresponding small area image of the second color image. Multiple detections,
The image deforming means, for each of the small area images, based on the corresponding points detected by the corresponding point detecting means, the small area images of the first color image so that the positions of the corresponding points coincide with each other. Generating a transformed small area image of the third color image;
The image deformation result determination means corresponds to the small area image of the second color image acquired by the image acquisition means and the third color image generated by the image deformation means for each of the small area images. Determining whether the amount of positional deviation of the corresponding point of the small area image is greater than or equal to a threshold,
The color reproduction processing means is acquired by the image acquisition means for each of the small area images when the image deformation result determination means determines that the amount of positional deviation of the corresponding points for the small area image is greater than or equal to a threshold value. Using the small area image of the second color image that has been processed, subject color reproduction processing in the small area image is performed, and the positional deviation amount of the corresponding point for the small area image by the image deformation result determination unit is a threshold value. If it is determined that the image is less than the image, the small area image of the second color image acquired by the image acquisition means and the small area image corresponding to the third color image generated by the image deformation means are used. And subject color reproduction processing in the small area image,
The image processing apparatus according to claim 1, wherein the image pasting processing unit generates one image from each small region image subjected to the color reproduction processing. Image acquisition means for acquiring a first color image captured using a color filter that transmits a part of wavelengths of colors that can be captured by the camera, and a second color image captured without using the color filter;
Image division processing means for dividing each of the first color image and the second color image acquired by the image acquisition means into small area images;
Corresponding point detection for detecting a plurality of corresponding points between the small area image of the first color image acquired by the image acquisition unit and the corresponding small area image of the second color image for each of the small area images. Means,
For each of the small area images, based on the corresponding points detected for the small area image by the corresponding point detection unit, the small area image of the first color image is deformed so that the positions of the corresponding points coincide with each other. Image deformation means for generating a small area image of the third color image,
For each of the small area images, the corresponding points of the small area image corresponding to the small area image of the second color image acquired by the image acquisition means and the third color image generated by the image deformation means. Image deformation result determination means for determining whether or not the amount of positional deviation is greater than or equal to a threshold value;
Correspondence of the first color image from the small area image of the second color image for each of the small area images determined by the image deformation result determination means that the amount of positional deviation of the corresponding point is equal to or greater than a threshold value Image estimating means for estimating the small area image;
The small area image of the first color image corresponding to the small area image determined by the image deformation result determining means that the positional deviation amount of the corresponding point is less than or more, and estimated by the image estimating means. Image combining means for generating an image obtained by combining the small area images of the first color image;
Color reproduction processing means for performing color reproduction processing of a subject using the second color image acquired by the image acquisition means and the image generated by the image combining processing means;
An image processing apparatus. For each of the small area images, further includes detected corresponding point number determining means for determining whether or not the number of the corresponding points detected for the small area image by the corresponding point detecting means is equal to or greater than a predetermined value.
For each of the small area images, the image deformation means matches the positions of the corresponding points when the number of corresponding points for the small area image is determined to be greater than or equal to a predetermined value for the small area image. Generating a small area image of the third color image obtained by deforming the small area image of the first color image,
The image estimation means further includes, for each of the small area images determined by the detection corresponding point number determination means that the number of corresponding points is less than a predetermined value, from the small area image of the second color image, The image processing apparatus according to claim 4, wherein the small area image corresponding to the first color image is estimated. An image processing method in an image processing apparatus including image acquisition means, corresponding point detection means, image deformation means, image deformation result determination means, and color reproduction processing means,
The image acquisition unit acquires a first color image captured using a color filter that transmits a part of wavelengths of colors that can be captured by the camera, and a second color image captured without using the color filter. And
A plurality of corresponding points between the first color image and the second color image acquired by the image acquisition unit are detected by the corresponding point detection unit;
Based on the plurality of corresponding points detected by the corresponding point detection unit, the image deformation unit generates a third color image obtained by deforming the first color image so that the positions of the corresponding points coincide with each other.
Whether or not the amount of positional deviation between the corresponding points of the second color image acquired by the image acquisition means and the third color image generated by the image deformation means by the image deformation result determination means is greater than or equal to a threshold value. Determine whether
When the color reproduction processing unit determines that the positional deviation amount of the corresponding point is greater than or equal to a threshold value by the image deformation result determination unit, the second color image acquired by the image acquisition unit is used to The second color image acquired by the image acquisition unit and the image deformation unit when the image deformation result determination unit determines that the amount of positional deviation of the corresponding point is less than a threshold value. An image processing method for performing color reproduction processing of a subject using the third color image generated by the method.   A program that causes a computer to function as each means that constitutes the image processing apparatus according to any one of claims 1 to 5.