JP2010025750A - Image processing apparatus, method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To separate spectral images acquired from multi-band images into images under a primary light source and images under a secondary light source by a simple technique. <P>SOLUTION: An image processing apparatus for processing multi-band images acquired by captured through the use of a plurality of band-pass filters (f1)-(f9) specifies the secondary light source (narrow spectral light source) different from the principal light source, acquires a spectral image J1 from a multi-band image by spectral estimation, and separates the spectral image J1 into an image J2 under the principal light source and an image J3 under the secondary light source on he basis of the spectrum of a specified secondary light source. It is thereby possible to perform white balance processing on the image J2 under the principal light source and the image J3 under the secondary light source. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program suitable for use in processing a multiband image captured using a plurality of bandpass filters.

  A multiband image processing apparatus such as a multiband camera can acquire a multiband image of a subject by photographing the subject using a plurality of bandpass filters having different transmission wavelength ranges. In recent years, multiband cameras having four or more bands have been put into practical use.

  In addition, there is spectral estimation for obtaining a spectral image from a multiband image in consideration of spectral characteristics of each bandpass filter in the multiband image processing apparatus. As the spectral estimation, methods such as a low-dimensional linear approximation method, a Wiener estimation method, and a multiple regression analysis method are known (for example, see Non-Patent Document 1).

  As a general process in a digital camera, there is a white balance process. White balance processing is image processing required to adjust the signal levels of R, G, and B and output the white portion of the subject as an achromatic white image. Even in a multiband image processing apparatus having four or more bands, processing corresponding to white balance processing is performed when obtaining a general image that matches human perception.

JP 2006-84401 A Tokudo Tsumura, Hideaki Haneishi, Yoichi Miyake, "Estimation of Spectral Reflectance from Multiband Images by Multiple Regression Analysis," Optics Vol. 27, No. 7, PP.384-391 (1998).

  When a subject exists under each of a plurality of light sources including a strong emission spectrum in a narrow band such as a bright line, the balance of the light sources irradiated by the subject is different. Therefore, when normal white balance processing (correction using one white balance gain) is performed, one subject is photographed with an appropriate white balance, but the other subject has a problem of color cast.

  As a countermeasure to this problem, a method of simulating an image under a specific light source by obtaining a light source ratio for each region or pixel in the screen (for example, see Patent Document 1) can be considered. Therefore, the light source ratio must be obtained, which is not a simple method.

  The present invention has been made in view of the above points, and enables a spectral image obtained from a multiband image to be separated into an image under a main light source and an image under a sub-light source by a simple method. With the goal.

An image processing apparatus according to the present invention is an image processing apparatus that processes a multiband image photographed using a plurality of bandpass filters, a sub-light source designating unit that designates a sub-light source different from the main light source, Based on the spectrum estimation unit for obtaining a spectrum image from the band image and the spectrum of the sub-light source designated by the sub-light source designation unit, the spectrum image obtained by the spectrum estimation unit is compared with the image under the main light source and the sub-light source. Image separation means for separating an image under a light source.
An imaging apparatus according to the present invention is an imaging apparatus that obtains a multiband image by performing imaging using a plurality of bandpass filters, the image processing apparatus according to the present invention, and a sub-light source designated by the sub-light source designating unit. And a focal position determining means for determining a focal position while avoiding an underlying subject.
An image processing method of the present invention is an image processing method for processing a multiband image photographed using a plurality of bandpass filters, a sub-light source designating process for designating a sub-light source different from the main light source, Based on the spectrum estimation process for obtaining the spectrum image from the band image and the spectrum of the sub-light source designated by the sub-light source designation process, the spectrum spectrum image obtained by the spectrum estimation process is compared with the image under the main light source and the sub-light source. And an image separation process for separating an image under a light source.
A program of the present invention is a program for causing a computer to execute processing of a multiband image captured using a plurality of bandpass filters, and a sub-light source designation process for designating a sub-light source different from the main light source; Based on the spectrum estimation process for obtaining the spectrum image from the multiband image and the spectrum of the sub-light source designated by the sub-light source designation process, the spectrum spectrum image obtained by the spectrum estimation process is an image under the main light source. And image separation processing for separating the image under the sub-light source into a computer.

  According to the present invention, a spectrum image obtained from a multiband image can be separated into an image under the main light source and an image under the sub light source by a simple method of designating the sub light source. Therefore, for example, white balance processing can be performed on the image under the main light source and the image under the sub light source, respectively, and color fog can be prevented from occurring.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. In the image processing apparatus of this embodiment, multiband image data is converted into RGB image data.

  FIG. 1 shows a state in which a scene in which a subject exists under a plurality of light sources is photographed by an imaging apparatus (multiband camera) 101 including an image processing apparatus to which the present invention is applied. A subject 104 exists under the light source 103, and a subject 106 exists under the light source 105. The light from the light source 105 does not reach the subject 104, and the light from the light source 103 does not reach the subject 106.

  The imaging device 101 is a monochrome camera having an imaging element that does not have a color filter. The filter 102 is a filter in which 9-band bandpass filters (f1) to (f9) are arranged on a turret. FIG. 2 shows the spectral transmittance of each of the bandpass filters (f1) to (f9). The turret rotates every time the image capturing is performed by the image capturing apparatus 101, and the image capturing is repeated 9 times, whereby multiband image data having the spectral data of the bandpass filters (f1) to (f9) is output.

  In addition, as shown in FIG. 3, multiband image data is output by disposing nine band bandpass filters (f1) to (f9) on a single-plate image sensor (for example, a CCD or CMOS sensor). You may do it. When a single-plate image sensor is used, multiband image data can be obtained by one shooting, so that isochronism in each band image can be maintained. In addition, since the space of the image processing apparatus can be reduced, the portability of the apparatus can be improved.

  In this embodiment, an example using a 9-band bandpass filter will be described. However, the number of bands to be used and the spectral transmittance characteristics of each bandpass filter are not limited thereto.

  FIG. 4 shows a flow of operations until a final image is obtained in the present embodiment. The user performs settings relating to the sub-light source in the image condition setting (S101), and performs photographing (S102). After shooting (S102), image processing (S103) is performed in the image processing apparatus, and the image file is saved (S104).

  Details of the image condition setting (S101), shooting (S102), and image processing (S103) will be described below.

<Image condition setting (S101)>
First, the image condition setting (S101) will be described. As shown in FIG. 5, the image processing setting (S1) includes white balance processing setting (S11) and sub-light source processing setting (S12). As the sub light source processing setting (S12), there are a sub light source setting (S121), an image separation intensity setting (S122), and an image composition intensity setting (S123).

“Sub-light source setting (S121)”
In the sub-light source setting (S121), a light source having a narrow band peak (narrow spectrum light source) such as a bright line spectrum light source is set. In the sub-light source setting (S121), the sub-light source is set using a setting method of default sub-light source selection (S1211), custom sub-light source setting (S1212), and automatic sub-light source setting (S1213).

-Default sub-light source selection (S1211)
A plurality of light sources can be selected simultaneously from the following preset light sources.
Mercury lamp (S12111): Set emission line spectral wavelength information of mercury lamp Sodium lamp (S12112): Set emission line spectral wavelength information of sodium lamp White LED1 (S12113): Bright line spectral wavelength information of red LED, blue LED, green LED Setting White LED 2 (S12114): Setting Bright Line Spectrum Wavelength Information of Blue Green LED and Yellow LED Here, only four types of light sources are illustrated, but any light source having a narrow band spectrum may be selectable.

-Custom auxiliary light source setting (S1212)
A narrow spectrum light source that the user wants to set as a sub-light source is designated by photographing a narrow spectrum light source or a subject under the narrow spectrum light source. Here, spectral estimation of the light source is performed, and in particular, wavelength information of the bright line spectrum is set.

Automatic automatic light source setting (S1213)
The main light source is estimated from the lens information (focused coordinates in the captured image) of the image pickup apparatus 101, the bright line spectrum light source in the region away from the focus coordinates is searched, and the wavelength information of the bright line spectrum is set as the sub light source. . Thereby, when the subject under the bright line spectrum light source is the main subject, image separation is not performed.

  This sub-light source setting (S121) is an example of the sub-light source designation process by the sub-light source designation means in the present invention.

“Image separation strength setting (S122)”
With this setting, image separation can be turned ON / OFF, and the separation intensity includes an image under the main light source (hereinafter referred to as the main light source image) and an image under the sub light source (hereinafter referred to as the sub light source image). The strength (ratio) at the time of separation can be set. When the separation strength is “0”, image separation is not performed (OFF), and when the separation strength is “100”, the image is completely separated (ON). This image separation strength setting (S122) is an example of processing by the image separation strength setting means referred to in the present invention.

“Image Composite Strength Setting (S123)”
Specifies how to process the separated images. Here, the intensity (ratio) for combining the separated main light source image and sub light source image is set. When the combined intensity is “0”, only the main light source image is used without using the image under the sub-light source. When the combined intensity is “50”, the sub-light source image and the main light source image are combined equally. When the combined intensity is “100”, only the sub-light source image is used without using the main light source image. This image composition intensity setting (S123) is an example of processing by the image composition intensity setting means referred to in the present invention.

  In this embodiment, the sub-light source processing setting (S12) is performed before shooting and the sub-light source is set. However, when storing unprocessed image data such as RAW image recording, the sub-light source processing setting (S12) is performed after shooting. ) Can also be performed.

  For example, the above setting is used as follows. When shooting astronomical objects in the city, the sky appears bright due to light damage caused by excessive illumination, etc., and the entire image becomes a bright and low-contrast image, and it may be difficult to capture relatively pale objects. In this case, an unnecessary light source that causes light pollution is set as a secondary light source in the secondary light source processing setting (S12), the separation strength is set to “100” (complete separation), and the composite image strength setting is set in the image separation strength setting (S122). By setting “0” (non-sub-light source image not used) in (S123), light having a wavelength corresponding to light pollution is deleted, and the optical bandpass filter can be used.

<Shooting (S102)>
Next, photographing (S102) will be described. In photographing (S102), photographing is performed using the multiband filter 102 of FIG. 1 described above to obtain a 9-band multiband image signal.

  Here, when performing autofocus, it is possible to focus on a more expected position by avoiding the subject under the sub-light source set in the sub-light source processing setting (S12). This is an example of processing by the focal position determining means in the present invention.

  When performing automatic exposure correction when calculating the exposure amount, the exposure amount is determined so as to avoid the influence of the sub-light source. Specifically, the exposure amount is adjusted depending on the sub light source processing method (image separation intensity, image synthesis intensity) set in the sub light source processing setting (S12). For example, when “100” (complete separation) is set in the image separation intensity setting (S122) of the sub-light source processing setting (S12), and “0” (sub-light source image not used) is set in the image composition intensity setting (S123). When the automatic exposure correction is performed as usual, there is a case where the image becomes dark as much as the influence of the auxiliary light source is removed. Therefore, under conditions where the sub-light source image is not used, an image with appropriate exposure can be obtained by performing correction so that an image brighter than usual can be obtained by automatic exposure correction. This is an example of processing by the exposure amount determining means in the present invention.

<Image processing (S103)>
Next, image processing (S103) will be described. Image processing in the image processing (S103) will be described with reference to the flowchart of FIG. In “spectral estimation” in step S201, the multiband image J0 obtained by photographing is converted into a spectral image J1. This is an example of spectral estimation processing by the spectral estimation means referred to in the present invention.

  Next, in step S202, if the separation strength of the image separation strength setting (S122) is “0”, the process proceeds to step S205, otherwise the process proceeds to step S203.

  In “image separation” in step S203, based on the spectrum of the secondary light source designated in the secondary light source setting (S121), the spectral image J1 is converted into a main light source image (spectral image) J2 and a secondary light source image (spectral image) J3. To separate. This is an example of image separation processing by the image separation means in the present invention.

  Next, in step S204, when the combined intensity of the image combined intensity setting (S123) is “100” and the main light source image J2 is processed, the combined intensity is “0”, and the auxiliary light source If the image J3 is to be processed, the process proceeds to step S207. Otherwise, the process proceeds to step S205.

  In “white balance processing” in step S205, white balance processing is performed on the main light source image J2 and the sub light source image J3, and the spectral images J2 and J3 are converted into RGB images (main light source image (RGB image) J4). Sub-light source image (RGB image) J5). Thereafter, in RGB processing 1 in step S206, image adjustment for composition is performed.

  In step S207, it is confirmed that the processing up to step S206 is completed for both the main light source image J2 and the sub light source image J3. If completed, the process proceeds to step S208.

  In “image composition” in step S208, the main light source image J4 and the sub light source image J5 are combined to create an RGB image J6. This is an example of image composition processing by the image composition means referred to in the present invention. Thereafter, final image quality adjustment is performed in RGB processing 2 in step S209.

  Details of each process of the flowchart of the image processing in FIG. 6 will be described below. In “spectral estimation” in step S201, a spectral spectrum estimated from multiband image data is obtained by using a spectral estimation method such as multiple regression analysis (see FIG. 7). By performing spectral estimation, spectral reflectance elements equal to or more than the number of bandpass filters can be obtained, and the accuracy of the obtained spectral reflectance depends on the spectral estimation method. Assuming that f represents a vector having nine elements corresponding to nine bandpass filters, and r represents a vector composed of 20 elements representing the spectral reflectance of the subject, the spectral estimation method is as follows: Means for obtaining a 20 × 9 transformation matrix C is given as in equation 1).

  The spectral image J1 can be obtained by performing the calculation of the above formula (1) for each pixel. If automatic sub-light source setting (S1213) is set, the sub-light source spectrum is estimated after spectral estimation. In the estimation, a region away from the in-focus coordinates is searched, and it is determined from the image that there is a spectral spectrum peculiar to a narrow spectrum light source such as a bright line.

  In “image separation” in step S203, a spectrum image is obtained using the spectrum of the secondary light source (see FIG. 9) set in the secondary light source processing setting (S12) and the separation intensity set in the image separation intensity setting (S122). The sub-light source spectrum is removed from J1.

Assume that the sub-light source spectrum intensity is S (λ) and is normalized by 0 to 1 as shown in FIG. Here, the spectrum image J1 after the “spectral estimation” in step S202 is I ₁ (x, y, λ), the spectrum of the main light source image J2 is I ₂ (x, y, λ), and the sub-light source image J3. When the spectrum is I ₃ (x, y, λ), I ₂ and I ₃ are obtained as shown in the following (formula 2) and (formula 3).

When attention is paid to a certain pixel in the spectrum image J1, the results of image separation using the sub-light source spectrum intensity S (λ) shown in FIG. 9 for the spectrum shown in FIG. 8 are shown in FIGS. Shown in FIG. 10 shows the spectral spectrum I ₂ (x, y, λ) of the main light source image J2, which is the result of (Expression 2). FIG. 11 shows the spectral spectrum I ₃ (x, y, λ) of the sub-light source image J3, which is the result of Expression (Expression 3).

  In the “white balance processing” in step S205, an RGB image is first created from the spectral image and then white balance correction is performed. In the spectral spectrum of the main light source image J2, when separating, the wavelength is applied like a skewer. Therefore, in the separated main light source image J2, the wavelength band missing due to the image separation is interpolated from the adjacent band. Specifically, using the information obtained by averaging the previous and subsequent band information as the missing wavelength band information, the R, G, and B values of each pixel are obtained as in the following (Formula 4). Note that P (λ) represents the spectral spectrum of the pixel. Further, r (λ), g (λ), and b (λ) represent color matching functions of the RGB color system used such as sRGB.

  Since the white balance processing is performed on an image that has already been separated according to the light source, it is possible to set the white balance according to each light source.

  In “RGB processing 1” in step S206, image processing is performed on the RGB image. As main image processing, γ correction processing, matrix processing considering the spectral spectrum and sub-light source spectrum of the multiband filter, and gradation adjustment processing for obtaining a natural image after image synthesis are performed.

  In “image composition” in step S208, the images J4 and J5 are composed using the composition intensity set in the image composition intensity setting (S123). In FIG. 1, since the subject 104 and the subject 106 exist under different light sources, when the light source 105 is a sub-light source set in the sub-light source setting, the subject 106 is not shown in the main light source image J2, and the subject 106 The image area is black. In this case, when the main light source image and the sub light source image are combined, an image with appropriate white balance can be obtained for both subjects.

  If the main light source includes the wavelength range of the sub-light source, the color tone of the RGB image J6 may be deteriorated if simply combined. Therefore, when there is an area where the luminance of the main light source image is high, the composition processing is performed by adjusting the composition ratio and reducing the composition ratio of the sub light source image.

  In “RGB processing 2” in step S209, image processing for final adjustment is performed on the combined RGB image J6. As main image processing, sharpness enhancement processing, saturation enhancement processing, and contrast enhancement processing are performed.

  According to the present invention, in addition to means for selecting a predetermined narrow spectrum light source in the same manner as the existing default white balance setting, a narrow spectrum light source can be obtained by capturing an image including a narrow band spectrum such as a bright line like a custom white balance. And a means for automatically setting a narrow spectrum light source other than the focal position using the coordinates of the focal position at the time of photographing, so that the narrow spectrum light source can be designated easily.

  In addition, even if an image has a color cast caused by a bright line light source in a part of the image, the color cast image area is separated, separately subjected to white balance processing, and then combined to adjust the white balance in the entire area. Images can be obtained.

  In addition, even if the image is fogged by a light source that includes bright lines throughout the image, as in celestial photography under light pollution, removing the spectral spectrum region of the bright line light source from the spectral spectrum will reduce the effect of the fog. An excluded image can be obtained.

  Further, by separating the spectral spectrum region of the bright line light source from the spectral spectrum, it is possible to obtain an image of only the subject under the narrow spectral light source.

  In addition, the luminance is reduced when the influence of the bright line light source is removed, but the luminance reduction can be prevented by performing the AE adjustment at the time of photographing.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. In this case, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the functions of the above-described embodiments are not limited to being realized by executing the program code read by the computer. For example, an OS (basic system or operating system) running on the computer performs part or all of the actual processing based on an instruction of the program code, and the functions of the above-described embodiments are realized by the processing. May be.

  Further, the program code read from the storage medium may be written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. In this case, after being written in the memory, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the function of the above-described embodiment is performed by the processing. Is realized.

It is a figure which shows the state which image | photographed the scene which a to-be-photographed object exists in each of several light sources with the camera which functions as an image processing apparatus to which this invention is applied. It is a characteristic view which shows the spectral transmittance of each band pass filter in an embodiment. It is a figure which shows the band pass filter arrange | positioned on the image pick-up element of a single plate. It is a flowchart showing the flow of operation until a final image is obtained in embodiment. It is a figure which shows the content of image processing setting. It is a flowchart for demonstrating the image processing in embodiment. It is a characteristic view which shows the spectral spectrum estimated from multiband image data, and an actual measurement spectrum. It is a characteristic view which shows a spectral spectrum at the time of paying attention to a certain pixel of a spectral spectrum image. It is a characteristic view which shows the spectrum of a sublight source. It is a characteristic view which shows the spectral spectrum in the pixel with an image under the main light source. It is a characteristic view which shows the spectral spectrum in the pixel with an image under a sublight source.

Explanation of symbols

101 Imaging device 102 Filter 103, 105 Light source 104, 106 Subject

Claims (14)

An image processing apparatus for processing a multiband image photographed using a plurality of bandpass filters,
Sub-light source designating means for designating a sub-light source different from the main light source,
A spectral estimation means for obtaining a spectral image from the multiband image;
Based on the spectrum of the sub-light source designated by the sub-light source designating means, image separation means for separating the spectral image obtained by the spectral estimation means into an image under the main light source and an image under the sub-light source An image processing apparatus.   The image processing apparatus according to claim 1, wherein the sub-light source designating unit designates a preset narrow spectrum light source.   The image processing apparatus according to claim 1, wherein the sub-light source designation unit designates a narrow spectrum light source by photographing a narrow spectrum light source or a subject under the narrow spectrum light source.   The image processing apparatus according to claim 1, wherein the sub-light source designation unit is designated by searching for a narrow spectrum light source in a region away from the in-focus coordinates in the captured image.   5. The image processing device according to claim 1, further comprising an image processing unit configured to perform image processing on each of the image under the main light source and the image under the sub light source separated by the image separation unit. Image processing device.   The image processing apparatus according to claim 5, wherein the image processing is white balance processing.   6. The image processing unit according to claim 5, wherein the image processing unit performs image processing by interpolating, from an adjacent band, a wavelength band lost due to the image separation among images under the main light source separated by the image separation unit. Image processing apparatus.   The image processing apparatus according to claim 1, further comprising an image separation intensity setting unit configured to set an intensity of image separation performed by the image separation unit.   The image processing apparatus according to claim 5, further comprising an image synthesis unit that synthesizes an image under a main light source and an image under a sub-light source that have been subjected to image processing by the image processing unit.   The image processing apparatus according to claim 9, further comprising an image composition intensity setting unit configured to set an image composition intensity by the image composition unit. An imaging device that obtains a multiband image by shooting using a plurality of bandpass filters,
The image processing apparatus according to any one of claims 1 to 10,
An imaging apparatus comprising: a focus position determining unit that determines a focus position while avoiding a subject under the sub light source designated by the sub light source designating unit.   The imaging apparatus according to claim 11, further comprising an exposure amount determining unit that determines an exposure amount so as to avoid the influence of the sub light source specified by the sub light source specifying unit. An image processing method for processing a multiband image photographed using a plurality of bandpass filters,
Sub-light source designation processing for designating a sub-light source different from the main light source,
Spectral estimation processing for obtaining a spectral image from a multiband image;
An image separation process for separating a spectral spectrum image obtained by the spectral estimation process into an image under the main light source and an image under the sub light source based on the spectrum of the sub light source designated by the sub light source designation process. A featured image processing method. A program for causing a computer to process a multiband image photographed using a plurality of bandpass filters,
Sub-light source designation processing for designating a sub-light source different from the main light source,
Spectral estimation processing for obtaining a spectral image from a multiband image;
Based on the spectrum of the sub-light source designated by the sub-light source designation process, the computer performs image separation processing for separating the spectral image obtained by the spectral estimation process into an image under the main light source and an image under the sub-light source. Program to let you.

Images