JP5300791B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program that generate a color reproduction image from a multiband image.

  In an image processing apparatus based on RGB (three primary colors) that is currently in widespread use, a white pixel or neutral gray pixel in an image is designated on a GUI (Graphical User Interface), and the RGB value of the pixel is determined. Correction processing for multiplying a signal value by a coefficient so that the maximum value (in the case of 8 bits, [R, G, B] = [255, 255, 255]) or the same value (R value = G value = B value) In addition, it is known to perform color adjustment (white balance) by performing gamma correction processing (see, for example, 3.3 White Balance in Non-Patent Document 1).

Tomoaki Higo, Daisuke Miyazaki, Katsushi Ikeuchi, “Real-time specular reflection removal based on dichroic reflection model”, Information Processing Society of Japan. CVIM, [Computer Vision and Image Media] 2006 (93), 211-218, 2006060908

  However, in a multiband image of four or more channels, when correction is performed so as to align the pixel values of the four channels by performing correction processing as described in Non-Patent Document 1, for example, subsequent color reproduction There is a problem that the color is greatly shifted in the processing of the result.

  The present invention has been made in view of such circumstances, and an image processing apparatus capable of preventing a color from being greatly shifted when a color reproduction image is generated from a multiband image having four or more channels. An object is to provide an image processing method and an image processing program.

  The present invention is an image processing apparatus for generating a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera, the spectral sensitivity information of the camera, and illumination light at the time of photographing The spectral reflectance estimation matrix obtained from the spectral distribution information of the subject, the spectral information of the subject, and the pixel value of the white point on the multiband image are input to calculate the spectral reflectance of the white point. Reflectance estimation means, spectral reflectance estimation matrix correction means for correcting the spectral reflectance estimation matrix based on the spectral reflectance of the white point, and white balance after adjustment based on the corrected spectral reflectance estimation matrix Spectral reflectance image generating means for generating a spectral reflectance image, and color reproduction processing means for generating a color reproduction image under the predetermined illumination environment from the spectral reflectance image.

  In the present invention, the spectral reflectance estimation matrix correction means divides a component corresponding to the wavelength of the spectral reflectance estimation matrix by the spectral reflectance, and sets the result as a corrected spectral reflectance estimation matrix. Features.

  In the present invention, the spectral reflectance estimation matrix correction means corrects the spectral sensitivity of each channel of the camera by applying a weighting factor to correct the spectral sensitivity of each channel of the camera so that the spectral reflectance becomes reference white. It is characterized by obtaining an estimation matrix.

  According to the present invention, the spectral reflectance estimation matrix correction unit corrects the spectral reflectance after correction by applying a weighting factor to each wavelength component of the illumination light spectrum at the time of photographing when the spectral reflectance becomes reference white. It is characterized by obtaining a rate estimation matrix.

  The present invention is an image processing method for executing image processing on a computer in order to generate a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera, wherein the computer , The spectral sensitivity information of the camera, the spectral distribution information of the illumination light at the time of shooting, the spectral reflectance estimation matrix obtained from the statistical information of the subject, and the pixel value of the white point on the multiband image A spectral reflectance estimation step of calculating a spectral reflectance of the white point; and a spectral reflectance estimation matrix correction step in which the computer corrects the spectral reflectance estimation matrix based on the spectral reflectance of the white point. And a spectral reflectance image generation step in which the computer generates a spectral reflectance image after white balance adjustment based on the corrected spectral reflectance estimation matrix; The computer, and having a color reproduction processing step of generating a color reproduction image under the predetermined illumination environment from the spectral reflectance image.

  The present invention is an image processing program for causing a computer to execute image processing in order to generate a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera. The spectral sensitivity information, the spectral distribution information of the illumination light at the time of shooting, the spectral reflectance estimation matrix obtained from the statistical information of the subject, and the pixel value of the white point on the multiband image are input, A spectral reflectance estimation step for calculating a spectral reflectance of a white point, a spectral reflectance estimation matrix correction step for correcting the spectral reflectance estimation matrix based on the spectral reflectance of the white point, and the spectral reflectance after the correction A spectral reflectance image generation step of generating a spectral reflectance image after white balance adjustment based on a rate estimation matrix, and color reproduction under the predetermined illumination environment from the spectral reflectance image Characterized in that to perform the color reproduction processing step of generating an image on the computer.

  According to the present invention, since white balance correction is performed with spectral reflectance, it is possible to generate a color reproduction image that is appropriately white balanced under the assumed observation illumination. Further, the white balance correction process using the spectral reflectance can simultaneously correct the photographing data including measurement errors such as the spectral sensitivity of the camera and the illumination light spectrum.

It is a block diagram which shows the structure of one Embodiment of this invention. 2 is a flowchart illustrating an operation of the image processing apparatus illustrated in FIG. 1. It is explanatory drawing which shows the principle of color reproduction.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numeral A denotes an image processing apparatus constituted by a computer apparatus. Reference numeral 1 denotes a multiband image storage unit in which multiband image data to be subjected to white balance correction processing is stored. Reference numeral 2 denotes an input unit configured by a keyboard, a pointing device, or the like. Reference numeral 3 denotes a display unit configured by a display device capable of displaying a color image. Reference numeral 4 denotes a white point designation unit that designates the position of the white point on the image and obtains the pixel value of the designated white pixel. Reference numeral 5 denotes spectral reflectance information obtained from the spectral sensitivity information of the camera when the image stored in the multiband image storage unit 1 is captured, the spectrum (spectral distribution) information of the illumination light at the time of capturing, and the statistical information of the subject. It is a spectral reflectance estimation matrix storage unit in which an estimation matrix is stored.

  Reference numeral 6 denotes a spectral reflectance estimator that receives the spectral reflectance estimation matrix and calculates the spectral reflectance. Reference numeral 7 denotes a spectral reflectance estimation matrix correction unit that inputs the calculated spectral reflectance and the spectral reflectance estimation matrix and corrects the spectral reflectance estimation matrix. Reference numeral 8 denotes a spectral reflectance image generation unit that inputs a corrected spectral reflectance estimation matrix and generates a spectral reflectance image after white balance adjustment. Reference numeral 9 denotes a color reproduction environment storage unit that stores illumination light spectrum information at the time of color reproduction and characteristic information of the display unit 3 that displays an image subjected to color reproduction. Reference numeral 10 denotes a color reproduction processing unit that inputs a spectral reflectance image after white balance adjustment and information stored in the color reproduction environment storage unit 9 to generate and output a color reproduction image. Reference numeral 11 denotes a color reproduction image storage unit that stores a color reproduction image output from the color reproduction processing unit 10.

  As shown in FIG. 3, the multiband image data stored in the multiband image storage unit 1 is the result of color separation of incident light on the sensor using three (three bands) or more color filters. It is stored as a pixel value in each pixel. Here, the value of each band is corrected so that the estimation results of the spectral (spectral) reflectance of the white point serving as a reference on the stored multiband image (three or more bands) are equal at all wavelengths. Rather than simply correcting the image values of the respective bands to be equal, it is a significant difference from the prior art that the white balance of the spectral reflectance of the estimation result using the pixel values is taken. In most cases, the spectral reflectance estimation result is often not white (reflectances at all wavelengths are substantially equal) simply by aligning the pixel values of each band. The present invention solves such problems.

  Next, the operation of the image processing apparatus A shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the operation of the image processing apparatus A shown in FIG. First, the white point designation unit 4 reads out the image data stored in the multiband image storage unit 1 and displays it on the display unit 3. Then, the operator operates the input unit 2 to identify the reference white point in the designated image. Then, the white point designating unit 4 outputs the pixel value of the identified white point pixel to the spectral reflectance estimating unit 6 (step S1).

Next, the spectral reflectance estimation unit 6 uses the spectral reflectance estimation matrix stored in the spectral reflectance estimation matrix storage unit 5 and the pixel value output from the white point designation unit 4 to use this white point. Is calculated (step S2). A number of methods have been proposed for obtaining the spectral reflectance estimation matrix, and one of them is the Wiener estimation method. When the wavelength is represented by λ, the spectral spectrum of the illumination light is expressed as W (λ), and the spectral reflectance of the subject is expressed as f (λ), the spectral spectrum I (λ) of the reflected light to be observed is expressed as follows: It is.
I (λ) = W (λ) · f (λ) (1)

Here, consider a case where an image is obtained with an N-band sensor (discrete system) instead of calculation with the number of dimensions of the wavelength level (continuous system). When considering only the visible light region, the spectral reflectance sampled at 1 nm intervals is expressed as f = [f ₃₈₀ , f ₃₈₁ ,..., F ₇₈₀ ] ^T. The sensor sensitivity of each band is expressed as a vector as S = [S ₁ (λ), S ₂ (λ),..., S _N (λ)] ^T. Assuming that the diagonal matrix whose diagonal component is equal to the illumination light spectrum (zero except for the physique component) is W, equation (1) is rewritten as a vector / matrix operation equation such as equation (2).
I = Wf. ... (2)

Signal c = [c ₁ (λ), c ₂ (λ),..., C _N (λ)] obtained from the camera ^T is expressed as C = SWf = Hf, and the inverse matrix M of the system matrix H = SW Can be obtained, the spectral reflectance vector can be calculated as follows. This inverse matrix M is referred to as a spectral reflectance estimation matrix (matrix that returns pixel values to spectral reflectance).
M = RH ^t {HRH ^t } ⁻¹ (3)

  The matrix M obtained by the equation (3) is called a Wiener estimation matrix, and includes a system matrix H (product of sensor spectral sensitivity and illumination spectrum) and subject statistical information R (teacher data correlation matrix, covariance matrix, etc. ).

  Next, the spectral reflectance estimation matrix correction unit 7 calculates the spectral reflectance estimation matrix stored in the spectral reflectance estimation matrix storage unit 5 and the spectral reflectance of the white point calculated by the spectral reflectance estimation unit 6. Then, the component corresponding to the wavelength of the spectral reflectance estimation matrix is divided by the spectral reflectance, and the result is output as a new corrected spectral reflectance estimation matrix (step S3).

Next, the spectral reflectance image generation unit 8 receives the corrected spectral reflectance estimation matrix, generates and outputs a spectral reflectance image after white balance adjustment (step S4). Then, the color reproduction processing unit 10 stores the color reproduction environment memory for the spectral reflectance f ^ (λ) of the subject calculated using the corrected spectral reflectance estimation matrix (^ is attached to the head of f). The illumination light spectrum W ′ (λ) of the observation environment stored in the unit 9 is multiplied, and the reflected light spectrum I ^ (λ) (^ is attached to the head of I) under the illumination at the time of observation (4) Calculated by the formula.
I ^ (λ) = W ′ (λ) · f ^ (λ) (4) (^ is attached to the head of I and f)

Subsequently, the color reproduction processing unit 10 is determined by an international lighting committee called a tristimulus value by multiplying the calculated reflected light spectrum by a color matching function (sensitivity characteristic of human eyes). Convert to CIE-XYZ three-dimensional data. In the case where the display unit 3 is a linear system, the relationship shown in the equation (5) can be obtained from the emission spectrum of each color for each primary color (eg, R, G, B).

By calculating this inverse matrix, equation (6) is obtained. When the CIE-XYZ of the previously calculated reflected light spectrum is input, the RGB values in the display device used for the display unit 3 whose input and output are linear are calculated. can do.

  By correcting the RGB values using the gamma characteristic (tone curve) of the display device, it is possible to reproduce the correct color on the display unit 3 when the subject is under the assumed observation illumination. The color reproduction processing unit 10 stores the RGB values obtained here as a color reproduction image under a predetermined observation illumination in the color reproduction image storage unit 11 and outputs the color reproduction image to the display unit 3 to display the color reproduction image. 3 is displayed. When the number of primary colors on the display unit 3 is 4 or more, it is necessary to perform more complicated conversion processing. However, the processing operation described above is very effective in generating data to be input to the color reproduction processing unit 10.

  Note that the spectral reflectance estimation matrix correction unit 7 described above has the spectral sensitivity of each channel of the camera so that the spectral reflectance calculated by the spectral reflectance estimation unit 6 is the reference white (the reflectance of each wavelength is 1). The spectral reflectance estimation matrix may be corrected by applying a weighting factor to. Further, the spectral reflectance estimation matrix correction unit 7 described above corrects the illumination light spectrum at the time of photographing so that the spectral reflectance calculated by the spectral reflectance estimation unit 6 becomes the reference white (the reflectance of each wavelength is 1). The spectral reflectance estimation matrix may be corrected by applying a weighting coefficient to each wavelength component.

  As described above, in the multiband image (number of channels ≧ 4), the white balance correction is not performed on the signal value obtained from the input device, but the white balance as a result of color reproduction using the obtained multiband is obtained. In order to perform an appropriate process so as to be appropriate, a process for achieving an appropriate white balance in the color reproduction result image (RGB image, CIE-Lab or CIE-XYZ image) is added. As a result, it is possible to solve the problem that the color is greatly shifted in the processing of the color reproduction result. In addition, the input image or the spectral reflectance estimation matrix, the spectral sensitivity of the camera, the illumination light spectrum, etc., so that the spectral reflectance (spectral reflectance) of the white point is 1 (or equal) for each wavelength. A correction process was added to. Accordingly, similarly, the problem that the color is greatly shifted in the processing of the color reproduction result can be solved.

  As described above, conventionally, white balance correction is generally performed on an input image (for example, RGB signal values) itself. However, in the present invention, a white of an estimated spectral reflectance is used instead of an input image. A correction process is performed on the spectral reflectance estimation matrix, the spectral sensitivity of the camera, the spectrum of the photographic illumination, etc. so that the balance is appropriate, so that a color reproduction image with an appropriate white balance under the assumed observation illumination The effect that it becomes possible to produce | generate is acquired. Further, the white balance correction process using the spectral reflectance can simultaneously correct the photographing data including measurement errors such as the spectral sensitivity of the camera and the illumination light spectrum.

  Note that a program for realizing the functions of the processing unit in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to perform color reproduction processing. You may go. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer system” includes a WWW system having a homepage providing environment (or display environment). The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  It can be applied to applications where it is essential to generate a color reproduction image from a multiband image of four or more channels.

  DESCRIPTION OF SYMBOLS 1 ... Multiband image memory | storage part, 2 ... Input part, 3 ... Display part, 4 ... White point designation | designated part, 5 ... Spectral reflectance estimation matrix memory | storage part, 6 ... Spectral Reflectance estimation unit, 7... Spectral reflectance estimation matrix correction unit, 8... Spectral reflectance image generation unit, 9... Color reproduction environment storage unit, 10.・ Color reproduction image storage

Claims (6)

An image processing apparatus for generating a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera,
Input spectral reflectance information of the camera, spectral distribution information of illumination light at the time of photographing, and a spectral reflectance estimation matrix obtained from the statistical information of the subject, and a pixel value of a white point on the multiband image. Spectral reflectance estimating means for calculating the spectral reflectance of the white point;
Spectral reflectance estimation matrix correction means for correcting the spectral reflectance estimation matrix based on the spectral reflectance of the white point;
Spectral reflectance image generation means for generating a spectral reflectance image after white balance adjustment based on the corrected spectral reflectance estimation matrix;
An image processing apparatus comprising: color reproduction processing means for generating a color reproduction image under the predetermined illumination environment from the spectral reflectance image.   The spectral reflectance estimation matrix correction means divides a component corresponding to the wavelength of the spectral reflectance estimation matrix by the spectral reflectance, and uses the result as a corrected spectral reflectance estimation matrix. Item 8. The image processing apparatus according to Item 1.   The spectral reflectance estimation matrix correction means obtains a corrected spectral reflectance estimation matrix by correcting the spectral sensitivity of each channel of the camera by applying a weighting coefficient so that the spectral reflectance becomes a reference white color. The image processing apparatus according to claim 1.   The spectral reflectance estimation matrix correction means corrects the spectral reflectance estimation matrix after correction by applying a weighting factor to each wavelength component of the illumination light spectrum at the time of photographing when the spectral reflectance becomes reference white. The image processing apparatus according to claim 1, wherein the image processing apparatus is obtained. An image processing method for executing image processing on a computer in order to generate a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera,
The computer includes a spectral reflectance estimation matrix obtained from spectral sensitivity information of the camera, spectral distribution information of illumination light at the time of photographing, and statistical information of the subject, and a pixel value of a white point on the multiband image And a spectral reflectance estimation step of calculating the spectral reflectance of the white point,
The computer corrects the spectral reflectance estimation matrix based on the spectral reflectance of the white point, a spectral reflectance estimation matrix correction step,
The computer generates a spectral reflectance image after white balance adjustment based on the corrected spectral reflectance estimation matrix, and a spectral reflectance image generation step,
A color reproduction processing step in which the computer generates a color reproduction image under the predetermined illumination environment from the spectral reflectance image. An image processing program for causing a computer to perform image processing in order to generate a color reproduction image under a predetermined illumination environment from a multiband image obtained by photographing a subject with a camera,
Input spectral reflectance information of the camera, spectral distribution information of illumination light at the time of photographing, and a spectral reflectance estimation matrix obtained from the statistical information of the subject, and a pixel value of a white point on the multiband image. A spectral reflectance estimation step of calculating a spectral reflectance of the white point;
Spectral reflectance estimation matrix correction step for correcting the spectral reflectance estimation matrix based on the spectral reflectance of the white point;
A spectral reflectance image generation step of generating a spectral reflectance image after white balance adjustment based on the corrected spectral reflectance estimation matrix;
An image processing program that causes the computer to perform a color reproduction processing step of generating a color reproduction image under the predetermined illumination environment from the spectral reflectance image.

Images