JP2016213623A - Image processing apparatus, image processing system, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image under arbitrary observation illumination light using a reflection component and a fluorescent component, without requiring estimation of the absorption characteristics of the fluorescent component, by a multiband image captured by performing spectroscopy with a wavelength width broader than conventional one.SOLUTION: A fluorescent component separation unit 40 separates a multiband image into a reflection component image and a fluorescent component image. A weighting factor calculation unit 44 calculates a weighting factor for the fluorescent component image of each band, by the shooting illumination light spectrum, wavelength selection characteristics of a second optical element, and observation illumination light spectrum. A multiband fluorescent component image generation unit 42 generates a multiband fluorescent component image on the basis of the fluorescent component image for each band of shooting illumination light, and the weighting factor. A fluorescent component color reproduction unit 28 generates a fluorescent component color reproduction image by calculating the fluorescent spectrum, a reflection component color reproduction unit 26 generates a reflection component color reproduction image by calculating a reflection spectrum. An image synthesis processing unit 30 generates a color reproduction result image by adding both color reproduction images, and outputs that image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing system, an image processing method, and an image processing program.

  As shown in non-patent literature, fluorescence characteristics are measured by using a single wavelength light or narrow-band monochromatic light as incident light and measuring the spectrum of the emitted light for each incident light while changing the wavelength of the incident light. There was a need. Recently, as shown in Non-Patent Documents 2 and 3, a technique for estimating spectral components of reflected light and fluorescence using spectral images taken under two types of illumination light has been developed.

A. Lam, and I. Sato, “Spectral modeling and relighting of reflective-fluorescent scenes”, Proc. CVPR, pp. 1452-1459, 2013 Kamiyama, Hirai, Horiuchi, Tominaga, "Spectral component estimation of fluorescent objects using visible light source and multiband camera", IEICE Transactions D, Vol. J95-D, No.3, pp. 638-644, 2012 Y. Fu, A. Lam, I. Sato, T. Okabe, and Y. Sato, “Separating Reflective and Fluorescent Components Using High Frequency Illumination in the Spectral Domain”, proc. ICCV, pp. 457-464, 2013

    In order to generate an image under any observation illumination light using the reflected light component (reflected component) and the fluorescent component obtained by the techniques of Non-Patent Documents 1 to 3, the spectral reflectance of the reflected component In addition to estimation, it is necessary to estimate the absorption characteristics and emission characteristics (fluorescence spectrum) relating to fluorescence. In order to estimate the fluorescence spectrum, it is necessary to split into narrowband light on the camera side as in the techniques described in Non-Patent Documents 1 to 3. In addition, in order to estimate the fluorescence absorption characteristics, as in the technique described in Non-Patent Document 1, it is necessary to split into narrowband light on the photographing illumination light source side in addition to the camera side. In the technique described in Non-Patent Document 3, although the photographing illumination light source is not split into narrowband light, it is necessary to control the shape of the illumination light spectrum using a special light source device.

  For this reason, it is necessary to estimate the fluorescence absorption characteristics.

  In addition, since the spectral bandwidth and transmitted light energy are in a trade-off relationship, the narrower the spectral bandwidth on the camera side and the photographing illumination light source side, the higher the sensitivity and the higher the signal / noise ratio. And a high-power light source. If a low-output light source is used, the exposure time of the camera at the time of image capture becomes longer, and the overall work time also becomes longer. Similarly, when the wavelength bandwidth on the camera side is narrow and the sensitivity of the camera is low, the work time becomes longer.

In addition, as a result of performing spectral analysis on narrowband light, it is necessary to analyze / process a huge number of images. For example, if the visible light range of 380 nm to 780 nm is dispersed at 10 nm intervals on the camera side, 41 images are required, and if the same is also spectrally separated on the photographing illumination light source side, a total of 41 ² = 1681 images are required.

  Furthermore, in order to obtain narrowband light at an arbitrary center wavelength, a special and expensive device is required.

  The present invention has been made in consideration of the above-mentioned problems, and it is possible to obtain a reflection component without requiring an estimation of the absorption characteristic of the fluorescence component by using a multiband image obtained by performing spectroscopy with a wider wavelength range than before. It is an object to provide an image processing apparatus, an image processing system, an image processing method, and an image processing program capable of generating an image under arbitrary observation illumination light using a fluorescent component and a fluorescent component.

In order to achieve the above object, the image processing apparatus of the present invention limits the wavelength of light incident on one camera, and includes a plurality of first optical elements each having a different band, which is a wavelength band of transmitted light, and Photographed by a multiband image photographing device including a plurality of second optical elements that limit the wavelength of photographing illumination light irradiated to the subject from the photographing illumination light source and each have a different band that is a wavelength band of the transmitted light. An acquisition unit that acquires a multiband image for each band of the photographic illumination light, and for each band of the photographic illumination light, the multiband image acquired by the acquisition unit is a band equal to the band of the photographic illumination light. A fluorescence component separation unit that separates a reflection component image that is an image and a fluorescence component image that is an image of a band having a longer center wavelength than the band of the imaging illumination light; The product of the illumination light spectrum of the photographic illumination light and the wavelength selection characteristic of the second optical element with respect to the band of the photographic illumination light, and the illumination light spectrum of the illumination light in the observation environment and the band of the photographic illumination light By dividing the product of the wavelength selection characteristics of the second optical element, a weighting factor calculation unit for calculating a weighting factor for the fluorescence component image of each band, and the photographing illumination light separated by the fluorescence component separation unit A multiband fluorescence component is obtained by obtaining a weighted linear sum of the fluorescence component images based on the fluorescence component image for each band and the weighting factor for the fluorescence component image of each band calculated by the weighting factor calculation unit. A multiband fluorescent component image generation unit that generates and outputs an image, and a multiband fluorescent component generated by the multiband fluorescent component image generation unit For each pixel of the image, a spectral spectrum estimation matrix is applied to calculate a fluorescence spectrum, and a fluorescence component color reproduction unit that generates a fluorescence component color reproduction image, and multiband reflection by a reflection component image separated by the fluorescence component separation unit A reflected component color reproduction unit that calculates a reflected light spectrum by applying a spectral reflectance estimation matrix and an illumination light spectrum of the illumination light of the observation environment for each pixel of the component image, and generates the reflected component color reproduction image, and the fluorescence An image composition processing unit that adds a component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image; and
Is provided.

  In addition, the image processing apparatus of the present invention includes a plurality of first optical elements that limit the wavelength of light incident on the camera and have a plurality of first optical elements that are different in wavelength bands of light to be transmitted and have different viewpoints. A multiband image photographing apparatus comprising: a plurality of second optical elements that limit the wavelength of photographing illumination light irradiated to a subject from the photographing illumination light source and have different bands that are wavelength bands of light to be transmitted; An acquisition unit that acquires a plurality of images taken for each band of shooting illumination light, and a reference image that is a predetermined image among the plurality of images, and the reference image for each of the images different from the reference image For each reference point on the image, a corresponding point detection unit for detecting each corresponding point on the image, and for each band of the photographing illumination light, in each of the images different from the reference image, Based on each corresponding point in the image with respect to each reference point on the reference image detected by the corresponding point detection unit, a deformation obtained by deforming the image so that the position of each reference point matches the position of each corresponding point An image deformation processing unit that generates a result image and generates a multiband image including each of the reference image and the deformation result image, and is generated by the image deformation processing unit for each band of the photographing illumination light. Fluorescence component separation that separates the multiband image into a reflection component image that is an image of a band equal to the band of the photographing illumination light and a fluorescence component image that is an image of a band having a longer center wavelength than the band of the photographing illumination light And the product of the illumination light spectrum of the photographic illumination light and the wavelength selection characteristic of the second optical element for the photographic illumination light band for each band of the photographic illumination light. A weighting factor calculation unit for calculating a weighting factor for the fluorescence component image of each band by dividing the product of the wavelength spectrum of the second optical element with respect to the band of the illumination light of the environment and the band of the photographing illumination light; , Based on the fluorescence component image for each band of the photographic illumination light separated by the fluorescence component separation unit and the weighting factor for the fluorescence component image of each band calculated by the weighting factor calculation unit. By obtaining a weighted linear sum of component images, a multiband fluorescence component image generation unit that generates and outputs a multiband fluorescence component image, and a multiband fluorescence component image generated by the multiband fluorescence component image generation unit For each pixel, calculate a fluorescence spectrum by applying a spectral spectrum estimation matrix, and generate a fluorescence component color reproduction image, For each pixel of the multiband reflection component image by the reflection component image separated by the fluorescence component separation unit, the reflected light spectrum is calculated by multiplying the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment, A reflection component color reproduction unit that generates a component color reproduction image, and an image synthesis processing unit that generates and outputs a color reproduction result image by adding the fluorescence component color reproduction image and the reflection component color reproduction image. .

  In the image processing apparatus of the present invention, the plurality of cameras include one color camera that captures a color image and one or more monochrome cameras that capture a monochrome image, and the plurality of first optical cameras. The element transmits a different band of light to each of the monochromatic cameras, and a spectral reflectance image having a spectral reflectance estimated for each pixel of the color image acquired by the acquisition unit as a pixel value, and Using the illumination light spectrum of the photographic illumination light in the state before the spectrum and the spectral sensitivity characteristics of each monochromatic camera, obtained for each monochromatic camera and obtained from the viewpoint of the color camera. The image processing apparatus may further include a single color image generation unit that generates a wax virtual single color image, and the corresponding point detection unit may use the single color image generated by the single color image generation unit as the reference image.

  In addition, the image processing system of the present invention limits the wavelength of light incident on one camera, and a plurality of first optical elements having different wavelength bands of transmitted light and photographing illumination light sources to a subject. A multiband image photographing device comprising a plurality of second optical elements that limit the wavelength of the illuminating illumination light to be irradiated and each have a different band, which is the wavelength band of the transmitted light, and the multiband image photographing device. And an image processing apparatus according to the present invention that acquires a multiband image for each band of photographing illumination light, and generates and outputs a color reproduction result image based on the acquired multiband image.

  In addition, the image processing system of the present invention includes a plurality of first optical elements that limit the wavelength of light incident on the camera and have a plurality of first optical elements that are different in wavelength bands of light to be transmitted and have different viewpoints. A multi-band image photographing apparatus comprising: a plurality of second optical elements that limit the wavelength of photographing illumination light irradiated to a subject from the photographing illumination light source and have different bands that are wavelength bands of transmitted light; The image of the present invention is obtained by acquiring a plurality of images for each band of photographic illumination light captured by the multiband image capturing device, and generating and outputting a color reproduction result image based on the acquired plurality of images. And a processing device.

  Further, in the image processing method of the present invention, the acquisition unit limits the wavelength of light incident on one camera, and includes a plurality of first optical elements each having a different band, which is a wavelength band of transmitted light, and photographing illumination. Photographing illumination, which is photographed by a multiband image photographing device that includes a plurality of second optical elements that limit the wavelength of photographing illumination light emitted from a light source to a subject and that have different wavelength bands of transmitted light. A step of acquiring a multiband image for each band of light, and a fluorescence component separating unit, for each band of the imaging illumination light, the multiband image acquired by the acquisition unit as a band of the imaging illumination light A step of separating a reflection component image that is an image of an equal band and a fluorescence component image that is an image of a band having a longer center wavelength than the band of the photographing illumination light; For each band of the photographic illumination light, the product of the illumination light spectrum of the photographic illumination light and the wavelength selection characteristic of the second optical element for the band of the photographic illumination light, and the illumination light spectrum of the illumination light of the observation environment and the A step of calculating a weighting factor for the fluorescence component image of each band by dividing the product of the wavelength selection characteristics of the second optical element with respect to the band of the imaging illumination light, and a multiband fluorescence component image generation unit comprising: Based on the fluorescence component image for each band of the photographing illumination light separated by the separation unit and the weighting factor for the fluorescence component image of each band calculated by the weighting factor calculation unit, the overlap of the fluorescence component image is calculated. A step of generating and outputting a multiband fluorescent component image by obtaining a mitsuki linear sum, and a fluorescent component color reproduction unit For each pixel of the multiband fluorescence component image generated by the split image generation unit, a step of calculating a fluorescence spectrum by applying a spectral spectrum estimation matrix to generate a fluorescence component color reproduction image; For each pixel of the multiband reflection component image by the reflection component image separated by the component separation unit, the reflected light spectrum is calculated by multiplying the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment, and the reflected component color A step of generating a reproduction image, and a step of the image composition processing unit adding the fluorescence component color reproduction image and the reflection component color reproduction image to generate and outputting a color reproduction result image.

  Further, in the image processing method of the present invention, the acquisition unit includes a plurality of first optical elements that limit the wavelength of light incident on the camera and have different bands, which are wavelength bands of light to be transmitted, and a viewpoint. A multi-band comprising: a plurality of cameras having different wavelengths; and a plurality of second optical elements that limit the wavelength of the photographic illumination light irradiated to the subject from the photographic illumination light source and that have different wavelength bands of light to be transmitted A step of acquiring a plurality of images for each band of the photographic illumination light captured by the image capturing device; and a corresponding point detection unit sets a predetermined image among the plurality of images as a reference image, and the reference image For each of the different images, detecting each corresponding point on the image for each reference point on the reference image, and an image transformation processing unit for each band of the photographic illumination light In each of the images different from the reference image, the position of each reference point and the position of each corresponding point based on each corresponding point in the image with respect to each reference point on the reference image detected by the corresponding point detection unit Generating a deformation result image obtained by deforming the image so as to coincide with each other, generating a multiband image including each of the reference image and the deformation result image, and a fluorescence component separation unit comprising: For each band, the multi-band image generated by the image deformation processing unit includes a reflection component image that is an image of a band equal to the band of the photographing illumination light, and a band having a longer center wavelength than the band of the photographing illumination light. And a weighting factor calculation unit for each band of the photographing illumination light and the illumination light spectrum of the photographing illumination light and the Dividing the product of the wavelength selection characteristic of the second optical element for the shadow illumination light band by the product of the illumination light spectrum of the illumination light in the observation environment and the wavelength selection characteristic of the second optical element for the imaging illumination light band A step of calculating a weighting factor for the fluorescence component image of each band, and a fluorescence component image for each band of the photographing illumination light, wherein the multiband fluorescence component image generation unit is separated by the fluorescence component separation unit, A step of generating and outputting a multiband fluorescence component image by obtaining a weighted linear sum of the fluorescence component images based on the weighting factors for the fluorescence component images of the respective bands calculated by the weighting factor calculation unit. And a fluorescence component color reproduction unit for each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit. A fluorescence spectrum is calculated by applying a toll estimation matrix to generate a fluorescence component color reproduction image, and a reflection component color reproduction unit is a pixel of a multiband reflection component image by a reflection component image separated by the fluorescence component separation unit A step of calculating a reflected light spectrum by multiplying a spectral reflectance estimation matrix and an illumination light spectrum of the illumination light of the observation environment, and generating a reflected component color reproduction image; and Adding a reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image.

  The image processing program of the present invention is for causing a computer to function as each part of the image processing apparatus of the present invention.

  According to the image processing apparatus, the image processing system, the image processing method, and the image processing program of the present invention, the absorption characteristic of the fluorescent component is estimated from the multiband image captured by performing the spectrum with a wider wavelength range than the conventional one. There is an effect that it is possible to generate an image under arbitrary observation illumination light by using the reflection component and the fluorescence component.

It is a block diagram which shows an example of schematic structure of the image processing system (image processing apparatus) of 1st Embodiment. It is a figure which shows the spectral transmission factor of the band pass filter of the multiband illumination part of 1st Embodiment, and a multiband image imaging part. It is a figure which shows the multiband image image | photographed with the multiband image imaging part of 1st Embodiment. 3 is a flowchart illustrating an example of a flow of image processing in the image processing apparatus according to the first embodiment. It is a figure which shows the estimation result under a fluorescent lamp, and the measurement result by a spectrometer. It is a figure which shows the estimation result under an incandescent lamp, and the measurement result by a spectrometer. It is a block diagram which shows an example of schematic structure of the image processing system (image processing apparatus) of 2nd Embodiment. It is a figure which shows the specific example of the multiband illuminating device used as the multiband illuminating part of 2nd Embodiment. It is a specific example of the 9-eye stereo camera used for the multiband image photographing unit of the second embodiment. It is a figure which shows the spectral transmittance of the band pass filter of the multiband illumination part of 2nd Embodiment, and a multiband image imaging part. It is a block diagram which shows an example of schematic structure of the image transformation process part of 2nd Embodiment. It is a flowchart which shows an example of the flow of the image processing in the image processing apparatus of 2nd Embodiment. It is a block diagram which shows an example of schematic structure of the image processing system (image processing apparatus) of 3rd Embodiment. It is a specific example of the 9-eye stereo camera used for the multiband image photographing unit of the third embodiment. 15 is a flowchart illustrating an example of a flow of image processing in the image processing apparatus according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that this embodiment does not limit the present invention.

[First Embodiment]
FIG. 1 is a configuration diagram illustrating an example of a schematic configuration of an image processing system 1 including an image processing apparatus 20 according to the present embodiment.

  The image processing system 1 according to the present embodiment includes a multiband image photographing device 10 and an image processing device 20.

  The multiband image photographing device 10 includes a photographing control unit 12, a multiband illumination unit 14, and a multiband image photographing unit 16.

  The photographing control unit 12 controls the operation timing of the multiband illumination unit 14 and the multiband image photographing unit 16. Therefore, the imaging control unit 12 outputs a control signal for controlling the operation timing to each of the multiband illumination unit 14 and the multiband image capturing unit 16. Note that the imaging control unit 12 according to the present embodiment is realized by a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The control signal is output by executing the stored program. Therefore, according to the multiband image photographing device 10, fully automatic photographing is possible.

  The multiband illumination unit 14 includes a photographing illumination light source and an optical element (for example, a band pass filter) that controls a wavelength band (band) through which light emitted from the photographing illumination light source is transmitted (all not shown). The multiband illumination unit 14 switches the wavelength characteristic of the optical element based on a control signal input from the imaging control unit 12. As a specific example, when the optical element is a bandpass filter, the multiband illumination unit 14 performs switching of the bandpass filter. Further, the multiband illumination unit 14 emits light from the imaging illumination light source based on a control signal input from the imaging control unit 12. Thereby, the multiband illumination unit 14 outputs photographing illumination light having a selected wavelength. Hereinafter, the wavelength of photographing illumination light (the wavelength of the bandpass filter when a bandpass filter is used as an optical element) is referred to as “light source wavelength”.

  The multiband image photographing unit 16 includes one monochrome camera and an optical element (for example, a bandpass filter) that controls the wavelength band of incident light (all not shown). The multiband image photographing unit 16 switches the wavelength characteristic of the optical element based on the control signal input from the photographing control unit 12. As a specific example, when the optical element is a bandpass filter, the multiband image photographing unit 16 performs switching of the bandpass filter. The multiband image capturing unit 16 captures an image of the subject based on the control signal input from the capturing control unit 12. Thereby, the multiband image photographing unit 16 outputs a multiband image corresponding to the photographing illumination light of each band. Hereinafter, the wavelength of the optical element of the multiband image photographing unit 16 (or a bandpass filter when a bandpass filter is used) is referred to as a “camera wavelength”.

  The multiband image photographing unit 16 sets exposure conditions so that the output signal value output from the camera is not saturated. Further, photographing may be performed only when the light source wavelength is equal to or less than the camera wavelength (light source wavelength ≦ camera wavelength).

  On the other hand, the image processing apparatus 20 includes an acquisition unit 21, a captured image storage unit 22, a multiband image generation unit 24, a reflection component color reproduction unit 26, a fluorescence component color reproduction unit 28, and an image composition processing unit 30. The image processing apparatus 20 receives a multiband image captured by the multiband image capturing unit 16 of the multiband image capturing apparatus 10. Further, the image processing apparatus 20 outputs a color reproduction result image (details will be described later). Note that the image processing apparatus 20 according to the present embodiment is realized by a computer including a CPU, a RAM, a ROM, and the like, and the CPU executes a program stored in the ROM so that a multiband image can be obtained. A color reproduction result image is generated and output.

  The acquisition unit 21 acquires a multiband image from the multiband image capturing device 10 and stores the acquired multiband image in the captured image storage unit 22. Therefore, a multiband image is input to the acquisition unit 21, and the input multiband image is output to the captured image storage unit 22.

  The captured image storage unit 22 stores the input multiband image. The captured image storage unit 22 also includes a spectral sensitivity characteristic of each color band in the camera of the multiband image capturing unit 16, a captured illumination light spectrum, and wavelength selection characteristics of each band used for the spectrum of the captured illumination light (bandpass filter). Is used, the spectral transmittance of the bandpass filter) is stored together with the multiband image. Note that the captured image storage unit 22 may be integrated with the multiband image generation unit 24. The captured image storage unit 22 may be outside the image processing apparatus 20.

  The multiband image generation unit 24 includes a fluorescence component separation unit 40, a multiband fluorescence component image generation unit 42, and a weight coefficient calculation unit 44. The multiband image generation unit 24 generates a multiband reflection component image and a multiband fluorescence component image from the multiband image captured by the multiband image capturing device 10.

  The fluorescence component separation unit 40 selects and extracts an image in which a reflected light component (hereinafter referred to as “reflection component”) is recorded and an image in which a fluorescence component is recorded from a multiband image.

  Specifically, the fluorescence component separation unit 40 includes a multiband image corresponding to the photographing illumination light of each band from the photographed image storage unit 22, the spectral sensitivity characteristics of each color band in the camera, the photographing illumination light spectrum, and the photographing illumination. The wavelength selection characteristic (spectral transmittance of the bandpass filter) of each band used for the light spectrum is acquired. Then, the fluorescence component separation unit 40 extracts and outputs an image of a color band (light source wavelength = camera wavelength) equal to the band for each band of the photographic illumination light as a multiband reflection component image. Further, the fluorescence component separation unit 40 extracts and outputs an image of a color band (light source wavelength <camera wavelength) having a wavelength longer than that of each band of photographing illumination light as a fluorescence component image.

  The weighting coefficient calculation unit 44 includes wavelength selection characteristics (spectral transmittance of the bandpass filter), imaging illumination light spectrum, and illumination light of the observation environment (hereinafter referred to as “observation illumination light”) in each band of the multiband illumination unit 14. ) Illumination light spectrum (hereinafter referred to as “observation illumination light spectrum”). Note that the wavelength selection characteristic (spectral transmittance of the bandpass filter) and the photographing illumination light spectrum may be acquired from the photographed image storage unit 22. The weighting coefficient calculation unit 44 calculates a weighting coefficient for the fluorescence component image of each band of photographing illumination light.

  A fluorescence component image is input from the fluorescence component separation unit 40 to the multiband fluorescence component image generation unit 42. In addition, the multiband fluorescent component image generation unit 42 receives a weighting factor for each band of photographing illumination light from the weighting factor calculation unit 44. The multiband fluorescence component image generation unit 42 multiplies each band of the photographic illumination light by multiplying the fluorescence component image for the band by the weighting coefficient for the band and obtains a weighted linear sum, thereby obtaining the multiband fluorescence component image. Generate and output.

  Hereinafter, the generation of the multiband fluorescent component image in the multiband fluorescent component image generation unit 42 will be described by generalizing the number of bands as N and formulating it. In the following description, a case where a band-pass filter is used as an optical element for spectroscopy will be described. Further, in the following formula, only the visible light wavelength region (380 nm to 780 nm) is handled, but it may be extended to include the ultraviolet light wavelength region and the infrared light wavelength region.

The signal value of the i-th (i-th band) image (1 ≦ i, j ≦ N) photographed under the j-th (j-th band) illumination is the imaging illumination light spectrum as I (λ) and the j-th band. Assuming that the spectral transmittance of the filter is T _j (λ), the spectral reflectance of the subject surface is R (λ), and the spectral sensitivity of the camera in the i-th band is S _i (λ), Can be represented.

Here, λ represents a wavelength. From the above equation (1), the reflection component r _i can be expressed as the following equation (2).

On the other hand, the fluorescence component f _{i, j} can be expressed by the following equation (3), where α (λ) is the fluorescence absorption characteristic and E (λ) is the emission spectrum.

  From the above, the fluorescent component can be expressed as the following equation (4).

Assuming that the bandwidth of the filter T _j (λ) of the _j-th band is sufficiently small, the above equation (3) can be rewritten as the following equation (5).

The second term of the above equation (5) represents the energy of the photographic illumination light that has passed through the filter T _j (λ) of the j-th band, and can be measured using a spectrometer.

Next, it is considered to reproduce an image of the observation illumination light spectrum I ′ (λ) under the observation illumination light. The fluorescence component f _{i, j} under the observation illumination light spectrum I ′ (λ) can be expressed by the following equation 6) from the above equation (5).

That is, the above equation (6) means that the fluorescent component under observation illumination light different from that at the time of photographing is obtained by a weighted linear sum of the photographed image. When the weighting coefficient w _j in the j-th band is expressed by the following equation (7), the multiband fluorescent component image can be obtained based on the following equation (8).

The weighting factor calculation unit 44 calculates the weighting factor w _j for the multiband fluorescence component image generation unit 42 to generate a multiband fluorescence component image based on the above equation (8). For each band (j) of the imaging illumination light of the multiband illumination unit 14, the weighting coefficient calculation unit 44 uses the wavelength selection characteristic (spectral transmission of the bandpass filter) for the imaging illumination light spectrum and the band (j) used for spectroscopy. The product of the observation illumination light spectrum and the wavelength selection characteristic (spectral transmittance of the bandpass filter) for the band (j) used for spectroscopy, and calculate the weighting factor w _j for the band (j). To do. Specifically, the weighting coefficient w _j is calculated by the above equation (7). The weight coefficient calculation unit 44 outputs the calculated weight coefficient w _j for each band (j) of the photographic illumination light to the multiband fluorescent component image generation unit 42.

  The reflection component color reproduction unit 26 receives the multiband reflection component image extracted by the fluorescence component separation unit 40 of the multiband image generation unit 24. In addition, the observation illumination light spectrum is input to the reflection component color reproduction unit 26. Further, a color matching function, a monitor primary color spectrum, or a primary colorimetric value may be input to the multiband image generation unit 24.

  The reflection component color reproduction unit 26 uses the multiband reflection component image to calculate a spectral reflectance estimation matrix by spectral reflectance estimation processing. Then, the reflection component color reproduction unit 26 generates a reflected light spectrum image by calculating the reflected light spectrum by applying the spectral reflectance estimation matrix and the observation illumination light spectrum for each pixel of the multiband reflected component image. The reflection component color reproduction unit 26 displays a colorimetric value image that does not depend on the monitor characteristics when the color matching function is used, and a linear gamma characteristic when the primary color spectrum or colorimetric value of the monitor is used. A primary color signal value image (RGB image) is calculated. Hereinafter, a reflected light spectrum image or colorimetric value image calculated by the reflection component color reproduction unit 26 or a primary color signal value image of a monitor having a linear gamma characteristic is referred to as a “reflection component color reproduction image”. A reflection component color reproduction image is output from the reflection component color reproduction unit 26.

A method for obtaining a spectral spectrum from the multiband reflection component image in the reflection component color reproduction unit 26 will be described. Here, the case where the winner estimation method is used will be described as a specific example. It is assumed that the spectral sensitivity of the camera is expressed as a matrix S = [S ₁ (λ), S ₂ (λ),..., S _N (λ)] ^T. A diagonal matrix in which each diagonal component is a photographic illumination light spectrum is defined as a matrix I. Assuming that the spectral reflectance is R, the observed spectral spectrum S _r can be expressed as the following equation (9).

  Since the output signal value r from the camera is expressed by the following equation (10), the spectral reflectance R can be estimated from the following equation (11) by the Wiener estimation method.

  In the above equation (11), R ^ is an estimation result of the spectral reflectance, and a matrix M is a Wiener estimation matrix (spectral reflectance estimation matrix) and is obtained from the matrix H. In the above equation (11), the matrix K is a matrix representing the statistical properties of the spectral reflectance of the subject. Here, for the sake of simplicity, a first-order Markov property is assumed, and the matrix K is defined as the following equation (12).

Using the spectral reflectance estimation result R ^ and the observation illumination light spectrum I ~, the reflection component S ^ _r under the observation illumination light can be expressed by the following equation (13).

  The reflection component color reproduction unit 26 estimates the spectral reflectance for each pixel using the Wiener estimation method as described above, and generates a reflection component color reproduction image having the estimated spectral reflectance as a pixel value.

  When the color reproduction result image output from the image processing device 20 is used as a display color reproduction result image for display on an RGB display, the reflection component color reproduction unit 26 uses the color matching function and the display device 3. The reflected light spectrum image may be converted into an RGB image using the spectral spectral distribution or colorimetric value relating to the primary color and the gamma characteristic of the display device. In this case, first, a colorimetric value is calculated by multiplying the reflected light spectrum and the color matching function, and the calculated colorimetric value is converted into an RGB value. Finally, a correction process according to the gamma characteristic of the display device is performed to obtain a reflection component color reproduction image for display.

  The multiband fluorescence component image generated by the multiband fluorescence component image generation unit 42 is input to the fluorescence component color reproduction unit 28. Further, the observation illumination light spectrum is input to the fluorescence component color reproduction unit 28. Further, the fluorescence component color reproduction unit 28, color matching function, monitor primary color spectrum, or primary color measurement value may be input.

  The fluorescence component color reproduction unit 28 uses the multiband fluorescence component image to calculate a spectral spectrum estimation matrix by spectral spectrum estimation processing. Then, the fluorescence component color reproduction unit 28 generates a fluorescence spectrum image by calculating a fluorescence spectrum by applying a spectrum estimation matrix for each pixel of the multiband fluorescence component image. The fluorescence component color reproduction unit 28 displays a colorimetric value image that does not depend on the monitor characteristic when the color matching function is used, and a linear gamma characteristic when the primary color spectrum or colorimetric value of the monitor is used. A primary color signal value image (RGB image) is calculated. Hereinafter, the fluorescence spectrum image or colorimetric value image calculated by the fluorescence component color reproduction unit 28 or the primary color signal value image of the monitor having linear gamma characteristics is referred to as a “fluorescence component image”. A fluorescence component image is output from the fluorescence component color reproduction unit 28.

Specifically, the fluorescence component color reproduction unit 28 performs processing similar to that of the reflection component color reproduction unit 26 described above, and estimates a fluorescence spectrum to generate a multiband fluorescence component image. When f ′ = [f ′ ₁ , f ′ ₂ ,..., f ′ _N ] ^T and using a diagonal matrix I ′ whose diagonal components are all 1, the following equation (14) gives Is represented by the following equation (15).

  When the color reproduction result image output from the image processing device 20 is used as a display color reproduction result image for display on an RGB display, the fluorescent component color reproduction unit 28 is similar to the reflection component color reproduction unit 26. In FIG. 5, the fluorescence spectrum image may be converted into an RGB image.

  A reflection component color reproduction image and a fluorescence component color reproduction image are input to the image composition processing unit 30. The image composition processing unit 30 adds the reflection component color reproduction image and the fluorescence component color reproduction image, and generates and outputs a color reproduction result image observed under observation illumination light. The color reproduction result image is one of a spectrum image, a colorimetric value image, and a primary color signal value image of a monitor having a linear gamma characteristic according to the reflection component color reproduction image and the fluorescence component color reproduction image.

  Specifically, the spectral spectrum observed under the observation illumination light I˜ is expressed by the following equation (16). The image composition processing unit 30 generates a color reproduction result image based on the following equation (16).

  When the reflected light spectrum image and the fluorescence spectrum image are input to the image composition processing unit 30, the image composition processing unit 30 observes the same as the reflection component color reproduction unit 26 and the fluorescence component color reproduction unit 28 described above. The illumination light spectrum image may be converted into an RGB image.

  Next, an outline of the flow of image processing in the image processing apparatus 20 of the present embodiment will be described.

  As the camera of the multiband image photographing unit 16, one monochrome camera (no signal value correction within the camera and linear gamma characteristics) was used. A color chart was used for the subject. In both the multiband illumination unit 14 and the multiband image capturing unit 16, eight bandpass filters having the spectral transmittance shown in FIG. 2 were used as optical elements. The center wavelengths of the bandpass filters are 420 nm, 450 nm, 490 nm, 530 nm, 570 nm, 610 nm, 650 nm, and 690 nm, respectively, and the half-value width is 20 nm for the bandpass filter whose center wavelength is 420 nm, and 40 nm for the other bandpass filters. The exposure time of each camera was set so that the image signal value was not saturated when the band pass filter of the camera and the photographic illumination light source were the same. The exposure time may be constant in all shooting processes, or may be set to an optimum value for each band of the shooting illumination light source of the multiband illumination unit 14 and corrected by image processing after shooting. Focus adjustment was performed when the band-pass filters on the camera side and the photographing illumination light source side were the same in order to suppress the influence of aberrations of the optical system. A total of 36 images as shown in FIG. 3 were taken as multiband images by the multiband image photographing device 10.

  After the multiband image capturing device 10 captures a multiband image for each band of the photographic illumination light, the image processing device 20 performs image processing. FIG. 4 is a flowchart showing an example of the flow of image processing in the image processing apparatus 20 of the present embodiment.

  In step S <b> 100, the acquisition unit 21 acquires a multiband image for each band of photographing illumination light from the multiband image photographing device 10. Then, the acquisition unit 21 stores the acquired multiband image in the captured image storage unit 22.

  In the next step S102, the fluorescence component separation unit 40 separates the fluorescence component for each band of the photographic illumination light, and records the reflection component image in which the reflection component is recorded and the fluorescence component from the multiband image. The selected fluorescent component image is selected and extracted.

  In the next step S104, the weighting factor calculation unit 44 performs imaging illumination based on the wavelength selection characteristics (spectral transmittance of the bandpass filter), imaging illumination light spectrum, and observation illumination light spectrum in each band of the multiband illumination unit 14. A weighting factor is calculated for the fluorescent component image of each band of light based on the above-described equation (7). Note that the timing for calculating the weighting coefficient is not limited to this embodiment, and may be calculated in advance.

  In the next step S106, the multiband fluorescence component image generation unit 42 is based on the fluorescence component image obtained in step S102 and the weighting factor for the fluorescence component image of each band of the imaging illumination light obtained in step S104. As described above, for each band of the photographic illumination light, the fluorescence component image for the band is multiplied by the weighting coefficient for the band to obtain a weighted linear sum, thereby obtaining the photographic illumination for the band of the fluorescence component image. A multiband fluorescent component image is generated by multiplying the image captured with light by the weighting coefficient of the band.

  In the next step S108, the fluorescence component color reproduction unit 28 calculates the fluorescence spectrum as described above on the basis of the multiband fluorescence component image obtained in step S106 and the spectral spectrum estimation matrix. A spectrum image (fluorescence component color reproduction image) is generated.

  In the next step S110, the reflection component color reproduction unit 26 reflects the reflected light as described above on the basis of the multiband reflection component image obtained in step S102, the spectral reflectance estimation matrix, and the observation illumination light spectrum. A reflected light spectrum image (reflection component color reproduction image) is generated by calculating the spectrum.

  In the next step S112, as described above, the image composition processing unit 30 generates a color reproduction result image by adding the fluorescence color reproduction image and the reflection component color reproduction image, and outputs the resultant image to a display device. The image processing in the form is finished.

  Using four types of fluorescent color cards of yellow, green, orange and pink as the subject, a xenon lamp is used as the photographing illumination light source of the multiband illumination unit 14 at the time of photographing, and under the fluorescent lamp and the incandescent lamp as observation illumination light The experiment result which performed the experiment which estimates the observation illumination light spectrum in was performed. In FIG. 5, the estimation result (estimation) under a fluorescent lamp and the measurement result (measurement) by a spectrometer are shown. Moreover, in FIG. 6, the estimation result under an incandescent lamp and the measurement result by a spectrometer are shown. According to FIGS. 5 and 6, it can be seen that the estimation result and the measurement result almost coincide with each other under the fluorescent lamp and the incandescent lamp.

  In addition, it cannot be overemphasized that the order of the process of step S104-108 which produces | generates the fluorescence component color reproduction image in the said image process, and the process of step S110 which produces | generates a reflection component color reproduction image is not limited to this Embodiment. .

  In the image processing of the present embodiment, the configuration of the plurality of optical elements (bandpass filters) included in the multiband image photographing unit 16 of the multiband image photographing device 10 is not particularly limited. For example, the plurality of optical elements may control a wavelength band (band) through which light emitted from the photographing illumination light source passes for each photographed image photographed by the camera. In the case of such a configuration, for example, the optical elements arranged in front of the lens of the camera may be sequentially switched and photographing may be performed for each optical element.

  The plurality of optical elements are, for example, wavelength bands (bands) through which light emitted from the photographing illumination light source passes for each pixel (including each pixel group unit of the plurality of pixels) of the photographed image photographed by the camera. May be controlled. In the case of such a configuration, for example, when the camera has an image sensor, a different optical element (bandpass filter) may be arranged for each pixel of the image sensor. As a specific example, a charge coupled device (CCD) Image sensor) The color filter on the image sensor may be the plurality of optical elements.

  The multiband image photographing unit 16 includes one monochrome camera and an optical element (for example, a bandpass filter) that controls the wavelength band of incident light (all not shown). The multiband image photographing unit 16 switches the wavelength characteristic of the optical element based on the control signal input from the photographing control unit 12. As a specific example, when the optical element is a bandpass filter, the multiband image photographing unit 16 performs switching of the bandpass filter. The multiband image capturing unit 16 captures an image of the subject based on the control signal input from the capturing control unit 12. Thereby, the multiband image photographing unit 16 outputs a multiband image corresponding to the photographing illumination light of each band. Hereinafter, the wavelength of the optical element of the multiband image photographing unit 16 (or a bandpass filter when a bandpass filter is used) is referred to as a “camera wavelength”.

  As described above, the multiband image capturing apparatus of the present embodiment includes the multiband illumination unit 14 and the multiband image capturing unit 16. The multiband image photographing unit 16 is an example of a single monochromatic camera and a plurality of first optical elements that limit the wavelength of light incident on the single camera and have different bands that are wavelength bands of light to be transmitted. A certain band pass filter is provided. In addition, the multiband illumination unit 14 restricts the wavelength of the photographic illumination light source and the photographic illumination light irradiated to the subject from the photographic illumination light source, and a plurality of second optical elements having different bands that are the wavelength bands of the transmitted light. A band-pass filter which is an example of

  The image processing apparatus 20 includes a multiband image generation unit 24. The multiband image generation unit 24 includes a fluorescence component separation unit 40, a multiband fluorescence component image generation unit 42, and a weight coefficient calculation unit 44. The fluorescence component separation unit 40, for each band of photographic illumination light, forms a multiband image, a reflection component image that is an image of a band equal to the band, and a fluorescence component image that is an image of a band having a longer center wavelength than the band. And to separate. The weighting factor calculation unit 44 is the product of the illumination light spectrum of the imaging illumination light and the wavelength selection characteristic of the second optical element for the band for each band of the imaging illumination light, and the illumination light spectrum of the illumination light in the observation environment A weighting factor corresponding to the fluorescent component image of the band is calculated by dividing the product of the wavelength selection characteristics of the second optical element with respect to the band. The multi-band fluorescence component image generation unit 42 is a fluorescence component image for each band of imaging illumination light separated by the fluorescence component separation unit 40 and a weighting coefficient for each band of imaging illumination light calculated by the weighting coefficient calculation unit 44. Based on the above, a multiband fluorescence component image is generated and output by obtaining a weighted linear sum of the fluorescence component images.

  The image processing apparatus 20 also includes a reflection component color reproduction unit 26, a fluorescence component color reproduction unit 28, and an image composition processing unit 30. The fluorescence component color reproduction unit 28 calculates a fluorescence spectrum by applying a spectral spectrum estimation matrix for each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit 42, and generates a fluorescence component color reproduction image. To do. The reflection component color reproduction unit 26 calculates the reflected light spectrum by multiplying the spectral reflectance estimation matrix and the observation illumination light spectrum for each pixel of the multiband reflection component image by the reflection component image separated by the fluorescence component separation unit 40. Then, a reflection component color reproduction image is generated. The image composition processing unit 30 adds the fluorescent component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image.

  Therefore, according to the image processing apparatus 20 of the present embodiment, the reflection component image can be obtained without estimating the light absorption characteristic of the fluorescent component by using the multiband image obtained by performing spectroscopy with a wider wavelength range than before. And a fluorescent component image can be used to generate an image under any observation illumination light.

[Second Embodiment]
Since the present embodiment includes the same configuration and operation as those of the first embodiment described above, the same configuration and operation are described as such and detailed description thereof is omitted.

  FIG. 7 is a configuration diagram illustrating an example of a schematic configuration of the image processing system 101 including the image processing apparatus 120 according to the present embodiment.

  The image processing system 101 of this embodiment includes a multiband image photographing device 110 and an image processing device 120.

  The multiband image photographing device 110 includes a multiband illumination unit 114 and a multiband image photographing unit 116.

  FIG. 8 shows a specific example of a multiband illumination device used as the multiband illumination unit 114 of this embodiment. The imaging illumination light source of the multiband illumination device 114A shown in FIG. 8 is a xenon lamp, and a filter turret capable of attaching a maximum of eight bandpass filters is built in the multiband illumination device 114A. The emitted light is irradiated to the subject through the lens via the fiber guide. In the case of the multiband illuminating device 114A shown in FIG. 8, two 8-band illuminating devices are connected by a fiber guide, and the subject is irradiated with the emitted light from the same lens.

  The multiband image photographing unit 116 includes a plurality of monochrome cameras. FIG. 9 shows a specific example of a nine-eye stereo camera used in the multiband image photographing unit 116 of the present embodiment. All nine cameras are monochrome cameras, and each camera is provided with a band-pass filter having a spectral transmittance shown in FIG. Nine bandpass filters having the same spectral transmittance are also attached to the multiband illumination device 114A.

  The exposure time of each camera was set so that the image signal value was not saturated when the band-pass filters of the imaging control unit 112 and the multiband illumination unit 114 were equal, and correction was performed after imaging. The pixel value of a standard white plate photographed under the same conditions was used for correcting the brightness of the photographed image. The focus adjustment of each camera was also performed when the wavelengths of the band-pass filters of the imaging control unit 112 and the multiband illumination unit 114 were equal in order to suppress the influence of aberrations of the optical system.

  As described above, the camera of the multiband image capturing unit 116 of the present embodiment uses nine cameras, and the positions of the respective cameras are different, so the position of the subject in the image captured by each camera is shifted. ing. Therefore, in the image processing apparatus 120 according to the present embodiment, other images are deformed so as to match the position of the subject based on the reference image selected from among the multiband images.

  The image processing apparatus 120 according to the present embodiment receives a multiband image having N bands taken by the multiband image photographing unit 116. As shown in FIG. 7, the image processing apparatus 120 according to the present embodiment includes a multiband image generation unit 124 instead of the multiband image generation unit 24 of the image processing apparatus 20 according to the first embodiment. Yes.

  The multiband image generation unit 124 further includes an image deformation processing unit 46 before the fluorescence component separation unit 40. FIG. 11 is a configuration diagram illustrating an example of a schematic configuration of the image deformation processing unit 46 of the present embodiment.

  As shown in FIG. 11, the image deformation processing unit 46 includes a reference image selection unit 50, a corresponding point detection unit 52, an image deformation parameter calculation unit 54, an image deformation parameter storage unit 56, and an image deformation processing unit 58. .

  The reference image selection unit 50 receives a plurality of images captured by the multiband image capturing unit 116 stored in the captured image storage unit 22. One reference image as a reference for detecting corresponding points is selected from images (reflection component images) having the same light source wavelength and camera wavelength (light source wavelength = camera wavelength). For example, as a reference image, an image captured by a camera installed at a desired position with respect to a subject, a fourth image having a shorter center wavelength in a 9-band image, and dispersion of luminance histogram distribution. Examples include an image having the maximum entropy and an image having the maximum entropy.

  From the reference image selection unit 50, a reference image that is one selected reflection component image and a deformation target image that is N−1 reflection component images that are different from the reference image and have the same light source wavelength and camera wavelength. Are output to the corresponding point detection unit 52. In addition, from the reference image selection unit 50, images having different light source wavelengths and camera wavelengths are output to the image deformation processing unit 58.

  The corresponding point detection unit 52 receives the reference image selected by the reference image selection unit 50 and the deformation target image which is N−1 reflection component images. The deformation target image may be sequentially input to the corresponding point detection unit 52 every time the image deformation parameter is stored in the image deformation parameter storage unit 56 described later.

  The corresponding point detection unit 52 searches for and detects corresponding points of the deformation target image corresponding to the reference points of the reference image. A method for searching for and detecting corresponding points is not particularly limited, and an existing method may be used. Examples of existing methods include NCC (Normalized Cross Correlation), Phase-only correlation (POC), SAD (sum of absolute differences), SSD (sum of squared difference), and histogram correlation. SIFT (Scale-Invariant Feature Transform) method, HOG (Histograms of Oriented Gradients) method and the like.

  The corresponding point detection unit 52 outputs the detected corresponding points and the positional relationship between the reference points of the reference image and the corresponding points on the deformation target image.

  The image deformation parameter calculation unit 54 receives the corresponding points detected by the corresponding point detection unit 52 and the positional relationship between the reference points of the reference image and the corresponding points on the deformation target image.

  The image deformation parameter calculation unit 54 uses the corresponding points detected by the corresponding point detection unit 52 to calculate image deformation parameters so that the reference points and the corresponding points coincide. Examples of the image deformation parameter include an affine transformation matrix and a lookup table used in the spline interpolation method. The method for calculating the image deformation parameter is not particularly limited, and an existing method may be used.

  In the present embodiment, the image deformation parameter calculation unit 54 calculates image deformation parameters for the deformation target images that are N−1 reflection component images. Therefore, the image deformation parameter calculation unit 54 outputs image deformation parameters for each band.

  The image transformation parameter storage unit 56 stores the input image transformation parameter calculated by the image transformation parameter calculation unit 54.

  The image deformation processing unit 58 receives the image deformation parameters and deformation target images of each band stored in the image deformation parameter storage unit 56.

  The image deformation processing unit 58 deforms the deformation target image based on the image conversion parameter. It should be noted that the same image deformation parameter can be applied to images taken with the same camera under different photographing illumination light sources. Therefore, for an image (fluorescence component image) in which the light source wavelength and the camera wavelength are different (light source wavelength ≠ camera wavelength), it is not necessary to perform corresponding point detection and image deformation parameter calculation, and the image deformation parameter storage unit 56 stores the image. Image deformation processing is performed using image deformation parameters having the same camera wavelength. From the image deformation processing unit 58, a multiband image including a reference image and a deformation result image (deformation result image) of the deformation target image of each band is output to the fluorescence component separation unit 40.

  In the present embodiment, when the subject is planar, a projective transformation method, which is one of linear methods, is used as a means in the image deformation parameter calculation unit 54 and the image deformation processing unit 58. Further, when the subject has a three-dimensional shape, a TPS (Thin-plate spline) method, which is one of nonlinear methods, is used. As another image deformation method, an image may be divided into small regions, and a linear method such as a projective transformation method or a nonlinear method such as a TPS method may be applied for each image. Alternatively, the stereo information obtained by each camera may be used to estimate the three-dimensional shape of the subject, the shape estimation result may be associated with the captured image, and image generation may be performed. Instead of using the three-dimensional shape estimation result from the photographed image, the shape measurement by the three-dimensional scanner may be performed simultaneously with the image photographing, and the obtained three-dimensional shape data and the photographed image may be associated with each other.

  Next, an outline of the flow of image processing in the image processing apparatus 120 of the present embodiment will be described.

  FIG. 12 is a flowchart illustrating an example of the flow of image processing in the image processing apparatus 120 according to the present embodiment.

  In the image processing of the present embodiment, the acquisition unit 21 acquires a multiband image and stores it in the captured image storage unit 22 in step S100, and then proceeds to step S202.

  In step S202, the reference image selection unit 50 selects a reference image as a reference for detecting corresponding points from images (reflection component images) having the same light source wavelength and camera wavelength. For example, as described above, an image taken by the central camera is selected as the reference image.

  In the next step S204, as described above, the corresponding point detection unit 52 searches for and detects corresponding points between the reference image and one deformation target image, in which the light source wavelength and the camera wavelength are equal. For example, a phase only correlation method is used for detecting corresponding points with sub-pixel accuracy.

  In the next step S206, the image deformation parameter calculation unit 54 calculates the image deformation parameter using the corresponding points detected by the corresponding point detection unit 52 as described above. The calculated image deformation parameter is stored in the image deformation parameter storage unit 56.

  In the next step S <b> 208, the image deformation parameter calculation unit 54 determines whether image deformation parameters for all bands of the monochromatic camera are stored in the image deformation parameter storage unit 56. If not yet stored, the determination is negative, and the process returns to step S204 to repeat the calculation and storage of the image deformation parameter. On the other hand, when it becomes affirmation determination, it transfers to step S210.

  In step S210, as described above, the image deformation processing unit 58 includes all the deformation target images including the deformation target images having different light source wavelengths and camera wavelengths (light source wavelength ≠ camera wavelength) based on the image conversion parameters. Perform deformation.

  After step S210, the process proceeds to step S102 of the above-described first embodiment, and after performing the processes of steps S102 to S112, the main image processing is terminated.

  In the present embodiment, as described above, the case where the multiband image capturing unit 116 includes a plurality of monochrome (single color) cameras has been described. However, the camera included in the multiband image capturing unit 116 is not limited thereto. Not. For example, as with the multiband image photographing unit 16 of the multiband image photographing device 10 according to the first embodiment, a single monochrome (monochrome) camera may be used. Even if there is only one camera as in the multiband image capturing unit 16 of the first embodiment, if the camera position or orientation of the multiband image capturing unit 16 changes for some reason, the present embodiment It is preferable to perform the above-described image processing (particularly, image deformation processing by the image deformation processing unit 58).

  Further, the multiband image photographing unit 116 may be provided with a color (RGB) camera as in a third embodiment to be described later. In this case, a monochromatic image obtained by converting a color image taken by a color camera into a black and white image may be used.

  In the present embodiment, as described above, a reference image is selected from images that have the same light source wavelength and camera wavelength among a plurality of captured images, and the image deformation parameter calculation unit 54 performs the deformation target image. Although the image deformation parameter is calculated for the image deformation parameter, the image deformation parameter is calculated for all of a plurality of captured images excluding the reference image, and the image deformation processing unit is calculated based on the calculated image deformation parameter. 58 may perform image deformation processing.

  As described above, the multiband image capturing apparatus 110 according to the present embodiment includes the multiband illumination unit 114 and the multiband image capturing unit 116. The multiband image photographing unit 116 includes a bandpass filter that is an example of a plurality of first optical elements that limit the wavelength of light incident on the camera and have different bands that are wavelength bands of light to be transmitted, and Equipped with multiple monochrome cameras with different viewpoints. The multiband illumination unit 114 is an example of a plurality of second optical elements that limit the wavelength of the photographic illumination light source and the photographic illumination light emitted from the photographic illumination light source to the subject and have different bands that are the wavelength bands of the transmitted light. A certain band pass filter is provided.

  The multiband image generation unit 124 in the image processing apparatus 120 of the present embodiment includes a fluorescence component separation unit 40, a multiband fluorescence component image generation unit 42, a weight coefficient calculation unit 44, and an image deformation processing unit 46. The image deformation processing unit 46 includes a reference image selection unit 50, a corresponding point detection unit 52, an image deformation parameter calculation unit 54, an image deformation parameter storage unit 56, and an image deformation processing unit 58.

  The reference image selection unit 50 selects a predetermined image as a reference image from among a plurality of captured images having the same light source wavelength and camera wavelength.

  The corresponding point detection unit 52 detects each corresponding point on the image with respect to each reference point on the reference image for each of the images different from the reference image and having the same light source wavelength and camera wavelength.

  The image deformation parameter calculation unit 54 calculates an image deformation parameter for each of the images that are different from the reference image and have the same light source wavelength and camera wavelength, and stores them in the image deformation parameter storage unit 56.

  The image transformation processing unit 58 is configured to detect the position of each reference point and the position of each reference point on the basis of each corresponding point in the image with respect to each reference point on the reference image. A deformation result image obtained by deforming the image so as to match the position of the corresponding point is generated, and a multiband image including each of the reference image and the deformation result image is generated.

  The fluorescence component separation unit 40 according to the present embodiment converts the multiband image deformed by the image deformation processing unit 58 for each band of photographing illumination light into a reflection component image that is an image of a band equal to the band, and The image is separated into a fluorescent component image that is an image of a band having a longer center wavelength than the band.

  Therefore, in the image processing apparatus 120 of the present embodiment as well, as in the first embodiment, the absorption characteristics of the fluorescent component are estimated from the multiband image captured by performing the spectrum with a wider wavelength range than before. It is not necessary to generate an image under any observation illumination light using the reflection component image and the fluorescence component image.

In addition, since a plurality of cameras can be used for shooting simultaneously, the number of shootings can be reduced as compared with the first embodiment.
[Third Embodiment]
Since this embodiment includes the same configuration and operation as each of the above-described embodiments, the same configuration and operation are described as such and detailed description thereof is omitted.

  FIG. 13 is a configuration diagram illustrating an example of a schematic configuration of the image processing system 201 according to the present embodiment. The image processing system 201 of the present embodiment is different from the second embodiment in a multiband image photographing unit 216 and an image deformation processing unit 246 of the image processing device 220.

  The multiband image photographing unit 216 according to the present embodiment includes one color (RGB) camera and a plurality of single color (black and white) cameras (as a specific example, eight). The multiband image photographing unit 216 includes an optical element that controls the wavelength band of incident light associated with each monochrome camera.

  FIG. 14 shows a specific example of a nine-eye stereo camera used in the multiband image photographing unit 216 of this embodiment. The center camera is an RGB camera, and the surrounding eight cameras are monochrome cameras. The angle of view was determined with an RGB camera, and a black and white camera was placed around it. In each monochrome camera, a bandpass filter having the spectral transmittance shown in FIG. 2 is attached in front of the lens.

  The exposure time of each camera was set so as not to saturate the image signal value when the wavelengths of the multiband illumination unit 14 and the multiband image photographing unit 216 bandpass filter were equal, and correction was performed after photographing. The pixel value of a standard white plate photographed under the same conditions was used for correcting the brightness of the photographed image. In addition, focus adjustment of each camera was also performed when the wavelengths of the band-pass filters of the multiband illumination unit 14 and the multiband image photographing unit 216 were equal in order to suppress the influence of aberration of the optical system.

  From the multiband image photographing unit 216, a multiband image corresponding to the photographing illumination light of each band and a color (RGB) image photographed with the photographing illumination light (the state before the spectrum), that is, the number of bands is (N-1 + 3). ) Multiband image is output.

  According to the multiband image photographing unit 216 of the present embodiment, since an 8-band image and an RGB image can be photographed with one shot, the number of photographing can be reduced to 8 times (36 times in the first embodiment). it can.

  As illustrated in FIG. 13, the image deformation processing unit 246 according to the present embodiment includes a reference image selection unit 250 and a corresponding point detection unit instead of the reference image selection unit 50 and the corresponding point detection unit 52 according to the second embodiment. 252. The image deformation processing unit 246 further includes a spectral spectrum estimation unit 60, a camera spectral sensitivity characteristic storage unit 62, and a monochrome image generation unit 64.

  The multiband image photographed by the multiband image photographing unit 216 stored in the photographed image storage unit 22 is input to the reference image selection unit 250 of the present embodiment.

  The reference image selection unit 250 selects, as a reference image, a color image of a color camera that has been shot under shooting illumination light that is not spectrally separated.

  A reference image is output from the reference image selection unit 250 to the spectral spectrum estimation unit 60. In addition, a deformation target image that is an image of a monochromatic camera in which the light source wavelength is equal to the camera wavelength (light source wavelength = camera wavelength) is output to the corresponding point detection unit 252. Further, the reference image selection unit 250 outputs a deformation target image (fluorescence component image) in which the light source wavelength and the camera wavelength are different (light source wavelength ≠ camera wavelength) to the image deformation processing unit 58.

  The camera spectral sensitivity characteristic storage unit 62 stores in advance the spectral sensitivity characteristics of each monochrome camera of the multiband image photographing unit 216.

  A reference image (a color image photographed under photographing illumination light that is not spectrally separated) is input from the reference image selection unit 250 to the spectral spectrum estimation unit 60. The spectral spectrum estimation unit 60 estimates the spectral reflectance for each pixel of the color image. The estimation method of the spectral reflectance is not particularly limited, and may be the same as the estimation method described above in the first embodiment. The spectral spectrum estimation unit 60 outputs a spectral reflectance image having the estimated spectral reflectance as a pixel value.

  The monochromatic image generation unit 64 is supplied with the spectral reflectance image generated by the spectral spectrum estimation unit 60 and the photographing illumination spectrum (the state before spectral separation). The monochromatic image generation unit 64 receives the spectral sensitivity characteristics of each monochromatic camera from the camera spectral sensitivity characteristic storage unit 62.

  The monochrome image generation unit 64 performs a shooting simulation for each monochrome camera (eight monochrome cameras in the present embodiment) using the spectral reflectance image, the shooting illumination spectrum, and the spectral sensitivity characteristics of the monochrome camera. A virtual single-color image that will be obtained when taken from the viewpoint of the camera is generated, and is used as a reference image for detecting a corresponding point with respect to the image to be deformed taken with a certain bandpass filter attached.

  From the monochromatic image generation unit 64, each monochromatic image at the viewpoint of the color camera is output to the corresponding point detection unit 252 as a reference image.

  The corresponding point detection unit 252 searches for and detects corresponding points between the input reference image generated by the monochrome image generation unit 64 and the deformation target image. The corresponding point detection method is not particularly limited. For example, the corresponding point detection method may be the same as the detection method in the corresponding point detection unit 52 described above in the second embodiment.

  From the corresponding point detection unit 252, the detected corresponding points and the positional relationship between the corresponding points on the reference image and the deformation target image are output to the image deformation parameter calculation unit 54.

  Since the image deformation of the deformation target image by the image deformation parameter calculation unit 54, the image deformation parameter storage unit 56, and the image deformation processing unit 58 is the same as that in the second embodiment, the description thereof is omitted. Note that the number of bands of the multiband image output from the image deformation processing unit 246 of the present embodiment to the fluorescence component separation unit 40 is N−1 unlike the second embodiment (number of bands: N). .

  Next, an outline of the flow of image processing in the image processing apparatus 220 of the present embodiment will be described.

  FIG. 15 is a flowchart showing an example of the flow of image processing in the image processing apparatus 220 of the present embodiment.

  In the image processing of the present embodiment, the acquisition unit 21 acquires a multiband image and stores it in the captured image storage unit 22 in step S100, and then the process proceeds to step S302.

  In step S302, the reference image selection unit 250 selects, as a reference image, a color image of a color camera that has been shot under shooting illumination light that is not spectrally separated.

  In the next step S304, the spectral spectrum estimation unit 60 estimates the spectral reflectance for each pixel of the color image as described above.

  In the next step S306, as described above, the monochromatic image generation unit 64 performs an imaging simulation using the spectral reflectance image, the imaging illumination spectrum, and the spectral sensitivity characteristics of the monochromatic camera, and generates a virtual monochromatic image.

  In the next step S308, the single color image generation unit 64 determines whether or not the processing in step S306 has been performed for all single color cameras. If not, a negative determination is made, and the process of step S306 is repeated. On the other hand, when it becomes affirmation determination, it transfers to step S310.

  In step S310, as described above, the corresponding point detection unit 252 searches for and detects corresponding points between one deformation target image in which the reference image, the light source wavelength, and the camera wavelength are equal, and then proceeds to step S206. . Steps S206 and S208 perform the same processing as in the second embodiment described above. If the determination in step S208 is affirmative, the process proceeds to step S210 described in the second embodiment, and step S210 is performed. After performing the above processing and the processing in steps S102 to S112 of the first embodiment described above, the present image processing is terminated.

  Note that a monochrome (monochrome) camera can be used instead of the color camera used as the camera of the multiband image photographing unit 216 in this embodiment. In this case, since the image captured by the monochrome camera used instead is a monochrome image, the above-described spectral spectrum estimation unit 60, camera spectral sensitivity characteristic storage unit 62, and monochrome image generation unit 64 are not necessary.

  As described above, in the image processing apparatus 220 according to the present embodiment, similarly to the image processing apparatus 120 according to the second embodiment, a multiband image captured by performing spectroscopy with a broader wavelength width than the conventional one, An image under any observation illumination light can be generated using the reflection component image and the fluorescence component image without requiring the estimation of the light absorption characteristics of the fluorescence component. Furthermore, since a plurality of cameras can be used for shooting simultaneously, the number of times of shooting can be reduced as compared with the first embodiment.

  In the reference image selection unit 50 according to the second embodiment and the reference image selection unit 250 according to the third embodiment, the image of the monochrome camera to be deformed is an image in which the light source wavelength and the camera wavelength are equal. Only may be input.

  In each of the above embodiments, the Wiener estimation method is used for the process of estimating the spectral reflectance and the fluorescence spectrum, but other methods may be used. For example, a method using a principal component calculated from a database of spectral reflectance or fluorescence component spectrum, a multiple regression analysis method, a PLS (partial least square) method (partial least square method), or the like may be used.

  Moreover, each said embodiment is an example, and a specific structure is not restricted to this embodiment, The design etc. of the range which does not deviate from the summary of this invention are included, and it can change according to a condition. Needless to say.

1, 101, 201 Image processing system 10, 110 Multiband image capturing device 14, 114 Multiband illumination unit 16, 116, 216 Multiband image capturing unit 20, 120, 220 Image processing device 21 Acquisition unit 24 Multiband image generation unit 26 reflection component color reproduction unit 28 fluorescence component color reproduction unit 30 image composition processing unit 40 fluorescence component separation unit 42 multiband fluorescence component image generation unit 44 weight coefficient calculation unit 46, 246 image deformation processing unit 50, 250 reference image selection unit 52 , 252 Corresponding point detection unit 54 Image deformation parameter calculation unit 56 Image deformation parameter storage unit 58 Image deformation processing unit

Claims (8)

Limiting the wavelength of light incident on a single camera, limiting the wavelength of illuminating illumination light irradiating a subject from a plurality of first optical elements, each having a different band, which is the wavelength band of transmitted light, and the photographic illumination light source An acquisition unit that acquires a multiband image for each band of imaging illumination light, which is captured by a multiband image capturing device including a plurality of second optical elements each having a different band that is a wavelength band of light to be transmitted; ,
For each band of the photographic illumination light, the multiband image acquired by the acquisition unit, a reflection component image that is an image of a band equal to the band of the photographic illumination light, and a center wavelength from the band of the photographic illumination light A fluorescent component separation unit that separates into a fluorescent component image that is an image of a long band,
For each band of the photographic illumination light, the product of the illumination light spectrum of the photographic illumination light and the wavelength selection characteristic of the second optical element for the band of the photographic illumination light, and the illumination light spectrum of the illumination light of the observation environment and the A weighting factor calculation unit that calculates a weighting factor for the fluorescent component image of each band by dividing the product of the wavelength selection characteristics of the second optical element with respect to the fluorescent component image of the band of the imaging illumination light;
Based on the fluorescence component image for each band of the photographing illumination light separated by the fluorescence component separation unit and the weighting factor for the fluorescence component image of each band calculated by the weighting factor calculation unit, the fluorescence component A multiband fluorescence component image generation unit that generates and outputs a multiband fluorescence component image by calculating a weighted linear sum of images,
For each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit, calculate a fluorescence spectrum by applying a spectral spectrum estimation matrix, and generate a fluorescence component color reproduction image;
For each pixel of the multiband reflection component image by the reflection component image separated by the fluorescence component separation unit, the reflected light spectrum is calculated by multiplying the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment, A reflection component color reproduction unit that generates a component color reproduction image;
An image composition processing unit that adds the fluorescence component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image; and
An image processing apparatus. The wavelength of the light incident on the camera is limited and the wavelength band of the light to be transmitted has a plurality of first optical elements that are different from each other. Each band of photographic illumination light that is photographed by a multiband image photographing device that includes a plurality of second optical elements that limit the wavelength of illuminating illuminating light to be irradiated and that are different in wavelength bands of light to be transmitted. An acquisition unit for acquiring a plurality of images for
Corresponding point detection for detecting a corresponding point on the image with respect to each reference point on the reference image for each image different from the reference image, using a predetermined image among the plurality of images as a reference image And
For each band of the photographic illumination light, in each of the images different from the reference image, each corresponding point in the image with respect to each reference point on the reference image detected by the corresponding point detection unit, Image deformation processing for generating a deformation result image obtained by deforming the image so that the position of the reference point matches the position of each corresponding point, and generating a multiband image including each of the reference image and the deformation result image And
For each band of the photographic illumination light, the multiband image generated by the image deformation processing unit is a reflection component image that is an image of a band equal to the band of the photographic illumination light, and a band of the photographic illumination light A fluorescence component separation unit that separates a fluorescence component image that is an image of a band having a long center wavelength;
For each band of the photographic illumination light, the product of the illumination light spectrum of the photographic illumination light and the wavelength selection characteristic of the second optical element for the band of the photographic illumination light, and the illumination light spectrum of the illumination light of the observation environment and the A weighting factor calculation unit that calculates a weighting factor for the fluorescence component image of each band by dividing the product of the wavelength selection characteristics of the second optical element with respect to the band of the imaging illumination light;
Based on the fluorescence component image for each band of the photographing illumination light separated by the fluorescence component separation unit and the weighting factor for the fluorescence component image of each band calculated by the weighting factor calculation unit, the fluorescence component A multiband fluorescence component image generation unit that generates and outputs a multiband fluorescence component image by calculating a weighted linear sum of images,
For each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit, calculate a fluorescence spectrum by applying a spectral spectrum estimation matrix, and generate a fluorescence component color reproduction image;
For each pixel of the multiband reflection component image by the reflection component image separated by the fluorescence component separation unit, the reflected light spectrum is calculated by multiplying the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment, A reflection component color reproduction unit that generates a component color reproduction image;
An image composition processing unit that adds the fluorescence component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image; and
An image processing apparatus. The plurality of cameras includes one color camera that captures a color image and one or more single-color cameras that capture a monochrome image, and the plurality of first optical elements correspond to each of the monochrome cameras. Transmit different bands of light,
Spectral reflectance image having the spectral reflectance estimated for each pixel of the color image acquired by the acquisition unit as a pixel value, the illumination light spectrum of the photographic illumination light in a state before spectral separation, and each monochromatic camera Using a spectral sensitivity characteristic, further comprising a single-color image generation unit for generating a virtual single-color image obtained for each single-color camera, which would be obtained from the viewpoint of the color camera;
The corresponding point detection unit uses the monochrome image generated by the monochrome image generation unit as the reference image.
The image processing apparatus according to claim 2. Limiting the wavelength of light incident on a single camera, limiting the wavelength of illuminating illumination light irradiating a subject from a plurality of first optical elements, each having a different band, which is the wavelength band of transmitted light, and the photographic illumination light source A plurality of second optical elements each having a different band, which is a wavelength band of light to be transmitted, and a multiband image photographing device,
The multi-band image with respect to each band of the illuminating light photographed by the multi-band image photographing device is acquired, and a color reproduction result image is generated and output based on the acquired multi-band image. Image processing apparatus,
An image processing system. The wavelength of the light incident on the camera is limited and the wavelength band of the light to be transmitted has a plurality of first optical elements that are different from each other. A multiband image photographing device comprising: a plurality of second optical elements that limit wavelengths of photographing illumination light to be irradiated and have different bands each being a wavelength band of light to be transmitted;
3. A plurality of images for each band of photographing illumination light captured by the multiband image capturing device are acquired, and a color reproduction result image is generated and output based on the acquired plurality of images. Item 3. The image processing apparatus according to Item 3,
An image processing system. The acquisition unit restricts the wavelength of light incident on one camera, and a plurality of first optical elements having different wavelength bands of light to be transmitted, and photographing illumination light irradiated to the subject from the photographing illumination light source Acquire multiband images for each band of illuminating illumination light captured by a multiband image capturing device that includes a plurality of second optical elements each having a different band, which is the wavelength band of light to be transmitted, limiting the wavelength. And steps to
For each band of the photographic illumination light, a fluorescent component separation unit obtains the multiband image acquired by the acquisition unit, a reflection component image that is an image of a band equal to the band of the photographic illumination light, and the photographic illumination Separating the fluorescence component image, which is an image of a band having a longer center wavelength than the light band;
The weighting factor calculation unit is a product of an illumination light spectrum of the photographing illumination light and a wavelength selection characteristic of the second optical element for the band of the photographing illumination light for each band of the photographing illumination light. Calculating a weighting factor for the fluorescence component image of each band by dividing the product of the wavelength selection characteristic of the second optical element with respect to the illumination light spectrum and the band of the imaging illumination light;
The multi-band fluorescence component image generation unit is divided by the fluorescence component separation unit, the fluorescence component image for each band of the photographing illumination light, and the weight for the fluorescence component image of each band calculated by the weight coefficient calculation unit Generating and outputting a multiband fluorescent component image by obtaining a weighted linear sum of the fluorescent component images based on the coefficients; and
A step in which a fluorescence component color reproduction unit calculates a fluorescence spectrum by multiplying a spectral spectrum estimation matrix for each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit, and generates a fluorescence component color reproduction image When,
The reflected component color reproduction unit applies the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment to each pixel of the multiband reflected component image by the reflected component image separated by the fluorescent component separating unit. Calculating a spectrum and generating a reflection component color reproduction image;
An image synthesis processing unit adding the fluorescence component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image; and
An image processing method including: The acquisition unit restricts the wavelength of light incident on the camera and has a plurality of first optical elements each having a different wavelength band of light to be transmitted, and a plurality of cameras having different viewpoints, and photographing illumination Photographing illumination, which is photographed by a multiband image photographing device that includes a plurality of second optical elements that limit the wavelength of photographing illumination light emitted from a light source to a subject and that have different wavelength bands of transmitted light. Acquiring a plurality of images for each band of light;
The corresponding point detection unit uses a predetermined image of the plurality of images as a reference image, and for each image different from the reference image, each corresponding point on the image with respect to each reference point on the reference image Detecting steps,
The image deformation processing unit corresponds to each band in the image with respect to each reference point on the reference image detected by the corresponding point detection unit in each image different from the reference image for each band of the photographing illumination light. Based on the points, a multi-band image is generated that generates a deformation result image obtained by deforming the image so that the position of each reference point matches the position of each corresponding point, and includes each of the reference image and the deformation result image A step of generating
A fluorescence component separation unit, for each band of the photographic illumination light, the multiband image generated by the image deformation processing unit, a reflection component image that is an image of a band equal to the band of the photographic illumination light; Separating the fluorescence component image, which is an image of a band having a longer center wavelength than the band of the imaging illumination light;
The weighting factor calculation unit is a product of an illumination light spectrum of the photographing illumination light and a wavelength selection characteristic of the second optical element for the band of the photographing illumination light for each band of the photographing illumination light. Calculating a weighting factor for the fluorescence component image of each band by dividing the product of the wavelength selection characteristic of the second optical element with respect to the illumination light spectrum and the band of the imaging illumination light;
The multi-band fluorescence component image generation unit is divided by the fluorescence component separation unit, the fluorescence component image for each band of the photographing illumination light, and the weight for the fluorescence component image of each band calculated by the weight coefficient calculation unit Generating and outputting a multiband fluorescent component image by obtaining a weighted linear sum of the fluorescent component images based on the coefficients; and
The fluorescence component color reproduction unit calculates a fluorescence spectrum by applying a spectral spectrum estimation matrix for each pixel of the multiband fluorescence component image generated by the multiband fluorescence component image generation unit, and generates a fluorescence component color reproduction image Steps,
The reflection component color reproduction unit reflects the spectral reflectance estimation matrix and the illumination light spectrum of the illumination light of the observation environment for each pixel of the multiband reflection component image based on the reflection component image separated by the fluorescence component separation unit. Calculating a light spectrum and generating a reflection component color reproduction image;
An image synthesis processing unit adding the fluorescence component color reproduction image and the reflection component color reproduction image to generate and output a color reproduction result image; and
An image processing method including:   The image processing program for functioning a computer as each part of the image processing apparatus of any one of Claims 1-3.